Method for detecting and/or tracking objects

p0001The present invention relates to a process for the recognition and tracking of objects that correspond to objects in a detection region of a sensor for electromagnetic radiation, and a device for carrying out the method.

p0002Methods for the detection and tracking of objects are known in principle. Typically, images of a detection range are thereby detected by a sensor for electromagnetic radiation, such as a laser scanner or a video camera at constant intervals. The pictures will then search for objects that match objects in the detection area. Was in an image an object first found, is wanted in subsequent cycles of the method for this object to track its location or change in position over time can. For this purpose is often predicated on the basis of the position and velocity of an object in a previous cycle its location for a current cycle or in a current picture to be in the picture discovered near the predicted position of the object elements, eg segments, the object assign from the previous cycle and to be able to detect its current location.

p0003Such methods are suitable for example for monitoring a region in front of and / or next to a motor vehicle. A potential advantage of such monitoring may be that sudden threats automatically detected and appropriate countermeasures can be taken. For this it is necessary, however, that objects can be quickly detected and tracked accurately.

p0004In the above-mentioned method of tracking an object is often to increase the accuracy of the detected object properties with increasing duration, because in the course of pursuing further information can be gathered about the object, which allow a better characterization of the object or the state of the object. Accordingly, inaccuracies can especially occur when new items appear in images. In the example of the motor vehicle, this may be for example, pedestrians, obscured by another object such as a parked on a roadway edge automotive, suddenly into the street and then be detected by the sensor. The same problem can occur when pedestrians are positioned in front of an object so that the object tracking method the pedestrian and the object resumes to an object so that the pedestrian is also recognized only when it has moved far enough away from the object.

p0005<patcit id="pcit0001" dnum="US6480615B1"><text>US 6,480,615 B1</text></patcit> describes a method for tracking of objects, in which the optical flow of an image frame using data from three successively recorded images k-1, k, k + 1 is determined. On the basis of a parameter S is decided whether the required for the determination of the optical flow gradient of the images k-1 and k or from the images K and K + may be used first Due to the possibility of using the gradient based on the images k-1 and k, can also be the optical flow in areas that are obscured by a foreground object temporarily can be determined.

p0006The present invention is based on the object, a method for detecting and tracking objects which correspond to objects in a detection range of a sensor for electromagnetic radiation, provide that allow for fast detection and tracking of objects, and a corresponding apparatus for performing the method to create.

p0007The object is achieved by a method having the features of claim 1.

p0008In the inventive method for the detection and tracking of objects in at least one detection area corresponding to objects at least one sensor for electromagnetic radiation, on the basis of repeatedly detected by the at least one sensor images of the detection range in consecutive cycles consecutive images are analyzed and it is each at least one aufgefundenes in one cycle on the basis of a corresponding image object in a later cycle wanted in a corresponding later image to track the object. It is a recognized on the basis of the results of an at least preliminary evaluation of a current image while at least one current cycle at least one determined in the current cycle part of a current image or in the current cycle upon detection of a new object and / or feature in a current cycle object associated with at least a part of an earlier image and / or at least one indication with respect to an earlier state of the object or of an appropriate object, which is determined using a corresponding earlier image in the current cycle.

p0009The object is further achieved by a device having the features of claim 13.

p0010The inventive apparatus for the recognition and tracking of objects has at least one formed to capture images of a coverage sensor for electromagnetic radiation and with the at least one sensor connected to data processing means, which is designed for carrying out the method according to the invention, and in particular means for evaluation of current images of the at least one sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to includes a previous state of the object or of an appropriate object using a corresponding earlier image in the current cycle and association of the indication with the part of the current image or the current object.

p0011The inventive method, which is carried out by means of the inventive device is used for detection and tracking of objects in a detection area. This can basically be stationary or moving, which is particularly the case may be, when the images are captured by a sensor attached to a vehicle.

p0012The images are acquired by at least one sensor for electromagnetic radiation. The sensor can be a purely passive sensor which receives only electromagnetic radiation. However, it may also be a sensor may be used, the directed across a source of particular radiated electromagnetic radiation which can be reflected from an illuminated by the radiation spot or zone on an object, and at least one corresponding to from an object point or range reflected radiation-sensitive sensor element has. Furthermore, combinations of sensors for electromagnetic radiation can be used which detect different types of images. The detection ranges of the sensors need then not necessarily overlap, but it is preferred that this is at least partially overlap.

p0013Under an image is understood to mean in particular a set of data, each representing the position of a point or area on one or more detected by the sensor object in at least one direction transverse to a direction of view of the sensor and at least one property of the object point or region , The location of the object point can, for example, upon detection of object points in a scan plane by an angle relative to an arbitrary but fixed predetermined reference direction, for example, the sensor may be given. Upon detection of object points or region in space, the layers can by layers of sensor elements of the sensor and possibly be given imaging properties. In the property may be, for example, the intensity and / or wavelength or color of the emitted optical radiation, the distance from the sensor or the speed in the radial direction relative to the sensor. For example, video images data to the intensity of radiated optical radiation as a property, and in respect of a sensor element which has detected the radiation of an object point or region, and thus in respect to a location, while distance images as properties of the detected object points or regions the distance from the sensor and as a position information angle at which the object point or region has been detected, may contain. Speed ​​images eventually contain data about the speed of the object points or ranges relative to the sensor.

p0014The images need not be detected directly by a single sensor. Rather, it is also possible, at least two temporally associated with each other images that have been recorded by at least two sensors for electromagnetic radiation to combine to form an overall image that represents a larger detection area or additional characteristics of object points or regions. For example, from two images that were recorded by two mutually spaced video cameras, distance images are formed in accordance with the imaging properties and the arrangement of the video cameras, which in addition to a distance of an object point or -area of one relative to de and n video cameras fixed reference point comprise data related to the intensity or color of each of the object point or range emitted light.

p0015Furthermore, different types can be combined images to a total image or processed separately or together as a function. Thus, preferably video and distance images can be processed together without being grouped together in a narrow sense to a set of image data.

p0016The images used in the inventive method are recorded at successive times, which preferably have a constant interval from each other. The sensor of the device according to the invention is therefore ready for repeatedly capturing images of the detection region. If for one cycle multiple images are used, they are preferably synchronously detected, ie their sensing times differ by a smaller period of time than that between successive scans by the sensor with the lowest detection rate substantially.

p0017The captured images are then analyzed in consecutive cycles, whereby in a given cycle, at least one corresponding image is evaluated. If multiple images are synchronously acquired substantially sufficient that one of the images is at least provisionally evaluated.

p0018In each case, at least wanted a aufgefundenes in a cycle based on at least a corresponding image object in a later cycle in a corresponding later image to trace it. For this purpose, conventional object recognition and -verfolgungsverfahren can be used in particular. In this case can be effected an allocation of pixels of the current cycle to recognized in a previous cycle or image objects.

p0019In particular, a prediction method may be used, in which on the basis of the position of at least one object in an earlier, in particular in a preceding cycle and the speed of a location in the current cycle is predicted and is searched in the vicinity of the predicted location for the object. New objects can be made of parts of an image, ie pixels or calculated in the cycle amounts of pixels, in particular segments are formed which could not be attributed to the already known objects.

p0020According to the invention it is now deviating from conventional object recognition and -verfolgungsverfahren provided that during at least one current cycle on the basis of the results of an at least preliminary evaluation of a current image of a detected at least one determined in the current cycle part of a current image in the current cycle or object is associated with at least a part of an earlier image and / or at least one indication with respect to an earlier state of the object or of an appropriate object, which is determined on the basis of a corresponding previous image.

p0021Are thereby more substantially synchronously captured images the same or different type is used, it is sufficient that a preliminary evaluation of these pictures is performed, the assignment can be parts of the same or another take place substantially synchronously captured image.

p0022This means that in a current cycle is not only a current image is evaluated. Rather, initially at least provisional or optionally partially evaluating a current image in the current cycle is performed. On the basis of these at least preliminary evaluation, ie in particular in dependence on the result of the at least preliminary evaluation, then a previous image for object tracking and determination of data relating to an object employed in which it in particular, a current immediately prior to the image captured image that is associated with a preceding cycle, can act.

p0023In the preliminary analysis can be particularly at least one criterion to be checked, reference to which it is decided whether an earlier image to be used or not. This means then that non-fulfillment of the criterion one else can take place known object recognition and tracking.

p0024In the at least preliminary evaluation of a part of a current image is determined, which may be, be a pixel or determined in the current cycle set of pixels, for example, at least one segment. Where in a current cycle more current images used while the image that has been preliminarily evaluated, not necessarily coincide with the image, a portion of which is determined needs. However, it can be detected and used as an object in the current image. Both the determination of the portion of the image and the recognition of the object can initially take place with process steps similar to those in conventional methods.

p0025The figure recorded in the at least preliminary evaluation part of the current picture or the object in the current cycle can be a part of the previous image are assigned, which is in turn a one pixel or one determined in the current cycle set of pixels, for example, at least can act segment, the previous image. The parts of the images may each have different numbers of pixels, need not be so that there is no one-to-one correspondence of image points given. For any first and / or second objects and / or parts of the first and / or second images is in this case understood to mean an allocation of a portion of a first image or a first object to a part of a second image or a second object such that the portion of the second image or the second object is determined, and for an appropriate part of a first part or a suitable first object is optionally selected and assigned to several alternatives.

p0026However, the portion of the current picture or the object in the current cycle can also be an indication be assigned, which refers to the object or a name corresponding object. This entry concerns may be values ​​of any state variables of an object or an object, such as position, velocity, shape, size, allocation to one of several object classes, according to their relevance to the object tracking typical characteristics are classified in the objects, or the presence of the object at all. To determine the indication this need not be the entire previous image used instead, the investigation may also be restricted to partial areas. However, for the determination always directly at least one pixel of the previous image, which can be preferably to have saved during the current cycle, and simultaneously uses the result of the at least preliminary evaluation of the current image.

p0027In any case, it is essential that a part of the earlier image or of an indication derived from the previous image are assigned to a part of the current image or object of the present cycle and not parts of the current image objects or information, the image from the former were recovered.

p0028This means a total of that for object tracking, depending on the preliminary evaluation information from a current image to a renewed and improved at least partial re-evaluation of the previous image are used, the results are in turn used in the current cycle. This information contained in the images can be better evaluated, which allows faster and more accurate object recognition and tracking.

p0029The data obtained by the method can then be output or stored to be used for example by the following means, for controlling a vehicle.

p0030To carry out the individual method steps of a data processing device is provided in the apparatus according to the invention, which is connected to the sensor for electromagnetic radiation for the transmission of images. The data processing device can be completely or partially as a non-programmable circuit, which increases the speed of execution. Preferably, however, the data processing device comprises a programmable processor for implementing the method according to the invention and, associated with memory, an output interface and an input to the sensor sections.

p0031In particular, means for evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and / or for the association of parts of an earlier image or means for determining comprises at least an indication relating to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and allocation of the indication on the part of the current image or the current object can be provided, in whole or in part, as a non-programmable electrical circuit can be formed, however, by a suitably programmed processor may be formed, preferably.

p0032Further developments and preferred embodiments of the invention are the drawings and the claims described in the specification.

p0033The sensor is a sensor for any electromagnetic radiation can be used. In a preferred embodiment of the device according to the invention is at least as a sensor, a radar sensor, in particular a spatially resolving radar sensor used. Accordingly, in the present process preferably spatially resolved images of a radar sensor are used. Radar sensors can detect particular radial velocities of objects directly relative to the radar sensor.

p0034Preferably, however, at least one sensor for optical radiation, ie, radiation in infrared, visible or ultraviolet region of the electromagnetic spectrum can be used. Such sensors generally have a better spatial resolution than radar sensors.

p0035Sensors for detecting distance images are particularly preferably used. For example, can be used with at least two video cameras, a stereo video camera system, which are arranged spaced apart in a plane in which a distance is detected by the stereo video camera system. Preferably, however, sensors are used in which at least one scanning beam is used for scanning with electromagnetic radiation at least one scanning plane. A plurality of scanning beams for substantially simultaneous scanning of strips can be used in a scan plane, or it may, preferably, a scanning are pivoted in the scanning plane, wherein in each case reflected back from objects radiation having a corresponding detector is received. In particular, appropriate laser scanner can be used by those with angular resolution at least one pulsed laser radiation beam and a sensing range pivotable from an object back radiation of the pulsed laser radiation beam can be received and based on the duration of a pulse of laser radiation beam to the object and back to the laser scanner a distance of the object can be determined , Such laser scanners are characterized by a good spatial resolution in conjunction with high detection rate. Particularly preferred laser scanners are used, which scan more like a fan superimposed scanning planes.

p0036Furthermore, it is preferred that in the inventive process video images are used, and that the inventive device for capturing video images having a video system which may comprise at least one video camera which an imaging optical system and a sensor system for spatially resolved reception of optical radiation, for example visible light and / or infrared radiation has. As a video camera can be used depending on the application, in particular a wide-angle angular or a panoramic camera. For monitoring of the far region, a video camera with a telephoto lens is particularly suitable. It can be used in principle, video cameras with zoom lenses. Furthermore, can be used as a sensor system, a black-and-white or gray scale system or a color-sensitive system. In this example, corresponding CCD or CMOS sensor elements can be used.

p0037Furthermore, combinations of sensors for electromagnetic radiation can be used to capture the same or different types of images. The detection ranges of the sensors need not be identical, it is sufficient that this overlap in the monitoring area.

p0038In order to improve object detection and tracking in the current cycle, it is preferred that is used to assign the part of the earlier image or of the indication with respect to the earlier state of the object to determine a further indication with respect to the current state of an object , The determined additional disclosure may be not to refer to the object for which the indication was determined with reference to its previous state. Rather, it may be any object of the current cycle. Thus, there is effectively a least partially recursively working process in which, starting from information from the at least preliminary evaluation of a current image first new information about an object or an object is obtained at a time corresponding to a previous image, in turn, to evaluate the the current image is used. Thus, the existing information is more effectively utilized in the pictures than in purely linear working method.

p0039Particularly in the treatment of newly detected objects, it is preferable that, when a new object is detected in a current cycle, a current location of the new object is determined in the current cycle, that based on the current location of the new object a previous location the new object in an earlier cycle is estimated and that the current value of a state variable of the new object is determined in the current cycle using the estimated earlier position of the new object. In particular, it can be estimated as a state variable, the velocity of the object, on the basis only of an image can not be determined otherwise.

p0040Considering that an object was initially not recognized in the previous cycle, a number of reasons may be. In one development of the inventive method it is preferable that it is checked based on the current location of the new object in the current cycle and a previous location of at least one other object in a previous cycle, whether in the evaluated in the previous cycle image corresponding to the new object object could be obscured by another object in the earlier cycle corresponding object. Under masking is understood to mean that the object points of the hidden object from the sensor used were undetectable, since areas other object were placed in the propagation path of the electromagnetic radiation from the object points to the sensor. This further facilitates the one occlusion detection in the previous cycle and allowed by estimating the size and position of the hidden portion of the detection range to estimate the location of a corresponding new object object at the time of the previous image.

p0041It is particularly preferred that for the new object in the current cycle, a current size and / or shape is determined, and that the current size and / or shape may be used in the check for masking. This can for example be ensured that a masking is only detected when a detectable by the sensor or a silhouette detectable by the sensor outline of the new object or of the corresponding object from the point of view of the sensor is smaller than the corresponding outline or corresponding silhouette of the other object. Moreover, the size and / or shape of the new object serve this initially classified into one of several classes of objects, each of which includes objects with characteristic for object recognition and tracking properties. For example, object classes for pedestrians, motor vehicles, road boundaries or the like can be provided. Depending on the object classes can then be estimated, whether the object in the available time between the detection of the current image and the previous image with the maximum value possible for objects of the relevant object class rate from a concealed by the other subject area detected in a position might have moved.

p0042From those obtained by the evaluation of the current image in the current cycle information that a new object exists in the detection area, that can be detected through the occlusion check whether this new object corresponding subject already to a previous time in the detection range of the sensor was present. In particular in this case, it is preferable that assuming a concealment of the current situation and preferably the current size and / or shape of the corresponding to the occluding object object an earlier position of the corresponding new object object in the earlier cycle and this results in a current speed the new object is estimated. The obtained in the current cycle information on the new object in the earlier cycle can thus be used to estimate its speed as a state variable in the current cycle of both the direction and in terms of size. This is for example particularly useful to recognize from parked cars undercover pedestrians emerge suddenly behind the car quickly and then possibly also to be able to respond.

p0043However, in an actual image newly discovered object could not have been recognized for a reason other than a concealment in the previous cycle. Thus, it is preferable that is wanted in an earlier image to the new object, and that, in finding the object in the previous image in the current cycle, a current value of at least one state variable of the object and / or other object using a past value of the state variable is determined for the new object and / or the other object in the earlier cycle. The search can be facilitated on the one hand, that the object's properties, such as its size and / or shape or outline of the current cycle are already known. On the other hand can be used that in the previous image, the new object must have been present, so that in the evaluation of the previous image in the earlier cycle optionally dispelled existing uncertainties in the allocation of portions of an image, for example, pixels or segments to objects can be.

p0044Such uncertainty may occur in particular when in the previous image is actually two different objects associated object points or regions as an object to be recognized as belonging. the new object finds in the previous picture again, so information can be both on the new object or the name corresponding an object and a subject of the other corresponding other, are now obtained for the previous cycle newly given object. In particular, the positions of which and, where appropriate sizes and / or shapes can be determined for the new object and the other object as a state variable in the previous cycle. Using appropriate data for the current cycle can then, for one for the in the current cycle at first new object already a speed are determined, and on the other for the other object, which corresponds to the other object in the previous cycle, a more accurate speed can be determined.

p0045By using the method according to this embodiment, for example, pedestrians can be detected particularly easily, who stayed out of view of the sensor immediately before an object at the roadside and then stepped on the road. Immediately with recognition of the pedestrian in a current image can then by the time tracing its speed both in terms of size as well as their direction can be estimated, so that valuable time can be obtained in detecting the pedestrian.

p0046The search for the new object in the earlier image can be done in different ways. In one development of the inventive method, it is preferable that the images are segmented by a predetermined segmentation method and at least one corresponding predetermined segmentation parameters, and that the finding of the new object in the earlier image at least, and a corresponding region of the former image with another segmentation method is segmented again / or a modified segmentation parameters. As a result, can be carried out a new allocation of the newly determined segments to corresponding objects. The new segmentation need not be carried out for the entire image, but may preferably be only on the location of the new object in the current cycle and appropriate, nearby other objects depending on their speeds extend certain area, so that the detection of other objects is not impaired. Both the segmentation process used and the possibly changed segmentation parameters can be adjusted, in particular situational, ie be selected depending on the locations, types and sizes, and / or the number of pixels corresponding to the new object and the other object.

p0047In a preferred development of the method in time mutually assigned distance and video images are used, which are arranged spaced from one another, for example by a laser scanner for the acquisition of range images and a video camera for capturing video images or through a video system for generating stereo images with at least two, each coupled video cameras can be recorded. For example, the video image processing is controlled to reduce the processing burden by evaluating corresponding distance images, ie there is a attention control of the video image processing by the laser scanner is only in those areas of the video image according to features and / or objects sought, include sections where in the corresponding distance image object points or regions corresponding pixels were detected. In particular, the regions may be formed by extending at right angles in a direction perpendicular to a scanning strip whose interface is defined by the detection range of the sensor for detecting the distance images in its position and width by at least one distance picture element or more distance picture elements. An object is only detected in the video image when it enters the detection range of the sensor for detecting the distance images. For example, to detect the distance images using a laser scanner which scans a level of coverage by means of a pivoted in a scanning laser beam, an object in a video image can be seen, although this is not yet in the distance image, for example due to obstruction at the level of the scanning plane can be seen.

p0048Therefore, it is preferable that time mutually assigned removal and video images are at least used a common portion of the coverage that is wanted in video images for objects and / or features only in sections, which are determined as a function of time associated distance picture elements that in a current cycle upon detection of a new object and / or feature of the new object and / or feature is traced in the video image in time to an earlier cycle, and that a position of corresponding to the new object and / or feature of the object in the previous cycle using the information on an object corresponding to the object and / or feature is determined in the earlier video image. The time allocated to each other distance and video images used are preferably synchronously acquired substantially, ie their sensing times differ by a smaller period of time than that between successive scans by the sensor with the lowest detection rate. The sections may, in particular through spaces of predetermined size and shape to the distance pixels, preferably orthogonal, added to a scanning plane, the aforementioned strips. For removing pixels in the range image ie a corresponding feature or an entire object can be searched in the video image in the corresponding section of the video image according to at least. Using only features that traceability is usually carried out more quickly. The determination of the position of corresponding to the new object or feature article in the previous cycle can in particular be carried out by an estimate of the position of the corresponding feature and / or object is preferably based in the video image. Furthermore, the speed in the current cycle can be determined from the position of the corresponding to the new object or feature article in the previous cycle and the location of the new object or feature in the current cycle. In this process variant can be particularly utilized that often have objects may be identified by a video camera when the usually only in the amount less extensive, that reflect shallower detection range, not, for example, by masking at the level in the range images, the sensing range for Entfemungsbilder recognizable. If an object in the distance image for the first time on, it is possible to detect a corresponding feature this in the previous cycle in the video image when the subject was covered only in the range image, but not in the video image. A velocity of the object can then be determined on the basis of the location of the feature in the video images. The occurrence of "blind spots" of the video image processing, which are caused by the attention control, can thus be reduced without considerable effort.

p0049In a process other time mutually assigned distance and video images are to be used at least a common portion of the detection range distance picture elements in an earlier distance image distance picture elements in a current distance image set by for at least one a distance picture element in the earlier distance image corresponding region and / or at least one a distance picture element in the former distance image corresponding characteristic of the corresponding previous video image and / or of the current video image, a displacement and / or shifting speed, in particular, an optical flow is determined, and, the displacement or displacement velocity, in particular the optical flow, the assignment of a distance picture element in the current distance image to the distance picture element in the earlier distance image used wherein for objects in the Entternungsbild, approaching a distance image capturing sensor and / or in a current cycle longer distance picture elements have as in a previous cycle, distance picture elements in the current cycle using the corresponding displacement or displacement velocity of the range or feature, in particular the corresponding optical flow, are assigned in the previous cycle distance picture elements. In the case of this other method. linked information from distance images with information from corresponding video images, thereby further data is obtained for at least one distance picture element which can be used in the processing of the distance picture elements. It is particularly exploited that with a movement of an object is usually in the range image and the video image captured areas are moved to the object. Assign the areas in the video image an intensity curve or corresponding features, the movement of the object is expressed approximately as a shift of the corresponding intensity characteristics or features in the video image. Under a shift is in this case a displacement between successive cycles understood but not necessarily need to directly follow one another. The movement information from the video images can then be assigned to the appropriate areas of the article or the corresponding distance picture elements.

p0050Preferably, a corresponding optical flow is determined for the determination of displacement or speed of displacement of the range. In the area used to define the optical flow on the one hand, an image plane are used in the video pixels of then also defined in the plane video image are. Preferably, however, one surface is used, which is used in a camera model for the treatment of a video image. In principle, it is sufficient if the optical flow can be determined for at least one distance picture element. In general, the case can occur namely that distance picture elements are recorded on subject areas which do not have intensity and structure can not be determined for which therefore also an optical flow. The optical flow is for example completely determined when the corresponding area of ​​the video image having a change of intensity in two linearly independent directions. Preferably, a corresponding tracking therefore applies to these areas.

p0051Method for calculating the optical flow are generally known and, for example, in the article "<nplcit id="ncit0001" npl-type="s"><text>. The computation of optical flow "by JL Barren and SS Beauchemin in ACM Computing Survey, Vol 27, No. 3, pp 433 -. 467 (1995)</text></nplcit> and in the article "<nplcit id="ncit0002" npl-type="s"><text>Performance of optical flow techniques "by JL Barren, DJ Flied, DJ and SS Beauchemin in International Journal of Computer Vision, 12 (1), pp 43-77 (1994)</text></nplcit> described.

p0052The optical flow needs while not necessarily be computed for the entire video image. It is sufficient that for the distance picture element corresponding object point or - if necessary - to determine its environment. The place in which used to define the optical flow area for which the optical flow has to be calculated, is from the position coordinates of the distance picture element, the relative position of the sensor used to detect the distance images to that used for capturing video images video system, the imaging geometry of the video system or a video camera therein and a model for this and the predetermined surface on which the optical flow is to be determined, derived. It can be used as a place, a corresponding video image point, however, it is also possible to subpixel accuracy to determine the optical flow at locations between video picture points or pixels of the video image. Keeping track of distance picture elements with the aid of the optical flow, distance picture elements are allocated in a later cycle in removing pixels in a previous cycle, is filed by the applicant German patent application with the official file reference<patcit id="pcit0002" dnum="DE10312249"><text>DE 103 12 249</text></patcit> described. There distance images are called depth-resolved images.

p0053Therefore, this used to be from the sensor removing objects or objects with a not greater number of distance picture elements in the current cycle process has the advantage that a prediction of the object layers need not be done because the use of the shift or the optical flow tracking individual Distance pixels allowed. The reverse procedure when approaching objects or objects that have in the current cycle longer distance picture elements than in a previous cycle has, on the one hand the advantage that an allocation of a greater number of distance picture elements to a smaller number of distance picture elements, in particular in situations where objects are in the current cycle closely adjacent to each other, can be carried out correctly with greater certainty. Secondly, the corresponding object more accurate speed are assigned because that detects the speed of the object, an averaging over corresponding speeds of associated distance picture elements can be used and in the current cycle, a larger number of such distance picture elements is available.

p0054In another development of the inventive method, it is preferable that the object recognition and tracking is carried out on the basis of video images, and that when the object detection, the detection of objects is carried out on the basis of filtered video images that comprise the filtering a reduced resolution, and that after detection of an object in the current cycle based on the location and optionally shape of the object is searched in the current filtered video image to an object corresponding to the object in an earlier video image whose resolution is higher than that of the filtered video image, and that respective data be used via the object in the evaluation of the current video frame at finding a corresponding object in the earlier video image. The object recognition and tracking can be optionally supported by the use of distance images. Video images In this development, mainly in order to save computational effort, initially detected subjected to reduction in the resolution of a filtering. When filtering, it may be any method in which the number of pixels of the video image is reduced. In the simplest case, for example, just a sub-sampling of the image are performed at the of square blocks of pixels, or pixels, each only a pixel or pixels are referred to by the whole assembly. a far object approaches the sensor used for detection of the video image, it may happen that at the first detection of the object very few pixels or pixels are in the filtered video image available. By analyzing the previous image with increased resolution, can optionally the object that could not be detected in the previous image only due to filtering, now be recognized, so that the newly discovered object in the current cycle despite the only low resolution a trajectory and speed can be allocated without requiring a considerable computational effort would be necessary.

p0055The invention is also a computer program with program code means to carry out the inventive method when the program is executed on a computer.

p0056Another subject of the invention is also a computer program product with program code means stored on a computer readable data carrier to carry out the inventive method when the computer program product runs on a computer.

p0057At a computer In this case, any data processing device, in particular a data processing means of the device according to the invention, understood, with which the method can be performed. In particular, this may include a digital signal processor and / or microprocessor exhibit, with which the process is carried out in whole or in parts.

p0058The invention is preferably provided with movable objects, in particular road traffic, suitable for monitoring of areas, wherein the acquisition of the images by at least can be carried out a fixed-position sensor and / or one held on a vehicle sensor.

p0059The invention will now be further illustrated by way of example with reference to the drawings. Show it:<dl id="dl0001"><dt>Fig. 1</dt><dd>a schematic plan view of a vehicle with a device for the recognition and tracking of objects according to a first preferred embodiment of the invention and a vehicle located in front of the object,</dd><dt>FIG. 2</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a first preferred embodiment of the invention,</dd><dt>FIGS. 3A and 3B</dt><dd>Cutouts detected from at successive points in time distance images of a detection range of a laser scanner of the apparatus in <figref idrefs="f0001">Fig. 1</figref> with a vehicle only partially shown and a moving pedestrian, </dd><dt>Fig. 4</dt><dd>the presentation in <figref idrefs="f0003">Fig. 3B</figref> with additional guides for explaining a calculation of the position or speed of the pedestrian,</dd><dt>Fig. 5</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a second preferred embodiment of the invention,</dd><dt>Figs. 6A to 6G</dt><dd>Excerpts from time sequentially sensed distance images of scenes with a vehicle and a at times ahead of the vehicle is only partially shown, moving pedestrians,</dd><dt>Fig. 7</dt><dd>a schematic plan view of a vehicle with a device for the recognition and tracking of objects according to a second preferred embodiment of the invention and a vehicle located in front of the object,</dd><dt>Fig. 8</dt><dd>a schematic, partial side view of the vehicle and the object in <figref idrefs="f0007">Fig. 7</figref>.</dd><dt>Fig. 9</dt><dd>a flow diagram is illustrated in the schematic of the flow of a method according to a third preferred embodiment of the invention;</dd><dt>FIG. 10A and 10B</dt><dd>Excerpts from a distance image and a corresponding video image with a vehicle and a moving pedestrian at a first detection time,</dd><dt>FIG. 11A and 11B</dt><dd>the images in the <figref idrefs="f0010">FIG. 10A and 10B</figref> corresponding of distance or video images at a later time of data,</dd><dt>FIG. 12A and 12B</dt><dd>a flow diagram is illustrated in the schematic of the flow of another method,</dd><dt>Fig. 13</dt><dd>a schematic, perspective view of a portion of a scan plane with a distance picture element, an image geometry for the calculation of an optical flow and at the end of the procedure in <figref idrefs="f0012">Fig. 12A</figref> and <figref idrefs="f0013">12B</figref> occurring layers of recorded and predicted distance pixels and video pixels and a corresponding one optical flow displacement vector,</dd><dt>Fig. 14</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a fourth preferred embodiment of the invention,</dd><dt>FIG. 15A, 15B</dt><dd>Excerpts from an unfiltered and a filtered video image with a schematically represented pedestrian at a first time, and </dd><dt>Fig. 16</dt><dd>an unfiltered image of the pedestrian in <figref idrefs="f0016">Fig. 15</figref> at an earlier time.</dd></dl>

p0060In <figref idrefs="f0001">Fig. 1</figref> carries a vehicle 10 on its front side a laser scanner 12 as well as a connected via a data line with the laser scanner 12 data processing means 14, which together with the laser scanner 12 is a device for the recognition and tracking of objects according to a first preferred embodiment of the invention. In the direction of travel in front of the vehicle is one in<figref idrefs="f0001">Fig. 1</figref> Person only very schematically shown 16 that is half viewed in the context of the invention for simplicity as an object.

p0061By means of the laser scanner 12 is in <figref idrefs="f0001">Fig. 1</figref> only in partial detection range 18 scanned, the 12 is positioned due to the mounting position of the laser scanner at the front of the vehicle 10 symmetrically to the longitudinal axis of the vehicle 10 and covers an angle of slightly more than 180 °. The detection range 18 is in<figref idrefs="f0001">Fig. 1</figref> only shown schematically and too small for better representation in particular in the radial direction.

p0062The laser scanner 12 scans its detection range 18 in basically known manner with a constant angular velocity circulating, pulsed laser radiation bundle 20 from which also τ circumferentially at constant intervals .DELTA.t at times_i α in fixed angular ranges around a mean angle_i it is detected whether the laser beam radiation 20 is of a point or region 22 of an object, such as the person 16, reflected. The index i runs here from 1 to the number of angular regions in the detection area 18. From this angle ranges is in<figref idrefs="f0001">Fig. 1</figref> shown only an angular range of the central angle α_i assigned. Here, the angular range is, however, shown for clarity in an exaggeratedly large.

p0063Due to the pivoting of the laser radiation beam 20 to the detection range 18 on the expansion of the laser radiation beam 20 is substantially two-dimensional and substantially forms, ie to the diameter of the laser radiation beam 20, a scanning plane.

p0064With the maturity of the votes of the laser scanner 12 laser radiation pulse of the laser scanner 12 to the object point 22 and back to the laser scanner 12, the distance d_i of the object point 22 is determined by the laser scanner 12th Therefore, the laser scanner 12 recorded as coordinates in a the object point 22 of the object or person 16 corresponding distance picture element the angle α_i and those established in this angular distance d_i, Ie the position of the object point 22 in polar coordinates. Each detected object point is therefore associated with a distance picture element.

p0065The amount of in which a scan detected distance image points forming a distance image in the sense of the present application.

p0066The laser scanner 12 scans the detection area 18 are each in successive scans with .DELTA.t intervals, so that a time sequence of samples and corresponding distance images. The processing of the distance images of the laser scanner 12 is performed by the data processing device 14th

p0067The data processing device 14 has to, inter alia, a for performing the method according to the invention programmed with a corresponding inventive computer program and a digital signal processor coupled to the digital signal processor memory device. In another embodiment of the device according to the invention the data processing device may also include a conventional processor to which a program stored in the data processing device according to the invention a computer program for executing the method according to the invention is carried out.

p0068The programmed data processing means is here in the sense of the invention means for the evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and allocation of the indication on the part of the current image or the current object ready.

p0069On the basis of the acquired of the laser scanner 12, the distance images in <figref idrefs="f0002">FIG. 2</figref> illustrated process is carried out according to a first preferred embodiment of the invention.

p0070In successive cycles respectively the steps S 10 to S28 are performed.

p0071First, a range image is in a current cycle in step S 10 by a scan of the detection area 18 detected and read into a memory in the data processing device 14th

p0072In step S 10 while pre-processing the distance image data is carried out, is conducted at the, possibly after correction of the data, a transformation of the position coordinates of the distance picture elements into a Cartesian, firmly connected to the vehicle 10 vehicle coordinate system.

p0073In step S 12, the distance image is then segmented. Here are formed in a conventional manner amounts of distance picture elements, which are characterized in that each distance pixel an amount of at least one other distance pixel same amount has a mean square distance which is less than a predetermined segmentation distance. A lot thus formed corresponds to one segment. In this case, a segment may also be formed by a single range image point having from all other distance picture elements of the current image a distance mean square distance which is greater than the predetermined segmentation interval.

p0074In step S 14, which is not performed in the very first cycle of the process, then an assignment of segments already identified in a previous cycle of the process objects, including well-known methods can be used per se. In this process each one predicted in step S28 of the previous cycle object location is used for each object known in the preceding cycle. In the current cycle a segment of the current cycle is associated with a respective object of the previous cycle when at least one of the distance picture elements of the segment of the object in the predicted object position at a distance which is smaller than one of an uncertainty of the prediction and the size and orientation of the object in the previous cycle dependent maximum distance.

p0075In step S 16 are then of segments that could be assigned to any known from the previous cycle objects, formed new items and determined from the positions of the segments constituting distance picture elements a location of the object in the current cycle.

p0076The above described steps S 10 to S 16 and the yet to be described steps S26 and S28 are not different from conventional object recognition and -verfolgungsverfahren.

p0077In contrast to these, however, in step S 18 after an examination of whether new objects were found, checks whether a an object newly formed corresponding object in a preceding distance image or cycle optionally detected in the previous cycle by a one in the previous cycle could be obscured object corresponding object.

p0078The processing of the current distance image in the steps S 10 to S 16 including the check in step S 18, whether a new object has been found is a least preliminary evaluation of the current image.

p0079For occlusion detection in the preceding distance image is first checked, which in the previous image detected objects in this exhibit is formed by a smooth curve through the corresponding distance picture elements a contour which is greater than that determined by the corresponding distance picture elements expansion of the new object in the current cycle.

p0080This is exemplified in the <figref idrefs="f0003">FIGS. 3A and 3B</figref> shown in which are shown in excerpts from successive distance images, a vehicle 24, which is only partially represented by the black solid line, and symbolized by a rectangle 26 pedestrians, the v moves at a speed relative to the vehicle 24th The individual dots indicate distance picture elements of the respective distance images again. In the<figref idrefs="f0003">FIGS. 3A and 3B</figref> is the laser scanner 12 at the origin, ie at the point (0,0), the coordinate axes of the Cartesian coordinate system are divided into arbitrary but fixed chosen distance units.

p0081In <figref idrefs="f0003">Fig. 3A</figref>, A preceding distance image, the pedestrian 26 is concealed from view of the laser scanner 12 of the vehicle 24 and therefore can not be detected in the range image. In<figref idrefs="f0003">Fig. 3B</figref>In which the current distance image is shown, the pedestrian has moved away 24 behind the vehicle 24 and is now of the laser scanner 12 first detected. The contours of the articles 24 and 26 corresponding objects are in<figref idrefs="f0003">FIGS. 3A and 3B</figref> each represented as a dashed line regression curve through the distance picture elements. It is easy to see that in<figref idrefs="f0003">Fig. 3B</figref> the contour of the vehicle 24 corresponding object is greater than that of the pedestrian 26 corresponding object.

p0082The further masking recognition is carried out preferably situationsadaptiv, ie. A function of the speeds of the already detected objects in the previous cycle In the example in the<figref idrefs="f0003">FIGS. 3A and 3B</figref> it is known that the vehicle 24 is resting, and that since no other object in the immediate vicinity of the newly detected, the pedestrian 26 corresponding object can be found or any other object having a smaller distance from it and the pedestrian corresponding object 26 due sizes can not match a very fast moving object, the object or pedestrian 26 was covered in the previous image with very high probability by the vehicle 24th

p0083In step S20, the location of the preceding distance image hidden, new objects corresponding objects as an indication relating to an earlier state of the respective new objects will be appreciated in the context of the invention. To the example of<figref idrefs="f0003">FIGS. 3A and 3B</figref> to estimate a minimum speed of the object or pedestrian 26, is used to estimate the situation in the preceding distance image in <figref idrefs="f0003">Fig. 3A</figref> Assume that the object or pedestrian 26 was just not visible to the detection time of the removal of the image, so that this object box surrounding in the current distance image whose size in the process for objects of type pedestrian is given, with its edge just not circulating of the laser radiation beam could be detected in 20th In this way, the results in<figref idrefs="f0003">Fig. 4</figref> represented by the dotted rectangle drawn estimated position of the pedestrian 26 in the preceding distance image or cycle, which is displaced by the displacement vector d compared with the situation in the current distance image or cycle.

p0084In step S22, the speed of the objects is then estimated by the change in position of an object in the previous image and the current image is determined, wherein for the new objects, as far as they were covered, the estimated position is used. The estimated speed of the objects is then obtained both in terms of the direction and the magnitude of the velocity by dividing the displacement vector given by the d position change by the time period At between successive scans of the detection area 18th

p0085While for already identified in the previous cycle discovered objects speeds can be expected only a slight error due to the present measurement data, results for new objects only one afflicted with a larger error estimate. This estimate provides faster but much more and especially security-related information than would be possible without recourse to the previous image.

p0086In step S24, then the determined object positions and sizes and the current distance image that has to be the then preceding distance image available in the following cycle is stored, the previous distance image is deleted.

p0087In step S26, the object positions and speeds are output to appropriate further processing facilities.

p0088In the step S28 is performed at the end of the current cycle, a prediction of new object positions for the following cycle. The prediction of the location as well as the determination of the uncertainty of the prediction can be carried out for example in a known manner by means of a Kalman filter.

p0089In that the new object or object in the current cycle is already a rate can be assigned to the initialization of the Kalman filter in this object with considerably more accurate data can be performed, which significantly facilitates the following object recognition and tracking.

p0090Then the next cycle with step S 10 can begin.

p0091A method for the recognition and tracking of objects according to a second preferred embodiment of the invention is shown schematically in the flow chart in <figref idrefs="f0004">Fig. 5</figref> shown. It differs from the method according to the first embodiment in that in a current cycle newly discovered objects are treated differently, so that the steps S 10 to S 16 and S22 to S28 are the same as in the above-described embodiment, and the explanations there correspondingly apply here. Accordingly, the apparatus for tracking and recognition of objects with the method according to the second preferred embodiment of the invention over the corresponding device in the first embodiment is modified only in that the worked off in the data processing device 14 program is modified accordingly, that is, in particular, that the means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assigning the item to the part of the current image or the current object modified are.

p0092In the process S 10 is whether any found to S 16 after the formation of new objects from previously assigned segments in step S16 in step S30, after review of the criteria in the current new items after the running as in the first embodiment, steps were , searching for the new objects corresponding objects or objects in the preceding distance image. It is assumed that the new object in the previous cycle or distance image could not be retrieved because it could not be recognized as an independent object due to the segmentation and object formation or segment-object association.

p0093An example of this is in the <figref idrefs="f0005 f0006">Figs. 6A to 6G</figref> illustrated in which excerpts are shown from successive distance images respectively. The laser scanner 12 is placed in the range images in turn at the origin (0,0) of the coordinate system corresponding to the Cartesian coordinate system of the previous embodiment.

p0094In the distance images, a pedestrian 26 moves from the point of view of the laser scanner 12 first immediately before a parked vehicle 24 at a speed v in order in the <figref idrefs="f0006">Fig. 6F and 6G</figref> Finally, as far removed from the vehicle 24, that it can be recognized as a separate object with the normal segmentation process. In the preceding distance images, however, the segmentation corresponding to the vehicle 24 removal pixels not the pedestrian part with the 26 corresponding distance picture elements.

p0095Based on the size of the article 26 and the corresponding new object in the current cycle, which in the example the distance image in <figref idrefs="f0006">Fig. 6F</figref> is evaluated, it is believed that it may have only a low speed, because the object has to be a pedestrian. To search in step S30, therefore the objects are determined for the situation adaptive segmentation, in which the pedestrian 26 corresponding new object in the current, the range image<figref idrefs="f0006">Fig. 6F</figref> corresponding cycle are closest. In our example, only the object corresponding to the vehicle 24. The steps up to this stage corresponds to a least preliminary evaluation of the current distance image in the sense of the invention.

p0096It is a part of the preceding distance image is then re-segmented. More specifically, the distance picture elements of the segments in the preceding distance image or cycle into new segments constituting the respective previously determined for segmentation objects in the previous cycles. In the example, these are all distance picture elements of the detail of the distance image in<figref idrefs="f0005">Fig. 6E</figref>,

p0097Given in the present embodiment, the applied in step S 12 segmentation method, but with a reduced segmentation spacing used. The segmentation spacing can be fixed. Preferably, the segmentation distance but, preferably the smallest, selected depending on the detected size of at least one of the detected objects, so that a situationally adapted segmentation can be done.

p0098there is a change in the number of segments, in step S32, a new allocation of segments is made to those affected by the re-segmentation objects. Here the position and speed in the previous cycle in conjunction with the determined in the current cycle orientation, shape and size as well as for the new object is the determined in the current cycle shape and size used for already known in the previous cycle properties.

p0099In step S34, it is then checked whether new or changed objects have been found in the preceding distance image. this is not the case, the method continues with step S22.

p0100Otherwise, in step S36, a position of the corresponding object to the new object and the modified object are determined in the preceding distance image. The new object is then treated as a known object, its location in the preceding cycle is known, what changes have to be made in data structures of the program and a corresponding Kalman filter for the previous cycle is initiated subsequently. Furthermore it is possible for the altered object a new prediction of the location for the current cycle in order to improve the accuracy of the Kalman filter.

p0101There are therefore in the current cycle object positions for the previous and current cycle for known in the preceding cycle and in step S32, newly discovered properties obtained, so that in step S22 can now be carried out to estimate the velocity of these objects, the object positions in the preceding distance image or cycle can be used.

p0102The subsequent steps S24 to S28 extend as in the first embodiment.

p0103Characterized in that the object or pedestrian 26 in the current cycle is already a speed can be associated with the initialization of the Kalman filter in the current cycle as a result, substantially already on the basis of data for this object from the previous cycle and thus overall with accurate data occur, which substantially facilitates the following object recognition and tracking.

p0104In this way it is possible, for example, one is assigned directly in front of a vehicle is parked moving pedestrian, who then takes to the road, after processing only a distance image in which he was recognized as a single object, a trajectory and hence speed.

p0105Basically, the method of the first two embodiments may be combined. For example, only a check in step S30 carried out and, if no new objects were found in step S34, a masking detection according to the first embodiment.

p0106In the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> is in a situation in <figref idrefs="f0001">Fig. 1</figref> corresponding situation a vehicle 10, which carries a device for the recognition and tracking of objects according to a third preferred embodiment of the invention.

p0107The apparatus for detection and tracking of objects different from the apparatus for the recognition and tracking of objects in the first embodiment in that in addition to the laser scanner 12, a video system 30 is provided which, as the laser scanner 12 also, via a corresponding data connection with a is connected over the data processing device 14 modified in the first embodiment, data processing device 14 '. respectively the same reference numerals are used for the other, like parts or features, and the explanations for the first embodiment apply accordingly.

p0108The video system 28 has a monocular video camera 30, which is to a conventional black and white video camera with a CCD area sensor 32 and an imaging device that in <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> is shown schematically as a simple lens 34, actually , but is composed of a lens system, and reflects incident light from a detection area 36 of the video system to the CCD area sensor 32nd An optical axis 38 of the video camera 30 is in a low, in<figref idrefs="f0008">Fig. 8</figref> exaggeratedly large angle shown to be caused by the pivoting plane of the laser radiation beam 20 scanning 40 of the laser scanner 12 to inclined.

p0109The CCD area sensor 32 includes in a matrix arranged on the photodetection elements, which are read out cyclically to form video images with video pixels. The video images for each pixel contained in each case initially the position of the photodetection elements in the matrix or other identifier for the photodetection elements, and respective one of the intensity of the received from the corresponding photodetection element corresponding light intensity value. The video images are recorded synchronously in this embodiment essentially with the distance images and thus at the same rate, is recognized when the well of the laser scanner 12 range images.

p0110Of an object, in the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> the person 16, outgoing light is imaged by the lens 34 onto the CCD area sensor 32nd This is illustrated in<figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> schematically indicated for the outline of the object or the person is only schematically shown 16 by the short dashed lines.

p0111From the distance of the CCD area sensor 32 and lens 34 and from the position and the imaging properties of the lens 34, for example, the focal length, can be made of the position of an object point, for example, the object point 22 on the person 16 are calculated on which place the CCD -Flächensensors 32 and which is mapped to the arranged as a matrix photodetection elements of the object point. Conversely, from the position of a photodetection element by the short dashed lines in<figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">Fig. 8</figref> an indicated beam are determined, in the object points or regions must lie from which sensed radiation can fall on the photodetection element. For this purpose, a corresponding camera model used. The example is a per se known pinhole model.

p0112A common detection area 42 is in the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> schematically approximately represented by a dotted line and, by the intersection of the detection region 18 of the laser scanner 12 and the detection region 36 of the video system 28th

p0113The data processing device 14 'differs from the data processing device 14 firstly by the fact that interfaces are provided to substantially synchronously, ie one current within a period, which is significantly smaller than the period .DELTA.t between successive samples of the detection region 18 of the laser scanner 12, distance image and a current video image of the video system 28 to read. On the other hand the processor is programmed with a computer program according to a third preferred embodiment of the invention for performing a method according to a third preferred embodiment of the invention. The data processing device 14 'therefore includes means for evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assigning the item to the part of the current image or with the current object in the sense of the invention.

p0114On the basis of of the laser scanner 12 and the video system 28 substantially synchronously detected distance images and video images, the in <figref idrefs="f0009">Fig. 9</figref> illustrated process is carried out according to the third preferred embodiment of the invention. In the method of this embodiment, features are detected and tracked in the video images, but the video image processing of computing time, made on the basis of the distance images, by looking for features only in those portions of the video image in which distance picture elements of the distance image and thus corresponding item ranges.

p0115First, in step S38, a current distance image and a current video image are read and preprocessed. The reading and preprocessing of images can be carried out independently for the two images in parallel, or in any order.

p0116The distance picture elements in the range image is not in addition to the position coordinates in the scanning plane 40 a through the scanning plane 40, in the corresponding to the distance picture elements object points are added certain location component to form a complete position coordinate set in three dimensions. With the position of a distance picture element defined by this coordinate position is referred to below.

p0117Furthermore, for possible correction of the video image, transforms the data of the video image in the vehicle coordinate system in which the distance picture elements are defined. For this purpose, a rectification of the video image data, for example, to eliminate distortions, and a transformation of the video pixels is performed on an image plane. By means of the camera model for the video camera 30 can then be in the vehicle coordinate system assigned to a respective level layers of the video pixels.

p0118The current distance image and the current video image will be saved for further use.

p0119In step S 12 is then, as in the first embodiment, the distance image is segmented.

p0120In the following step S40 features are detected in the video image, said of the video image only ranges are used, in which, according to the distance image, a one segment must be detectable in the distance image corresponding object. More specifically, the detection is carried out only in running essentially perpendicular to the scanning plane of the laser scanner 12, a respective segment of the distance image corresponding stripe of the video image having the distance picture elements in the segments corresponding video pixels. The strips are defined such that they extend orthogonally in a direction perpendicular to the scanning plane 40 of the laser scanner 12 over the entire extent of the video image, and have in the orthogonal direction, a width which is determined so that all distance picture elements in each of a segment corresponding video pixels in each strip are. This video pixels can be determined by use of the camera model. Furthermore, the strip in width at each predetermined edge regions on both sides of each outermost video pixels. Since other parts of the video image are initially ignored, done so far an attention control of the video image processing.

p0121In this step still an assignment in the current video image is performed found dead features known from the previous cycle characteristics using a predicted in step S49 of the previous cycle in each case for the current cycle position of the respective features. In this case, positions of the features in the current cycle are determined simultaneously.

p0122In step S41 it is then checked whether new features were found that could not be assigned to known from the previous cycle characteristics.

p0123Are there new features found are searched in step S42 according to the new features corresponding features in the preceding video image by backtracking. For this purpose, known methods of video image processing can be used.

p0124In step S44, the position of a corresponding new characteristic feature in the preceding video image is then determined.

p0125S46 speeds of characteristics on the basis of the positions of features in the preceding video image and the positions of the features in the current video image then estimated in step.

p0126The determined feature positions and velocities are then output in step S26 after storing for further use.

p0127In step S48, the current video image will be saved after the previous video image has been deleted.

p0128In step S49, new feature layers now be predicted, so that an assignment of features in step S40 will be facilitated in the following cycle to objects. For this purpose, for example, the now well-known speeds or a Kalman filter may be used.

p0129Starting from the at least preliminary evaluation of the range and video image in the steps S38, S 12 and S40 is performed so a subsequent or additional analysis of a preceding video image. The results of this analysis, the located feature or its location in the preceding video image are then used in the current cycle to determine the speed of the current feature.

p0130The use of the method for detecting sudden protruding behind parked vehicles pedestrians is in the <figref idrefs="f0010">FIG. 10A and 10B</figref> and 11A and 11B. The<figref idrefs="f0010">Fig. 10A</figref> and <figref idrefs="f0011">11A</figref> each show a section of consecutive, sensed by the laser scanner 12 distance image at the height of the scanning plane 40, while the <figref idrefs="f0010">Fig. 10B</figref> and <figref idrefs="f0011">11B</figref> show areas of corresponding substantially synchronously acquired with the video system 28 video images. In these Figures, the scanning plane 40 of the laser scanner 12 is shown by a dotted line. Furthermore, it is equivalent to<figref idrefs="f0010">FIG. 10A and 10B</figref> Cartesian coordinate system used in the previous embodiments.

p0131In the <figref idrefs="f0010">Fig. 10A</figref> and <figref idrefs="f0011">11A</figref> the vehicle are shown schematically by a rectangle 44 and the pedestrian by a rectangle 46th

p0132First, is the pedestrian, as in the <figref idrefs="f0010">FIG. 10A and 10B</figref> shown, of the laser scanner 12 can not be detected because it is hidden by the vehicle 44th Since the detection area 36 of the video system 28 but in a direction perpendicular to the scanning plane 40 a larger opening angle than the detection range 18 of the laser scanner 12, can already be seen in the corresponding video image of the pedestrian 46th Due to the attention control this is not detected in the video image, as it in the distance image (see.<figref idrefs="f0010">Fig. 10A</figref>) Is obscured.

p0133Since the pedestrian 46 in the example with a speed v moves to the right, he is no longer covered after some time by the vehicle 44, which in the <figref idrefs="f0011">FIG. 11A and 11B</figref> is shown. Now the pedestrian can be detected 46 in the range image so that it is detected in the video image via the attention control. While in a conventional video image processing nothing would now known about this except the location of the pedestrian 46, is in the process of the embodiment of the pedestrian 46 time traced in the video images, which he in the in<figref idrefs="f0010">Fig. 10B</figref> shown video images can be recognized. The resulting finding made in the previous video image now allows to determine the speed of the first recognized only in the current video image pedestrian 46 even at its first detection or recognition. Since the tracking occurs only when the discovery of new features, the speed of execution of the method is only slightly reduced on average.

p0134In another method for the recognition and tracking of objects in substantially the same apparatus for the recognition and tracking of objects as in the third embodiment is used, however, the programming of the data processing means 14 'is also changed according to the modified method. This also means that the funds for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and assigning the item to the part of the are the current image or the current object modified.

p0135With this in <figref idrefs="f0012">Fig. 12A</figref> and <figref idrefs="f0013">B</figref> schematically illustrated methods are pursued distance picture elements in successively sensed distance images using optical flow in respective areas in substantially synchronously detected with the distance images video images.

p0136Here a mapping of objects and representative thereof distance picture elements is used to one another, so that a tracking of objects can be done by following the corresponding object associated distance picture elements. Regarding details of some steps of the present process and in relation to possible variants of the first embodiment and the other embodiments in the German patent application with the assignee of the present application with the official file reference<patcit id="pcit0003" dnum="DE10312249"><text>DE 103 12 249</text></patcit> referenced.

p0137In step S50, a current cycle, first a current distance image and a current video image are detected and read into the data processing device in which are preprocessed these images corresponding to step S38 in the previous embodiment. The transformation of the video pixels, however, been modified for easier calculation of the optical flow over the step S38 in the previous embodiment, that the transformation of the video pixels on an image plane used to define or calculate an optical flow 48 (see FIG.<figref idrefs="f0014">Fig. 13</figref>) is carried out. For one inspection in steps S52 and S60 projection of distance picture elements in the image plane for simplicity, a principle to the expert uses known modified Pinhole or ground glass model of the video camera 30, represented by the position of an optical center 50 and the image plane 48 is defined, which serves as a surface for defining or determining the optical flow. The location of the optical center 50 is using the imaging geometry of the video camera 30, in particular the position relative to the laser scanner 12 and the focal length of the lens 34 is determined. The pinhole camera model is modified for ease of illustration, that the image plane 48 to a relative to the video system 28 fixed to the vehicle 10 position between the optical center 50 and the object points, in<figref idrefs="f0014">Fig. 13</figref> the points 52 and 58 is, and seen by point reflection of the actual image plane of the optical center 50 of these.

p0138In step S52, which is appropriate in the first cycle of the process, all distance picture elements of the current cycle immediately preceding cycle are initially determined which correspond to 12 objects removed from the sensor or laser scanner. For simplicity, these distance picture elements are referred to as receding objects corresponding distance picture elements. Using the camera model corresponding distance picture elements are then from the current cycle immediately preceding cycle corresponding positions or coordinates of corresponding video image points in the image plane 48 is calculated and stored for use in the current cycle of this receding objects. As in<figref idrefs="f0014">Fig. 13</figref> geometrically illustrates the situation results in each case by the intersection 54 a passing through the distance picture element, in Figure the distance picture element 52, and the optical center 50 degrees with the image plane 48th

p0139In step S54, which is also omitted in the first cycle, are then projected for all, moving away from the laser scanner 12 objects, hereinafter also referred to corresponding distance picture elements of the preceding cycle or corresponding intersections corresponding current optical flow vectors just as optical flows, on the basis of the video image obtained from the immediately preceding cycle and of the current cycle and converted by multiplication with the cycle time or the inverse of the sampling frequency in a displacement vector. In<figref idrefs="f0014">Fig. 13</figref> shown for the distance pixel 52, for at the intersection 54, an optical flux vector is determined, which lies in the image plane 48 and the after scaling with the cycle time a starting position at the intersection 54 displacement vector 56 results. The optical flow is determined according to a differential method, in this example, the in "<nplcit id="ncit0003" npl-type="s"><text>Performance of optical flow techniques "by JL Barren, DJ fleed and SS Beauchemin, International Journal of Computer Vision, 12 (1), pp 43-77 (1994)</text></nplcit> method described by Lukas and Canade.

p0140For all receding objects corresponding distance picture elements for which an optical flow was determined, is then determined in step S56, a predicted by the optical flow position of a corresponding object point in the current cycle in the scan 40th Geometric in<figref idrefs="f0014">Fig. 13</figref> illustrates the intersection is determined to 57 a plane passing through the optical center 50 and the end point of the displacement vector 56 lines with the scanning 40th

p0141In step S58 are current distance picture elements in <figref idrefs="f0014">Fig. 13</figref> assigned for example the current distance picture element 58, the current distance image to the predicted positions and corresponding distance picture elements of the preceding image or cycle where possible. Allocation criteria can be used for example, that the one current distance picture element is assigned, whose squared distance between the predicted position and the actual current situation is minimal compared with those of other current distance picture elements.

p0142In step S60, then the unassigned distance picture elements of the current cycle, which therefore do not have to comply by the laser scanner 12 removed objects, projected onto the image plane 48, which is analogous to the projection in the step S52.

p0143S62 are then for all projected, not yet associated distance picture elements of the current cycle in step calculated analogously to step S54 optical flows.

p0144For all not associated distance picture elements of the current cycle for which an optical flow was determined, a retraced position of a corresponding object point in the scan plane 40 is now determined in step S64. The determination of the retraced position takes place analogous to the determination of the predicted position in step S56, but using as the optical flow vector is a displacement of oppositely directed appropriate vector.

p0145In step S66, then an assignment of distance picture elements of the previous cycle that do not correspond from the sensor removed objects to be predetermined, retraced positions and corresponding not yet associated distance picture elements of the current cycle in step S64. The assignment can be made for an appropriate scheme, as in step S58, but now predetermined distance picture elements of the current cycle and the corresponding documents retraced distance picture elements of the preceding distance image are assigned.

p0146So removing the pixels of the previous cycle and the current cycle and the image in this way are associated with each other, whereby the allocation for moving away from the laser scanner 12 objects is performed such that current distance picture elements are associated with distance picture elements of the preceding image, while the remaining distance picture elements of the preceding image that are not assigned 12 removing objects from the sensor or laser scanner, the not yet associated distance picture elements of the current cycle and the distance image or assigned their back tracked positions.

p0147In step S68, a segmentation of the current distance of the image according to the segmentation in step S 12 will now be given of the first embodiment.

p0148In step S70 a segment-object association is then carried out, wherein the assignment of distance picture elements of the previous image and the current distance of the image distance as well as the association between distance picture elements of the previous distance image and objects of the previous cycle are used.

p0149In step S72, object properties, in particular their positions and velocities, determined and output.

p0150The process can then be continued with step S50 in the next cycle. A prediction can be eliminated by the use of optical flow.

p0151In other variants of the method the projection in another manner can be carried out, examples of this are described in the aforementioned patent application.

p0152By treating other distance picture elements that do not match from the laser scanner 12 removed objects, a clearer association of distance picture elements is achieved. Because in the current cycle are due to the imaging geometry of the laser scanner 12 which scans its detection range 18 radially, such objects more each distance picture elements correspond as in the previous cycle. It may then an allocation of more distance picture elements of the current cycle to prevent, for instance a predetermined distance image point of the previous cycle in which may occur for themselves the laser scanner approaching objects the event that more distance picture elements of the current image would be associated with a predicted distance picture element, which, however, can bring a complex allocation method because of the ambiguity.

p0153In a method according to a fourth preferred embodiment of the invention, an object recognition and tracking takes place only on the basis of video images. A corresponding device for object recognition and tracking according to a fourth preferred embodiment of the invention therefore comprises a as in the foregoing embodiment and the foregoing other methods used video system, the video camera is connected to a data processing device as in the previous embodiment and the preceding other methods is however programmed to perform the method according to the fourth preferred embodiment. This also means that the funds for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and assigning the item to the part of the are the current image or the current object modified. The laser scanner 12 is omitted.

p0154With this in <figref idrefs="f0015">Fig. 14</figref> illustrated process is first detected a current video image in a current cycle in step S74 and read. In this case, can take place, a corresponding pre-processing the video image data as in the foregoing embodiment and the foregoing other methods.

p0155In step S76, the captured video image of the image resolution reducing filtering is subjected to increase the processing speed, which is in the present embodiment, that of 4x4 blocks of video picture elements, or pixels, only the pixels is used in the lower left corner.

p0156In step S78, then objects are detected in the sub-sampled video image. To this end, conventional methods of object recognition and tracking can be used in video images.

p0157In step S80 will now be given an assignment of the detected objects to objects from the previous cycle. The assignment is done in the example based on the predicted object positions for the current cycle, which have been determined in the preceding cycle in a step S94, by means of known association methods.

p0158In step S82, it is checked whether new objects were detected, none of the objects of the preceding cycle were assigned. If no new objects detected, the method continues with step S90.

p0159Otherwise, first in step S84 in the preceding video image, which is not, or in the example only half as much as sub-sampled the image generated in step S76, search for an object corresponding to a new object.

p0160Due to the increased resolution, objects in the preceding video image thus detected that would otherwise not be seen due to undersampling.

p0161This is illustrated in <figref idrefs="f0016">FIG. 15A and 15B</figref> and 16 shown again for yourself the video system approaching person. In<figref idrefs="f0016">Fig. 15A</figref> the person is shown schematically in a current video image full resolution, with black rectangles representing video pixels or pixels that match the person. Through the above-mentioned sub-sampling results in the in<figref idrefs="f0016">Fig. 15B</figref> shown undersampled video image in which the person is only represented by four individual pixels. Has now the person approached from a distance of the video camera, so they can in the preceding video image at full resolution, for example, in<figref idrefs="f0016">Fig. 16</figref> have size shown, wherein after undersampling no pixel over remain, by means of which a corresponding object could be detectable.

p0162In the method used here, however, wanted in the previous image in full resolution and not in the lowered resolution after the new object, which can be therefore found. Since the new object or the object can occur only in the vicinity of the detected in the current video image object, the calculation time for the recognition of the object in the preceding video image is greatly reduced, nevertheless.

p0163In step S88, the amount equivalent to the new objects objects are tracked in the preceding video image in the current cycle in what is easily possible because the object has been detected already in the undersampled current video image.

p0164Due to the determined positions of the objects in the preceding video image and thus in the previous cycle and the current cycle S90 velocities of objects can now be calculated by estimating or subtraction in step.

p0165In step S92, the current video image is then stored, the previous video image is deleted.

p0166In step S94, an output of the object positions and speeds takes place for use in the following applications.

p0167new object positions for the following cycle will be on the basis of the object positions and velocities then in step S96, predicted, near which will search for objects in the video image of the next cycle. To this end, a Kalman filter can be reused.

p0168The method according to the fourth embodiment allows the one hand by the sub-scan speed of execution. However, for the other, it is possible for approaching, new and thus critical for a vehicle objects, determine their speeds at its first appearance already using a previous video image, so that in an emergency situation, the reaction time can be shortened by one cycle duration.

<u>LIST OF REFERENCE NUMBERS</u>

p0169<dl id="dl0002" compact="compact"><dt>10</dt><dd>vehicle</dd><dt>12</dt><dd>laser scanner</dd><dt>14, 14 '</dt><dd>Data processing device</dd><dt>16</dt><dd>object</dd><dt>18</dt><dd>detection range</dd><dt>20</dt><dd>Laser radiation beam</dd><dt>22</dt><dd>Subject point</dd><dt>24</dt><dd>vehicle</dd><dt>26</dt><dd>pedestrian</dd><dt>28</dt><dd>video system</dd><dt>30</dt><dd>video camera</dd><dt>32</dt><dd>CCD area sensor</dd><dt>34</dt><dd>lens</dd><dt>36</dt><dd>detection range</dd><dt>38</dt><dd>optical axis</dd><dt>40</dt><dd>scan</dd><dt>42</dt><dd>common detection area</dd><dt>44</dt><dd>vehicle</dd><dt>46</dt><dd>person</dd><dt>48</dt><dd>image plane</dd><dt>50</dt><dd>optical center</dd><dt>52</dt><dd>Subject point</dd><dt>54</dt><dd>intersection</dd><dt>56</dt><dd>displacement vector </dd><dt>57</dt><dd>intersection</dd><dt>58</dt><dd>Subject point</dd></dl><dl id="dl0003" compact="compact"><dt>v</dt><dd>velocity vector</dd><dt>d</dt><dd>displacement vector</dd></dl>