Method for recognising and following objects

The present invention relates to a process for the recognition and tracking of objects that correspond to objects in a detection region of a sensor for electromagnetic radiation, and a device for carrying out the method.

Methods for the detection and tracking of objects are known in principle. Typically, images of a detection range are thereby detected by a sensor for electromagnetic radiation, such as a laser scanner or a video camera at constant intervals. The pictures will then search for objects that match objects in the detection area. Was in an image an object first found, is wanted in subsequent cycles of the method for this object to track its location or change in position over time can. For this purpose is often predicated on the basis of the position and velocity of an object in a previous cycle its location for a current cycle or in a current picture to be in the picture discovered near the predicted position of the object elements, eg segments, the object assign from the previous cycle and to be able to detect its current location.

Such methods are suitable for example for monitoring a region in front of and / or next to a motor vehicle. A potential advantage of such monitoring may be that sudden threats automatically detected and appropriate countermeasures can be taken. For this it is necessary, however, that objects can be quickly detected and tracked accurately.

In the above-mentioned method of tracking an object is often to increase the accuracy of the detected object properties with increasing duration, because in the course of pursuing further information can be gathered about the object, which allow a better characterization of the object or the state of the object. Accordingly, inaccuracies can especially occur when new items appear in images. In the example of the motor vehicle, this may be for example, pedestrians, obscured by another object such as a parked on a roadway edge automotive, suddenly into the street and then be detected by the sensor. The same problem can occur when pedestrians are positioned in front of an object so that the object tracking method the pedestrian and the object resumes to an object so that the pedestrian is also recognized only when it has moved far enough away from the object.

<patcit id="pcit0001" dnum="US6480615B1"><text>US 6,480,615 B1</text></patcit> describes a method for tracking of objects, in which the optical flow of an image frame using data from three successively recorded images k-1, k, k + 1 is determined. On the basis of a parameter S is decided whether the required for the determination of the optical flow gradient of the images k-1 and k or from the images K and K + may be used first Due to the possibility of the use of gradients. based on the images k-1 and k, while an optical flow in areas that are covered by a foreground object temporarily be determined.

The present invention is based on the object, a method for detecting and tracking objects which correspond to objects in a detection range of a sensor for electromagnetic radiation, provide that allow for fast detection and tracking of objects.

The object is achieved by a method having the features of claim 1.

In one method, which is an object of this divisional application is not, for the detection and tracking of objects, the objects in at least one detection area of ​​at least one sensor for electromagnetic radiation are, on the basis of repeatedly using the at least one sensor captured images of the detection region are in successive cycles successive images is evaluated and it is searched in each case at least one aufgefundenes in a cycle based on a corresponding image object in a later cycle in a corresponding later image to track the object. It is a recognized on the basis of the results of an at least preliminary evaluation of a current image while at least one current cycle at least one determined in the current cycle part of a current image or in the current cycle upon detection of a new object and / or feature in a current cycle object associated with at least a part of an earlier image and / or at least one indication with respect to an earlier state of the object or of an appropriate object, which is determined using a corresponding earlier image in the current cycle.

The apparatus for detection and tracking of objects has at least one formed to capture images of a coverage sensor for electromagnetic radiation and with the at least one sensor connected to data processing means, which is designed for carrying out the method according to the invention, and in particular means for evaluation of current images of said at least a sensor means for the determination of parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to a earlier state of the object or a name corresponding include cross-stands using a corresponding earlier image in the current cycle and allocation of the indication on the part of the current image or the current object.

The procedure, which is carried out by means of the device, used for detection and tracking of objects in a detection area. This can basically be stationary or moving, which is particularly the case may be, when the images are captured by a sensor attached to a vehicle.

The images are acquired by at least one sensor for electromagnetic radiation. The sensor can be a purely passive sensor which receives only electromagnetic radiation. However, it may also be a sensor may be used, the directed across a source of particular radiated electromagnetic radiation which can be reflected from an illuminated by the radiation spot or zone on an object, and at least one corresponding to from an object point or range reflected radiation-sensitive sensor element has. Furthermore, combinations of sensors for electromagnetic radiation can be used which detect different types of images. The detection ranges of the sensors need then not necessarily overlap, but it is preferred that this is at least partially overlap.

Under an image is understood to mean in particular a set of data, each representing the position of a point or area on one or more detected by the sensor object in at least one direction transverse to a direction of view of the sensor and at least one property of the object point or region , The location of the object point can, for example, upon detection of object points in a scan plane by an angle relative to an arbitrary but fixed predetermined reference direction, for example, the sensor may be given. Upon detection of object points or regions in space, the layers can by layers of sensor elements of the sensor and possibly be given imaging properties. In the property may be, for example, the intensity and / or wavelength or color of the emitted optical radiation, the distance from the sensor or the speed in the radial direction relative to the sensor. For example, video images data to the intensity of radiated optical radiation as a property, and in respect of a sensor element which has detected the radiation of an object point or region, and thus in respect to a location, while distance images as properties of the detected object points or regions the distance from the sensor and as position information may include an angle at which the object point or region has been detected. Speed ​​images eventually contain data about the speed of the object points or ranges relative to the sensor.

The images need not be detected directly by a single sensor. Rather, it is also possible, at least two temporally associated with each other images that have been recorded by at least two sensors for electromagnetic radiation to combine to form an overall image that represents a larger detection area or additional characteristics of object points or regions. For example, from two images that were recorded by two mutually spaced video cameras, distance images are formed in accordance with the imaging properties and the arrangement of the video cameras, which in addition to a distance of an object point or -area of ​​one relative to the video cameras fixed reference point and data have in terms of the intensity and color of each of the object point or range the light beams.

Furthermore, different types can be combined images to a total image or processed separately or together as a function. Thus, preferably video and distance images can be processed together without being grouped together in a narrow sense to a set of image data.

The images used in the method are detected at successive times, which preferably have a constant distance in time from one another. The sensor of the device is therefore ready for repeatedly capturing images of the detection region. If for one cycle multiple images are used, they are preferably synchronously detected, ie their sensing times differ by a smaller period of time than that between successive scans by the sensor with the lowest detection rate substantially.

The captured images are then analyzed in consecutive cycles, whereby in a given cycle, at least one corresponding image is evaluated. If multiple images are synchronously acquired substantially sufficient that one of the images is at least provisionally evaluated.

In each case, at least wanted a aufgefundenes in a cycle based on at least a corresponding image object in a later cycle in a corresponding later image to trace it. For this purpose, conventional object recognition and -verfolgungsverfahren can be used in particular. In this case can be effected an allocation of pixels of the current cycle to recognized in a previous cycle or image objects.

In particular, a prediction method may be used, in which on the basis of the position of at least one object in an earlier, in particular in a preceding cycle and the speed of a location in the current cycle is predicted and is searched in the vicinity of the predicted location for the object. New objects can be made of parts of an image, ie pixels or calculated in the cycle amounts of pixels, in particular segments are formed which could not be attributed to the already known objects.

Now it is provided deviating from conventional object recognition and -verfolgungsverfahren that during at least one current cycle on the basis of the results of an at least preliminary evaluation of a current image to at least one determined in the current cycle part of a current image or a detected object in the current cycle , is assigned to at least a part of an earlier image and / or at least one indication with respect to an earlier state of the object or of an appropriate object, which is determined on the basis of a corresponding previous image.

Are there more substantially synchronously captured images the same or different type is used, it is sufficient that a preliminary analysis of these images is done, the assignment may be parts of the same or another, done substantially synchronously captured image.

This means that in a current cycle is not only a current image is evaluated. Rather, initially at least provisional or optionally partially evaluating a current image in the current cycle is performed. On the basis of these at least preliminary evaluation, ie in particular in dependence on the result of the at least preliminary evaluation, then a previous image for object tracking and determination of data relating to an object employed in which it in particular, a current immediately prior to the image captured image that is associated with a preceding cycle, can act.

In the preliminary analysis can be particularly at least one criterion to be checked, reference to which it is decided whether an earlier image to be used or not. This means then that non-fulfillment of the criterion one else can take place known object recognition and tracking.

In the at least preliminary evaluation of a part of a current image is determined, which may be, be a pixel or determined in the current cycle set of pixels, for example, at least one segment. Where in a current cycle more current images used while the image that has been preliminarily evaluated, not necessarily coincide with the image, a portion of which is determined needs. However, it can be detected and used as an object in the current image. Both the determination of the portion of the image and the recognition of the object can initially take place with process steps similar to those in conventional methods.

The figure recorded in the at least preliminary evaluation part of the current picture or the object in the current cycle can be a part of the previous image are assigned, which is in turn a one pixel or one determined in the current cycle set of pixels, for example, at least can act segment, the previous image. The parts of the images may each have different numbers of pixels needs to be so that no one-to-one mapping of pixels added. For any first and / or second objects and / or parts of the first and / or second images is in this case understood to mean an allocation of a portion of a first image or a first object to a part of a second image or a second object such that the portion of the second image or the second object is determined, and for an appropriate part of a first part or a suitable first object is optionally selected and assigned to several alternatives.

However, the portion of the current picture or the object in the current cycle can also be an indication be assigned, which refers to the object or a name corresponding object. This entry concerns may be values ​​of any state variables of an object or an object, such as position, velocity, shape, size, allocation to one of several object classes, according to their relevance to the object tracking typical characteristics are classified in the objects, or the presence of the object at all. To determine the indication this need not be the entire previous image used instead, the investigation may also be restricted to partial areas. However, for the determination always directly at least one pixel of the previous image, which can be preferably to have saved during the current cycle, and simultaneously uses the result of the at least preliminary evaluation of the current image.

In any case, it is essential that a part of the earlier image or of an indication derived from the previous image are assigned to a part of the current image or object of the present cycle and not parts of the current image objects or information, the image from the former were recovered.

This means a total of that for object tracking, depending on the preliminary evaluation information from a current image to a renewed and improved at least partial re-evaluation of the previous image are used, the results are in turn used in the current cycle. This information contained in the images can be better evaluated, which allows faster and more accurate object recognition and tracking.

The data obtained by the method can then be output or stored to be used for example by the following means, for controlling a vehicle.

To carry out the individual method steps of a data processing device is provided in the device which is connected to the sensor for electromagnetic radiation for the transmission of images. The data processing device can be completely or partially as a non-programmable circuit, which increases the speed of execution. Preferably, however, the data processing device comprises a programmable processor for implementing the method according to the invention and, associated with memory, an output interface and an input to the sensor sections.

In particular, means for evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and / or for the association of parts of an earlier image or means for determining comprises at least an indication relating to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and allocation of the indication on the part of the current image or the current object can be provided, in whole or in part, as a non-programmable electrical circuit can be formed, however, by a suitably programmed processor may be formed, preferably.

Further developments and preferred embodiments are described in the specification and the drawings.

The sensor is a sensor for any electromagnetic radiation can be used. In a preferred embodiment of the device as a sensor, at least one radar sensor, in particular a spatially resolving radar sensor used. Accordingly, in the method preferably spatially resolved images of a radar sensor are used. Radar sensors can detect particular radial velocities of objects directly relative to the radar sensor.

Preferably, however, at least one sensor for optical radiation, ie, radiation in infrared, visible or ultraviolet region of the electromagnetic spectrum can be used. Such sensors generally have a better spatial resolution than radar sensors.

Sensors for detecting distance images are particularly preferably used. For example, can be used with at least two video cameras, a stereo video camera system, which are arranged spaced apart in a plane in which a distance is detected by the stereo video camera system. Preferably, however, sensors are used in which at least one scanning beam is used for scanning with electromagnetic radiation at least one scanning plane. A plurality of scanning beams for substantially simultaneous scanning of strips can be used in a scan plane, or it may, preferably, a scanning are pivoted in the scanning plane, wherein in each case reflected back from objects radiation having a corresponding detector is received. In particular, appropriate laser scanner can be used by those with angular resolution at least one pulsed laser radiation beam and a sensing range pivotable from an object back radiation of the pulsed laser radiation beam can be received and based on the duration of a pulse of laser radiation beam to the object and back to the laser scanner a distance of the object can be determined , Such laser scanners are characterized by a good spatial resolution in conjunction with high detection rate. Particularly preferred laser scanners are used, which scan more like a fan superimposed scanning planes.

Furthermore, it is preferred that in the method, video images are used and that the device for capturing video images having a video system which may comprise at least one video camera and an imaging optical system and a sensor system for spatially resolved reception of optical radiation, for example visible light / or infrared radiation has. As a video camera can be used depending on the application, in particular a wide-angle or panoramic camera. For monitoring of the far region, a video camera with a telephoto lens is particularly suitable. It can be used in principle, video cameras with zoom lenses. Furthermore, can be used as a sensor system, a black-and-white or gray scale system or a color-sensitive system. In this example, corresponding CCD or CMOS sensor elements can be used.

Furthermore, combinations of sensors for electromagnetic radiation can be used to capture the same or different types of images. The detection ranges of the sensors need not be identical, it is sufficient that this overlap in the monitoring area.

In order to improve object detection and tracking in the current cycle, it is preferred that the assignment of the part of the earlier image or of the indication with respect to the earlier state of the object to determine a further indication with respect to the current state of an object used becomes. The determined additional disclosure may be not to refer to the object for which the indication was determined with reference to its previous state. Rather, it may be any object of the current cycle. Thus, there is effectively a least partially recursively working process in which, starting from information from the at least preliminary evaluation of a current image first new information about an object or an object is obtained at a time corresponding to a previous image, in turn, to evaluate the the current image is used. Thus, the existing information is more effectively utilized in the pictures than in purely linear working method.

Particularly in the treatment of newly detected objects, it is preferable that, when a new object is detected in a current cycle, a current location of the new object is determined in the current cycle, that based on the current location of the new object a previous location the new object in an earlier cycle is estimated and that using the geschätz the current value of a state variable of the new object is determined in the current cycle th former location of the new object. In particular, it can be estimated as a state variable, the velocity of the object, on the basis only of an image can not be determined otherwise.

Considering that an object was initially not recognized in the previous cycle, a number of reasons may be. In a further development of the method it is preferable that it is checked based on the current location of the new object in the current cycle and a previous location of at least one other object in a previous cycle, whether in the evaluated in the previous cycle image of the new object corresponding object could be obscured by another object in the earlier cycle corresponding object. Under masking is understood to mean that the object points of the hidden object from the sensor used were undetectable, since areas other object were placed in the propagation path of the electromagnetic radiation from the object points to the sensor. This further facilitates the one occlusion detection in the previous cycle and allowed by estimating the size and position of the hidden portion of the detection range to estimate the location of a corresponding new object object at the time of the previous image.

It is particularly preferred that for the new object in the current cycle, a current size and / or shape is determined, and that the current size and / or shape may be used in the check for masking. This can for example be ensured that a masking is only detected when a detectable by the sensor or a silhouette detectable by the sensor outline of the new object or of the corresponding object from the point of view of the sensor is smaller than the corresponding outline or corresponding silhouette of the other object. Moreover, the size and / or shape of the new object serve this initially classified into one of several classes of objects, each of which includes objects with characteristic for object recognition and tracking properties. For example, object classes for pedestrians, motor vehicles, road boundaries or the like can be provided. Depending on the object classes can then be estimated, whether the object in the available time between the detection of the current image and the previous image with the maximum value possible for objects of the relevant object class rate from a concealed by the other subject area detected in a position might have moved.

From those obtained by the evaluation of the current image in the current cycle information that a new object exists in the detection area, that can be detected through the occlusion check whether this new object corresponding subject already to a previous time in the detection range of the sensor was present. In particular in this case, it is preferable that assuming a concealment of the current situation and preferably the current size and / or shape of the corresponding to the occluding object object an earlier position of the corresponding new object object in the earlier cycle and this results in a current speed the new object is estimated. The obtained in the current cycle information on the new object in the earlier cycle can thus be used to estimate its speed as a state variable in the current cycle of both the direction and in terms of size. This is for example particularly useful to recognize from parked cars undercover pedestrians emerge suddenly behind the car quickly and then possibly also to be able to respond.

However, in an actual image newly discovered object could not have been recognized for a reason other than a concealment in the previous cycle. Thus, it is preferable that is wanted in an earlier image to the new object, and that, in finding the object in the previous image in the current cycle, a current value of at least one state variable of the object and / or other object using a past value of the state variable is determined for the new object and / or the other object in the earlier cycle. The search can be facilitated on the one hand, that the object's properties, such as its size and / or shape or outline of the current cycle are already known. On the other hand can be used that in the previous image, the new object must have been present, so that in the evaluation of the previous image in the earlier cycle optionally dispelled existing uncertainties in the allocation of portions of an image, for example, pixels or segments to objects can be.

Such uncertainty may occur in particular when in the previous image is actually two different objects associated object points or regions as an object to be recognized as belonging. the new object finds in the previous picture again, so information can be both on the new object or the name corresponding an object and a subject of the other corresponding other, are now obtained for the previous cycle newly given object. In particular, the positions of which and, where appropriate sizes and / or shapes can be determined for the new object and the other object as a state variable in the previous cycle. Using appropriate data for the current cycle can then, for one for the in the current cycle at first new object already a speed are determined, and on the other for the other object, which corresponds to the other object in the previous cycle, a more accurate speed can be determined.

By using the method according to this embodiment, for example, pedestrians can be detected particularly easily, who stayed out of view of the sensor immediately before an object at the roadside and then stepped on the road. Immediately with recognition of the pedestrian in a current image can then by the time tracing its speed both in terms of size as well as their direction can be estimated, so that valuable time can be obtained in detecting the pedestrian.

The search for the new object in the earlier image can be done in different ways. In a further development of the method, it is preferable that the images are segmented by a predetermined segmentation method and at least one corresponding predetermined segmentation parameters, and that the finding of the new object in the earlier image of at least a corresponding portion of the previous image with a different segmentation methods and / or is segmented again with a changed segmentation parameters. As a result, can be carried out a new allocation of the newly determined segments to corresponding objects. The new segmentation need not be carried out for the entire image, but may preferably be only on the location of the new object in the current cycle and appropriate, nearby other objects depending on their speeds extend certain area, so that the detection of other objects is not impaired. Both the segmentation process used and the possibly changed segmentation parameters can be adjusted, in particular situational, ie be selected depending on the locations, types and sizes, and / or the number of pixels corresponding to the new object and the other object.

In a preferred development of the process time associated with one another distance and video images are used, which are arranged spaced from one another, for example by a laser scanner for the acquisition of range images and a video camera for capturing video images or through a video system for generating stereo images with at least two, mutually coupled video cameras can be recorded. For example, the video image processing is controlled to reduce the processing burden by evaluating corresponding distance images, ie there is a attention control of the video image processing by the laser scanner is only in those areas of the video image according to features and / or objects sought, include sections where in the corresponding distance image object points or regions corresponding pixels were detected. In particular, the regions may be formed by extending at right angles in a direction perpendicular to a scanning strip whose interface is defined by the detection range of the sensor for detecting the distance images in its position and width by at least one distance picture element or more distance picture elements. An object is only detected in the video image when it enters the detection range of the sensor for detecting the distance images. For example, to detect the distance images using a laser scanner which scans a level of coverage by means of a pivoted in a scanning laser beam, an object in a video image can be seen, although this is not yet in the distance image, for example due to obstruction at the level of the scanning plane can be seen.

Therefore, it is preferable that time mutually assigned distance and video images are at least used a common portion of the coverage that is wanted in video images for objects and / or features only in sections, which are determined as a function of time associated distance picture elements that in a current cycle upon detection of a new object and / or feature of the new object and / or feature is traced in the video image in time to an earlier cycle, and that a position of corresponding to the new object and / or feature of the object in the previous cycle using the information on an object corresponding to the object and / or feature is determined in the earlier video image. The time allocated to each other distance and video images used are preferably synchronously acquired substantially, ie their sensing times differ by a smaller period of time than that between successive scans by the sensor with the lowest detection rate. The sections may, in particular through spaces of predetermined size and shape to the distance pixels, preferably orthogonal, added to a scanning plane, the aforementioned strips. For removing pixels in the range image ie a corresponding feature or an entire object can be searched in the video image in the corresponding section of the video image according to at least. Using only features that traceability is usually carried out more quickly. The determination of the position of corresponding to the new object or feature article in the previous cycle can in particular be carried out by an estimate of the position of the corresponding feature and / or object is preferably based in the video image. Furthermore, the speed in the current cycle can be determined from the position of the corresponding to the new object or feature article in the previous cycle and the location of the new object or feature in the current cycle. In this process variant can be particularly utilized that often have objects may be identified by a video camera when the usually only in the amount less extensive, that reflect shallower detection range, not, for example, by masking at the level in the range images, the sensing range for the distance images are recognizable. If an object in the distance image for the first time on, it is possible to detect a corresponding feature this in the previous cycle in the video image when the subject was covered only in the range image, but not in the video image. A velocity of the object can then be determined on the basis of the location of the feature in the video images. The occurrence of "blind spots" of the video image processing, which are caused by the attention control, can thus be reduced without considerable effort.

Another embodiment, the method corresponds to the inventive method, which is defined in Claim first In this method, one at a time associated with removal and video images of at least one joint portion of the detection range used in an earlier distance image distance picture elements associated with distance picture elements in a current distance image by for at least one a distance picture element in the earlier distance image corresponding region and / or at least one a distance picture element is in the former distance image corresponding characteristic of the corresponding previous video image and / or of the current video image, a displacement and / or shifting speed, in particular an optical flow determined and the shift or displacement velocity, in particular the optical flow, the assignment of a distance picture element in the current range image is used to the distance picture element in the earlier distance image, for objects in the distance image, approaching a distance image capturing sensor and / or in a current cycle have longer distance picture elements than in a previous cycle, distance picture elements in the current cycle using the corresponding displacement or displacement velocity of the range or feature, in particular the corresponding optical flow, are assigned in the previous cycle distance picture elements. In this embodiment, information from range images are combined with information from corresponding video images, whereby additional data is obtained to at least one distance picture element that can be used in the processing of the distance picture elements. It is particularly exploited that with a movement of an object is usually in the range image and the video image captured areas are moved to the object. Assign the areas in the video image an intensity curve or corresponding features, the movement of the object is expressed approximately as a shift of the corresponding intensity characteristics or features in the video image. Under a shift is in this case a displacement between successive cycles understood but not necessarily need to directly follow one another. The movement information from the video images can then be assigned to the appropriate areas of the article or the corresponding distance picture elements.

Preferably, a corresponding optical flow is determined for the determination of displacement or speed of displacement of the range. In the area used to define the optical flow on the one hand, an image plane are used in the video pixels of then also defined in the plane video image are. Preferably, however, one surface is used, which is used in a camera model for the treatment of a video image. In principle, it is sufficient if the optical flow can be determined for at least one distance picture element. In general, the case can occur namely that distance picture elements are recorded on subject areas which do not have intensity and structure can not be determined for which therefore also an optical flow. The optical flow is for example completely determined when the corresponding area of ​​the video image having a change of intensity in two linearly independent directions. Preferably, a corresponding tracking therefore applies to these areas.

Method for calculating the optical flow are generally known and, for example, in the article "<nplcit id="ncit0001" npl-type="s"><text>The computation of optical flow "by JL Barren and SS Beauchemin in ACM Computing Survey, Vol 27, No. 3, pp 433 -.. 467 (1995</text></nplcit>) And in the article "<nplcit id="ncit0002" npl-type="s"><text>Performance of optical flow techniques "by JL Barren, DJ Flied, DJ and SS Beauchemin in International Journal of Computer Vision, 12 (1), pp 43-77 (1994</text></nplcit>) Described.

The optical flow needs while not necessarily be computed for the entire video image. It is sufficient that for the distance picture element corresponding object point or - if necessary - to determine its environment. The place in which used to define the optical flow area for which the optical flow has to be calculated, is from the position coordinates of the distance picture element, the relative position of the sensor used to detect the distance images to that used for capturing video images video system, the imaging geometry of the video system or a video camera therein and a model for this and the predetermined surface on which the optical flow is to be determined, derived. It can be used as a place, a corresponding video image point, however, it is also possible to subpixel accuracy to determine the optical flow at locations between video picture points or pixels of the video image. Keeping track of distance picture elements with the aid of the optical flow, distance picture elements are allocated in a later cycle in removing pixels in a previous cycle, is filed by the applicant German patent application with the official file reference<patcit id="pcit0002" dnum="DE10312249"><text>DE 103 12 249</text></patcit> described. There distance images are called depth-resolved images.

Therefore, this used to be from the sensor removing objects or objects with a not greater number of distance picture elements in the current cycle process has the advantage that a prediction of the object layers need not be done because the use of the shift or the optical flow tracking individual Distance pixels allowed. The reverse procedure when approaching objects or objects that have in the current cycle longer distance picture elements than in a previous cycle has, on the one hand the advantage that an allocation of a greater number of distance picture elements to a smaller number of distance picture elements, in particular in situations where objects are in the current cycle closely adjacent to each other, can be carried out correctly with greater certainty. Secondly, the corresponding object more accurate speed are assigned because that detects the speed of the object, an averaging over corresponding speeds of associated distance picture elements can be used and in the current cycle, a larger number of such distance picture elements is available.

In another development of the method, it is preferable that the object recognition and tracking is carried out on the basis of video images, and that when the object detection, the detection of objects is carried out on the basis of filtered video images that comprise the filtering a reduced resolution, and a search for an object is detected in the current cycle based on the position and possibly the form of the object in the current filtered video image according to one corresponding to the object article in a previous video image whose resolution is higher than that of the filtered video image, and that in finding a corresponding object in the earlier video image corresponding data be used via the object in the evaluation of the current video image. The object recognition and tracking can be optionally supported by the use of distance images. Video images In this development, mainly in order to save computational effort, initially detected subjected to reduction in the resolution of a filtering. When filtering, it may be any method in which the number of pixels of the video image is reduced. In the simplest case, for example, just a sub-sampling of the image are performed at the of square blocks of pixels, or pixels, each only a pixel or pixels are referred to by the whole assembly. a far object approaches the sensor used for detection of the video image, it may happen that at the first detection of the object very few pixels or pixels are in the filtered video image available. By analyzing the previous image with increased resolution, can optionally the object that could not be detected in the previous image only due to filtering, now be recognized, so that the newly discovered object in the current cycle despite the only low resolution a trajectory and speed can be allocated without requiring a considerable computational effort would be necessary.

The invention is also a computer program with program code means to carry out the inventive method when the program is executed on a computer.

Another subject of the invention is also a computer program product with program code means stored on a computer readable data carrier to carry out the inventive method when the computer program product runs on a computer.

Under a computer in this case, any data processing device, in particular a data processing device of the apparatus is understood to mean the process of the invention can be performed. In particular, this may include a digital signal processor and / or microprocessor exhibit, with which the process is carried out in whole or in parts.

The invention is preferably provided with movable objects, in particular road traffic, suitable for monitoring of areas, wherein the acquisition of the images by at least can be carried out a fixed-position sensor and / or one held on a vehicle sensor.

The invention will now be further illustrated by way of example with reference to the drawings. Show it:<dl id="dl0001"><dt>Fig. 1</dt><dd>a schematic plan view of a vehicle with a device for the recognition and tracking of objects according to a first preferred embodiment of the invention and a vehicle located in front of the object,</dd><dt>FIG. 2</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a first preferred embodiment of the invention,</dd><dt>FIGS. 3A and 3B</dt><dd>Cutouts detected from at successive points in time distance images of a detection range of a laser scanner of the apparatus in <figref idrefs="f0001">Fig. 1</figref> with a vehicle only partially shown and a moving pedestrian,</dd><dt>Fig. 4</dt><dd>the presentation in <figref idrefs="f0003">Fig. 3B</figref> with additional guides for explaining a calculation of the position or speed of the pedestrian,</dd><dt>Fig. 5</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a second preferred embodiment of the invention,</dd><dt>Figs. 6A to 6G</dt><dd>Excerpts from time sequentially sensed distance images of scenes with a vehicle and a at times ahead of the vehicle is only partially shown, moving pedestrians,</dd><dt>Fig. 7</dt><dd>a schematic plan view of a vehicle with a device for the recognition and tracking of objects according to a second preferred embodiment of the invention and a vehicle located in front of the object,</dd><dt>Fig. 8</dt><dd>a schematic, partial side view of the vehicle and the object in <figref idrefs="f0007">Fig. 7</figref>.</dd><dt>Fig. 9</dt><dd>a flow diagram is illustrated in the schematic of the flow of a method according to a third preferred embodiment of the invention;</dd><dt>FIG. 10A and 10B</dt><dd>Excerpts from a distance image and a corresponding video image with a vehicle and a moving pedestrian at a first detection time,</dd><dt>FIG. 11A and 11B</dt><dd>the images in the <figref idrefs="f0010">FIG. 10A and 10B</figref> corresponding of distance or video images at a later time of data,</dd><dt>FIG. 12A and 12B</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a fourth preferred embodiment of the invention,</dd><dt>Fig. 13</dt><dd>a schematic, perspective view of a portion of a scan plane with a distance picture element, an image geometry for the calculation of an optical flow and at the end of the procedure in <figref idrefs="f0012">Fig. 12A</figref> and <figref idrefs="f0013">12B</figref> occurring layers of recorded and predicted distance pixels and video pixels and a corresponding one optical flow displacement vector,</dd><dt>Fig. 14</dt><dd>a flow diagram in which schematically the flow of a method is illustrated according to a fifth preferred embodiment of the invention,</dd><dt>FIG. 15A, 15B</dt><dd>Excerpts from an unfiltered and a filtered video image with a schematically represented pedestrian at a first time, and</dd><dt>Fig. 16</dt><dd>an unfiltered image of the pedestrian in <figref idrefs="f0016">Fig. 15</figref> at an earlier time.</dd></dl>

In <figref idrefs="f0001">Fig. 1</figref> carries a vehicle 10 on its front side a laser scanner 12 as well as a connected via a data line with the laser scanner 12 data processing means 14, which together with the laser scanner 12 is a device for the recognition and tracking of objects according to a first preferred embodiment of the invention. In the direction of travel in front of the vehicle is one in<figref idrefs="f0001">Fig. 1</figref> Person only very schematically shown 16 that is half viewed in the context of the invention for simplicity as an object.

By means of the laser scanner 12 is in <figref idrefs="f0001">Fig. 1</figref> only in partial detection range 18 scanned, the 12 is positioned due to the mounting position of the laser scanner at the front of the vehicle 10 symmetrically to the longitudinal axis of the vehicle 10 and covers an angle of slightly more than 180 °. The detection range 18 is in<figref idrefs="f0001">Fig. 1</figref> only shown schematically and too small for better representation in particular in the radial direction.

The laser scanner 12 scans its detection range 18 in basically known manner with a constant angular velocity circulating, pulsed laser radiation bundle 20 from which also τ circumferentially at constant intervals .DELTA.t at times_i α in fixed angular ranges around a mean angle_i it is detected whether the laser beam radiation 20 is of a point or region 22 of an object, such as the person 16, reflected. The index i runs here from 1 to the number of angular regions in the detection area 18. Of these angular ranges is in<figref idrefs="f0001">Fig. 1</figref> shown only an angular range of the central angle α_i assigned. Here, the angular range is, however, shown for clarity in an exaggeratedly large.

Due to the pivoting of the laser radiation beam 20 to the detection range 18 on the expansion of the laser radiation beam 20 is substantially two-dimensional and substantially forms, ie to the diameter of the laser radiation beam 20, a scanning plane.

With the maturity of the votes of the laser scanner 12 laser radiation pulse of the laser scanner 12 to the object point 22 and back to the laser scanner 12, the distance d_i of the object point 22 is determined by the laser scanner 12th Therefore, the laser scanner 12 recorded as coordinates in a the object point 22 of the object or person 16 corresponding distance picture element the angle α_i and those established in this angular distance d_i, Ie the position of the object point 22 in polar coordinates. Each detected object point is therefore associated with a distance picture element.

The amount of in which a scan detected distance image points forming a distance image in the sense of the present application.

The laser scanner 12 scans the detection area 18 are each in successive scans with .DELTA.t intervals, so that a time sequence of samples and corresponding distance images. The processing of the distance images of the laser scanner 12 is performed by the data processing device 14th

The data processing device 14 has to, inter alia, a for performing the method according to the invention programmed with a corresponding inventive computer program and a digital signal processor coupled to the digital signal processor memory device. In another embodiment of the apparatus, the data processing device may also include a conventional processor to which a program stored in the data processing device according to the invention a computer program for executing the method according to the invention is carried out.

The programmed data processing means is here in the sense of the invention means for the evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and allocation of the indication on the part of the current image or the current object ready.

On the basis of the acquired of the laser scanner 12, the distance images in <figref idrefs="f0002">FIG. 2</figref> illustrated process is carried out according to a first preferred embodiment of the invention.

In successive cycles respectively the steps S10 to S28 are performed.

First, a range image is in a current cycle in step S10 by a scan of the detection area 18 detected and read into a memory in the data processing device 14th

In step S10, while a pre-removal image data is carried out, is conducted at the, possibly after correction of the data, a transformation of the position coordinates of the distance picture elements into a Cartesian, firmly connected to the vehicle 10 vehicle coordinate system.

In step S12, the distance image is then segmented. Here are formed in a conventional manner amounts of distance picture elements, which are characterized in that each distance pixel an amount of at least one other distance pixel same amount has a mean square distance which is less than a predetermined segmentation distance. A lot thus formed corresponds to one segment. In this case, a segment may also be formed by a single range image point having from all other distance picture elements of the current image a distance mean square distance which is greater than the predetermined segmentation interval.

In step S14, which is not performed in the very first cycle of the process, then an assignment of segments already identified in a previous cycle of the process objects, including well-known methods can be used per se. In this process each one predicted in step S28 of the previous cycle object location is used for each object known in the preceding cycle. In the current cycle a segment of the current cycle is associated with a respective object of the previous cycle when at least one of the distance picture elements of the segment of the object in the predicted object position at a distance which is smaller than one of an uncertainty of the prediction and the size and orientation of the object in the previous cycle dependent maximum distance.

In step S16, then in segments that could be assigned not known from the previous cycle objects, formed new items and determined from the positions of the segments constituting distance picture elements a location of the object in the current cycle.

The above described steps S10 to S16 and the steps to be described S26 and S28 are not different from conventional object recognition and -verfolgungsverfahren.

In contrast to these, however, after an examination of whether new objects were found, checked in step S18, whether in the previous cycle a an object newly formed corresponding object in a preceding distance image or cycle optionally identified by a one in the previous cycle object corresponding object could be obscured.

The processing of the current distance image in the steps S10 to S16 including the examination in step S18, whether a new object has been found is an at least preliminary evaluation of the current image.

For occlusion detection in the preceding distance image is first checked, which in the previous image detected objects in this exhibit is formed by a smooth curve through the corresponding distance picture elements a contour which is greater than that determined by the corresponding distance picture elements expansion of the new object in the current cycle.

This is exemplified in the <figref idrefs="f0003">FIGS. 3A and 3B</figref> shown in which are shown in excerpts from successive distance images, a vehicle 24, which is only partially represented by the black solid line, and symbolized by a rectangle 26 pedestrians, the v moves at a speed relative to the vehicle 24th The individual dots indicate distance picture elements of the respective distance images again. In the<figref idrefs="f0003">FIGS. 3A and 3B</figref> is the laser scanner 12 at the origin, ie at the point (0,0), the coordinate axes of the Cartesian coordinate system are divided into arbitrary but fixed chosen distance units.

In <figref idrefs="f0003">Fig. 3A</figref>, A preceding distance image, the pedestrian 26 is concealed from view of the laser scanner 12 of the vehicle 24 and therefore can not be detected in the range image. In<figref idrefs="f0003">Fig. 3B</figref>In which the current distance image is shown, the pedestrian has moved away 24 behind the vehicle 24 and is now of the laser scanner 12 first detected. The contours of the articles 24 and 26 corresponding objects are in<figref idrefs="f0003">FIGS. 3A and 3B</figref> each represented as a dashed line regression curve through the distance picture elements. It is easy to see that in<figref idrefs="f0003">Fig. 3B</figref> the contour of the vehicle 24 corresponding object is greater than that of the pedestrian 26 corresponding object.

The further masking recognition is carried out preferably situationsadaptiv, ie. A function of the speeds of the already detected objects in the previous cycle In the example in the<figref idrefs="f0003">FIGS. 3A and 3B</figref> it is known that the vehicle 24 is resting, and that since no other object in the immediate vicinity of the newly detected, the pedestrian 26 corresponding object can be found or any other object having a smaller distance from it and the pedestrian corresponding object 26 due sizes can not match a very fast moving object, the object or pedestrian 26 was covered in the previous image with very high probability by the vehicle 24th

In step S20, the location of the preceding distance image hidden, new objects corresponding objects as an indication relating to an earlier state of the respective new objects will be appreciated in the context of the invention. To the example of<figref idrefs="f0003">FIGS. 3A and 3B</figref> to estimate a minimum speed of the object or pedestrian 26, is used to estimate the situation in the preceding distance image in <figref idrefs="f0003">Fig. 3A</figref> Assume that the object or pedestrian 26 was just not visible to the detection time of the removal of the image, so that this object box surrounding in the current distance image whose size in the process for objects of type pedestrian is given, with its edge just not circulating of the laser radiation beam could be detected in 20th In this way, the results in<figref idrefs="f0003">Fig. 4</figref> represented by the dotted rectangle drawn estimated position of the pedestrian 26 in the preceding distance image or cycle, which is displaced by the displacement vector d compared with the situation in the current distance image or cycle.

In step S22, the speed of the objects is then estimated by the change in position of an object in the previous image and the current image is determined, wherein for the new objects, as far as they were covered, the estimated position is used. The estimated speed of the objects is then obtained both in terms of the direction and the magnitude of the velocity by dividing the displacement vector given by the d position change by the time period At between successive scans of the detection area 18th

While for already identified in the previous cycle discovered objects speeds can be expected only a slight error due to the present measurement data, results for new objects only one afflicted with a larger error estimate. This estimate provides faster but much more and especially security-related information than would be possible without recourse to the previous image.

In step S24, then the determined object positions and sizes and the current distance image that has to be the then preceding distance image available in the following cycle is stored, the previous distance image is deleted.

In step S26, the object positions and speeds are output to appropriate further processing facilities.

In the step S28 is performed at the end of the current cycle, a prediction of new object positions for the following cycle. The prediction of the location as well as the determination of the uncertainty of the prediction can be carried out for example in a known manner by means of a Kalman filter.

In that the new object or object in the current cycle is already a rate can be assigned to the initialization of the Kalman filter in this object with considerably more accurate data can be performed, which significantly facilitates the following object recognition and tracking.

Then the next cycle with step S10 can begin.

A method for the recognition and tracking of objects according to a second preferred embodiment of the invention is shown schematically in the flow chart in <figref idrefs="f0004">Fig. 5</figref> shown. It differs from the method according to the first embodiment in that in a current cycle newly discovered objects are treated differently, so that the steps S10 to S16 and S22 to S28 are the same as in the above-described embodiment, and the explanations there accordingly apply here. Accordingly, the apparatus for tracking and recognition of objects with the method according to the second preferred embodiment of the invention over the corresponding device in the first embodiment is modified only in that the worked off in the data processing device 14 program is modified accordingly, that is, in particular, that the means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assigning the item to the part of the current image or the current object modified are.

In the method, after the running as in the first embodiment, steps S10 to S16 the following the formation of new objects from previously assigned segments in step S16 in step S30, after checking the criterion whether or was ever found in the current a new object, by new objects corresponding objects or objects in the preceding distance image wanted. It is assumed that the new object in the previous cycle or distance image could not be retrieved because it could not be recognized as an independent object due to the segmentation and object formation or segment-object association.

An example of this is in the <figref idrefs="f0005 f0006">Figs. 6A to 6G</figref> illustrated in which excerpts are shown from successive distance images respectively. The laser scanner 12 is placed in the range images in turn at the origin (0,0) of the coordinate system corresponding to the Cartesian coordinate system of the previous embodiment.

In the distance images, a pedestrian 26 moves from the point of view of the laser scanner 12 first immediately before a parked vehicle 24 at a speed v in order in the <figref idrefs="f0006">Fig. 6F and 6G</figref> Finally, as far removed from the vehicle 24, that it can be recognized as a separate object with the normal segmentation process. In the preceding distance images, however, the segmentation corresponding to the vehicle 24 removal pixels not the pedestrian part with the 26 corresponding distance picture elements.

Based on the size of the article 26 and the corresponding new object in the current cycle, which in the example the distance image in <figref idrefs="f0006">Fig. 6F</figref> is evaluated, it is believed that it may have only a low speed, because the object needs to be a pedestrian. To search in step S30, therefore the objects are determined for the situation adaptive segmentation, in which the pedestrian 26 corresponding new object in the current, the range image<figref idrefs="f0006">Fig. 6F</figref> corresponding cycle are closest. In our example, only the object corresponding to the vehicle 24. The steps up to this stage corresponds to a least preliminary evaluation of the current distance image in the sense of the invention.

It is a part of the preceding distance image is then re-segmented. More specifically, the distance picture elements of the segments in the preceding distance image or cycle into new segments constituting the respective previously determined for segmentation objects in the previous cycles. In the example, these are all distance picture elements of the detail of the distance image in<figref idrefs="f0005">Fig. 6E</figref>,

Given in the present embodiment, the applied in step S12 segmentation method, but with a reduced segmentation spacing used. The segmentation spacing can be fixed. Preferably, the segmentation distance but, preferably the smallest, selected depending on the detected size of at least one of the detected objects, so that a situationally adapted segmentation can be done.

there is a change in the number of segments, in step S32, a new allocation of segments is made to those affected by the re-segmentation objects. Here the position and speed in the previous cycle in conjunction with the determined in the current cycle orientation, shape and size as well as for the new object is the determined in the current cycle shape and size used for already known in the previous cycle properties.

In step S34, it is then checked whether new or changed objects have been found in the preceding distance image. this is not the case, the method continues with step S22.

Otherwise, in step S36, a position of the corresponding object to the new object and the modified object are determined in the preceding distance image. The new object is then treated as a known object, its location in the preceding cycle is known, what changes have to be made in data structures of the program and a corresponding Kalman filter for the previous cycle is initiated subsequently. Furthermore it is possible for the altered object a new prediction of the location for the current cycle in order to improve the accuracy of the Kalman filter.

There are therefore in the current cycle object positions for the previous and current cycle for known in the preceding cycle and in step S32, newly discovered properties obtained, so that in step S22 can now be carried out to estimate the velocity of these objects, the object positions in the preceding distance image or cycle can be used.

The subsequent steps S24 to S28 extend as in the first embodiment.

Characterized in that the object or pedestrian 26 in the current cycle is already a speed can be associated with the initialization of the Kalman filter in the current cycle as a result, substantially already on the basis of data for this object from the previous cycle and thus overall with accurate data occur, which substantially facilitates the following object recognition and tracking.

In this way it is possible, for example, one is assigned directly in front of a vehicle is parked moving pedestrian, who then takes to the road, after processing only a distance image in which he was recognized as a single object, a trajectory and hence speed.

Basically, the method of the first two embodiments may be combined. For example, only a check in step S30 carried out and, if no new objects were found in step S34, a masking detection according to the first embodiment.

In the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> is in a situation in <figref idrefs="f0001">Fig. 1</figref> corresponding situation a vehicle 10, which carries a device for the recognition and tracking of objects according to a third preferred embodiment of the invention.

The apparatus for detection and tracking of objects different from the apparatus for the recognition and tracking of objects in the first embodiment in that in addition to the laser scanner 12, a video system 30 is provided which, as the laser scanner 12 also, via a corresponding data connection with a is connected over the data processing device 14 modified in the first embodiment, data processing device 14 '. respectively the same reference numerals are used for the other, like parts or features, and the explanations for the first embodiment apply accordingly.

The video system 28 has a monocular video camera 30, which is to a conventional black and white video camera with a CCD area sensor 32 and an imaging device that in <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> is shown schematically as a simple lens 34, but actually consists of a lens system, and reflects incident light from a detection area 36 of the video system to the CCD area sensor 32nd An optical axis 38 of the video camera 30 is in a low, in<figref idrefs="f0008">Fig. 8</figref> exaggeratedly large angle shown to be caused by the pivoting plane of the laser radiation beam 20 scanning 40 of the laser scanner 12 to inclined.

The CCD area sensor 32 includes in a matrix arranged on the photodetection elements, which are read out cyclically to form video images with video pixels. The video images for each pixel contained in each case initially the position of the photodetection elements in the matrix or other identifier for the photodetection elements, and respective one of the intensity of the received from the corresponding photodetection element corresponding light intensity value. The video images are recorded synchronously in this embodiment essentially with the distance images and thus at the same rate, is recognized when the well of the laser scanner 12 range images.

Of an object, in the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> the person 16, outgoing light is imaged by the lens 34 onto the CCD area sensor 32nd This is illustrated in<figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> schematically indicated for the outline of the object or the person is only schematically shown 16 by the short dashed lines.

From the distance of the CCD area sensor 32 and lens 34 and from the position and the imaging properties of the lens 34, for example, the focal length, can be made of the position of an object point, for example, the object point 22 on the person 16 are calculated on which place the CCD -Flächensensors 32 and which is mapped to the arranged as a matrix photodetection elements of the object point. Conversely, from the position of a photodetection element by the short dashed lines in<figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">Fig. 8</figref> an indicated beam are determined, in the object points or regions must lie from which sensed radiation can fall on the photodetection element. For this purpose, a corresponding camera model used. The example is a per se known pinhole model.

A common detection area 42 is in the <figref idrefs="f0007">Fig. 7</figref> and <figref idrefs="f0008">8th</figref> schematically approximately represented by a dotted line and, by the intersection of the detection region 18 of the laser scanner 12 and the detection region 36 of the video system 28th

The data processing device 14 'differs from the data processing device 14 firstly by the fact that interfaces are provided to substantially synchronously, ie one current within a period, which is significantly smaller than the period .DELTA.t between successive samples of the detection region 18 of the laser scanner 12, distance image and a current video image of the video system 28 to read. On the other hand the processor is programmed with a computer program according to a third preferred embodiment of the invention for performing a method according to a third preferred embodiment of the invention. The data processing device 14 'therefore includes means for evaluation of current images of the sensor, means for determining parts of a current image as a result of an at least preliminary evaluation of the current image or for the recognition of an object in the current cycle and means for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or of an appropriate object on the basis of a corresponding previous image and assigning the item to the part of the current image or with the current object in the sense of the invention.

On the basis of of the laser scanner 12 and the video system 28 substantially synchronously detected distance images and video images, the in <figref idrefs="f0009">Fig. 9</figref> illustrated process is carried out according to the third preferred embodiment of the invention. In the method of this embodiment, features are detected and tracked in the video images, but the video image processing of computing time, made on the basis of the distance images, by looking for features only in those portions of the video image in which distance picture elements of the distance image and thus corresponding item ranges.

First, in step S38, a current distance image and a current video image are read and preprocessed. The reading and preprocessing of images can be carried out independently for the two images in parallel, or in any order.

The distance picture elements in the range image is not in addition to the position coordinates in the scanning plane 40 a through the scanning plane 40, in the corresponding to the distance picture elements object points are added certain location component to form a complete position coordinate set in three dimensions. With the position of a distance picture element defined by this coordinate position is referred to below.

Furthermore, for possible correction of the video image, transforms the data of the video image in the vehicle coordinate system in which the distance picture elements are defined. For this purpose, a rectification of the video image data, for example, to eliminate distortions, and a transformation of the video pixels is performed on an image plane. By means of the camera model for the video camera 30 can then be in the vehicle coordinate system assigned to a respective level layers of the video pixels.

The current distance image and the current video image will be saved for further use.

In step S12, then, as in the first embodiment, the distance image is segmented.

In the following step S40 features are detected in the video image, said of the video image only ranges are used, in which, according to the distance image, a one segment must be detectable in the distance image corresponding object. More specifically, the detection is carried out only in running essentially perpendicular to the scanning plane of the laser scanner 12, a respective segment of the distance image corresponding stripe of the video image having the distance picture elements in the segments corresponding video pixels. The strips are defined such that they extend orthogonally in a direction perpendicular to the scanning plane 40 of the laser scanner 12 over the entire extent of the video image, and have in the orthogonal direction, a width which is determined so that all distance picture elements in each of a segment corresponding video pixels in each strip are. This video pixels can be determined by use of the camera model. Furthermore, the strip in width at each predetermined edge regions on both sides of each outermost video pixels. Since other parts of the video image are initially ignored, done so far an attention control of the video image processing.

In this step still an assignment in the current video image is performed found dead features known from the previous cycle characteristics using a predicted in step S49 of the previous cycle in each case for the current cycle position of the respective features. In this case, positions of the features in the current cycle are determined simultaneously.

In step S41 it is then checked whether new features were found that could not be assigned to known from the previous cycle characteristics.

Are there new features found are searched in step S42 according to the new features corresponding features in the preceding video image by backtracking. For this purpose, known methods of video image processing can be used.

In step S44, the position of a corresponding new characteristic feature in the preceding video image is then determined.

S46 speeds of characteristics on the basis of the positions of features in the preceding video image and the positions of the features in the current video image then estimated in step.

The determined feature positions and velocities are then output in step S26 after storing for further use.

In step S48, the current video image will be saved after the previous video image has been deleted.

In step S49, new feature layers now be predicted, so that an assignment of features in step S40 will be facilitated in the following cycle to objects. For this purpose, for example, the now well-known speeds or a Kalman filter may be used.

Starting from the at least preliminary evaluation of the distance and the video image in the steps S38, S12 and S40 is performed so a subsequent or additional analysis of a preceding video image. The results of this analysis, the located feature or its location in the preceding video image are then used in the current cycle to determine the speed of the current feature.

The use of the method for detecting sudden protruding behind parked vehicles pedestrians is in the <figref idrefs="f0010">FIG. 10A and 10B</figref> and 11A and 11B. The<figref idrefs="f0010">Fig. 10A</figref> and <figref idrefs="f0011">11A</figref> each show a section of consecutive, sensed by the laser scanner 12 distance image at the height of the scanning plane 40, while the <figref idrefs="f0010">Fig. 10B</figref> and <figref idrefs="f0011">11B</figref> show areas of corresponding substantially synchronously acquired with the video system 28 video images. In these Figures, the scanning plane 40 of the laser scanner 12 is shown by a dotted line. Furthermore, it is equivalent to<figref idrefs="f0010">FIG. 10A and 10B</figref> Cartesian coordinate system used in the previous embodiments.

In the <figref idrefs="f0010">Fig. 10A</figref> and <figref idrefs="f0011">11A</figref> the vehicle are shown schematically by a rectangle 44 and the pedestrian by a rectangle 46th

First, is the pedestrian, as in the <figref idrefs="f0010">FIG. 10A and 10B</figref> shown, of the laser scanner 12 can not be detected because it is hidden by the vehicle 44th Since the detection area 36 of the video system 28 but in a direction perpendicular to the scanning plane 40 a larger opening angle than the detection range 18 of the laser scanner 12, can already be seen in the corresponding video image of the pedestrian 46th Due to the attention control this is not detected in the video image, as it in the distance image (see.<figref idrefs="f0010">Fig. 10A</figref>) Is obscured.

Since the pedestrian 46 in the example with a speed v moves to the right, he is no longer covered after some time by the vehicle 44, which in the <figref idrefs="f0011">FIG. 11A and 11B</figref> is shown. Now the pedestrian can be detected 46 in the range image so that it is detected in the video image via the attention control. While in a conventional video image processing nothing would now known about this except the location of the pedestrian 46, is in the process of the embodiment of the pedestrian 46 time traced in the video images, which he in the in<figref idrefs="f0010">Fig. 10B</figref> shown video images can be recognized. The resulting finding made in the previous video image now allows to determine the speed of the first recognized only in the current video image pedestrian 46 even at its first detection or recognition. Since the tracking occurs only when the discovery of new features, the speed of execution of the method is only slightly reduced on average.

In a method for the detection and tracking of objects according to a fourth preferred embodiment of the invention is essentially the same device for the detection and tracking of objects as used in the third embodiment, is but also changed the programming of the data processing device 14 'according to the modified method , This also means that the funds for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and assigning the item to the part of the are the current image or the current object modified.

With this in <figref idrefs="f0012 f0013">FIG. 12A and B</figref> schematically illustrated methods are pursued distance picture elements in successively sensed distance images using optical flow in respective areas in substantially synchronously detected with the distance images video images.

Here a mapping of objects and representative thereof distance picture elements is used to one another, so that a tracking of objects can be done by following the corresponding object associated distance picture elements. Regarding details of some steps of the present process and in relation to possible variants of the first embodiment and the other embodiments in the German patent application with the assignee of the present application with the official file reference<patcit id="pcit0003" dnum="DE10312249"><text>DE 103 12 249</text></patcit> referenced.

In step S50, a current cycle, first a current distance image and a current video image are detected and read into the data processing device in which are preprocessed these images corresponding to step S38 in the previous embodiment. The transformation of the video pixels is however modified to in the foregoing embodiment, for ease of calculation of the optical flow over the step S38, the s transformation of video pixels on a used to define or calculate an optical flow image plane 48 (see FIG.<figref idrefs="f0014">Fig. 13</figref>) is carried out. For one inspection in steps S52 and S60 projection of distance picture elements in the image plane for simplicity, a principle to the expert uses known modified Pinhole or ground glass model of the video camera 30, represented by the position of an optical center 50 and the image plane 48 is defined, which serves as a surface for defining or determining the optical flow. The location of the optical center 50 is using the imaging geometry of the video camera 30, in particular the position relative to the laser scanner 12 and the focal length of the lens 34 is determined. The pinhole camera model is modified for ease of illustration, that the image plane 48 to a relative to the video system 28 fixed to the vehicle 10 position between the optical center 50 and the object points, in<figref idrefs="f0014">Fig. 13</figref> the points 52 and 58 is, and seen by point reflection of the actual image plane of the optical center 50 of these.

In step S52, which is appropriate in the first cycle of the process, all distance picture elements of the current cycle immediately preceding cycle are initially determined which correspond to 12 objects removed from the sensor or laser scanner. For simplicity, these distance picture elements are referred to as receding objects corresponding distance picture elements. Using the camera model corresponding distance picture elements are then from the current cycle immediately preceding cycle corresponding positions or coordinates of corresponding video image points in the image plane 48 is calculated and stored for use in the current cycle of this receding objects. As in<figref idrefs="f0014">Fig. 13</figref> geometrically illustrates the situation results in each case by the intersection 54 a passing through the distance picture element, in Figure the distance picture element 52, and the optical center 50 degrees with the image plane 48th

In step S54, which is also omitted in the first cycle, are then projected for all, moving away from the laser scanner 12 objects, hereinafter also referred to corresponding distance picture elements of the preceding cycle or corresponding intersections corresponding current optical flow vectors just as optical flows, on the basis of the video image obtained from the immediately preceding cycle and of the current cycle and converted by multiplication with the cycle time or the inverse of the sampling frequency in a displacement vector. In<figref idrefs="f0014">Fig. 13</figref> shown for the distance pixel 52, for at the intersection 54, an optical flux vector is determined, which lies in the image plane 48 and the after scaling with the cycle time a starting position at the intersection 54 displacement vector 56 results. The optical flow is determined according to a differential method, in this example, the in "<nplcit id="ncit0003" npl-type="s"><text>Performance of optical flow techniques "by JL Barren, DJ fleed and SS Beauchemin, International Journal of Computer Vision, 12 (1), pp 43-77 (1994</text></nplcit>) Method described by Lukas and Canade.

For all receding objects corresponding distance picture elements for which an optical flow was determined, is then determined in step S56, a predicted by the optical flow position of a corresponding object point in the current cycle in the scan 40th Geometric in<figref idrefs="f0014">Fig. 13</figref> illustrates the intersection is determined to 57 a plane passing through the optical center 50 and the end point of the displacement vector 56 lines with the scanning 40th

In step S58 are current distance picture elements in <figref idrefs="f0014">Fig. 13</figref> assigned for example the current distance picture element 58, the current distance image to the predicted positions and corresponding distance picture elements of the preceding image or cycle where possible. Allocation criteria can be used for example, that the one current distance picture element is assigned, whose squared distance between the predicted position and the actual current situation is minimal compared with those of other current distance picture elements.

In step S60, then the unassigned distance picture elements of the current cycle, which therefore do not have to comply by the laser scanner 12 removed objects, projected onto the image plane 48, which is analogous to the projection in the step S52.

S62 are then for all projected, not yet associated distance picture elements of the current cycle in step calculated analogously to step S54 optical flows.

For all not associated distance picture elements of the current cycle for which an optical flow was determined, a retraced position of a corresponding object point in the scan plane 40 is now determined in step S64. The determination of the retraced position takes place analogous to the determination of the predicted position in step S56, but using as the optical flow vector is a displacement of oppositely directed appropriate vector.

In step S66, then an assignment of distance picture elements of the previous cycle that do not correspond from the sensor removed objects to be predetermined, retraced positions and corresponding not yet associated distance picture elements of the current cycle in step S64. The assignment can be made for an appropriate scheme, as in step S58, but now predetermined distance picture elements of the current cycle and the corresponding documents retraced distance picture elements of the preceding distance image are assigned.

So removing the pixels of the previous cycle and the current cycle and the image in this way are associated with each other, whereby the allocation for moving away from the laser scanner 12 objects is performed such that current distance picture elements are associated with distance picture elements of the preceding image, while the remaining distance picture elements of the preceding image that are not assigned 12 removing objects from the sensor or laser scanner, the not yet associated distance picture elements of the current cycle and the distance image or assigned their back tracked positions.

In step S68 will now be a segmentation of the current distance of the image according to the segmentation in step S12 of the first embodiment.

In step S70 a segment-object association is then carried out, wherein the assignment of distance picture elements of the previous image and the current distance of the image distance as well as the association between distance picture elements of the previous distance image and objects of the previous cycle are used.

In step S72, object properties, in particular their positions and velocities, determined and output.

The process can then be continued with step S50 in the next cycle. A prediction can be eliminated by the use of optical flow.

In other variants of the method according to the fourth embodiment, the projection in a different manner can be carried out, examples of this are described in the referenced patent application in other embodiments, which are hereby expressly incorporated by reference in the description.

By treating other distance picture elements that do not match from the laser scanner 12 removed objects, a clearer association of distance picture elements is achieved. Because in the current cycle are due to the imaging geometry of the laser scanner 12 which scans its detection range 18 radially, such objects more each distance picture elements correspond as in the previous cycle. It may then an allocation of more distance picture elements of the current cycle to prevent, for instance a predetermined distance image point of the previous cycle in which may occur for themselves the laser scanner approaching objects the event that more distance picture elements of the current image would be associated with a predicted distance picture element, which, however, can bring a complex allocation method because of the ambiguity.

In a method according to a fifth preferred embodiment of the invention, an object recognition and tracking takes place only on the basis of video images. A corresponding device for object recognition and tracking according to a fifth preferred embodiment of the invention therefore comprises a used as in the previous two embodiments, video system, the video camera is connected to a data processing device as in the previous two embodiments, which, however, for implementing the method fifth after preferred embodiment is programmed. This also means that the funds for the association of parts of an earlier image and / or for determining at least one indication with respect to an earlier state of the object or a name corresponding object on the basis of a corresponding previous image and assigning the item to the part of the are the current image or the current object modified. The laser scanner 12 is omitted.

With this in <figref idrefs="f0015">Fig. 14</figref> illustrated process is first detected a current video image in a current cycle in step S74 and read. In this case, can take place as in the previous two embodiments, a corresponding pre-processing the video image data.

In step S76, the captured video image of the image resolution reducing filtering is subjected to increase the processing speed, which is in the present embodiment, that of 4x4 blocks of video picture elements, or pixels, only the pixels is used in the lower left corner.

In step S78, then objects are detected in the sub-sampled video image. To this end, conventional methods of object recognition and tracking can be used in video images.

In step S80 will now be given an assignment of the detected objects to objects from the previous cycle. The assignment is done in the example based on the predicted object positions for the current cycle, which have been determined in the preceding cycle in a step S94, by means of known association methods.

In step S82, it is checked whether new objects were detected, none of the objects of the preceding cycle were assigned. If no new objects detected, the method continues with step S90.

Otherwise, first in step S84 in the preceding video image, which is not, or in the example only half as much as sub-sampled the image generated in step S76, search for an object corresponding to a new object.

Due to the increased resolution, objects in the preceding video image thus detected that would otherwise not be seen due to undersampling.

This is illustrated in <figref idrefs="f0016">FIG. 15A and 15B</figref> and 16 shown again for yourself the video system approaching person. In<figref idrefs="f0016">Fig. 15A</figref> the person is shown schematically in a current video image full resolution, with black rectangles representing video pixels or pixels that match the person. Through the above-mentioned sub-sampling results in the in<figref idrefs="f0016">Fig. 15B</figref> shown undersampled video image in which the person is only represented by four individual pixels. Has now the person approached from a distance of the video camera, so they can in the preceding video image at full resolution, for example, in<figref idrefs="f0016">Fig. 16</figref> have size shown, wherein after undersampling no pixel over remain, by means of which a corresponding object could be detectable.

The method used here is, however, wanted in the previous image in full resolution and not in the lowered resolution after the new object, which can be therefore found. Since the new object or the object can occur only in the vicinity of the detected in the current video image object, the calculation time for the recognition of the object in the preceding video image is greatly reduced, nevertheless.

In step S88, the amount equivalent to the new objects objects are tracked in the preceding video image in the current cycle in what is easily possible because the object has been detected already in the undersampled current video image.

Due to the determined positions of the objects in the preceding video image and thus in the previous cycle and the current cycle S90 velocities of objects can now be calculated by estimating or subtraction in step.

In step S92, the current video image is then stored, the previous video image is deleted.

In step S94, an output of the object positions and speeds takes place for use in the following applications.

new object positions for the following cycle will be on the basis of the object positions and velocities then in step S96, predicted, near which will search for objects in the video image of the next cycle. To this end, a Kalman filter can be reused.

The method according to the fifth embodiment allows the one hand by the sub-scan speed of execution. However, for the other, it is possible for approaching, new and thus critical for a vehicle objects, determine their speeds at its first appearance already using a previous video image, so that in an emergency situation, the reaction time can be shortened by one cycle duration.

LIST OF REFERENCE NUMBERS

<dl id="dl0002" compact="compact"><dt>10</dt><dd>vehicle</dd><dt>12</dt><dd>laser scanner</dd><dt>14, 14 '</dt><dd>Data processing device</dd><dt>16</dt><dd>object</dd><dt>18</dt><dd>detection range</dd><dt>20</dt><dd>Laser radiation beam</dd><dt>22</dt><dd>Subject point</dd><dt>24</dt><dd>vehicle</dd><dt>26</dt><dd>pedestrian</dd><dt>28</dt><dd>video system</dd><dt>30</dt><dd>video camera</dd><dt>32</dt><dd>CCD area sensor</dd><dt>34</dt><dd>lens</dd><dt>36</dt><dd>detection range</dd><dt>38</dt><dd>optical axis</dd><dt>40</dt><dd>scan</dd><dt>42</dt><dd>common detection area</dd><dt>44</dt><dd>vehicle</dd><dt>46</dt><dd>person</dd><dt>48</dt><dd>image plane</dd><dt>50</dt><dd>optical center</dd><dt>52</dt><dd>Subject point</dd><dt>54</dt><dd>intersection</dd><dt>56</dt><dd>displacement vector</dd><dt>57</dt><dd>intersection</dd><dt>58</dt><dd>Subject point</dd></dl><dl id="dl0003" compact="compact"><dt>v</dt><dd>velocity vector</dd><dt>d</dt><dd>displacement vector</dd></dl>