Method of segmenting three-dimensional scenes.

0001The invention relates to a method for segmenting three-dimensional scenes for controlling an unmanned free-moving vehicle as well as a device and circuit arrangement for carrying out the method.

0002Aids are used to control free-moving unmanned vehicles, which the vehicle can use to orient itself by visual observation. In the simplest case, such aids can be strips arranged on the guideway or on the edge of the guideway; it can also be information carriers that are embedded in the floor, with which the vehicle communicates and can control its further movement in direction and speed from the information obtained in this way. In this respect, the movement of an unmanned free-moving vehicle is possible both inside and outside.

0003Difficulties in the movement of a free-moving unmanned vehicle arise, however, in that obstacles occur in the driving space in front of the vehicle, which should be free in itself, which are formed, for example, by objects parked without authorization or by people crossing the road; third-party vehicles can also be present in the driving space and become an obstacle. In closed rooms or halls, just like outdoors, hanging objects, e.g. B. Crane hooks that are present in the area of ​​the driving space, hinder the unmanned, freely moving vehicle.

0004In order to avoid such disabilities, a wide variety of protective devices have hitherto been arranged in and on freely moving unmanned vehicles or on the edges of the driving spaces. These consist, for example, of physical lane boundaries against which the unmanned free-running vehicle can collide in the event of incorrect control, the collision causing the propulsion of the vehicle to be switched off. Such protective devices require that the unmanned free moving vehicle is moved at a very low speed in order to be stopped without having to fear significant damage to the obstacle or the vehicle. In particular, this precautionary measure applies if it cannot be ruled out that people can be in the driving space of the unmanned, freely moving vehicle.

0005In the case of docking maneuvers of such vehicles at docking points, for example for loading or unloading goods or before starting such vehicles, continuous monitoring of the driving space in the current direction of travel of the vehicle is likewise not possible, so that docking maneuvers must be carried out with other aids, for example by means of a Forced operation of the unmanned vehicle shortly before the docking point. The same applies at the start of the journey of an unmanned mobile vehicle, in which the previous optical devices do not make it possible to detect moving objects in front of the vehicle in order to abort a journey that is already beginning before a rear-end collision.

0006Proceeding from this, the invention has for its object to continuously ensure a contactless monitoring of the driving space without a delay in the speed of the vehicle, with which clear collisions are avoided and also continuous docking maneuvers are to be carried out, and an obstacle that occurs is clearly recognizable even before the start of the journey. This object is achieved by the inventive features of method claim 1. By recording a three-dimensional scene from at least two different perspectives and transforming the binocular area of ​​at least one of these recordings to the perspective of the other recording, based on an area, and by subsequently comparing the recording according to the invention to which the transformation relates the transformed recording; or where both recordings are transformed into an imaginary perspective and the comparison is then carried out, the three-dimensional scene is segmented in such a way that only what is present outside the surface is recognized and that control commands can be derived in the event of an inequality. As a result, the segmentation of the binocular area is carried out within a practically continuous scanning cycle, that is to say in real time, so that during the practically continuous segmentation neither the vehicle speed has to be reduced, nor does a continuous docking maneuver have to be interrupted and the starting process can also be carried out continuously.

0007In an embodiment of the subject matter of the invention, it is proposed in claim 2 to simultaneously record the three-dimensional scene from at least two different perspectives. With this type of recording, in particular shortly before the start of an unmanned free-moving vehicle, the driving space in front of the vehicle is monitored and objects penetrating there immediately, such as crane hooks or a person moving through it, are immediately recognized. In this case, the starting process can be delayed until the starting place appears free due to further recordings.

0008Such a static way of looking at things, which happens in real time, makes it possible for the first time to determine with an unmanned vehicle before it starts, when a plurality of pictures of the three-dimensional scene are taken, whether a static object would hinder the journey or whether a moving object would hinder the road of the vehicle crosses.

0009Another embodiment of the subject matter of the invention is to record the three-dimensional scene in succession from at least two different perspectives. This configuration serves on the one hand to observe the driving space in front of the vehicle, during the starting process and on the other hand as routine monitoring of the driving space in front of the vehicle while driving.

0010Another inventive solution to the problem is protected in claim 4 and is characterized in that the three-dimensional scene is recorded at least twice in succession with the vehicle moving under the same perspective with respect to the coordinate system of the vehicle, that an orthogonal plan view of the from each shot binocular area of ​​the surface is formed, from which a velocity vector field is calculated, and that when the vector amounts change, a control command is issued to the unmanned vehicle. This inventive procedure likewise suppresses elements painted on the floor and does not lead to any incorrect messages being sent to the vehicle; moreover, the speed amount display indicates the extent of an object beyond the surface and it can be concluded from this that the height of the object and at the same time a signal is sent to the vehicle if this height could become a danger to the vehicle. The phases of the vector field are normalized in particular by the orthogonal top view, which surprisingly leads to a particularly simple evaluation of the vector field, after which only the amounts of the speed vectors occur.

0011In an embodiment of the invention according to claims 1, 2 and 3, protection is given in claim 5 that objects present on the surface, the vertical extension of which is smaller than a predetermined extension than belonging to the surface, are not segmented. As a result, the real-time evaluation of the contactless observation is further simplified according to the invention, since this makes objects on the surface visible, for example, but due to the small extension in the Z direction, that is to say perpendicular to the surface which is also the driving surface of the unmanned vehicle, are not taken into account.

0012The measure protected in claim 6 that existing objects on the surface whose speed vector in the horizontal direction is smaller than a speed vector specified in this direction than belonging to the surface are not segmented. This also serves to refine the control to be passed on in real time from the driving level, or to simplify only minimally superior objects; this is the case, for example, when pictures are painted on the floor or flat objects such as checker plates, cardboard paper of the roadway that do not hinder the driving of an unmanned vehicle.

0013Claim 7 provides a particularly expedient circuit arrangement for segmenting three-dimensional scenes after ascertaining the outer and inner orientation of an optical system for controlling an unmanned free-moving vehicle, in which, according to the invention, the gray values ​​coming from a binocular area act on camera targets provided with pixels and according to the invention not all pixels of the targets are queried. The inventive selection of a part from all acted on pixels enables rapid evaluation, namely evaluation in real time for the first time.

0014Claim 8 shows another circuit arrangement for segmenting three-dimensional scenes after determining the outer and inner orientation of an optical system for controlling an unmanned free-moving vehicle, in which, according to the invention, the gray values ​​recorded from a selected area act on a camera target provided with pixels, According to the invention, not all pixels of the target are queried. Thus, even in the case of the selection of an area, rapid evaluation in real time, that is to say the gray values ​​of the selected area, is achieved. It does not matter which coordinate system the pixels arranged on the targets or the target are arranged according to.

0015In claim 9, an embodiment of the inventive circuit arrangement is protected, in which the pixels are arranged in a right-angled coordinate system on the target; according to the invention, not all of the pixels arranged in the x or y direction of the target are queried there.

0016In yet another embodiment of the inventive circuit arrangement, protection is given in claim 10 that the pixels are scanned at least one-dimensionally, with non-scanned pixels being present between the scanned pixels. Another embodiment provides that the pixels are scanned at least one-dimensionally, with non-interrupted pixels being present between uninterrupted groups of queried pixels, and that an average value is passed on from each uninterrupted group of pixels. As a result of the inventive design of the circuit arrangement, according to the invention a number of pixels is selected from the target area of ​​the camera that is smaller than the number of pixels of the entire target area, so that a very quick query option is achieved which at the same time contains the considerable advantage, that the gray value detection of the individual pixels provided with tolerances is improved by a kind of averaging and, on the other hand, in particular as a measuring device for carrying out the method according to claim 1 in the transformation of an image from one camera to the image from the other camera, definitely changes in gray value, which indicate an obstacle let be clearly recognized. This makes it possible for the first time that the images are transformed in rapid succession, so that results that can be evaluated immediately are available during the individual scanning cycle.

0017In order to increase the speed of the evaluation, protection is provided as a configuration of the circuit arrangement in claim 12 that the selection of the interrogation takes place dynamically depending on an overall processing time. By dynamically adapting the selection, it is possible to coarsen the selection of the query in areas in which there definitely cannot be any obstacles, but immediately refine the selection of the query to detect this obstacle when an obstacle appears.

0018With claim 13, the use of the method for monitoring a changing state is protected according to the invention; For example, for access monitoring, or for contactless room and field monitoring against unauthorized entry, or for monitoring a manufacturing process.

0019The method is explained in more detail using exemplary embodiments on the basis of a schematic sketch which shows the unmanned vehicle, including the recording devices in a view and a top view, and in connection with correspondingly obtained recordings. It is pointed out that the receiving devices do not have to be arranged symmetrically to the center line of the unmanned moving vehicle, but that these receiving devices can also have a different position relative to one another, so that the area can be not only the driving area of ​​the vehicle, but also that to be segmented Objects can also be arranged on a surface, which runs as a docking surface, for example, perpendicular or at a certain angle to the driving surface.

0020Using other figures, the example of the simultaneous recording of the binocular area is used to explain how, after transforming one image in the direction of view of the other and then subtracting both scenes from one another as a result, with an undisturbed surface, extinction takes place, while if an object is present on the Surface a fault occurs.

0021In yet another representation, the moving vehicle is shown at a fixed angle to the horizontal driving surface observed, an orthogonal plan view of the driving surface being formed from this recording and each part segmented out of the driving surface being divided by a difference in the speed vector to the uniform one Represents the speed of the driving surface.

0022In the drawing, the method according to the invention and a circuit arrangement according to the invention are explained in more detail using schematic exemplary embodiments: Show it:<ul id="ul0001" list-style="none"><li>Figure 1 a the top view of a moving unmanned vehicle.</li><li>Figure 1 b is a side view of the moving unmanned vehicle.</li><li>Figures 2 to 9 results of an observed driving space.</li><li>Figure 10 is a moving unmanned vehicle with an obstacle in the driving space.</li><li>11 to 15, results of an observation according to FIG. 10.</li><li>Figure 16 shows a circuit arrangement</li><li>17 shows a representation of the transformation from a camera 1 to a camera 2, as well as a mathematical derivation, for the representations disclosed in FIGS. 1 to 9.</li><li>FIG. 18 shows a transformation to a vertical virtual camera, as explained in more detail in FIGS. 11 to 15, and a mathematical derivation.</li></ul>

0023An unmanned vehicle 1 standing on wheels 5 will scan the driving space in chronological order by means of two cameras 2 and 3 , according to the exemplary embodiment according to FIG. 1 a and FIG. 1 b, for example shortly before its start. Both cameras 2 and 3 cover a binocular area 6 in the area that is to be segmented. In the exemplary embodiment, this is a carriageway 7. Instead of the carriageway 7, however, there can also be a docking station which is arranged vertically on the carriageway 7, for example it can be a ramp on which the unmanned vehicle must dock so that goods to be transported can be driven onto the unmanned vehicle.

0024FIG. 2 shows, for example, the binocular area 6 from the point of view of the camera 2, while FIG. 3 shows the binocular area 6 from the point of view of the camera 3. If the recording of the camera 3 is now transformed into the perspective of the camera 2, FIG. 4 results in FIG. 4. As can be seen from a comparison between FIG. 2 and FIG. 4 in the binocular area 6, the lines arranged there on the roadway are congruent and thus are deleted when subtracted, so that the undisturbed image of the binocular area appears in FIG. 5.

0025In Figures 6 and 7, an obstacle is shown on the driving surface 7. If the binocular area of ​​camera 3 shown in FIG. 7 is now transformed into the perspective of camera 2, the markings in the difference image disappear, but the object itself remains, as can be seen in FIG. 9, as a disturbance of the binocular area. The subtraction of the two images has made it possible to uncover an obstacle on a surface, the obstacle being segmented from the images.

0026In order to carry out such segmentations, however, it is necessary that the position of the cameras in relation to the vehicle is first calibrated in order to then be able to correctly locate images on the target of the camera with respect to vehicle 1. The prior art already knows how to calibrate cameras (see R. Lenz, lens error-corrected calibration of semiconductor cameras with standard lenses for high-precision 3-D measurements in real time, ninth DAGM Symposium Spt./Okt. 1987)

0027If a series of such recordings is now made when the vehicle is standing, shortly before its departure, then in the static case of the stationary vehicle a moving something, be it a person or a crane hook, can be recognized and the direction of travel can be selected either that the evasive vehicle emerges from the obstacle area, for example the crane hook, or it can be waited until the crane hook is removed from the binocular area 6.

0028If, according to FIG. 10, the unmanned vehicle moves on the road 7 at a certain speed in accordance with the arrow 9, the road 7, according to the exemplary embodiment, covers a certain spatial section on the road 7 by a single camera 10. Images of section 11 taken by this camera 10, cf. 11, are now fed to a computer for evaluating the speed vectors; the section that is moved relative to the one camera and has a cuboid thereon is thereby determined. This results in a group of speed vectors of different sizes from the driving surface, if this is provided with markings, and also the cuboid 12 shown in FIG. 11 is represented by speed vectors of different sizes. This results in a difficult determination of the obstacle according to changing speed vectors.

0029However, if, according to FIG. 13, an orthogonal top view of the cuboid 12 according to FIG. 11 is calculated in a computer (not shown), a virtual image is created which is the same as an image of an imaginary camera orthogonally observing the driving surface 7 on the three-dimensional scene. It can be seen from the transformed image according to FIG. 13 that the square grid used to mark the surface for the observer who is moving along presents itself as velocity vectors of the same size, while the cuboid segmented from the surface, depending on its increasing vertical height to the driving surface 7, increases the speed vector sizes receives. With such an observation, depending on the speed vector size, it can thus be estimated in the simplest manner which obstacles can be passed over without damage, for example a protruding rail body, and from which objects, for deceleration or stopping, control commands must be passed on to the control of the unmanned vehicle, to decelerate and ultimately stop the vehicle, or to avoid the obstacle.

0030FIG. 16 shows a circuit arrangement for the flow of signals from a video image recording to control signals, which effects vehicle control of the vehicle shown in FIGS. 1 a, 1 b and 10, for example. Video signals 20 are digitized by the cameras 2, 3, which are embodied as CCD cameras in the exemplary embodiment, supplied to a video acquisition unit 21 in line-alternating fashion, for example with 8 bits, and written into a video memory 23 in real time via a digital interface 22. Instead of one video acquisition unit, several can also occur. At the end of the video memory 5, the information to be processed in real-time video is available so that a digital computer 24 has access to the video memory 23 via a computer bus 26 in such a way that from there only selected pixels with their gray value information, of which an average is still formed, are taken over be able to carry out perspective transformations and subsequent comparison and evaluation practically continuously there, in order to be able to pass on evaluable information via a vehicle controller 25 to the vehicle shown in FIGS. 1 a, 1 b and 10. The control signals derived in the digital computer 24 reach the vehicle control 25 via a line 26, so that from here the evasive, acceleration or braking maneuvers can be passed on to the vehicle and can also be checked at the same time.

0031According to FIG. 17, it is assumed that the external and internal orientation of a visual system, in the exemplary embodiment cameras, is known, and it is assumed that the vehicle is moving on one plane. Assuming that the level is object-free (i.e. there may be two-dimensional structures, but no upright components), the images of the level on the camera targets are clear. Under this boundary condition (Z = 0) applies for the projection of a pixel P '(x', y ', - c_small) from camera 2 or camera 3 to a point P_e(X_e, Y_e, 0) in the plane:<maths id="math0001" num=""><img file="EP0360880A1_D0001.tif" /></maths>

0032For the projection of point P_e on the corresponding pixel P˝ (x˝, y˝, -c_k2) = (x_t˝, y_t˝) of the target of camera 2 applies:<maths id="math0002" num=""><img file="EP0360880A1_D0002.tif" /></maths>

0033The rotation matrix A is a function of Euler's angles, which describes the rotation of the camera coordinates compared to the world coordinates.

0034According to Figure 18, it is also assumed that the inner and outer orientation of the camera is known.

0035From the equation for the central projection follows: P: R³ → R², (x, y, z) → (x, y): = f / z (x, y) (G1.5) For a linear translational motion of a camera m, x = (x, y, z) are replaced by x = x₀ - mt. For the projection of a point in the world, the shift in the image is calculated as: m_I. : = -J_p(x) (Eq.6a) = -f / z (m₁ - m₃ / z, m₂ - ym₃ / z) (Eq.6b) m₁, m₂ and m₃ are the movement components of the own movement m, and J_p the Jacobi matrix of P. A world coordinate system is defined for the projection of a pixel onto the plane of movement, the origin of which lies in the point of intersection of the optical axis through the plane of movement. 0: = (0, 0, h / sin β) P: = (1, 0, 0) q: = (0, cos β, sin β) r: = (0, sin β, cos β) (Eq. 7) For a point x in the plane of movement_H applies: x_H(p, r): = 0 + pp + rr (Eq. 8) Equations 5 to 8 are used to calculate the projection of a target point onto the movement plane: IP: R² → R² (x ′, y ′) → (p, r): = h / (f sin β - y′cos β) (x ′, y ′ / sin β) (Eq. 9) and the flow field of the undisturbed plane of motion to: m_I.(x ′, y ′) = -J_p(x_H(IP (x ′, y ′))) m (Eq. 10) The perspective transformation IP parallelizes the direction and normalizes the magnitude of the flow fields of surfaces that are parallel to the plane of motion. The speed of the own movement is equal to the smallest speed vector in the transformed image. The position for any point (p, q, r) in the perspective-transformed image is calculated as: IP o P: R³ → R² (p, q, r) → (p ′, r ′) = h / (h - q) (p, r + q cot β) (Eq. 11) and its shift due to camera movement too m_I. = J_IPO_P(p, q, r) (m_p, 0, m_r)^T= h / (h - q) (m_p, m_r).

0036It is assumed that R³ represents space while R² represents a plane.