Method for drift correction of an optical device

The invention relates to a method for correcting the drift in an optical device according to the preamble of claim 1 and a microscope with a device for correcting the drift according to the preamble of claim 12.

Optical devices, in particular microscopes, can also be regarded as mechanical structures which, due to technical limitations, such as the accuracy in the manufacture of the housing or possible inaccuracies in the fit when assembling the individual parts, appear to be quite stable macroscopically, but nevertheless show movements microscopically. These movements are often thermally induced. Usually these movements are called drift. In general, however, it must be noted that drift is usually only an observed characteristic that is perceived as a virtual movement of this object over time when long-term observation of immovable parts of an object with a camera or a confocal scanner. This apparent movement of the object can be perceived by the user in optical devices and often leads to complaints or difficulties in evaluating the images of examined objects.

Drift occurs as a visible result of the interaction of all parts of an optical device. For example, one sheet can expand due to heating and another can contract and move or even deform another component due to the resulting forces. However, the result is only perceived as a relatively small change in the x, y and z directions.

A clever mechanical construction can be used to avoid drift, the drift often being able to be reduced at least to a negligible extent.

With increasing resolution and enlargement, however, it is becoming increasingly difficult to achieve such a reduction in drift. This is because ever smaller movements of the optical device relative to the object have to be recognized or avoided, which often also requires cost-intensive measures. In addition, there is a tendency in today's optical devices to use ever cheaper materials, with metal parts often being replaced by plastic parts. However, this often results in new and also difficult to estimate effects that negatively influence the drift.

A laser scanning microscope is known from German published patent application DE 199 59 228, which comprises a temperature sensor, the signals for focus correction of which are based on stored reference values. The measured temperature change is converted into a corresponding change to be carried out in at least one component (moving the table, positioning the piezo, deforming the mirror, etc.) of the microscope. The temperature compensation can also be done via a saved table or curve. With this method, only the z coordinate, ie the focus, can be kept constant. A migration of the sample within the xy plane defined by the table surface cannot be compensated with this.

German patent DE 195 301 36 C1 also describes a microscope with focus stabilization. The temperature is stabilized by two metal rods with different coefficients of thermal expansion. One rod is connected to the rack for the focus drive, the other rod to the microscope stage. The focus is stabilized exclusively by mechanical means individually tailored to the microscope.

The proposed drift corrections also presuppose that the drift is caused by a change in the recorded temperature. However, temperature changes that are very small but lead to a large drift cannot be adequately recorded and corrected. Drift that is attributable to other causes, such as a change in the installation conditions or the interaction of the parts of the microscope already described above, cannot be detected with this.

The object of the present invention is to propose a method for correcting the drift in optical devices which is independent of the cause of the drift.

According to the invention, this object is achieved by a method for correcting the drift in an optical device with the features according to claim 1.

Basically, with the method according to the invention, two temporally successive images of an immovable object are initially recorded. Both images are subdivided with a grid, so that blocks are obtained on both images, the center of which is defined by coordinates, the so-called movement hypotheses. A block of the second image is then selected and compared with the blocks of the first image until the most similar block, the so-called target block, is found in the first image. If the comparison shows that the coordinates of the block found in the first image match the coordinates of the block of the second image, the block has not changed its position and there is no drift. However, if the target block is located at a different location, a vector can be determined that describes the displacement of this block as a drift.

Although moving objects can also be used in principle, it is advantageous to select immovable objects. In the case of moving objects, it must be taken into account that a drift by the object movement could also be simulated here, which can only be taken into account again, ie can be calculated out, if the movement pattern of the object is known. However, this should be the case in very few cases of the objects to be examined.

If the block of the second picture is compared with all blocks of the first picture, the so-called target block can be determined, for which it applies that it is most similar to the starting block, that is to say the block of the second picture. The vector which characterizes the drift can then be determined from the knowledge of the block coordinates.

A number of methods can be used for the comparison of the blocks, with the aid of which it can be determined whether the blocks compared with one another are similar to one another. However, an assessment must also be made that allows a statement about how high the degree of similarity is, in order to be able to later decide which is the target block, which block is the most similar to the starting block.

In order to increase the accuracy, it is possible to further subdivide the target block into sub-blocks and to carry out the process again for these sub-blocks. This can then continue until a better similarity can no longer be determined. In this case, the drift then results from a sum of individual steps that result from the inclusion of the sub-blocks.

The correction of the drift in the microscope can then take place, for example, in that the drift is calculated as an apparent movement from the determined images.

A microscope according to the invention thus has a device for recording a first and a second image. A device for correcting the drift is also provided. This device is equipped with a unit for dividing the first and second images into blocks. A unit for comparing a block of the second image with the blocks of the first image makes it possible to determine and evaluate the similarity between the blocks.

The described microscope and the method for correcting the drift have the advantage that the drift can be precisely detected and corrected for very small values. Furthermore, the detected drift is independent of its cause and in particular is not restricted to pure temperature changes.

The direction of the drift can also be determined and correspondingly corrected in the xy direction.

Further advantages and advantageous embodiments of the invention are the subject of the following figures and their descriptions.

They show in detail:

<dl id="dl0001"><dt>Fig. 1.</dt><dd>is a schematic view of an example of the microscope with a device for detecting and correcting the drift</dd><dt>Fig. 2</dt><dd>the basic procedure for determining the drift</dd><dt>Fig. 3</dt><dd>a schematic diagram of a motion estimator</dd><dt>Fig. 4</dt><dd>a schematic diagram of a motion estimator with subdivision into sub-blocks</dd><dt>Fig. 5</dt><dd>a schematic representation of the drift correction in a conventional microscope</dd><dt>Fig. 6</dt><dd>a schematic diagram of an alternative drift correction in a confocal microscope</dd></dl>

In Fig. 1, a microscope is shown schematically as an optical device 2. In the exemplary embodiment shown here, the microscope 2 is assigned a computer 4 with a display 6 and an input means 8, and a control and monitoring unit 10 for controlling the various microscope functions. The control and monitoring unit 10 comprises a memory 9 and a microprocessor 11. It goes without saying that the microscope 2 can take any conceivable shape and configuration and the illustration in FIG. 1 should not be interpreted as a limitation. The microscope 2 comprises a stand 12 on which at least one eyepiece 14, at least one objective 16 and a microscope stage 18 adjustable in all three spatial directions is provided. An object 30 to be examined or treated microscopically can be placed on the microscope stage 18. As represented schematically by the coordinate system, the xz direction runs in the drawing plane. In this representation, the y direction is perpendicular to the plane of the drawing. In the exemplary embodiment shown here, the microscope 2 comprises a revolver 15 to which the plurality of objectives 16 are attached. One of the 16 is in a working position and defines an optical axis 13. An adjustment button 20 is provided on both sides of the stand 12, with which the microscope stage 18 can be adjusted in height (in the z direction) relative to the objective 16 in the working position. The microscope stage 18 of the microscope 2 can be adjusted with a first motor 21 in the x direction, with a second motor 22 in the y direction and with a third motor 23 in the z direction. The control of the first, second and third motors 21, 22 and 23 takes place via the control and monitoring unit 10. A camera 25 is connected to the microscope 2 and takes a picture of the object 30 observed with the objective 16. The camera 25 is connected to the control and monitoring unit 10 via a first electrical connection 26. Likewise, the control and monitoring unit 10 is connected to the microscope 2 via a second electrical connection 27, via which signals from the microscope 2 to the control and monitoring unit 10 and signals from the control and monitoring unit 10 to the microscope 2 are supplied. It goes without saying that the camera 25 can be a video camera or a CCD camera. In the memory 9, those supplied by the camera 25 and possibly. data calculated by the microprocessor 11 are stored. These data include values ​​of two successive images of the object 30 and, if appropriate, the comparison values ​​of these images. In the exemplary embodiment shown in FIG. 1, the control and monitoring unit 10 is accommodated in an external electronics box 42 connected to the microscope 2.

As already described, the entirety of the components of the microscope 2 can cause a drift. The basic method for determining the drift according to the invention is shown in FIG. 2. Initially, two images of an area of ​​interest of the object 30 which are of successive time are recorded, this area usually being abbreviated as ROI (region of interest). The ROI is preferably selected by the user in step 32. In a further step, this ROI is then used to record a first image at a first point in time T (n-1) and preferably at a second point in time T (n) in pixel coordinates. The ROI is preferably selected as a cuboid. In a device for calculating the drift 38, the data obtained from the first and second points in time are processed and the current drift d (n) is calculated therefrom. Image values ​​are thus determined for the discrete times T (n-1) and T (n), which are, for example, as intensity values ​​I (x, y, T (n-1)) and I (x, y, T (n)) can be represented. These are fed to the device for calculating the drift 38 in steps 34 for I (x, y, T (n)) and 36 for I (x, y, T (n-1)). A prerequisite for carrying out this method is that there is an immovable object within the ROI and that the ROI has a recognizable image content, in particular is therefore not completely black. If there is only one moving object in the ROI, the intrinsic movement of the object simulates a drift that can only be further evaluated if the chronological sequence of the movement of the object is known, which is not the case in the majority of cases .

A motion estimator is used to calculate the current drift d (n), in which the movement of the object is carried out by comparing blocks based on their similarity. For this purpose, the first and the second image of the selected ROI are first divided into blocks. A block of the first image is then compared with all blocks of the second image, which is used to determine the degree of similarity between the compared blocks. The image section that is most similar to the scene from the last image is therefore searched for. As a measure of similarity for the similarity between two blocks, for example the minimum quadratic sum for a given ROI and a drift vector d to be evaluated can be. The minimum quadratic sum MSE can be represented as follows:<maths id="math0001" num=""><img file="EP1548485A1_D0001.tif" /></maths>

The principle of this motion estimator is shown in FIG. 3, it being necessary to determine the set of all possible displacement vectors.

For this purpose, as can be seen in FIG. 3 a), a predefined block 42 is defined from the predefined ROI 40, which is at least half the size of the predefined ROI 40. This first block 42 is positioned in the middle of the predefined ROI 40. With the center B1 and its corners B2, B3, B4 and B5, the first block 42 thus defines five movement hypotheses, as shown in FIG. 3b). The centers B1 to B5 form 5 movement hypotheses for a possible movement that may have taken place between the first and the second image. These five movement hypotheses must now be evaluated, ie the probability of a movement in the direction B1 to B5 is determined by comparison. For each movement hypothesis, an MSE is determined accordingly, so that the movement hypotheses MSE (1) to MSE (5) are determined according to the above equation. The determined MSE (2) and MSE (5) are shown schematically in Fig. 3c). As already mentioned, however, the calculation must be carried out for all movement hypotheses B1-B5.

As soon as all movement hypotheses have been tested, it can be determined which block is the target block, ie which block has the lowest MSE. The current drift vector d (n) is thus established for this target block. Now you can proceed in the same way in this target block. For this purpose, as shown in FIG. 4 using the example of block B4, the original target block size is halved. Then in the space of the target block B4 again, as in connection with Fig. 3rd already described, five sub-blocks generated. Again, all five movement hypotheses are tested and the sub-target block is determined based on the test. The current drift vector d (n) is thus again determined for the sub-target block thus determined, and further sub-blocks UUB can then be used accordingly.

If this method is successively continued recursively, a number of drift vectors, the sum of which represents the total drift, is obtained by determining the respective target blocks. The area of ​​the ROI is tiled with hypotheses. For example, if you start with an ROI that has a pixel size of 14x14, the process can be continued until a pixel size of 2x2 is reached for the smallest block. This means that the drift vector is determined after a maximum of 25 operations of successively smaller blocks.

The drift determined in this way is then compensated for in the microscope. For this purpose, the drift z. B. in a subsequent calculation step, which is carried out in particular in a computing unit of the microscope. For example, the control and monitoring device 10 of the microscope 2 can be used.

A possible overall sequence of the method for a conventional microscope 2 is shown again in FIG. 5. This consists in first selecting an ROI on the screen 6 in step 32. After selection, the described drift compensation algorithm runs. It is essential, however, that the user selects an immovable structure of object 30. In a shifter 48, a shift rule is used which is adapted to the algorithm used, by which amount and which direction is to be shifted for the next movement estimation step. This essentially depends on the position of the target block determined. The shift then takes place in step 46. The motion estimator is then used to calculate the current drift d (n), in which the movement of the object is based on its similarity by comparing the blocks B1 to B5 already described. For this purpose, the intensity values ​​of the first image 34 and the second image 36 of the selected ROI for the blocks to be compared are compared in the device for calculating the drift 38, and the minimum deviation is thus determined. This allows you to find the image section that is most similar to the scene from the last image. The resulting current drift vector d (n) is fed to an integrator 44, in which the total drift D (n) resulting from the sum of all individual drifts d (n) is determined. This total drift D (n) is stored in step 42 for later correction.

Since the total drift D (n) of the microscope 2 is now known, it can be taken into account in the representation of any objects, including moving objects, by calculating them out of the image in a subsequent step after the object has been recorded.

An alternative procedure for drift correction in a confocal microscope is shown in FIG. 6. The imaging in such microscopes is usually carried out using so-called galvanometers, which align a light beam falling on the object in such a way that it illuminates the object line by line. For this purpose, the galvanometer is actively controlled so that all points of the object can be collected. This active control can now be integrated in the process of determining and correcting drift within the scope of the degrees of freedom of movement of the galvanometer, the shifter 48 shown in FIG. 5 being dispensed with. This function is integrated into the control of the gananometer. After an ROI has been selected in step 32, the current drift d (n) is again determined in the device for calculating the drift 38. The current drift d (n) is fed to the integrator 44, which determines the current total drift D (n) by integration. This result is transferred to the galvano control 50, so that the overall drift can be taken into account when positioning the mirrors. This means that the algorithm always works in the same block. However, the scanner that scans the object captures images that have been shifted in the same way.

In principle, it is also possible to redesign the control in a confocal microscope in such a way that two different images are recorded in successive order. For this purpose, the first image is obtained directly from the sample itself, while the second image is obtained from the intermediate level. The overall sequence of the images obtained can then be generated in such a way that a reference image is taken from the intermediate image level in a successive sequence of sample images and is used to determine the drift. In this case there is definitely an immovable object and the recording can be limited to the number of pixels relevant for the drift determination.

<u>Reference list</u>

<dl id="dl0002" compact="compact"><dt>2</dt><dd>microscope</dd><dt>4</dt><dd>computer</dd><dt>6</dt><dd>Display</dd><dt>8</dt><dd>Input means</dd><dt>9</dt><dd>Storage</dd><dt>10</dt><dd>Control and monitoring unit</dd><dt>11</dt><dd>microprocessor</dd><dt>12</dt><dd>tripod</dd><dt>13</dt><dd>optical axis</dd><dt>14</dt><dd>eyepiece</dd><dt>15</dt><dd>revolver</dd><dt>16</dt><dd>Lenses</dd><dt>18</dt><dd>Microscope stage</dd><dt>20</dt><dd>Adjustment knob</dd><dt>21</dt><dd>first engine</dd><dt>22</dt><dd>second engine</dd><dt>23</dt><dd>third engine</dd><dt>25</dt><dd>camera</dd><dt>26</dt><dd>first electrical connection</dd><dt>27</dt><dd>second electrical connection</dd><dt>30</dt><dd>object</dd><dt>32</dt><dd>ROI selection</dd><dt>34</dt><dd>Intensity value of the first image</dd><dt>36</dt><dd>Intensity values ​​of the second image</dd><dt>38</dt><dd>Device for calculating the drift</dd><dt>40</dt><dd>ROI (region of interest)</dd><dt>42</dt><dd>Save the overall drift</dd><dt>44</dt><dd>Integrator</dd><dt>46</dt><dd>shift</dd><dt>48</dt><dd>Shifter</dd><dt>50</dt><dd>Galvanometer control</dd><dt>d (n)</dt><dd>current drift</dd><dt>D (n)</dt><dd>Total drift</dd></dl>