Method for calibrating a stereoscopic photography device

IS 2 550 195 T3

DESCRIPTION

Calibration procedure for a stereoscopic camera recorder

The present invention concerns the field of stereoscopic camera recorders and more precisely their calibration.

Stereoscopic camera recorders are tending to become more widespread and can now be marketed at a reasonable cost. Due to this, these devices make their appearance in numerous devices for use by the general public such as cam- scopes, photographic devices and their incorporation in devices such as mobile phones or portable game consoles is seen. Formerly reserved for professional equipment, these are currently made with the help of low-cost components.

The general architecture of these devices is illustrated by Fig. 1. These are made up of a support 1.1 integrated in the product that carries two cameras 1.2 and 1.3. These chambers are typically a few centimeters apart and ideally should be strictly identical. These must also have their optical axes 1.5 and 1.6 ideally parallel. When these conditions are met, it is then possible to take pictures of a 1.4 scene with the help of the system. A left image taken by camera 1.2 and a right image taken by camera 1.3 are obtained. The objects in scene 1.4 located far enough away to be considered as at infinity are then perfectly superimposable in the two images. The closest objects suffer a parallax that translates into a horizontal displacement in translation between the two images. This translation is all the more important the closer the object is to the cameras, keeping the distance between the cameras less.

In this area of ​​products for the general public produced on a large scale and at low cost, the quality requirements are lower than in the professional field. For this reason, the cameras used are not strictly identical and the positioning of the two cameras also suffers from fairly wide tolerances.

Furthermore, the appearance of the general public of the products considered implies a great variability in the course of time of the environment of the devices that can be subjected to shocks and large variations in temperature. These events have consequences on the physical parameters of the sensors and their positioning.

All these phenomena have the consequence that it is difficult to obtain reliable stereoscopic information from the left and right images taken by the cameras.

The invention aims to solve the preceding problems by proposing a calibration procedure for a stereoscopic camera recorder that calculates a set of image correction parameters. These parameters are classified according to an order of importance. In a first time the first order correction parameters are estimated, in a second time the second order correction parameters. The first order parameters are tuned taking into account the estimation values ​​of the second order parameters. Advantageously, a measurement of the relevance of the scene is carried out prior to the actual calibration.

The document “Camera Calibration with Distortion Models and Accuracy Evaluation”, Juyang Weng et al., IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 14. no. 10, 1992, pages 965-980, discloses such a correction of the parameters classified in two orders: the first order concerns the rotation and translation parameters, the second order concerns the distortion correction parameters. This document does not disclose a hierarchy in four orders.

The invention concerns a method of calibrating a stereoscopic camera recorder according to claim 1.

According to a particular embodiment of the invention, the defects that are chosen to be corrected are chosen from: a displacement in global translation of the image; a horizontal and vertical trapezoidal deformation; a rotation; a zoom factor and a barrel or cushion distortion.

According to one embodiment of the invention, the first order defect consists of a global translational displacement of the image; the second order defect consists of a horizontal and vertical trapezoidal deformation; the third-order defect consists of a rotation and the fourth-order defect consists of a zoom factor.

According to a particular embodiment of the invention, the method further comprises a step of calculating a relevance index of the scene.

According to a particular embodiment of the invention, the calculation of a relevance index of the scene comprises: a step of calculating the histograms of the right and left images; a step for calculating the parallaxes in a plurality of areas of the left and right images; a step calculating an index of details of the scene as a function of the extension of the histograms and of the correlation indices between the areas of the right and left images; a stage of calculating a distance index that will be the greater the more uniformly

ES 2 550 195 T3 small are the indices of parallaxes and a stage of formation of the index of relevance as a function of the index of details and the index of distance.

The aforementioned characteristics of the invention, as well as others, will become apparent more clearly on reading the following description of an embodiment, the aforementioned description being made in relation to the attached drawings, in which:

Fig 1 illustrates the general architecture of a stereoscopic camera recorder.

Fig 2 illustrates the flow chart of the calibration procedure according to an embodiment of the invention.

Fig 3 illustrates the flow chart of the scene relevance measurement step according to an embodiment of the invention.

Fig 4 illustrates clipping of the image used in one embodiment of the relevance measurement.

Fig 5 illustrates the fast correlation calculation procedure used in one embodiment of the relevance measurement.

A real stereoscopic camera system is subject to a certain number of physical imperfections that have more or less important consequences on the captured images. The physical positioning of the cameras on the support is never perfectly aligned, which means that the optical axes are not perfectly parallel. Thus, in order to exploit stereoscopy, it is important to correct the effects due to these physical imperfections.

The invention is based on the fact that the different effects to be corrected do not all have the same impact on stereoscopy. Some are more important than others. Thus, several orders are distinguished in the various distortions found in the images captured with respect to the images that would be obtained by an ideal system. We speak of first-order, second-order, third-order and fourth-order distortions.

It can be seen that, according to the manufacturing process of the cameras, the amplitude and therefore the order of magnitude and therefore the correction of each distortion can vary. Thus, it is advantageous to change this order of correction depending on the need for correction at the time. Certain distortions do not appear in certain cameras, you can advantageously avoid correcting them. Other distortions "switch" naturally: the estimation and correction of rotation and zoom can be carried out independently, therefore in an indifferent order.

The calibration procedure therefore consists of estimating the correction parameters that must be applied to the captured images to obtain images that are the closest that can be obtained to the images that would be obtained by a camera system that does not have any defect.

To do this, a scene is captured that must have good properties. First, the scene must be far enough apart so that the left and right images overlap. It has been seen that the points of an object in the distance overlap in the two images. The defects of the camera system will distort this overlap that you want to find. The scene must also propose a level of detail that allows the search for an area in the image. An area of ​​uniform color and brightness would not allow the distortions to be corrected to be measured. This must also be free of nearby objects, whose parallaxes would disturb the measurements.

The first order defect consists of a global translational displacement of the image. This is due to the fact that the optical axes not being perfectly parallel, the scene portion captured by the left camera 1.2 does not exactly correspond to the scene portion captured by the right camera 1.3. It is therefore sought to determine a horizontal translation parameter AX and a vertical translation parameter AY that allow the two images to be readjusted. This determination consists of calculating, during a step 2.1, the displacement for which the images are correlated. To do this, the exemplary embodiment calculates a correlation coefficient between the right image and the left image for a set of translation values. For example, the sum of the absolute values ​​of the brightness differences at each point in the image is calculated. The translation values ​​for which this sum is minimal give us a first estimate of the first-order parameters.

The second order defect consists of a trapezoidal, horizontal and vertical deformation, due to the perspective deformation induced by the non-parallelism of the optical axes of the two cameras. This deformation results in a relationship other than one between the left edge and the right edge of the image for the horizontal keystone and between the top edge and the bottom edge for the vertical keystone. These parameters are estimated during step 2.2 by calculating a correlation coefficient for various tested trapezoid values. Advantageously, the trapezoid being due to a perspective effect resulting from a parallelism defect of the optical axes of the cameras and causing this defect an associated translational displacement, the first-order corrections of translational displacement and the second-order corrections are associated. order linked to the trapezoid for the estimation of these parameters. A translation and the associated trapezoid effect are then applied to an image during tests aimed at estimating the values ​​of these parameters for which the correlation between the images is the highest.

IS 2 550 195 T3

The third order is the taking into account of a possible rotation of one camera with respect to the other around its optical axis. This rotation causes a rotation of one of the two images with respect to the other around the center of the image located in the path of the optical axis of the camera. During stage 2.3, this rotation factor is estimated by calculating the rotation coefficient for different values ​​of this rotation factor and the value for which the highest correlation is obtained is retained.

The fourth order concerns a viewing angle disparity between the two cameras. This disparity causes a zoom factor between the two images. During step 2.4, the value of this zoom factor is estimated by correlation.

Advantageously, a fifth order can be considered. These are the effects due to optical distortions of camera lenses. Depending on the case, each camera can induce a barrel distortion or a pincushion distortion. These distortions make the image appear as projected onto a concave or convex surface. During stage 2.5, these distortions are estimated by correlation.

Distortions can for example be estimated and corrected by a first order approximation in the form f (R) = R + aR³ with R the distance or radius between each pixel and the optical center located a priori in the center of the image and f (R) the new position of this same pixel after correction of the distortion. The sign of the correction factor a, which is 0 for a perfect target, determines whether the deformation is cushion or barrel.

By comparison, the difference in distortions between the two cameras can therefore be estimated by "deforming" one so that it corresponds as closely as possible to the other.

The important thing is not so much to correct the deformations at all of the cameras, as to minimize the disparities between the left and right cameras which, in turn, induce true visual discomfort in stereoscopy.

It is found that it is illusory to seek to correct an effect of a given order if the effects of a higher order have not previously been corrected. It is also found that the correction of an effect of a given order influences the estimation of the effects of a higher order. To improve the estimation of these parameters, the following algorithm is then adopted. An estimation of the order 1 parameters is made, step 2.1, then it is passed to the estimation of the order 2 parameters, step 2.2, on corrected images using the estimated order 1 parameters. A first estimate of the parameters of order 2 is then obtained. The estimate of the parameters of order 1 is then refined by applying a correction to the images with the parameters of order 2. The estimation of the parameters of order 2 is then refined with the new values ​​obtained for the parameters of order 1. Advantageously, this loop is continued until a convergence of the estimation of the parameters is obtained. These parameters are then used to estimate the parameters of order 3 and so on. According to this embodiment, the estimation of the parameters of a given order is used to carry out a new estimation of the parameters of a higher order.

It is understood that the actual defects that are chosen to correct as well as the order in which these defects are corrected may vary depending on the systems. The cited defects and the cited order represent only one embodiment of the invention.

In this way, a set of parameters is obtained that allows correcting the images obtained by approaching the images that would be obtained if the camera recorder were ideal. These correction parameters can then be used directly during shooting for immediate correction or linked to the images for later correction. These are usable either for a single photographic type shot or during a series of shots in a video type application.

It has been seen that the efficiency of this calibration procedure depends in part on the scene chosen for the taking of views used for the estimation of these parameters. This scene must be sufficiently far away, include a level of detail that allows reliable correlations and not contain nearby objects whose parallax would disturb the correlation operations. Since this calibration procedure can be carried out on demand throughout the service life of the apparatus, it can be put into practice by a user who is not very skilled in the art. To assist this user in choosing a relevant scene to perform the calibration procedure, a step of measuring an index of the relevance of the scene is added to this procedure. Advantageously, the result of this measurement step is displayed on the screen in the form of a relevance index. This index can be viewed as a relevance note or in graphical form. For example, a relevancy bar whose length is proportional to the relevancy index can be displayed on the screen of the apparatus. Advantageously, the color of this bar can be used to determine zones for this relevance index, for example a green bar if the index is sufficient for a quality calibration, orange if the index is usable but insufficient to ensure the best calibration and red if it is unusable.

The flow chart in FIG. 3 illustrates the procedure for calculating the relevance index according to the embodiment of the invention. A first stage 3.1 consists of a calculation of the histograms of the right and left images. The extent of this histogram gives a first index in relation to the level of detail of the scene. Indeed, a scene that is too uniform gives a tight histogram while a well-extended histogram is the signal that the scene has a variability of colors and luminances that a priori allow a good correlation.

IS 2 550 195 T3

A second stage consists of measuring a grid of parallaxes in the scene from the captured images. To do this, the image is divided into a plurality of zones. This division can be any. According to the exemplary embodiment of the invention, the division is made according to a grid such as that illustrated in Fig 4. Image 4.1 is in this case divided into nine zones 4.2 of equal size.

In such a way that the necessary calculation times are minimized, a parallax index is calculated for each zone according to the method illustrated in Fig 5. According to this embodiment, zone 5.2 is brought to a single line 5.4 by an operation such as the sum of the columns 5.3. The line thus obtained is compared by correlation with the equivalent line of the other image. The correlation is done, for example, by adding the absolute values ​​of the differences. This sum gives a correlation index. The translation that gives the minimum of these correlation indices is taken as the parallax index.

Advantageously, the calculation of these correlation indices is used to also calculate the mean value of the correlation indices and then the difference between the minimum value retained and this mean value. This difference is another index of the level of detail in the image and its adequacy with the reliability of the correlation. Indeed, if a minimum correlation index is obtained that is clearly different from the mean value of the correlation indices obtained for the close translation values, it can be said that the correlation is reliable and that the level of detail is therefore relevant. On the contrary, if the minimum value does not neatly cut other values ​​of the correlation index, then the level of detail does not allow a reliable correlation.

Thus, at the exit of step 3.2, two values ​​are obtained, a parallax index which is a translation value corresponding to a minimum correlation index and a correlation reliability index corresponding to the difference between the minimum correlation index and the mean value of the correlation indices. The correlation reliability index is used together with the histogram extension index to calculate an overall image detail index during a 3.3 step. For example, a weighted average of the correlation reliability indices for each zone and the spread index of the histogram is calculated.

The parallax indices of the different zones are used to calculate a distance index during stage 3.4. These parallax indices are a good approximation of the distance from the scene. With calibration not taking place yet, no reliable depth information can be derived, however a good idea is still obtained. Advantageously, the calibration process can be applied successively, which makes it possible at each calibration stage to take advantage of input images already "pre-calibrated", resulting in a gain in precision. The information on the distribution of the values ​​of the parallax indices is also used. If these values ​​have a great variability, this means that there are planes at different depths in the image and therefore of the objects in the foreground. This aspect has a negative impact on the calibration. A relevant scene for calibration is a scene that produces uniformly small values ​​of parallax indices for all zones. A distance index is therefore calculated which will be the greater the more uniformly small the parallax indices are.

The relevance index is finally calculated as a function of the details index obtained by stage 3.3 and the distance index obtained in stage 3.4. Advantageously, this index is a weighted average between the two indices.

Thanks to the described procedure, it is seen that it is possible for a user to carry out a calibration of a stereoscopic camera recorder at any time, this calibration allowing an exploitation of the stereoscopy by correcting defects in the system. Advantageously, the user is guided during the choice of the scene for calibration. This procedure can be used in any type of device, especially for the general public, such as photography devices, video cameras, telephones or game consoles.