Depth map output device

The present invention relates to a depth map output device that outputs a depth map that is distribution information of a distance to a subject.

An imaging device configured to individually handle light fluxes from different regions of the exit pupil of a photographing optical system is known (for example, Patent Document 1). Such an imaging device adopts a configuration in which a pixel array composed of a plurality of pixels is arranged behind each microlens constituting the microlens array. With such a configuration, since the luminous fluxes from different regions of the exit pupil are incident on different pixels, it is possible to handle these luminous fluxes individually. By treating these luminous fluxes individually, for example, it is possible to synthesize image data focused on a subject located at an arbitrary shooting distance with only one imaging. As another example, it is possible to obtain a plurality of images having parallax by one imaging.

On the other hand, Patent Document 2 describes, as a modification 4, an imaging device that creates a depth map that is distribution information of a distance to a subject. This imaging device creates a left-viewpoint image and a right-viewpoint image by the same method as the imaging device described in Patent Document 1 described above. After that, the phase difference between the two images is calculated by a known stereo matching process, and a depth map is created based on this phase difference.

<p><patcit num="1"><text>Japanese Unexamined Patent Publication No. 2007-4471</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2009-165115</text></patcit></p>

<p> The imaging device described in Patent Document 2 has a problem that an accurate distance to a subject cannot be calculated in the vicinity of a place where the perspective of the subject changes (for example, a boundary between a close subject and a background).</p>

<p> The invention according to claim 1 comprises an imaging optical system for forming a subject image on a predetermined imaging surface, a plurality of positive lenses arranged in a two-dimensional manner in the vicinity of the imaging surface, and a plurality of positive lenses. A plurality of light receiving elements, each of which is arranged at least three on the rear side of the positive lens corresponding to each, and at least two of the at least three light receiving elements are arranged side by side in the first direction, and at least. At least two of the three light receiving elements are arranged side by side in a second direction different from the first direction, and a plurality of first light receiving elements arranged in the first direction from a plurality of positive lenses. For each of the first selection means for selecting the positive lens array and the plurality of first positive lens sequences selected by the first selection means, the first direction corresponding to each of the positive lenses constituting the first positive lens array. With the first distance calculation means that calculates the output of at least two light receiving elements arranged in the same direction and calculates a value representing the distance to the subject in which the image is formed on the positive lens in the center of the first positive lens row. , A first depth map for creating a first depth map, which is data in which values representing a plurality of distances calculated by the first distance calculation means are arranged in a two-dimensional manner according to the position of a positive lens corresponding to the value. A mapping means, a second selection means for selecting a plurality of second positive lens sequences arranged in a second direction from a plurality of positive lenses, and a plurality of second positive lens sequences selected by the second selection means. For each, the output of at least two light receiving elements arranged in the second direction corresponding to each of the positive lenses constituting the second positive lens row is calculated, and the positive lens in the center of the second positive lens row is calculated. The second distance calculation means for calculating the value representing the distance to the subject on which the image is formed and the value representing a plurality of distances calculated by the second distance calculation means are set to the positions of the positive lenses corresponding to the values. The second depth map creation means that creates the second depth map, which is the data arranged in a two-dimensional manner, and the first depth map and the second depth map are combined and the third depth map is output. It is equipped with a depth map output means for executing the compositing process, and at least two light receiving elements arranged in the first direction and a second light receiving element corresponding to each of the plurality of positive lenses.At least two light receiving elements arranged in the direction are depth map output devices characterized in that light fluxes from different regions of the exit pupil of the imaging optical system are received through the positive lens.</p>

<p> According to the present invention, it is possible to calculate a value representing an accurate distance to a subject even in the vicinity of a portion where the perspective of the subject changes.</p>

<figref num="1">It is a figure which shows the structure of the stereo image pickup apparatus according to 1st Embodiment of this invention.</figref><figref num="2">It is a schematic diagram for demonstrating the horizontal selection function by a control circuit 40.</figref><figref num="3">It is a figure explaining the signal sequence used for calculating the defocus amount.</figref><figref num="4">It is a figure which shows the state of calculating the phase difference.</figref><figref num="5">It is a figure which shows the microlens ML which the defocus amount is calculated by the horizontal origin distance calculation function and the vertical origin distance calculation function.</figref><figref num="6">It is a figure which shows the range of the depth map which is finally output.</figref><figref num="7">It is a figure explaining the defocus amount in the lateral origin depth map.</figref>

(First Embodiment) FIG. 1 is a diagram showing a configuration of a stereo imaging device according to the first embodiment of the present invention. The stereo image pickup apparatus 1 includes a left side image pickup unit 100 and a right side image pickup unit 200 for photographing a subject, and a control circuit 40 including a microprocessor and the like for controlling these two image pickup units.

The left image pickup unit 100 includes a photographing lens 11, a microlens array 12 arranged behind the photographing lens, and an imaging element 13 arranged behind the microlens array 12. Normally, the distance between the microlens array 12 and the image sensor 13 is determined so that the light receiving surface of the image sensor 13 is near the focal position of the microlens ML constituting the microlens array 12, but in FIG. 1, it is actually for explanation. It is drawn wider than.

The microlens array 12 has a plurality of microlenses ML arranged in a two-dimensional manner. A plurality of light receiving elements are arranged two-dimensionally on the image pickup surface of the image pickup element 13 facing the microlens array 12. These light receiving elements are grouped by a fixed number, and each group (light receiving element arrangement 30) corresponds to each microlens ML. Hereinafter, this group is referred to as a light receiving element array. In the present embodiment, the light receiving element array 30 composed of 25 light receiving elements arranged in 5 rows and 5 columns is one group corresponding to each microlens ML.

The individual light receiving elements constituting the light receiving element array 30 are arranged at positions substantially conjugate with the exit pupil of the photographing lens 11. As a result, the individual light receiving elements constituting the light receiving element array 30 receive the luminous flux from different regions of the exit pupil of the photographing lens 11 via the microlens ML corresponding to the light receiving element array 30. The subject light incident on a certain microlens ML may be incident as stray light on another light receiving element array 30 adjacent to the light receiving element array 30 corresponding to the microlens ML. In order to prevent such stray light, a light-shielding mask is provided at the boundary between the microlens MLs, and a partition member for partitioning each light-receiving element array 30 is provided between the microlens array 12 and the image sensor 13. Is desirable. In FIG. 1, the illustration of such a member is omitted.

An image pickup control circuit 14 and a video circuit 15 are connected to the image pickup element 13. The image pickup control circuit 14 drives the image pickup element 13 to control the exposure time, photoelectric conversion, and the like. The video circuit 15 amplifies the electric signal output by the image sensor 13 and performs analog-to-digital conversion, and outputs the digital signal to the control circuit 40. The control circuit 40 creates image data representing a subject image based on this digital signal. In the arbitrary focus shooting mode described later, the resolution of the image data created by the control circuit 40 is limited by the number of microlens MLs in the microlens array 12. For example, when the microlens array 12 is composed of K microlenses ML in the horizontal direction and M microlenses ML in the vertical direction, the maximum number of pixels of the image data created by the control circuit 40 is K × M.

The right image pickup unit 200 includes a photographing lens 21 and an image pickup element 23 arranged behind the photographing lens 21. The individual light receiving elements constituting the image pickup element 23 have the same size as the light receiving element of the image pickup element 13. Unlike the left image pickup unit 100, the right image pickup unit 200 does not have a microlens ML corresponding to a plurality of light receiving elements. The positional relationship between the photographing lens 11 of the left imaging unit 100 and the front main surface of the microlens array 12 is the positional relationship between the photographing lens 21 of the right imaging unit 200 and the imaging surface (light receiving surface) of the imaging element 23 and the optics. Is equivalent.

An image pickup control circuit 24 and a video circuit 25 are connected to the image pickup element 23 in the same manner as the image pickup element 13 of the left side image pickup unit 100. The image pickup control circuit 24 drives the image pickup element 23 to control the exposure time, photoelectric conversion, and the like. The video circuit 25 amplifies the electric signal output by the image sensor 23 and performs analog-to-digital conversion, and outputs the digital signal to the control circuit 40. The control circuit 40 can create image data representing a subject image based on this digital signal. In the present embodiment, the number of pixels is larger in the image data created based on the digital signal output from the right image pickup unit 200 than in the image data synthesized from the digital signal output from the left image pickup unit 100. Will increase. This is because the resolution of the former image data is limited by the number of microlens MLs, whereas the latter image data is not limited to such a limitation.

The photographing lens 11 and the photographing lens 21 have optically equivalent configurations. An optical low-pass filter (not shown), an infrared cut filter, or the like is provided on the image pickup surface of the image pickup element 13 and the image pickup element 23. Although the photographing lens 11 and the photographing lens 21 are schematically shown as one lens in FIG. 1, they may be composed of a plurality of lenses.

The left image pickup unit 100 and the right image pickup unit 200 described above are arranged so that the optical axes of the photographing lens 11 and the photographing lens 21 are separated by 60 mm, which is slightly shorter than the average eye width of an adult. That is, the baseline length S in FIG. 1 is 60 mm.

The control circuit 40 is connected to the image pickup control circuit 14 and the image circuit 15 of the left image pickup unit 100 and the image pickup control circuit 24 and the image circuit 25 of the right image pickup unit 200, and controls these circuits. The control circuit 40 also has a horizontal selection function, a horizontal origin distance calculation function, a horizontal origin depth map creation function, a vertical selection function, a vertical origin distance calculation function, a vertical origin depth map creation function, and a depth map composition function. It has each function of and. Each of these functions will be described in detail later.

(Explanation of operation mode of stereo imager) The stereo imaging device 1 includes an input device (not shown) such as a keypad. The user can transmit four kinds of instruction signals to the control circuit 40 by this input device. The four types of instruction signals are a normal shooting instruction signal, an arbitrary focus shooting instruction signal, a stereo shooting instruction signal, and a depth map creation instruction signal. The control circuit 40 that has received any kind of instruction signal controls the left side image pickup unit 100 and the right side image pickup unit 200 according to the instruction signal.

The control circuit 40 that has received the normal shooting instruction signal starts the operation in the normal shooting mode. In this mode, the control circuit 40 controls the right image pickup unit 200 or the left image pickup unit 100, and receives a digital signal (image signal) output from the image pickup unit. Then, one image data is created by applying known image processing to this image signal. The control circuit 40 stores the image data thus created in a storage medium (not shown) such as a memory card, or displays it on a display device (not shown) such as a liquid crystal monitor. The image data obtained from the left image pickup unit 100 in this mode can be used to synthesize image data corresponding to an arbitrary focal plane by causing a personal computer or the like to perform a known process described later in Patent Document 1. Are suitable.

The control circuit 40 that has received the arbitrary focus shooting instruction signal starts the operation in the arbitrary focus shooting mode. In this mode, the control circuit 40 controls the left imaging unit 100 and receives a digital signal output from the imaging unit. Then, by applying a known process described in, for example, Patent Document 1 to this digital signal, image data corresponding to an arbitrary aperture value and an arbitrary focal plane is created. The control circuit 40 stores the image data thus created in the storage medium or displays it on the display device as in the case of the normal shooting mode.

The control circuit 40 that has received the stereo shooting instruction signal starts operation in the stereo shooting mode. In this mode, the control circuit 40 controls both the left image pickup unit 100 and the right image pickup unit 200, receives digital signals output from each of the two image pickup units, and creates two image data. These two image data are a pair of stereo image data having parallax. The control circuit 40 creates image data from the digital signal output by the right image pickup unit 200 in the same manner as in the normal shooting mode. On the other hand, the image data created from the digital signal output by the left image pickup unit 100 is formed by extracting the output of one light receiving element for each light receiving element array 30 and arranging them. At that time, the output of the light receiving element closer to the direction of the image pickup unit 200 on the right side than the central portion of the light receiving element array 30 is extracted from each light receiving element array 30. By extracting in this way, stereo image data having a baseline length wider than the distance S between the left image pickup unit 100 and the right image pickup unit 200 can be obtained. This stereo image data is data that allows a user to perform stereoscopic viewing by displaying it on, for example, a scan backlight type stereoscopic display device.

The image data thus created based on the output of the left image pickup unit 100 has a smaller number of pixels than the image data created based on the output of the right image pickup unit 200. The control circuit 40 matches the resolutions of these two image data by applying an interpolation process to the former and a thinning process to the latter.

The control circuit 40 that has received the depth map creation instruction signal starts the operation in the depth map creation mode. In this mode, the control circuit 40 controls only the left imaging unit 100 and receives the digital signal output from the imaging unit. Then, after creating the image data as in the case of the arbitrary focus shooting mode, the depth map (described later) is created. The control circuit 40 stores the above-mentioned image data and depth map in, for example, a storage medium or displays them on a display device.

(Explanation of depth map) The control circuit 40 creates a depth map using the digital signal output from the left image pickup unit 100. The depth map is distribution information of the distance to the subject. In the present embodiment, for each of the microlens MLs constituting the microlens array 12, the defocus amount, which is a value representing the distance to the subject in which the image is formed before and after the microlens ML, is set to the microlens ML. Data arranged in a two-dimensional manner according to the position is called a depth map. It is optically clear that the distance to the subject is determined by the amount of defocus, the focal length of the photographing lens 11, and the focus adjustment surface information. Here, the defocus amount is a numerical value representing the displacement between the image formation position of the subject image and a predetermined reference plane (focus detection plane).

The depth map includes the horizontal selection function, the horizontal origin distance calculation function, the horizontal origin depth map creation function, the vertical selection function, the vertical origin distance calculation function, the vertical origin depth map creation function, and the depth of the control circuit 40. It is created by each function of the map composition function. Each of these functions will be described below.

(Explanation of horizontal selection function) FIG. 2 is a schematic diagram for explaining the horizontal selection function by the control circuit 40. In the following description, it is assumed that the microlens array 12 is composed of K microlenses ML in the horizontal direction and M microlenses ML in the vertical direction. The horizontal selection function is a function of selecting a plurality of horizontal lens rows arranged in the horizontal direction from the plurality of microlens MLs constituting the microlens array 12.

In the horizontal selection function, the microlens array 12 is first divided into groups 50 for each horizontal row by the control circuit 40 as shown in FIG. 2 (a). This creates M groups 50. In FIG. 2, the microlenses ML constituting the group 50 divided in this way are assigned numbers starting from 1 in order from the left for explanation.

Next, as shown in FIG. 2B, the control circuit 40 extracts a predetermined number of microlens MLs from the left end of the group 50 from the right side of the shift allowance determined by the minimum value of N described later. In this embodiment, the predetermined number is 5. That is, the control circuit 40 extracts five microlens MLs (from the third microlens ML to the seventh microlens ML) from the third from the left end of the group 50. In the horizontal selection function, the five microlens MLs extracted in this way are selected as one horizontal lens row 50a. The control circuit 40 then extracts five microlens MLs, from the fourth microlens ML from the left to the seventh microlens ML from the left, as shown in FIG. 2 (c), and a new horizontal lens sequence. Select as 50b. The control circuit 40 repeats this process while shifting the target microlens ML one by one to the right, and selects a plurality of horizontal lens rows from the group 50 so that some of the microlens MLs overlap with each other. To do.

Similarly, the control circuit 40 selects a plurality of vertical lens rows arranged in the vertical direction for the vertical selection function. The only difference between the horizontal selection function and the vertical selection function is whether the arrangement direction of the microlens ML and the light receiving element is the horizontal direction or the vertical direction.

(Explanation of lateral origin distance calculation function) The lateral origin distance calculation function calculates a value representing the distance to the subject in which the image is formed on the microlens ML at the center of the horizontal lens row for each of the plurality of horizontal lens rows selected by the horizontal selection function. It is a function. As described above, since the defocus amount is used as a value representing the distance to the subject in the present embodiment, the defocus amount is calculated by the lateral origin distance calculation function.

In the present embodiment, the procedure for calculating the defocus amount is defined so that the defocus amount becomes positive when the actual imaging position is on the subject side of the focus detection surface. Therefore, the direction in which the subject moves away from the stereo imaging device 1 is the direction in which the defocus amount increases.

FIG. 3 is a diagram illustrating a signal sequence used for calculating the defocus amount. The numbers assigned to the microlens ML in FIG. 3 are the numbers assigned in order from the leftmost microlens ML in the horizontal row of the microlens array 12. In the lateral origin distance calculation function, the control circuit 40 first creates two signal sequences corresponding to each lateral lens array 50. These two signal sequences are created by using the output of a specific light receiving element among the light receiving elements included in the light receiving element array 30. The control circuit 40 of the present embodiment creates two signal sequences using the outputs of the light receiving element a on the left side of the center and the light receiving element b on the right side of the center in the light receiving element array 30 shown in FIG.

The control circuit 40 extracts the output of the specific light receiving element from the light receiving element array 30 corresponding to the microlens ML included in the horizontal lens row 50. For example, in FIG. 3, the output of the light receiving element a and the output of the light receiving element b are extracted from each of the five microlenses ML constituting the horizontal lens row 50. The control circuit 40 arranges the outputs of the light receiving element a extracted in this manner in the order of the microlens ML to create the first signal train 51. The control circuit 40 arranges the outputs of the light receiving elements b in the same manner to create the second signal train 52. In the following description, the output of the light receiving element a corresponding to the nth microlens ML is represented by a (n), and the output of the light receiving element b is represented by b (n). That is, the first signal string 51 is composed of the signal values of a (3) to a (7), and the second signal string 52 is composed of the signal values of b (3) to b (7).

The control circuit 40 has a phase difference between the first signal train 51 created for each horizontal lens row as described above and the second signal train 52 created for the horizontal lens train located around the horizontal lens row. Is calculated to calculate the defocus amount. As a specific defocus calculation method, for example, a known method described in JP-A-60-37513 is used.

FIG. 4 is a diagram showing how the phase difference is calculated. When determining the defocus amount for the horizontal lens row 50a shown in FIG. 4, the control circuit 40 first has a first signal row 51 corresponding to the horizontal lens row 50a and a second signal row 52 corresponding to the same horizontal lens row 50a. Calculate the correlation value of. Next, the correlation value with the second signal string 53 corresponding to the horizontal lens row to the right of one, the correlation value with the second signal string 54 corresponding to the horizontal lens row to the right of two, etc. are set to the horizontal lens. Calculated for each of a plurality of horizontal lens rows located in a certain range on the left and right of row 50a. The same calculation is performed for the signal sequence corresponding to the lateral lens array to the left of the lateral lens array 50a. The difference in position between the horizontal lens row on which the defocus amount is calculated and the horizontal lens row on which the correlation value is calculated is called the shift number. When calculating the defocus amount, the control circuit 40 calculates the correlation value C (N) defined by the following equation (1) while changing the shift number N within a predetermined range (for example, -5 to +5). That is, the correlation values C (-5), C (-4), C (-3), ..., C (3), C (4), and C (5) are calculated. However, Fig. 4 shows an example in which the number of shifts N is changed in the range of -2 to +2 for simplicity. Here, pL is the number of the microlens ML corresponding to the signal value at the left end of the first signal string 51, and qL is the number of the microlens ML corresponding to the signal value at the right end of the first signal string 51.

<maths num="1"><img file="JP2011237215A_D0001.tif" /></maths>

Of the correlation values C (N) calculated in this way, the minimum value is called C0. Further, N in which C (N) = C0, that is, the number of shifts in which the correlation value is minimized is called N0. From this shift number N0, the control circuit 40 calculates a more precise shift number Na by the following equation (2). Here, Cr is C (N0-1) and Cf is C (N0 + 1).

<maths num="2"><img file="JP2011237215A_D0002.tif" /></maths>

The control circuit 40 adds a predetermined correction amount const according to the position of the focus detection surface to the shift number Na obtained by Eq. (2), and calculates the image shift amount Δn on the focus detection surface. The final defocus amount Df can be calculated by multiplying the above image shift amount Δn by a constant Kf depending on the detection opening angle. That is, the defocus amount Df = Δn × Kf.

Here, equation (2) does not always calculate a reliable shift number Na. That is, the calculation result by Eq. (2) may be the unreliable shift number Na. Therefore, the control circuit 40 calculates the reliability parameter R representing the reliability of the shift number Na by the following equation (3) after calculating the shift number Na. The reliability parameter R becomes smaller as the reliability of the shift number Na becomes higher. When the reliability parameter R is larger than a predetermined value, the control circuit 40 determines that it is impossible to calculate the defocus amount from the horizontal lens array.

<maths num="3"><img file="JP2011237215A_D0003.tif" /></maths>

The defocus amount calculated above is the defocus amount with respect to the position of the microlens ML located at the center of the horizontal lens row on which the defocus amount is calculated. The control circuit 40 calculates the defocus amount for all the horizontal lens rows selected by the horizontal selection function by the procedure described above. As a result, the defocus amount of all the microlens MLs included in the microlens array 12 is calculated for the microlens ML excluding the microlens MLs near the left and right ends.

FIG. 5 is a diagram showing a microlens ML in which the defocus amount is calculated by the horizontal origin distance calculation function and the vertical origin distance calculation function. The amount of defocus is not calculated for the range shown by the diagonal lines in FIG. Therefore, in FIG. 5A, the defocus amount is calculated for the range 56 excluding the left and right ends of the microlens ML.

Similarly for the vertical origin distance calculation function, the control circuit 40 is a subject in which an image is formed on the microlens ML in the center of the vertical lens row for each of the plurality of vertical lens rows selected by the vertical selection function. Calculate the distance to. The range 57 shown in FIG. 5 (b) is the range in which the defocus amount is calculated by the vertical origin distance calculation function. The only difference between the horizontal origin distance calculation function and the vertical origin distance calculation function is whether the arrangement direction of the microlens ML and the light receiving elements used is the horizontal direction or the vertical direction.

(Explanation of horizontal depth map creation function) The lateral-derived depth map creation function sets the defocus amount, which is a value representing the distance to the subject corresponding to each microlens calculated by the lateral-derived distance calculation function, according to the position of the microlens ML corresponding to the distance. It is a function to create a laterally-derived depth map by arranging them in a two-dimensional manner. The control circuit 40 creates a laterally-derived depth map by arranging the amount of defocus in each microlens ML according to the position on the microlens array 12. Similarly, the control circuit 40 creates a vertical-derived depth map by arranging a plurality of defocus amounts calculated by the vertical-derived distance calculation function in a two-dimensional manner for the vertical-derived depth map creating function.

When the control circuit 40 determines that it is impossible to calculate the defocus amount, the control circuit 40 obtains the defocus amount by interpolating it from the defocus amount of the surrounding (up / down / left / right) microlens ML. , Create a horizontal depth map and a vertical depth map.

(Explanation of depth map composition function) The depth map composition function is a function that executes a composition process of synthesizing a horizontally-derived depth map and a vertically-derived depth map and outputting a final depth map. Specifically, the control circuit 40 describes the defocus amount on the horizontal depth map and the vertical depth for each microlens ML in which both the horizontal depth map and the vertical depth map include the defocus amount. Compare with the defocus amount on the map, and adopt the larger defocus amount as the defocus amount in the final depth map.

FIG. 6 is a diagram showing the range of the depth map that is finally output. As shown in FIG. 6, the control circuit 40 outputs a depth map in which the amount of defocus for the microlens ML within the range 58 excluding the upper, lower, left and right ends of the microlens array 12 is arranged. The reason for performing the above composition is that the amount of defocus at the boundary between a short-distance subject and a long-distance subject is easily affected by the short-distance subject. This point will be described below.

FIG. 7 is a diagram illustrating the amount of defocus in the laterally-derived depth map. The boundary line of each light receiving element array 30 is superimposed on the captured image 60 shown in FIG. 7 (a). The shaded area in the captured image 60 indicates that a distant subject is captured. Here, in the sub-row 61 composed of each light receiving element arrangement 30 at the lowermost end, the light receiving element arrangements PP (6) and PP (12) are located at the boundary between the near subject and the distant subject.

Here, since PP (5) corresponds to a distant subject, it is desirable that the amount of defocus at this location is large. However, as shown in FIG. 7 (b), the defocus amount is calculated including the output of the light receiving element corresponding to the left and right microlens ML. That is, in calculating the defocus amount of PP (5), a (3) to a (7) are used in this example in the equation (1). However, here a (n) is the output extracted from PP (n). Therefore, the signal value a (6) or the like has an influence on this defocus amount. As a result, the actually calculated defocus amount becomes smaller than the ideal value. A graph of the ideal defocus amount and the actually calculated defocus amount is shown in Fig. 7 (c). The actually calculated defocus amount 63 (solid line graph in Fig. 7 (c)) is the ideal defocus amount 62 (dotted line in Fig. 7 (c)) near the boundary between a distant subject and a near subject. It tends to be smaller than the graph).

The reason why the control circuit 40 creates the final depth map by synthesizing the horizontal-derived depth map and the vertical-derived depth map is to eliminate such a problem near the boundary between the distant subject and the near subject. Is. That is, even if a distant subject and a near subject are adjacent to each other in the horizontal direction, if they are not adjacent to each other in the vertical direction, the problem at the boundary as described above can be solved. .. The same applies to the case where they are adjacent in the vertical direction and not in the horizontal direction.

According to the stereo image pickup apparatus according to the first embodiment described above, the following effects can be obtained. (1) The control circuit 40 selects a plurality of lateral lens rows arranged in the lateral direction from the plurality of microlens MLs included in the microlens array 12, and selects the horizontal lens row for each of the selected horizontal lens rows. The output of the light receiving element corresponding to each of the constituent microlens MLs is calculated, and the values representing the distances to the subject in which the images are formed before and after the microlens ML in the center of the horizontal lens row are calculated, and these are also calculated. Create a horizontal depth map by arranging the values of. Then, similarly, after creating a vertical-derived depth map in the vertical direction, the horizontal-derived depth map and the vertical-derived depth map are combined and the final depth map is output. Since this is done, it is possible to calculate a value representing an accurate distance to the subject even in the vicinity of a portion where the perspective of the subject changes.

(Second embodiment) The stereo imaging device according to the second embodiment has the same configuration as the stereo imaging device 1 according to the first embodiment shown in FIG. In addition to this, the control circuit 40 has a function of creating a depth map corresponding to the captured image created by the right image pickup unit 200 from the depth map created by the left image pickup unit 100. This function will be described below.

The control circuit 40 of the present embodiment that has received the depth map creation instruction signal creates a captured image and a depth map using the left image pickup unit 100, and at the same time creates a captured image using the right image pickup unit 200. At this time, the photographing lens 11 and the photographing lens 21 are kept in focus on the same surface of the field of view. The control circuit 40 detects information on the focus adjustment surface when imaging is performed by an encoder (not shown) or the like attached to the photographing lens 11 and the photographing lens 21. Here, the information on the focus adjusting surface is the subject distance in which the photographing lens 11 and the photographing lens 21 are in focus, and is information such as 1 m or 3 m, for example.

The control circuit 40 then creates a captured image and a depth map using the left imaging unit 100, as in the first embodiment. After that, the control circuit 40 calculates the distance to the subject for each point constituting the depth map from the created depth map and the above-mentioned information on the focus adjustment surface. For example, when the focal length of the photographing lens 11 is 50 mm and the subject distance is 1 m, the defocus amount becomes 0 for the subject located 1 m ahead of the photographing lens 11. Further, for example, when the defocus amount is 0.5 mm, the subject distance is calculated to be 1.25 m. Similarly, even in the case of other defocus amounts, it is possible to calculate the distance to the subject according to the value.

In parallel with this, the control circuit 40 also creates a captured image using the right image pickup unit 200. Hereinafter, the captured image created by using the left imaging unit 100 will be referred to as a left captured image, and the captured image created by using the right imaging unit 200 will be referred to as a right captured image.

After that, the control circuit 40 calculates how much each point of the left side shot image moves laterally in the right side shot image based on the distance to the subject for each point calculated by the above process. From the positional relationship between the left imaging unit 100 and the right imaging unit, it is obvious that each point of the left imaging image moves only to the left and right in the right imaging image, and does not move up and down. Further, the amount of movement at this time is smaller for the pixels of the subject located farther from the stereo image pickup device 1, and is larger for the pixels of the subject closer to the stereo image pickup device 1.

Finally, the control circuit 40 moves each point on the depth map in the horizontal direction according to the amount of movement in the horizontal direction obtained by the above process. That is, the defocus amount corresponding to a specific point in the left-side captured image is defined as the defocus amount corresponding to the left-side point. At this time, if a plurality of defocus amounts correspond to one point, the smaller defocus amount is prioritized. In addition, the defocus amount estimated from the defocus amount at the adjacent point on the right side corresponds to the point where none of the defocus amounts correspond. This estimated value is estimated, for example, by extrapolating from the distribution of the defocus amount on the right side of the point to be estimated.

At this time, it is considered that the "point where none of the defocus amounts correspond" is the portion of the left image pickup unit 100 that is in the shadow of the subject on the front side. That is, it means that the subject is behind the subject on the front side.

As described above, the control circuit 40 creates a depth map corresponding to the right image pickup unit 200 from the depth map created based on the left image pickup unit 100. The new depth map created here is smaller than the resolution of the right-hand shot image. Therefore, when actually using the created depth map, it is necessary to enlarge the depth map according to the right-side captured image or reduce the right-side captured image according to the depth map. A known image enlargement technique may be used to enlarge the depth map.

According to the stereo image pickup apparatus according to the second embodiment described above, the following effects can be obtained. (1) The control circuit 40 captures the left side image based on the amount of deviation caused by the parallax between the left image captured image created by using the left image pickup unit 100 and the right image captured image created by using the right image pickup unit 200. The depth map corresponding to the unit 100 is corrected, and the depth map corresponding to the image taken on the right side is created. Since this is done, it becomes possible to use a more accurate depth map corresponding to the right-hand shot image having a higher number of pixels than the left-hand shot image, and the convenience is improved.

The following modifications are also within the scope of the present invention, and one or more of the modifications can be combined with the above-described embodiment.

(Modification example 1) As shown in FIG. 1, in the image pickup device 13 according to the first embodiment, each light receiving element array 30 exists independently. That is, each light receiving element array 30 is clearly separated. Instead of doing so, the light receiving element may be continuously present on the image pickup surface of the image pickup element 13, such as the image pickup element 23. That is, the image sensor 13 may have the same configuration as the image sensor 23.

(Modification example 2) In the present invention, if at least two light receiving elements are arranged along the two directions for each microlens ML, the depth map creation process can be performed. Therefore, the present invention can be applied to an apparatus in which the light receiving element array 30 is composed of three light receiving elements. Further, in the first embodiment, each light receiving element array 30 is composed of 5 columns and 5 rows of light receiving elements, but if the number of light receiving elements constituting the light receiving element array 30 is 3 or more, any number of light receiving elements can be used. There may be.

(Modification example 3) The first signal string and the second signal string used for calculating the defocus amount may be created from the outputs of light receiving elements other than the light receiving element a and the light receiving element b shown in FIG. Further, a signal sequence may be created from the calculated values of the outputs of a plurality of light receiving elements for each light receiving element array. For example, the average value of the outputs of a plurality of light receiving elements may be calculated, or the weighted average may be calculated with a predetermined weight.

(Modification example 4) The correlation value C (N) between the first signal sequence and the second signal sequence may be calculated by the following equation (4) instead of the equation (1). Here, the function Floor (x) is a so-called floor function that returns the largest integer less than or equal to x.

<maths num="4"><img file="JP2011237215A_D0004.tif" /></maths>

(Modification example 5) In the first embodiment, the horizontal lens row and the vertical lens row are always composed of a fixed number of microlens MLs. This may be such that some lens sequences consist of a smaller number of microlens MLs. Specifically, the control circuit 40 first extracts the output of the light receiving element one by one from the light receiving element array 30 corresponding to each microlens ML. Then, intermediate image data in which the extracted outputs are arranged two-dimensionally is created. By applying the lateral differential filter to the intermediate image data, the rate of change in the amount of light of the subject image in the lateral direction is calculated. Similarly, if a vertical differential filter is applied, the rate of change in the amount of light of the subject image in the vertical direction is calculated. The control circuit 40 refers to the rate of change of the amount of light of the subject image in the corresponding directions in the horizontal selection function and the vertical selection function, and is selected at a place where the rate of change is larger than a predetermined threshold value and a place around it. The lens array is selected so that the lens array is composed of less microlens ML than usual. By doing so, even when the boundary between a distant subject and a near subject exists in an oblique direction, the adverse effect of a nearby adjacent subject as explained in FIG. 7 is reduced and more accurate. It becomes possible to calculate the defocus amount.

(Modification example 6) In the depth map composition function, the control circuit 40 may refer to an element other than the magnitude of the defocus amount. For example, the synthesis process may be executed in consideration of the strength of the correlation between the horizontal depth map and the vertical depth map.

(Modification 7) The two imaging units may be arranged vertically instead of horizontally. Further, the distance between the two imaging units is not limited to the numerical values exemplified in the above-described embodiment.

(Modification example 8) The reliability parameter R of the defocus amount may be calculated by an equation other than the above equation (3). For example, the denominator on the right side of the above equation (3) may be the reliability parameter R.

The present invention is not limited to the above-described embodiment as long as the features of the present invention are not impaired, and other embodiments considered within the scope of the technical idea of the present invention are also included within the scope of the present invention. ..

1 Stereo imager 11, 21 Shooting lens 12 microlens array 13, 23 Image sensor 14, 24 Imaging control circuit 15, 25 video circuit 30 Light receiving element array 40 control circuit 100 Left imaging unit 200 Right imaging unit ML micro lens