Three-dimensional imaging apparatus

The present invention relates to a monocular three-dimensional imaging technique for acquiring a plurality of images having parallax using one optical system and one image sensor.

In recent years, the high functionality and high performance of digital cameras and digital movies that use image sensors such as CCD and CMOS have been remarkable. In particular, advances in semiconductor manufacturing technology have led to a high degree of miniaturization and high integration. As an image sensor, the number of pixels has increased from 1 million pixels to 10 million pixels, and the image quality of the captured image has been dramatically improved. In addition, the display device is also a thin liquid crystal or plasma display, which saves space and realizes high resolution and high contrast performance. The trend toward higher quality images is progressing from 2D images to 3D images as target images, and although polarized glasses are required these days, high-quality 3D display devices have begun to be developed. ..

With respect to the three-dimensional imaging technology, a typical simple configuration is to capture an image for the right eye and an image for the left eye by using an imaging system composed of two cameras. Since such a so-called two-lens imaging method technology uses two cameras, the imaging device may become large and costly. Therefore, a method of using one camera is being researched. For example, Patent Document 1 introduces a method using two polarizing plates whose polarization directions are orthogonal to each other and a rotating polarizing filter. FIG. 10 shows the configuration of the imaging system in this method.

In FIG. 10, 11 is a 0-degree polarized polarizing plate, 12 is a 90-degree polarized polarizing plate, 13 is a reflecting mirror, and 14 is the light transmitted through the polarizing plate 12 and the light reflected by the reflecting mirror 13 through the polarizing plate 11. A half mirror that transmits and reflects, 15 is a circular polarizing filter, 16 is a driving device that rotates a circular polarizing filter, 3 is an optical lens, and 9 is an imaging device that captures an image formed by an optical lens.

With the above configuration, the incident light passes through polarizing plates 11 and 12 arranged at different locations, and then their optical axes are aligned by a reflector and a half mirror, and one image is taken through a circular polarizing filter and an optical lens. Imaged by the device. The imaging principle of this method is that by rotating a circular polarizing filter, the light incident on the two polarizing plates is captured at different timings, and two images having parallax are imaged.

However, in the above method, since images at different positions are captured by time division while rotating a circular polarizing filter, there is a problem that images having parallax cannot be captured at the same time and mechanical drive is used, so that there is a problem in durability. obtain. In addition, since all the incident light is received by the polarizing plate and the polarizing filter, there is also a problem that the amount of received light is reduced by 50% or more.

In contrast to the above method, Patent Document 2 introduces a method of simultaneously capturing images having parallax without using mechanical drive. In the method of this patent document, two incident regions are created, light incident from those regions is collected and imaged by one image sensor, but the mechanical drive unit is not provided. FIG. 11 shows the configuration of the imaging system of this method, and the imaging principle will be described below. In FIG. 11, polarizing plates 11 and 12, whose polarization directions are orthogonal to each other, a reflecting mirror 13, an optical lens 3, and an image pickup element 1 are arranged, 10 is a pixel of the image pickup element, and 17 and 18 are one-to-one with the pixel of the image pickup element. The polarizing filter 17 has the same characteristics as the polarizing plate 11, and the polarizing filter 18 has the same characteristics as the polarizing plate 12. The polarizing filters 17 and 18 are arranged alternately and arranged on all pixels.

With the above configuration, the incident light passes through the polarizing plates 11 and 12, passes through the reflector 13 and the optical lens 3, and forms an image on the image sensor 1. Regarding the photoelectric conversion of the image formation, the light transmitted through the polarizing plate 11 and incident is photoelectrically converted by the pixels immediately below it through the polarizing filter 17, and the light transmitted through the polarizing plate 12 and incident is transmitted through the polarizing filter 18 and directly below the pixels. It is photoelectrically converted with. Here, assuming that the image of the incident light from the polarizing plate 11 is the image for the right eye and the image of the incident light from the polarizing plate 12 is the image for the left eye, the image obtained from the pixel group immediately below the polarizing filter 17 is the image for the right eye. Then, the image obtained from the pixel group immediately below the polarizing filter 18 is the image for the left eye.

After all, the method shown in Patent Document 2 has a resolution by alternately arranging polarizing filters having different characteristics on the pixels of the image sensor instead of using the rotating circular polarizing filter shown in Patent Document 1. Is halved, but the image for the right eye and the image for the left eye can be obtained at the same time.

However, although the above technique can obtain two images having a parallax with one image sensor, the amount of light is reduced because the incident light is transmitted through the polarizing plate, and the amount of light is also reduced when it is transmitted through the polarizing filter. Therefore, the sensitivity of the image is greatly reduced.

As another approach to the problem of image sensitivity reduction, Patent Document 3 shows a method of mechanically switching between imaging of two images having parallax and normal image imaging. The configuration of the imaging system in this method is shown in FIG. 12, and the basic principle of imaging is described. In FIG. 12, 19 has two light-transmitting portions 20 and 21, a light passing portion that transmits incident light from the optical lens 3 only through those transmitting portions, and 22 is a light passing portion that transmits light from the polarized light transmitting portions 20 and 21. The optical filter tray of the light receiving part, which is a set of the specific component transmission filter 23 and the color filter 24 to be separated, 25 removes the light passing part 19 and the specific component transmission filter 23 from the optical path and inserts the color filter 24 into the optical path. Alternatively, it is a filter drive unit that performs the opposite operation.

In this method, a filter driving unit is operated, a light passing unit and a specific component transmission filter are used for capturing an image having parallax, and a color filter is used for normal shooting. In imaging an image with parallax, it is basically the same as that shown in Patent Document 2, and the sensitivity of the image is greatly reduced. However, in normal imaging, the light passing portion is removed from the optical path, and the image is taken. By inserting a color filter instead of the specific component transmission filter, a color image without a decrease in sensitivity can be obtained.

<p><patcit num="1"><text>Japanese Patent Application Laid-Open No. 62-291292</text></patcit><patcit num="2"><text>Japanese Patent Application Laid-Open No. 62-217790</text></patcit><patcit num="3"><text>Japanese Unexamined Patent Publication No. 2001-016611</text></patcit></p>

<p> After all, in the conventional technique, two images having parallax can be captured by a monocular camera, but the sensitivity of each image is lowered. Further, in order to obtain more parallax information for the same subject for the purpose of improving the accuracy of the depth information of the subject, it is necessary to move the camera to take an image or rotate the camera itself to take an image. In any case, a mechanism to move the camera itself is required.</p><p> In view of the above problems, an object of the present invention is to provide an imaging technique that does not require a mechanism for moving the camera itself and can obtain more parallax information than the parallax information obtained from two images having parallax. To do. In the following description, a plurality of images having parallax are referred to as "multi-viewpoint images".</p>

<p> The three-dimensional image pickup device of the present invention has a light transmitting portion having at least two polarizers, a solid-state image sensor that receives light transmitted through the light-transmitting portion, and an image forming an image on the imaging surface of the solid-state imaging device. The unit includes a unit and a rotation drive unit that rotates the light transmitting unit with the direction of the optical axis of the incident light as the direction of the rotation axis. The light transmitting portion includes a first polarizer and a second polarizer having a transmitting axis having an angle of α (0 ° &lt;α 90 °) with respect to the transmitting axis of the first polarizing element. Have. The solid-state image sensor includes a plurality of pixel blocks each including a first pixel and a second pixel, a first polarizing filter arranged to face the first pixel in each pixel block, and each pixel. In the block, a second polarizing filter, which is arranged to face the second pixel and has a transmission axis at an angle of β (0 ° &lt;β 90 °) with respect to the transmission axis of the first polarizing filter, have. The first polarizing filter is arranged so as to receive light transmitted through the first polarizing element and light transmitted through the second polarizing element, and the second polarizing filter passes the first polarizing element. It is arranged so as to receive the transmitted light and the light transmitted through the second polarizer.</p><p> In a preferred embodiment, the light transmitting portion has a transparent region that transmits incident light regardless of the polarization direction, each pixel block includes a third pixel, and the third pixel is the first. The light transmitted through the polarizer, the light transmitted through the second polarizer, and the light transmitted through the transparent region are received, and a photoelectric conversion signal corresponding to the received light is output.</p><p> In a preferred embodiment, the transmission rate when unpolarized light is incident on the first polarizing element, the second polarizing element, the first polarizing filter, and the second polarizing filter is T1, and the transmission of the first polarizing filter is defined as T1. The transmittance when the polarized light vibrating in the direction of the axis is incident on the first polarizing filter and the transmittance when the polarized light vibrating in the direction of the transmission axis of the second polarizing filter is incident on the second polarizing filter. Let T2 be, and let φ be the angle formed by the direction of the transmission axis of the first polarizing element with respect to the direction of the transmission axis of the first polarizing filter.</p><p><maths num="1"><img file="JP2011237646A_D0001.tif" /></maths></p><p>The rotation angle of the light transmitting portion is set so that the value of is not 0.</p><p> In a preferred embodiment</p><p><maths num="2"><img file="JP2011237646A_D0002.tif" /></maths></p><p>Satisfying the relationship, φ is 0 φ &lt;π / 2-α, π / 2 + β &lt;φ &lt;3π / 2-α, 3π / 2 + β &lt;φ &lt;2π It is set to one of the ranges of.</p><p> In a preferred embodiment, 80 ° α 90 ° is satisfied.</p><p> In a preferred embodiment, each pixel block further comprises a fourth pixel, and the solid-state image sensor emits light of a first color component arranged to face the third pixel contained in each pixel block. It has a first color filter for transmitting light and a second color filter for transmitting light of a second color component arranged so as to face the fourth pixel included in each pixel block.</p><p> In a preferred embodiment, in each pixel block, the first pixel, the second pixel, the third pixel, and the fourth pixel are arranged in a matrix, and the first pixel is The second pixel is arranged in the first row and the first column, the second pixel is arranged in the second row and the second column, the third pixel is arranged in the first row and the second column, and the fourth pixel is arranged in the second row and the first column. It is located in.</p><p> In a preferred embodiment, one of the first color filter and the second color filter transmits light of a yellow component, and the other of the first color filter and the second color filter is of a cyan component. Allows light to pass through.</p><p> In a preferred embodiment, where φ is the angle formed by the direction of the transmission axis of the first polarizing element with respect to the direction of the transmission axis of the first polarizing filter, φ = φ1 (0 ° φ1 &lt;360 °). Imaging is performed in each of the first state where the above is true and the second state where φ = φ1 + 180 °.</p><p> In a preferred embodiment, the image processing unit further includes an image processing unit, which uses photoelectric conversion signals output from the first pixel and the second pixel to show an image showing the difference between two images having a parallax. To form.</p><p> The image forming method of the present invention is used in the three-dimensional imaging apparatus of the present invention, and a step of acquiring a first photoelectric conversion signal from the first pixel and a photoelectric conversion signal from the second pixel to a second pixel are obtained. It includes a step of acquiring and a step of forming an image showing the difference between two images having a parallax based on the first photoelectric conversion signal and the second photoelectric conversion signal.</p>

<p> According to the three-dimensional image pickup apparatus of the present invention, the image pickup device has at least two polarization regions with respect to the incident region of light, and the image pickup element has at least two types of pixel groups in which a polarizing filter is arranged. Therefore, images from the two incident regions are captured by two types of pixel groups. This is the same as capturing different incident light information with sensors with different characteristics, and the relationship between two outputs for two inputs can be expressed by a specific mathematical formula. Therefore, on the contrary, it is possible to calculate two input information from two output results. That is, the difference information between the plurality of viewpoint images can be obtained by obtaining the image information from the two polarized regions and then performing the difference processing between them. Further, since the three-dimensional image pickup apparatus of the present invention has a rotation drive unit that rotates the incident region of light, it is possible to perform imaging by changing the location of the polarization region. As a result, there is an effect that the depth information of the subject whose viewpoint is changed can be obtained. This makes it possible to improve the accuracy of the depth information of the subject.</p>

<figref num="1">Overall configuration diagram of the image pickup apparatus according to the first embodiment of the present invention</figref><figref num="2">Schematic diagram showing how light is incident on the solid-state image sensor according to the first embodiment of the present invention.</figref><figref num="3">Front view of the translucent plate according to the first embodiment of the present invention</figref><figref num="4">Basic pixel configuration diagram of the image pickup unit of the solid-state image pickup device according to the first embodiment of the present invention.</figref><figref num="5">Relationship diagram of rotation angle φ and | D | in the first embodiment of the present invention</figref><figref num="6">Basic color configuration diagram of the image pickup unit of the solid-state image pickup device according to the second embodiment of the present invention.</figref><figref num="7">Relationship diagram of rotation angle φ and | D | in the second embodiment of the present invention</figref><figref num="8">Basic pixel configuration diagram of another solid-state image sensor according to the first embodiment of the present invention</figref><figref num="9">Front view of another light-transmitting plate according to the first embodiment of the present invention.</figref><figref num="10">Configuration diagram of the imaging system in Patent Document 1</figref><figref num="11">Configuration diagram of the imaging system in Patent Document 2</figref><figref num="12">Configuration diagram of the imaging system in Patent Document 3</figref>

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Elements that are common to all figures have the same reference numerals.

(Embodiment 1) FIG. 1 is a configuration diagram of an image pickup apparatus according to the first embodiment of the present invention. 1 is a solid-state image sensor that performs photoelectric conversion, 2 is a light-transmitting plate that has a partially polarized region, 2A is a rotation drive unit that rotates the light-transmitting plate 2 with the direction of the optical axis as the direction of the rotation axis, and 3 is the incident light. Circular optical lens for imaging, 4 is an infrared cut filter, 5 is a signal generator and image signal receiver that generates the original signal used to drive the solid-state image sensor and receives the signal from the solid-state image sensor, 6 Is an element drive unit that creates a signal for driving a solid-state image sensor, 7 is a multi-viewpoint image that processes the image signal, an image showing the difference between the multi-viewpoint images (difference image), and there is no parallax and there is no problem in sensitivity. An image processing unit that generates an image (normal image), and 8 is an image interface unit that sends out the generated multi-viewpoint image, difference image, and image signal of the normal image to the outside. In the following description, the multi-viewpoint image and the difference image may be collectively referred to as an "image showing parallax".

The light transmitting plate 2 has a polarizing region in which two polarizing elements are arranged and a transparent region in which light is transmitted regardless of the polarization direction. The solid-state image sensor 1 (hereinafter, may be referred to as an image sensor) is typically a CCD or CMOS sensor, and is manufactured by a known semiconductor technology. A plurality of pixels (light sensing cells) are arranged two-dimensionally on the imaging surface of the solid-state imaging device 1. Each pixel is typically a photodiode, and outputs an electric signal (photoelectric conversion signal) according to the amount of incident light by photoelectric conversion. The image processing unit 7 has a memory for storing various information used for image processing and an image signal generation unit for generating an image signal for each pixel based on the data read from the memory.

With such a configuration, the incident light is imaged on the image pickup surface of the solid-state image sensor 1 through the light transmitting plate 2, the optical lens 3, and the infrared cut filter 4, and is photoelectrically converted by the solid-state image sensor 1. The image signal generated by the photoelectric conversion is sent to the image processing unit 7 through the image signal receiving unit 5, where a multi-viewpoint image, a difference image, and a normal image having no parallax and having no problem in sensitivity are generated. By rotating the light transmitting plate 2 by the rotation driving unit 2A, the positions of the two polarization regions on the light transmitting plate 2 can be changed. Further, the rotation drive unit 2A operates by receiving a signal generation and a command signal from the image signal reception unit 5 via the element drive unit 6.

FIG. 2 schematically shows how the incident light passes through the light transmitting plate 2 and the optical lens 3 and is incident on the image pickup surface of the solid-state image pickup device 1. In FIG. 2, components other than the translucent plate 2, the optical lens 3, the solid-state image sensor 1, and the rotation drive unit 2A are omitted. Further, for the solid-state image sensor 1, only a part of the image pickup surface is shown. As shown, the translucent plate 2 has polarization regions P (1), P (2) and transparent regions P (3). Here, the directions of the transmission axes of the polarization regions P (1) and P (2) are different from each other. Further, the plurality of pixels arranged on the image pickup surface of the solid-state image pickup device 1 constitute a plurality of pixel blocks having three pixels as one unit. The three pixels included in one pixel block are called W1, W2, and W3. In the present embodiment, the polarizing filters 50a and 50b are arranged so as to face the pixels W1 and W2, respectively. The directions of the transmission axes of the polarizing filters 50a and 50b are different from each other. The corresponding polarizing filter is not arranged in the pixel W3.

It should be noted that the arrangement relationship of each component shown in the drawing is merely an example, and the present invention is not limited to this arrangement relationship. For example, the optical lens 3 may be arranged farther from the image pickup element 1 than the light transmitting plate 2 as long as an image can be formed on the image pickup surface, or a plurality of optical lenses 3 may be arranged. Further, the optical lens 3 and the translucent plate 2 do not have to be independent components, and both may be configured as one integrated optical element. Further, in FIG. 2, the pixels W1, W2, and W3 are arranged in order along the direction (X direction) parallel to the line segment connecting the polarization regions P (1) and P (2) of the light transmitting plate 2. It is drawn as if it were, but it does not necessarily have to be arranged that way. On the image pickup surface of the image pickup device 1, a plurality of pixels are arranged in a direction (Y direction) perpendicular to the paper surface of FIG.

Hereinafter, the configuration of the light transmitting plate 2 and the pixel configuration of the solid-state image sensor 1 will be described in more detail. In the following description, the same coordinate system as in FIG. 2 is used.

FIG. 3 is a front view of the translucent plate 2 in the present embodiment. The shape of the light transmitting plate 2 is the same circular shape as that of the optical lens 3. In the light transmitting plate 2, two polarizing regions P (1) and P (2) having different directions of transmission axes are arranged apart from each other in the X direction. In the translucent plate 2, the region other than P (1) and P (2) is the transparent region P (3). The direction of the transmission axis of the polarization region P (1) coincides with the X direction when the light transmitting plate 2 is not rotated. The direction of the transmission axis of the polarization region P (2) is tilted by an angle α (0 ° &lt;α 90 °) with respect to the direction of the transmission axis of the polarization region P (1).

In the image pickup apparatus of this embodiment, the light transmitting plate 2 can be rotated by the rotation driving unit 2A. When the rotation angle of the light transmitting plate 2 is θ (0 ° θ &lt;360 °), the angle formed by the transmission axis of the polarization region P (1) with respect to the X direction is θ, and the polarization region P (2). The angle formed by the direction of the transmission axis of) with respect to the X direction is θ + α. Although the shape of the light transmitting plate 2 is circular in FIG. 3, it does not necessarily have to be circular. Further, the shapes of the polarized light regions P (1) and P (2) do not necessarily have to be rectangular, and may be any shape. However, it is preferable that the areas and shapes of the polarized light regions P (1) and P (2) are the same.

FIG. 4 shows one pixel block on the image pickup surface of the image pickup device 1. A plurality of pixels having a basic configuration of 3 rows and 1 column are arranged on the imaging surface. As described above, the basic configuration of pixels includes pixels W1 and W2 in which two polarizing filters 50a and 50b having different polarization directions are arranged, and pixels W3 in which nothing is arranged. In one pixel block, W1, W2, and W3 are arranged along the Y axis. Regarding the direction of the transmission axis of the polarizing filter, the transmission axis of the polarizing filter 50a in the 1st row and 1st column is tilted by an angle γ (0 ° γ 90 °) with respect to the X direction, and the polarizing filter 50b in the 2nd row and 1st column The transmission axis is tilted by an angle γ + β (0 ° &lt;β 90 °) with respect to the X direction.

With the above configuration, each pixel on the imaging surface passes through the polarization regions P (1), P (2), and the transparent region P (3) and receives the light focused by the optical lens 3. Hereinafter, the photoelectric conversion signal in each pixel will be described.

First, the photoelectric conversion signal of the pixel W3 in which the polarizing filter is not arranged will be described. The pixel W3 receives the incident light through the light transmitting plate 2, the optical lens 3, and the infrared cut filter 4, and outputs a photoelectric conversion signal according to the received light. Here, let T1 be the transmittance when the incident light passes through the polarization regions P (1) and P (2) of the light transmitting plate 2. The signal amount assuming that the light incident on the polarized light regions P (1) and P (2) and the transparent region P (3) is not dimmed and is photoelectrically converted by the image sensor 1 is added with a subscript s. Expressed as Ps (1), Ps (2), and Ps (3), respectively, the photoelectric conversion signal S3 in the pixel W3 is expressed by the following equation 1.

(Equation 1) S3 = T1 (Ps (1) + Ps (2)) + Ps (3) Next, the photoelectric conversion signals of the pixels W1 and W2 in which the polarizing filter is arranged will be described. Since the polarizing filters 50a and 50b are arranged so as to face the pixels W1 and W2, the amount of light incident on the pixels W1 and W2 is basically smaller than the amount of light incident on the pixels W3. Here, the transmittance when the non-polarized light passes through the polarizing filter 50a or 50b is T1 as in the transmittance in the polarizing regions P (1) and P (2). Further, the transmittance at which polarized light vibrating in the same direction as the transmission axis of each polarizing filter passes through the polarizing filter is defined as T2. When the translucent plate 2 is rotated by the rotation drive unit 2A by an angle θ, the signals S1 and S2 corresponding to the photoelectric conversion amount in the pixels W1 and W2 are represented by the following equations 2 and 3, respectively.

(Equation 2) S1 = T1 (T2 (Ps (1) | cos (θ-γ) | + Ps (2) | cos (θ + α-γ) |) + Ps (3)) (Equation 3) S2 = T1 (T2 (Ps (1) | cos (θ-γ-β) | + Ps (2) | cos (θ + α-γ-β) |) + Ps (3)) Here, if φ = θ-γ, Equations 2 and 3 are represented by the following Equations 4 and 5, respectively.

(Equation 4) S1 = T1 (T2 (Ps (1) | cosφ | + Ps (2) | cos (φ + α) |) + Ps (3)) (Equation 5) S2 = T1 (T2 (Ps (1) | cos (φ-β) | + Ps (2) | cos (φ + α-β) |) + Ps (3)) φ = θ-γ is the relative rotation angle of the light transmitting plate 2 with respect to the direction of the transmission axis of the polarizing filter 50a. Here, when Ps (3) is eliminated from the above equations 1, 4, and 5, and Ps (1) and Ps (2) are calculated, Ps (1) and Ps (2) are as follows, respectively. It is expressed by Equation 6 and Equation 7.

<maths num="3"><img file="JP2011237646A_D0003.tif" /></maths>

<maths num="4"><img file="JP2011237646A_D0004.tif" /></maths>

Here, | D | in the denominator in equations 6 and 7 is a determinant expressed by the following equation 8.

<maths num="5"><img file="JP2011237646A_D0005.tif" /></maths>

The signals Ps (1) and Ps (2) indicating the image by the light transmitted through the polarization regions P (1) and P (2) and incident on the imaging surface are obtained from S1, S2, and S3 by the equations 6 and 7. be able to. Ps (1) and Ps (2) correspond to two images with different viewpoints, and information on the depth of the subject can be obtained by obtaining the difference between them. In the present embodiment, the difference image is obtained by the difference between Ps (1) and Ps (2). Hereinafter, the signal indicating the difference image is referred to as Ds. Further, the signal Ps (3) indicating the image by the light transmitted through the transparent region can be obtained by substituting Ps (1) and Ps (2) represented by the equations 6 and 7 into the equation 1.

As described above, according to the image pickup apparatus of the present embodiment, it is possible to obtain a plurality of viewpoint images, a difference image, and a normal image without parallax. By rotating the light transmitting plate 2 by the rotation driving unit 2A, the positions of the polarized light regions P (1) and P (2) can be changed, and the parallax state can be changed for imaging. A plurality of parallax information can be obtained by taking an image at different positions of the polarized light regions P (1) and P (2) and performing the above processing. As a result, it is possible to improve the accuracy of the depth information of the subject as compared with the case of obtaining the depth information of the subject based on one parallax information.

If the value of the determinant | D | shown in Equation 8 is 0, the denominators of Ps (1) and Ps (2) expressed in Equations 6 and 7 are also 0, so Ps (1) and Ps. (2) cannot be obtained. Therefore, in the image pickup apparatus of the present embodiment, the rotation angle θ of the light transmitting plate 2 is set so that the determinant | D | shown in the equation 8 does not become 0.

Hereinafter, a preferable rotation range of the light transmitting plate 2 will be described. Equation 8 is expressed by the following equation 9 when k = T1 / T2.

<maths num="6"><img file="JP2011237646A_D0006.tif" /></maths>

In equation 9, if each cos term is positive, the absolute value can be removed without changing the sign, and equation 9 can be expressed by the following equation 10.

<maths num="7"><img file="JP2011237646A_D0007.tif" /></maths>

By transforming Equation 10, the following Equation 11 is obtained.

<maths num="8"><img file="JP2011237646A_D0008.tif" /></maths>

Further, in Equation 9, if all the cos terms are negative, the positive and negative signs can be reversed and the absolute value can be removed, and Equation 9 is expressed by Equation 12.

<maths num="9"><img file="JP2011237646A_D0009.tif" /></maths>

Since the image pickup apparatus in this embodiment satisfies 0 &lt;α 90 ° and 0 &lt;β 90 °, the rotation angle θ of the transmissive plate 2 should be set within the range of 0 ° to 360 °. Considering that, equations 11 and 12 are always positive and do not become 0 as long as the following condition (expression 13) is satisfied.

(Equation 13) cos (α / 2) cos (β / 2)&gt; k (= T1 / T2) However, Equation 13 is obtained from the precondition that each cos term is all positive or all negative in Equation 9. That is, this precondition is that cosφ, cos (φ + α), cos (φ-β), and cos (φ + α-β) are all positive or all negative. Therefore, in all ranges except the range of φ from (90 ° -α) to (90 ° + β) and the range from (270 ° -α) to (270 ° + β), Equation 9 | D | represented by is always positive and cannot be 0. In other words, | D | is always positive as long as it satisfies any of 0 ° φ &lt;90 ° -α, 90 ° + β &lt;φ &lt;270 ° -α, 270 ° + β &lt;φ &lt;360 °. , Does not become 0.

As an example of each parameter value in this embodiment, for example, T1 = 0.45, T2 = 0.9, α = 60 °, β = 60 ° can be set. Since γ does not appear in the above formula group, there is no problem with any value, but here, γ = 0 is set. Under the above conditions, the left side of Eq. 13 becomes 3/4 and the right side becomes 1/2, so the relational expression of Eq. 13 holds. Figure 5 shows the dependence of the value of | D | on the angle φ in this case. As shown in FIG. 5, when at least φ is in the range of 0 φ &lt;30 °, 150 ° &lt;φ &lt;210 °, 330 ° &lt;φ &lt;360 °, there is no problem with multiple viewpoint images, difference images, and parallax. It can be seen that no normal image is obtained.

As described above, the image pickup apparatus of the present embodiment includes a translucent plate 2 having two polarization regions P (1) and P (2) and one transparent region P (3). Each pixel block on the image pickup surface of the image pickup device 1 includes pixels W1 and W2 in which two polarizing filters 50a and 50b having different transmission axis directions are arranged, and pixels W3 in which nothing is arranged. .. The angle formed by the direction of the transmission axis of the polarizing region P (1) and the direction of the transmission axis of the polarization region P (2) is α, and the direction of the transmission axis of the polarizing filter 50a and the direction of the transmission axis of the polarizing filter 50b form. When the angle is β, the difference φ between the rotation angle θ of the transmissive plate 2 and the angle γ formed by the direction of the transmission axis of the polarizing filter 50a with respect to the X direction is from (90 ° -α) to (90 ° + β). As long as it is within the range excluding the range up to) and the range from (270 ° -α) to (270 ° + β), it is possible to obtain a multi-viewpoint image, a difference image, and a normal two-dimensional image without polarization. .. In particular, the smaller the polarization regions of P (1) and P (2), the more sensitively no problem can be obtained in a two-dimensional image. Further, by rotating the light transmitting plate 2 so that the value of φ is within the above allowable range and calculating the difference image each time, there is an effect that the depth information of the subject whose viewpoint is changed can be obtained. This makes it possible to improve the accuracy of the depth information of the subject.

In the above example, α = 60 ° and β = 60 °, but α and β are not limited to these values. The angles φ, α, β are set to satisfy at least 0 ° φ &lt;90 ° -α, 90 ° + β &lt;φ &lt;270 ° -α, 270 ° + β &lt;φ &lt;360 °. If so, since Equation 13 holds, a plurality of viewpoint images and a difference image can be obtained without any problem. Even if φ is set to either 90 ° -α φ 90 ° + β or 270 ° -α φ 270 ° + β, the determinant of Equation 9 | D | As long as the value of is not 0, a multi-viewpoint image and a difference image can be obtained without any problem.

As described above, the rotation drive unit 2A in the present embodiment operates by receiving a signal generation and a command signal from the image signal reception unit 5 from the element drive unit 6, but the present invention is limited to such an embodiment. I can't. For example, the light transmitting plate 2 may be rotated by manually moving the rotation driving unit 2A.

In the present embodiment, a two-dimensional image having no problem in sensitivity is obtained from light transmitted through only the transparent region P (3) by calculation between pixels, but the present invention is not limited to this. A two-dimensional image may be obtained by using all the light transmitted through the regions P (1), P (2), and P (3). In other words, a two-dimensional image may be generated by synthesizing the signals represented by Ps (1), Ps (2), and Ps (3).

In the above description, the light transmitting plate 2 is provided with two polarization regions (polarizers), but three or more polarization regions may be provided. Further, the direction of the transmission axis of the polarization region P (1) in the state of θ = 0 ° does not have to coincide with the X direction, and may be any direction.

Further, in the example shown in FIG. 4, the shapes of the pixels W1, W2, and W3 are square, and the pixels W1, W2, and W3 are arranged adjacent to each other in the Y direction. Not limited. The shape of each pixel may be any shape, and the pixels W1, W2, and W3 do not necessarily have to be adjacent to each other in the Y direction. However, it is preferable that the pixels are close to each other.

In the image pickup apparatus of the present embodiment, as shown in FIG. 2, the light transmitting plate 2 and the image pickup surface of the image pickup device 1 are arranged in parallel. However, they do not necessarily have to be arranged in parallel. For example, by arranging an optical element such as a mirror or a prism between the two, the light transmitting plate 2 and the image pickup surface of the image pickup element 1 can be configured to be located on a plane intersecting each other. When such a configuration is adopted, the angles α and β are assumed to be parallel to each other when the light transmitting plate 2 and the image pickup surface of the image pickup element 1 are parallel to each other in consideration of the change in the optical path due to the optical element. It may be determined with reference to the direction of the transmission axis of the polarization region P (1).

In the above description, the imaging device is configured to simultaneously obtain a multi-viewpoint image, a difference image, and a normal image. However, the present invention is not limited to such a configuration, and may be configured to acquire a plurality of viewpoint images and a difference image without acquiring a normal image. When the image pickup apparatus is configured for such a purpose, the pixel W3 in the above description is unnecessary, and a light-shielding region that does not transmit light is provided instead of the transparent region P (3).

6 and 7 show an example of the configuration of the translucent plate 2 in the imaging device that acquires a multi-viewpoint image and a difference image without acquiring a normal image, and an example of a basic pixel configuration, respectively. The regions other than the polarized light regions P (1) and P (2) in the light transmitting plate 2 are light-shielding regions. Even in such an imaging device, the light transmitting plate 2 can be rotated around the optical axis by the rotation driving unit 2A. Further, on the image pickup surface of the image pickup device 1, a plurality of pixel blocks are arranged in units of pixel blocks including pixels W1 and W2.

With the above configuration, the photoelectric conversion signals S1 and S2 output from the pixels W1 and W2, respectively, can be represented by the following equations 14 and 15, respectively.

(Equation 14) S1 = T1T2 (Ps (1) cos α + Ps (2) cos (α-θ)) (Equation 15) S2 = T1T2 (Ps (1) cos β + Ps (2) cos (β-θ)) From equations 14 and 15, Ps (1) and Ps (2) are represented by the following equations 16 and 17, respectively.

<maths num="10"><img file="JP2011237646A_D0010.tif" /></maths>

<maths num="11"><img file="JP2011237646A_D0011.tif" /></maths>

Here, | D | is a determinant represented by the following equation 18.

<maths num="12"><img file="JP2011237646A_D0012.tif" /></maths>

Further, the difference image Ds is expressed by the following equation 19 by taking the difference between Ps (1) and Ps (2).

<maths num="13"><img file="JP2011237646A_D0013.tif" /></maths>

As shown by equations 16 to 19, the signals Ps (1), Ps (2), and the signal Ds indicating the difference image can be obtained from the photoelectric conversion signals S1 and S2 in the pixels W1 and W2. When acquiring each image showing parallax by this configuration, it is preferable that the angles θ, α, β, and γ are set so that the value of the determinant | D | in Equation 18 does not become close to 0. By performing imaging by changing the rotation angle θ of the translucent plate 2 even with such an imaging device, it is possible to obtain a plurality of sets of a plurality of viewpoint images.

(Embodiment 2) Next, a second embodiment of the present invention will be described with reference to FIGS. 8 and 9. The image pickup device of the present embodiment is different from the image pickup device of the first embodiment in the basic pixel configuration of the image pickup device 1 and the method of acquiring an image showing parallax. Hereinafter, only the differences from the image pickup apparatus of the first embodiment will be described, and the description of common matters will be omitted.

FIG. 8 shows the basic pixel configuration on the image pickup surface of the solid-state image sensor 1 in the present embodiment. In the present embodiment, the pixel is composed of a plurality of pixel blocks having a basic configuration of 2 rows and 2 columns, and a color element (color filter) or a polarizing filter is arranged facing each pixel. The color element in this embodiment is a known color filter that transmits only light having a wavelength range of a specific color component. In the following description, a color filter that transmits only the light of the color component C will be referred to as a C element.

Regarding the color element, the cyan element (Cy) is placed facing the pixel in the 1st row and 1st column, the yellow element (Ye) is placed facing the pixel in the 2nd row and 2nd column, and the 1st row and 2nd column and No color element is placed in the pixels in the 2nd row and 1st column. A polarizing filter whose transmission axis direction coincides with the X direction is placed in the pixels in the first row and second column, and the transmission axis direction forms an angle of 45 ° with respect to the X direction in the pixels in the second row and first column. A polarizing filter is arranged. The pixel array is a square array, and the direction of the line segment connecting the centers of the two polarizing filters arranged facing the two pixels W1 and W2 is at an angle of 45 ° with respect to the X direction. There is.

On the other hand, the shape of the light transmitting plate 2 is the same as the shape of the light transmitting plate 2 in the first embodiment. However, in the present embodiment, the angle formed by the direction of the transmission axis of the polarization region P (1) and the direction of the transmission axis of the polarization region P (2) is set to 90 °. That is, in this embodiment, α = 90 °, β = 45 °, γ = 0 °, and θ = φ. Also, T1 = 0.45 and T2 = 0.9.

In the imaging device of the present embodiment, the translucent plate 2 is rotated by the rotation drive unit 2A with φ1 as an arbitrary angle, and imaging is performed in each of the states of φ = φ1 and φ = φ1 + 180 °. Perform the processing operation shown in Form 1. For example, imaging is performed in the states of φ = 0 ° and φ = 180 °, and a multi-viewpoint image and a difference image in each state are acquired. In this embodiment, it is possible to take an image at a rotation angle other than the permissible rotation range shown in the first embodiment, but even if it is outside the permissible rotation range, | D | shown in Equation 9 should not be 0. There is no problem if α, β, and φ are set. Conditions for α, β, and φ shown in Embodiment 1: 0 ° φ &lt;90 ° -α, 90 ° + β &lt;φ &lt;270 ° -α, 270 ° + β &lt;φ &lt;360 ° are just equations. It guarantees that | D | of 9 does not become 0, and exists if | D | does not become 0 even if the value is out of the guaranteed range.

The main features of the image pickup apparatus of this embodiment are the following three points. The first point is that there is no pixel W3 shown in the first embodiment. The second point is that the directions of the transmission axes of the polarization regions P (1) and P (2) are orthogonal to each other, and by rotating the light transmitting plate 2, there are two, φ = φ1 and φ = φ1 + 180 °. Imaging is performed in each state. The third point is that the image sensor is colorized.

First, the first point of the above characteristics will be described. In the pixel configuration shown in FIG. 8, since the pixel W3 in the first embodiment is not provided, the calculation shown in the first embodiment cannot be performed originally. However, when capturing a nearly achromatic subject, Cy + Ye = W + G. Therefore, if the RGB received signal ratios are Kr, Kg, and Kb, the signal corresponding to the pixel signal S3 from the pixel W3 is It is considered that the sum of the signals obtained through the Cy element and the Ye element is Scy + Sye multiplied by (Kr + Kg + Kb) / (Kr + 2Kg + Kb). Therefore, a signal corresponding to the pixel signal S3 is obtained by performing such an operation. By doing so, if | D | represented by Equation 9 is not 0, a multi-viewpoint image and a difference image can be created by the same processing as in the case of the first embodiment. The received signal ratios Kr, Kg, and Kb are recorded in advance on the storage medium inside the image pickup apparatus as internal parameters of the image pickup apparatus.

Next, the second point of the feature of this embodiment will be described. FIG. 9 shows the dependence of the value of | D | on the rotation angle θ (= φ) of the translucent plate in this embodiment. From FIG. 9, it can be seen that | D | does not become 0 even if the translucent plate 2 is rotated so that φ becomes, for example, 0 ° or 180 °. As another example, | D | does not become 0 even when the transmissive plate 2 is rotated so that φ becomes 150 ° or 330 °. That is, if | D | does not become 0 in either the state of φ = φ1 and the state of φ = φ1 + 180 °, the arithmetic processing shown in the first embodiment can be performed in both states, and the polarization region P. Images of light incident on each of (1), P (2), and P (3) can be calculated.

In the present embodiment, there are two states of φ = φ1 and φ = φ1 + 180 ° with respect to the signals Ps (1) and Ps (2) representing the image by the light incident on the polarization regions P (1) and P (2). A difference image is obtained based on the two sets of data obtained by imaging each with. In this process, the first image information is obtained by performing square root processing on the result of square-adding the image Ps (1) at φ = φ1 and the image Ps (2) at φ = φ1 + 180 °. Similarly, the second image information is obtained by performing square root processing on the result of square-adding the image Ps (2) at φ = φ1 and the image Ps (1) at φ = φ1 + 180 °. The image Ps (1) at φ = φ1 and the image Ps (2) at φ = φ1 + 180 ° correspond to the images with polarization characteristics of 0 and 90 degrees observed at the same position, respectively. When the image is taken, the polarization characteristics of the subject can be canceled by the above processing. Normally, a subject having polarization characteristics is imaged using orthogonal filters, and the light energy of the subject is measured by adding the squares of the respective polarization components to obtain a value. Therefore, the same applies to the present embodiment. Execute the process.

Finally, the third feature of the present embodiment will be described. For color images, the signal amount corresponding to the light transmitted through the cyan element and photoelectrically converted is Scy, the signal amount corresponding to the light transmitted through the yellow element and photoelectrically converted is Sye, and the photoelectric conversion signal in the pixels W1 and W2 is defined as the photoelectric conversion signal. Let Sw be the amount of signal obtained by adding. Then, the red color information Sr is obtained by (Sw-Scy), the blue color information Sb is obtained by (Sw-Sye), and the green color information is obtained by (Sw-Sr-Sb). As a result, the decrease in the amount of light is suppressed only by the amount of decrease due to the polarization regions P (1) and P (2) of the translucent plate 2, and a color image in which the decrease in the amount of incident light is significantly suppressed can be obtained.

As described above, according to the present embodiment, first, the directions of the transmission axes of the polarization regions P (1) and P (2) are orthogonal to each other, and φ1 is an arbitrary angle, and φ = φ1 and φ = φ1 +. Imaging is performed in two states of 180 °. Second, on the imaging surface of the solid-state image sensor 1, pixels are configured with 2 rows and 2 columns as the basic unit, and cyan elements (Cy) are arranged facing the pixels in the 1st row and 1st column, and the 2nd row and 2nd column. A yellow element (Ye) is placed facing the pixel, and a polarization filter whose transmission axis direction coincides with the X direction is placed facing the pixel in the 1st row and 2nd column (γ = 0), and 2nd row and 1st column. A polarizing filter is arranged so that the direction of the transmission axis faces the pixel of the eye at an angle of 45 ° (β = 45 °) with respect to the X direction. With the above configuration and the rotation operation by the rotation drive unit 2A, it is possible to obtain a multi-viewpoint image, a difference image, and a color image even if the subject has polarization characteristics. By making the sizes of the polarized light regions P (1) and P (2) sufficiently smaller than those of the transparent region P (3), there is an effect that a color image in which the decrease in sensitivity is significantly suppressed can be obtained.

In the above description, the parameters γ and β that define the direction of the transmission axis of the polarizing filter and the transmittances T1 and T2 in the polarizing region are limited, but these are not limited to the above values. In the present embodiment, the value of | D | in Equation 9 does not become 0, the directions of the transmission axes of the two polarization regions P (1) and P (2) are orthogonal to each other, and the translucent plate has an angle of φ1 and an angle of φ1. It suffices if it is configured to image in each state of + 180 °.

Further, the color filter in the present embodiment does not necessarily have to be a cyan element and a yellow element. As the two types of color filters, it is sufficient that a color filter that transmits the first color component and a color filter that transmits the second color component are arranged. For example, it is possible to adopt a configuration in which a red element and a blue element are used as a color filter and a red signal and a green signal can be directly obtained as pixel signals.

In the present embodiment, the angle α formed by the direction of the transmission axis of the polarization region P (2) with respect to the direction of the transmission axis of the polarization region P (1) is 90 °, but α must be exactly 90 °. However, some errors may be included. α is preferably set to satisfy 70 ° α 90 °, and more preferably set to satisfy 80 ° α 90 °. Further, the present invention is not limited to α = 90 °, and parallax information can be obtained based on Equations 6 and 7 even when α 90 °.

The pixels do not necessarily have to be arranged in a square grid, and the shape of each pixel does not have to be square. If one pixel block is composed of four pixels, polarizing filters with different transmission axis directions are arranged facing the two pixels, and different color filters are arranged facing the other two pixels. It is possible to obtain the effect of this embodiment.

Further, instead of the pixel configuration shown in FIG. 8, another pixel configuration may be used. For example, even if the pixel configuration shown in FIG. 4 or 7 is used, a multi-viewpoint image and a difference can be obtained for a subject having polarization characteristics by performing imaging in two states where the rotation angles of the light transmitting plates 2 differ by 180 °. You can get an image.

In the above embodiments 1 and 2, the image pickup apparatus may be configured to acquire either a plurality of viewpoint images or a difference image. For example, the image pickup apparatus may acquire only a plurality of viewpoint images, and another arithmetic processing unit connected to the image pickup apparatus by wire or wirelessly may obtain the difference image. Further, the imaging device may acquire only the difference image, and another device may acquire the plurality of viewpoint images.

Further, in the above-described first and second embodiments, a parallax image (disparity map) showing the magnitude of the displacement of the positions of the corresponding points on the image can be obtained from the plurality of viewpoint images. The depth information of the subject can be obtained from this parallax image.

The three-dimensional image pickup device according to the present invention is effective for all cameras using a solid-state image pickup device. For example, it is effective for consumer cameras such as digital still cameras and digital video cameras, and industrial solid-state surveillance cameras.

1 Solid-state image sensor 2 Translucent plate 2A rotary drive unit 3 Optical lens 4 Infrared cut filter 5 Signal generation and image signal receiver 6 element drive unit 7 Image processing unit 8 Image interface section 9 Imaging device 10 pixels 110 degree polarized polarizing plate 12 90 degree polarized polarizing plate 13 retroreflector 14 Half mirror 15 Circular polarizing filter 16 Drive device that rotates the polarizing filter 17, 18 polarizing filter 19 Light passage 20, 21 Polarized transmission 22 Receiver optical filter tray 23 Specific component transmission filter 24 color filters 25 Filter drive 50a, 50b polarizing filter