Image processing device, image processing method, and imaging device

This technique makes it possible to acquire an image in which the number of pixels is not reduced while generating a highly accurate depth map for an image processing device, an image processing method, and an imaging device.

In recent years, as the price of 3D printers has been reduced, there has been a demand for a means for easily acquiring a three-dimensional shape.

There are an active method and a passive method as means for acquiring the three-dimensional shape of the subject. The active method is, for example, a method of irradiating a subject with light and acquiring a three-dimensional shape based on the reflected light from the subject, and is not an easy method in terms of power consumption, component cost, and the like. In contrast to this active method, the passive method is a method of acquiring a three-dimensional shape without irradiating the subject with light, and is a simpler method than the active method. In the passive method, for example, a method of generating a depth map by finding the correspondence between images using a stereo camera, or a method of acquiring polarized images in a plurality of directions and generating a normal map are used.

In the passive method, it is known that the method using a stereo camera cannot acquire the depth of a flat portion of a subject. Further, it is known that a method using polarized images in a plurality of directions can acquire a relative surface shape of a subject but cannot acquire an absolute distance. Further, it is known that the method using polarized images in a plurality of directions has 180 degree indefiniteness in the azimuth angle of the normal of the subject. Therefore, in Patent Document 1, by arranging polarization filters having different polarization directions in each pixel of the image sensor mounted on each camera of the stereo camera, a depth map is acquired by the stereo camera and a normal map is obtained by polarization imaging. Is acquired at the same time. Further, in Patent Document 1, it is possible to solve the 180-degree indefiniteness of the normal map and obtain the absolute distance by referring to the depth map.

<p num="0005"><patcit num="1"><text>International Publication No. 2009/147814</text></patcit></p>

<p num="0006"> By the way, Patent Document 1 uses a configuration in which four pixels are used as an image unit and a polarizer having a different polarization direction is provided in each of the four pixels. Therefore, in such a configuration, the number of pixels of the polarized image in a specific polarization direction becomes (1/4) of the number of pixels of the original image sensor, and it is not possible to generate a highly accurate depth map. In addition, as the number of pixels decreases, the quality of a normal image also deteriorates.</p><p num="0007"> Therefore, an object of this technique is to provide an image processing device, an image processing method, and an imaging device capable of acquiring an image in which the number of pixels does not decrease while generating a high-precision depth map.</p>

<p num="0008"> The first aspect of this technique is a first image generated by a first imaging unit having a pixel configuration including pixels having different polarization directions, and a second imaging unit having a pixel configuration different from that of the first imaging unit. A depth map generation unit that generates a depth map by a matching process using the second image generated by the above, and at least one of the first image or the second image generated by the depth map generation unit. The integration process of the normal map generation unit that generates a normal map based on the polarization state of the polarized image, the depth map generated by the depth map generation unit, and the normal map generated by the normal map generation unit. It is in an image processing apparatus having a map integration unit to perform.</p><p num="0009"> In this technique, in this technique, a first image generated by a first imaging unit having a pixel configuration including pixels having different polarization directions and a second image generated by a second imaging unit having a pixel configuration different from that of the first imaging unit. A depth map is generated by matching processing using the image of. For example, the first image is an image generated by the first imaging unit including pixels having three or more polarization directions, and the second image is a second image composed of pixels having no polarization characteristics. In the case of an image generated by the unit, a non-polarized image is generated from the first image, and matching processing is performed using the edge-extracted images of the unpolarized image and the second image.</p><p num="0010"> The first image is composed of a first pixel group consisting of pixels having polarization characteristics and a second pixel group consisting of pixels in a polarization direction different from the first pixel group or pixels having no polarization characteristics. It is an image generated by the first imaging unit having a pixel configuration, and the second image is composed of pixels having a polarization direction different from that of the first image at a position corresponding to the first pixel group. A first pixel configuration composed of three pixel groups and a fourth pixel group composed of pixels in the polarization direction equal to the second pixel group at positions corresponding to the second pixel group or pixels having no polarization characteristics. In the case of the image generated by the image pickup unit 2, by using the image of the second pixel group in the first image and the image of the fourth pixel group in the second image, between images having the same polarization direction. Alternatively, matching processing is performed between images that do not have polarization characteristics.</p><p num="0011"> In addition, a normal map is generated based on the polarization state of a polarized image in which at least one of the first or second images has three or more polarization directions. For example, when the first image is an image generated by the first imaging unit including pixels having three or more polarization directions, a normal map is generated based on the first image.</p><p num="0012"> The first image is composed of a first pixel group consisting of pixels having polarization characteristics and a second pixel group consisting of pixels in a polarization direction different from the first pixel group or pixels having no polarization characteristics. It is an image generated by the first imaging unit having a pixel configuration, and the second image is a third pixel composed of pixels whose polarization direction is different from that of the first image at a position corresponding to the first pixel group. This is an image generated by the second imaging unit having a pixel configuration composed of a group and a fourth pixel group composed of pixels having the same configuration as the second pixel group at a position corresponding to the second pixel group. In the case, based on the amount of discrepancy between the first image and the second image, the phase of the image of the first pixel group in the first image and the image of the third pixel group in the second image are matched and polarized. A polarized image having a plurality of directions is generated, and a normal map is generated based on the polarization state of the polarized image.</p><p num="0013"> Further, when the second pixel group and the fourth pixel group are images having no polarization characteristic, the image of the second pixel group is generated by the interpolation process using the image of the first pixel group, and the third pixel group is generated. The image of the fourth pixel group is generated by the interpolation process using the image of the pixel group of, and the polarized image is generated using the image after the interpolation.</p><p num="0014"> Furthermore, the generated depth map and normal map are integrated, and the depth value not shown in the depth map is calculated from the depth value shown in the depth map and the shape determined based on the normal map. Then, a depth map with an accuracy higher than the generated depth map is generated.</p><p num="0015"> Further, the first image pickup unit that generates the first image and the second image pickup unit that generates the second image may be provided in the image processing device, and the first image pickup unit and the second image pickup unit may be provided. One of the image pickup units may be provided in the external device so that the image processing device acquires the image generated by the image pickup unit provided in the external device by communication.</p><p num="0016"> The second aspect of this technique is a depth map generator, which comprises a first image generated by a first image pickup unit having a pixel configuration including pixels having different polarization directions, and the first image pickup unit and the pixel configuration. A step of performing a matching process using a second image generated by a different second imaging unit to generate a depth map, and a normal map generation unit at least of the generated first or second image. The image processing method includes a step of generating a normal map based on the polarization state of any of the polarized images, and a step of integrating the depth map and the normal map in the map integration unit.</p><p num="0017"> The third aspect of this technique is a first imaging unit having a pixel configuration including pixels having different polarization directions, a second imaging unit having a pixel configuration different from that of the first imaging unit, and the first imaging unit. The imaging apparatus includes an image processing unit that performs image processing using the first image generated by the above and the second image generated by the second imaging unit.</p><p num="0018"> A fourth aspect of this technique is an image process including a first image pickup unit having a pixel configuration including pixels having different polarization directions and a transmission unit that transmits a first image generated by the first image pickup unit. A receiving unit that receives the first image from the apparatus, a second imaging unit having a pixel configuration different from that of the first imaging unit, a first image received by the receiving unit, and the second image unit. The image processing apparatus includes an image processing unit that performs image processing using the second image generated by the imaging unit.</p>

<p num="0019"> According to this technique, a first image generated by a first imaging unit having a pixel configuration including pixels having different polarization directions and a second imaging unit having a pixel configuration different from that of the first imaging unit are generated. A depth map is generated by the matching process using the second image. Also, a normal map is generated based on the polarization state of at least one of the polarized images of the first or second image. Further, the generated depth map and the normal map are integrated. Therefore, it is possible to acquire an image in which the number of pixels does not decrease while generating a high-precision depth map. It should be noted that the effects described in the present specification are merely exemplary and not limited, and may have additional effects.</p>

<figref num="1">It is a block diagram which shows the functional structure of an image processing apparatus.</figref><figref num="2">It is a figure which illustrated the structure of the 1st Embodiment.</figref><figref num="3">It is a figure which illustrated the pixel composition of the image sensor which constitutes an image pickup part.</figref><figref num="4">It is a figure for demonstrating the calculation of the distance to a subject.</figref><figref num="5">It is a figure for demonstrating the generation operation of a polarized image.</figref><figref num="6">It is a figure which illustrated the relationship between the brightness and the polarization angle.</figref><figref num="7">It is a figure which illustrated the relationship between the degree of polarization and the zenith angle.</figref><figref num="8">It is a flowchart which shows the processing operation of 1st Embodiment.</figref><figref num="9">It is a flowchart which shows the processing operation of the depth map generation part.</figref><figref num="10">It is a figure for demonstrating the integration process of a map.</figref><figref num="11">It is a figure which illustrated the pixel structure of the image sensor which constitutes the image pickup part in the 1st modification.</figref><figref num="12">It is a flowchart which shows the processing operation of the depth map generation part in the 1st modification.</figref><figref num="13">It is a figure which illustrated the structure of the 2nd modification.</figref><figref num="14">It is a figure for demonstrating the operation of the normal map generation processing part.</figref><figref num="15">It is a flowchart which shows the processing operation of the 2nd modification.</figref><figref num="16">It is a figure which illustrated the structure of the 2nd Embodiment.</figref><figref num="17">It is a figure which illustrated the pixel composition of the image sensor which constitutes an image pickup part.</figref><figref num="18">It is a figure which showed the image supplied from the imaging unit and the image after the interpolation process.</figref><figref num="19">It is a figure for demonstrating the phase adjustment process.</figref><figref num="20">It is a flowchart which shows the processing operation of the 2nd Embodiment.</figref><figref num="21">It is a flowchart which shows the generation process of a depth map.</figref><figref num="22">It is a flowchart which shows the generation process of a multi-polarized image.</figref><figref num="23">It is a figure which illustrated the pixel structure of the image sensor which constitutes the image pickup part in the 1st modification.</figref><figref num="24">It is a figure which illustrated the pixel composition of the image sensor which constitutes the image pickup part in the 2nd modification.</figref><figref num="25">It is a figure which illustrated the appearance of the 3rd Embodiment.</figref><figref num="26">It is a figure which illustrated the structure of the 3rd Embodiment.</figref>

Hereinafter, modes for implementing the present technology will be described. The explanation will be given in the following order. 1. About image processing equipment 2. First embodiment 2-1. Configuration and operation of the first embodiment 2-2. First variant of the first embodiment 2-3. Second variant of the first embodiment 3. Second embodiment 3-1. Configuration and operation of the second embodiment 3-2. First modification of the second embodiment 3-3. Second modification of the second embodiment 4. Third embodiment

&lt;1. About image processing equipment&gt; FIG. 1 is a block diagram showing a functional configuration of an image processing device of the present technology. The image processing device 10 has a depth map generation unit 15, a normal map generation unit 17, and a map integration unit 19.

The depth map generation unit 15 generates a depth map from a multi-viewpoint image. The multi-viewpoint image is a non-polarized image based on the first image generated by the first imaging unit having a pixel configuration including pixels having different polarization characteristics, and a second imaging unit having a pixel configuration different from that of the first imaging unit. Use an unpolarized image based on the second image generated by. Further, the depth map generation unit 15 may use a polarized image based on the first image and a polarized image based on the second image having the same polarization direction as the polarized image. The depth map generation unit 15 performs matching processing using, for example, an unpolarized image of the right viewpoint and an unpolarized image of the left viewpoint, or a polarized image of the right viewpoint and a polarized image of the left viewpoint having the same polarization direction, and performs matching processing for each pixel. Generate a depth map that stores the depth value. The depth map generation unit 15 outputs the generated depth map to the map integration unit 19.

The normal map generation unit 17 generates a normal map from polarized images in a plurality of directions. A polarized image in a plurality of directions is a polarized image having three or more polarized directions as described later. The normal map generation unit 17 uses a first image composed of pixels having three or more polarization directions generated by the first imaging unit having a pixel configuration including pixels having different polarization characteristics. Further, the normal map generation unit 17 includes a first image including pixels having a plurality of polarization directions generated by a first image pickup unit having a pixel configuration including pixels having different polarization characteristics, and a first image pickup unit and pixels. A second image containing a plurality of pixels whose polarization direction is different from that of the first image is used, which is generated by the second imaging unit having a different configuration. The normal map generation unit 17 generates a normal map in which normal information is stored for each pixel by using a polarized image having three or more polarization directions. The normal information of the normal map is information that can acquire the surface shape of the subject by integrating the normal information, and the surface shape of the subject is a relative value and does not include information on the distance to the subject. .. The normal map generation unit 17 outputs the generated normal map to the map integration unit 19. Further, the normal map generation unit 17 may generate a normal map in which the indefiniteness of 180 degrees is solved by using the depth map generated by the depth map generation unit 15.

The map integration unit 19 integrates the depth map generated by the depth map generation unit 15 and the normal map generated by the normal map generation unit 17, and is equal to or greater than the depth map generated by the depth map generation unit 15. Generate a depth map with accuracy. For example, when the depth value cannot be acquired in the depth map, the map integration unit 19 determines the surface shape of the subject corresponding to the depth unacquired region by using the normal map. The map integration unit 19 estimates the depth value of the unacquired depth region based on the determined surface shape and the acquired depth value, so that the depth map has an accuracy higher than that of the depth map generated by the depth map generation unit 15. To generate.

&lt;2. First Embodiment&gt; Next, the first embodiment of the image processing apparatus will be described. In the first embodiment, the first imaging unit has a pixel configuration including pixels having different polarization characteristics, and the second imaging unit having a pixel configuration different from that of the first imaging unit does not have the polarization characteristics. The case where it is composed of is described.

&lt;2-1. Configuration and operation of the first embodiment&gt; FIG. 2 illustrates the configuration of the first embodiment. The image processing device 20 includes imaging units 21, 22, a depth map generation unit 25, a normal map generation unit 27, and a map integration unit 29. The image pickup units 21 and 22 correspond to a stereo camera, and may be provided separately from the image processing device 20.

The imaging unit 21 corresponds to a first imaging unit having a pixel configuration including pixels having different polarization characteristics. Further, the image pickup unit 22 corresponds to a second image pickup unit composed of pixels having no polarization characteristic.

FIG. 3 illustrates the pixel configuration of the image sensor that constitutes the image pickup unit. Note that FIG. 3 shows a part of the image sensor. Further, FIG. 3A shows the pixel configuration of the image sensor 210 constituting the image pickup unit 21, and FIG. 3B shows the pixel configuration of the image sensor 220 constituting the image pickup unit 22.

The image sensor 210 of the image pickup unit 21 has a configuration in which a polarizing filter is arranged for each pixel. For example, as shown in FIG. 3A, the polarizing filter of the image sensor 210 has four polarization directions (the polarization directions are indicated by arrows), and the image pickup unit 21 displays polarized images in four directions. can get. The imaging unit 21 outputs the generated polarized image to the depth map generation unit 25 and the normal map generation unit 27.

The image sensor 220 of the image pickup unit 22 is composed of pixels of a single color (for example, white) in which a polarizing filter is not arranged. For example, as shown in FIG. 3B, the image sensor 220 is not provided with a polarizing filter, and the imaging unit 22 can obtain an unpolarized image. The imaging unit 22 outputs the generated unpolarized image to the depth map generation unit 25.

The depth map generation unit 25 has a preprocessing unit 251 and a depth map generation processing unit 255.

The preprocessing unit 251 generates a matching image to be used for the matching process from each of the polarized image supplied from the imaging unit 21 and the unpolarized image supplied from the imaging unit 22. As described above, since the image supplied from the imaging unit 21 is a polarized image transmitted through the polarizing filter, it is compared with the unpolarized image generated by the imaging unit 22 using the image sensor in which the polarizing filter is not arranged. The brightness is decreasing. Therefore, the preprocessing unit 251 generates a matching image so that the matching process corresponding to the difference in the brightness level can be performed. The preprocessing unit 251 filters the polarized image supplied from the imaging unit 21 to generate an unpolarized image. The preprocessing unit 251 can generate pixel values of an unpolarized image by, for example, performing an averaging filter processing of 2 pixels × 2 pixels and calculating the average value of the pixel values in the polarization directions in the four directions.

Next, the preprocessing unit 251 performs edge extraction processing on the unpolarized image obtained by filtering the polarized image supplied from the imaging unit 21 and the unpolarized image supplied from the imaging unit 22. To generate each edge extraction image. The preprocessing unit 251 outputs each generated edge extraction image as a matching image to the depth map generation processing unit 255. As described above, since the preprocessing unit 251 uses the edge extraction image as the matching image, the depth map generation processing unit 255 can perform the matching processing without being affected by the difference in the brightness level.

The depth map generation processing unit 255 performs matching processing using the matching image and generates a depth map. As the matching method, any method such as area-based matching or feature-based matching template matching may be used. The depth map generation processing unit 255 executes matching processing and calculates the distance to the subject at each pixel position (hereinafter referred to as depth value) based on the amount of deviation of the corresponding pixel position. FIG. 4 is a diagram for explaining the calculation of the distance to the subject. Note that FIG. 4 illustrates a case where the imaging unit 21 and the imaging unit 22 are arranged on the left and right in the same posture. Here, the left imaging unit is used as a reference imaging unit, and the right imaging unit is used as a reference imaging unit. Further, the distance (base length) between the reference positions of the imaging unit is "LB", and the focal length of the imaging unit is "f". In this case, the position X of the subject in the reference imaging unit_LOn the other hand, the position X of the subject in the reference imaging unit_RIf is deviated by "Ld", the distance "Zp" to the subject can be calculated based on Eq. (1).

<maths num="1"><img id="000003" he="24" wi="150" file="JP2015114307A_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The depth map generation processing unit 255 generates a depth map by associating the calculated distance (depth value) with the pixels of the captured image. The depth map generation processing unit 255 outputs the generated depth map to the map integration unit 29.

The normal map generation unit 27 has a normal map generation processing unit 275. The normal map generation processing unit 275 generates a normal map based on a plurality of polarized images in the polarization directions supplied from the imaging unit 21. FIG. 5 is a diagram for explaining the operation of generating a polarized image. As shown in FIG. 5, the subject OB is illuminated using the light source LT, and the subject OB is imaged by the imaging unit CM via the polarizing plate PL. In this case, it is known that the brightness of the subject OB of the polarized image generated by the imaging unit CM changes according to the rotation of the polarizing plate PL. Here, the highest brightness when the polarizing plate PL is rotated is Imax, and the lowest brightness is Imin. Further, when the x-axis and the y-axis in the two-dimensional coordinates are the plane directions of the polarizing plate PL, the angle on the xy plane with respect to the x-axis when the polarizing plate PL is rotated is defined as the polarization angle υ. When the polarizing plate PL is rotated 180 degrees, it returns to the original polarized state and has a period of 180 degrees. In the case of the diffuse reflection model, the polarization angle υ when the maximum brightness Imax is observed is defined as the azimuth angle φ. With such a definition, the brightness I observed when the polarizing plate PL is rotated can be expressed by Eq. (2). Note that FIG. 6 illustrates the relationship between the brightness and the polarization angle.

<maths num="2"><img id="000004" he="23" wi="170" file="JP2015114307A_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In Eq. (2), the polarization angle υ is clear when the polarized image is generated, and the maximum brightness Imax, the minimum brightness Imin, and the azimuth angle φ are variables. Therefore, since the normal map generation processing unit 275 has three variables, the normal map generation processing unit 275 performs fitting to the function shown in Eq. (2) using the brightness of the polarized image having three or more polarization directions, and the brightness and polarization. The azimuth angle φ, which is the maximum brightness, is determined based on the function indicating the relationship between the angles.

Further, the object surface normal is expressed in a polar coordinate system, and the normal information is defined as an azimuth angle φ and a zenith angle θ. The zenith angle θ is the angle from the z-axis to the normal, and the azimuth φ is the angle in the y-axis direction with respect to the x-axis as described above. Here, the degree of polarization ρ can be calculated by performing the calculation of Eq. (3) also using the minimum brightness Imin and the maximum brightness Imax obtained by rotating the polarizing plate PL.

<maths num="3"><img id="000005" he="24" wi="150" file="JP2015114307A_D0001.tif" img-format="tif" img-content="drawing" /></maths>

It is known from Fresnel's equation that the relationship between the degree of polarization and the zenith angle has the characteristics shown in FIG. 7, for example, and the zenith angle θ can be discriminated from the characteristics shown in FIG. 7 based on the degree of polarization ρ. The characteristics shown in FIG. 7 are examples, and the characteristics change depending on the refractive index of the subject.

Therefore, the normal map generation processing unit 275 obtains the relationship between the brightness and the polarization angle from the polarization direction and the brightness of the polarized image based on the polarized image having three or more polarization directions, and determines the azimuth angle φ which is the maximum brightness. To do. Further, the normal map generation processing unit 275 calculates the degree of polarization ρ using the maximum brightness and the minimum brightness obtained from the relationship between the brightness and the polarization angle, and based on the characteristic curve showing the relationship between the degree of polarization and the zenith angle. The zenith angle θ corresponding to the calculated degree of polarization ρ is determined. In this way, the normal map generation processing unit 275 obtains the normal information (azimuth φ and zenith angle θ) of the subject for each pixel position based on the polarized image having three or more polarization directions, and obtains the normal map. Generate. The normal map generation processing unit 275 outputs the generated normal map to the map integration unit 29.

The map integration unit 29 integrates the depth map and the normal map. Based on the subject surface shape shown in the normal map and the depth value shown in the depth map, the map integration unit 29 traces the subject surface shape starting from the pixel from which the depth value is obtained, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. Further, the map integration unit 29 includes the estimated depth value in the depth map supplied from the depth map generation unit 25 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 25. ..

FIG. 8 is a flowchart showing the processing operation of the first embodiment. In step ST1, the imaging unit 21 generates the first image. The imaging unit 21 generates polarized images in a plurality of polarization directions as a first image. Further, in step ST2, the imaging unit 22 generates a second image. The imaging unit 22 generates an unpolarized image as a second image.

In step ST3, the depth map generator 25 generates a depth map. FIG. 9 is a flowchart showing the processing operation of the depth map generation unit.

In step ST11, the depth map generator 25 generates an unpolarized image. The preprocessing unit 251 of the depth map generation unit 25 performs averaging filter processing or the like on the first image, that is, the polarized image in a plurality of polarization directions, and generates a non-polarized image.

In step ST13, the depth map generation unit 25 performs edge extraction processing. The preprocessing unit 251 of the depth map generation unit 25 performs edge extraction processing on the unpolarized image generated by the imaging unit 22 which is the second image and the unpolarized image generated in step ST11 to obtain a matching image. Generate. By performing the edge extraction process in this way, the depth map generation unit 25 can generate a matching image that is not affected by the difference in the brightness levels of the first image and the second image.

In step ST14, the depth map generator 25 performs the matching process. The depth map generation processing unit 255 of the depth map generation unit 25 performs matching processing using the matching image generated from the first image and the matching image generated from the second image. Further, the depth map generation unit 25 generates a depth map showing the depth value for each pixel based on the matching processing result.

In step ST4 of FIG. 8, the normal map generation unit 27 generates a normal map. The normal map generation unit 27 uses the first image to discriminate between the azimuth angle φ and the zenith angle θ for each pixel to generate a normal map.

In step ST5, the map integration unit 29 performs the map integration process. Based on the depth value shown in the depth map and the subject surface shape shown in the normal map, the map integration unit 29 traces the subject surface shape starting from the pixel from which the depth value is obtained, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. In addition, the map integration unit 29 includes the estimated depth value in the depth map.

FIG. 10 is a diagram for explaining the map integration process. For the sake of simplicity, for example, an integrated process for one line will be described. As shown in FIG. 10 (A), the subject OB is imaged by the imaging units 21 and 22, and the depth map generation unit 25 shows the depth map and the normal map generation unit 27 in FIG. 10 (B). It is assumed that the normal map shown in (C) of (C) is obtained. Further, in the depth map, for example, it is assumed that the depth value for the leftmost pixel is "2 (meters)", and the depth value is not stored in the other pixels indicated by "x". The map integration unit 29 estimates the surface shape of the subject OB based on the normal map. Here, it can be determined that the second pixel from the left end corresponds to an inclined surface approaching the directions of the imaging units 21 and 22 from the subject surface corresponding to the leftmost pixel based on the normal direction of this pixel. Therefore, the map integration unit 29 estimates the depth value of the second pixel from the left end by tracing the surface shape of the subject OB starting from the leftmost pixel, and sets it to, for example, "1.5 (meters)". In addition, the map integration unit 29 stores the estimated depth value in the depth map. It can be determined that the third pixel from the left end corresponds to the surface facing the imaging units 21 and 22 based on the normal direction of this pixel. Therefore, the map integration unit 29 estimates the depth value of the third pixel from the left end by tracing the surface shape of the subject OB starting from the leftmost pixel, and sets it as, for example, "1 (meter)". In addition, the map integration unit 29 stores the estimated depth value in the depth map. It can be determined that the fourth pixel from the left end corresponds to the inclined surface in the direction away from the subject surface corresponding to the third pixel from the left end with the imaging units 21 and 22. Therefore, the map integration unit 29 estimates the depth value of the fourth pixel from the left end by tracing the surface shape of the subject OB starting from the leftmost pixel, and for example, "1. 5 (meters) ". In addition, the map integration unit 29 stores the estimated depth value in the depth map. Similarly, the depth value of the fifth pixel from the left end is estimated and stored in the depth map as, for example, "2 (meters)".

In this way, the map integration unit 29 integrates the depth map and the normal map, and estimates the depth value by tracing the surface shape based on the normal map starting from the depth value of the depth map. Therefore, the map integration unit 29 can make up for the missing depth value even if a part of the depth value is missing in the depth map shown in FIG. 10 (B) generated by the depth map generation unit 25. It will be possible. Therefore, it is possible to generate the depth map shown in FIG. 10 (D), which is more accurate than the depth map shown in FIG. 10 (B).

As described above, according to the first embodiment, even for a subject region where it is difficult to obtain a depth value by the matching process, the depth is obtained by using a normal map generated based on polarized images in a plurality of polarization directions. The value can be estimated. Therefore, it is possible to generate a high-precision depth map having a accuracy higher than that of the depth map generated by the depth map generation unit 25, that is, a depth map in which the depth value is stored for each pixel of the subject area. Further, since a high-precision depth map can be generated without performing processing with four pixels as an image unit, it is possible to acquire an image in which the number of pixels does not decrease while generating a high-precision depth map.

&lt;2-2. First modification of the first embodiment&gt; In the above-described embodiment, the configuration in which the image sensor 22 uses an image sensor composed of pixels of a single color is illustrated, but an image sensor composed of pixels of a plurality of colors may be used. Next, as a first modification of the first embodiment, a case where an image sensor in which red, blue, and green pixels are arranged in a Bayer array is used in the imaging unit 22 will be described. The configuration of the image processing device is the same as that shown in FIG.

FIG. 11 illustrates the pixel configuration of the image sensor constituting the imaging unit in the first modification. Note that FIG. 11 shows a part of the image sensor. Further, FIG. 11A shows the pixel configuration of the image sensor 210 constituting the image pickup unit 21, and FIG. 11B shows the pixel configuration of the image sensor 221 constituting the image pickup unit 22. In addition, "R" indicates a red pixel, "G" indicates a green pixel, and "B" indicates a blue pixel.

The image sensor 210 of the image pickup unit 21 has a configuration in which a polarizing filter is arranged for each pixel. For example, as shown in FIG. 3A, the polarizing filter of the image sensor 210 has four polarization directions (the polarization directions are indicated by arrows), and the image pickup unit 21 displays polarized images in four directions. can get. The imaging unit 21 outputs the generated polarized image to the depth map generation unit 25 and the normal map generation unit 27.

As shown in (B) of FIG. 11, for example, the image sensor 221 of the image pickup unit 22 has a configuration in which no polarizing filter is arranged and the pixels (R, G, B) of the three primary colors are arranged in a Bayer arrangement. The image pickup unit 22 can obtain an unpolarized image. The imaging unit 22 outputs the generated unpolarized image to the depth map generation unit 25.

The depth map generation unit 25 has a preprocessing unit 251 and a depth map generation processing unit 255.

The preprocessing unit 251 generates a matching image to be used for the matching process from each of the polarized image supplied from the imaging unit 21 and the unpolarized image supplied from the imaging unit 22. As described above, since the image supplied from the imaging unit 21 is a polarized image transmitted through the polarizing filter, it is compared with the unpolarized image generated by the imaging unit 22 using the image sensor in which the polarizing filter is not arranged. The brightness is decreasing. Therefore, the preprocessing unit 251 generates a matching image so that the matching process corresponding to the difference in the brightness level can be performed. The preprocessing unit 251 filters the polarized image supplied from the imaging unit 21 to generate an unpolarized image. The preprocessing unit 251 can generate pixel values of an unpolarized image by, for example, performing an averaging filter processing of 2 pixels × 2 pixels and calculating the average value of the pixel values in the polarization directions in the four directions.

Next, since the image sensor 221 used in the imaging unit 22 has a configuration in which the pixels of the three primary colors are arranged in a Bayer array, the preprocessing unit 251 performs demosaic processing to generate a luminance image. The preprocessing unit 251 refers to the unpolarized image after filtering the polarized image supplied from the imaging unit 21 and the luminance image obtained by performing demosaic processing on the unpolarized image supplied from the imaging unit 22. Edge extraction processing is performed to generate each edge extraction image. The preprocessing unit 251 outputs each generated edge extraction image as a matching image to the depth map generation processing unit 255. As described above, since the preprocessing unit 251 uses the edge extraction image as the matching image, the depth map generation processing unit 255 can perform the matching processing without being affected by the difference in the brightness level.

The depth map generation processing unit 255 generates a depth map by performing matching processing using the matching image as described above. Further, the normal map generation unit 27 generates a normal map based on the polarized images in the plurality of polarization directions supplied from the imaging unit 21 as described above.

The map integration unit 29 integrates the depth map and the normal map. Based on the subject surface shape shown in the normal map and the depth value shown in the depth map, the map integration unit 29 traces the subject surface shape starting from the pixel from which the depth value is obtained, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. Further, the map integration unit 29 includes the estimated depth value in the depth map supplied from the depth map generation unit 25 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 25. ..

FIG. 12 is a flowchart showing the processing operation of the depth map generation unit in the first modification. In step ST21, the depth map generator 25 generates an unpolarized image. The preprocessing unit 251 of the depth map generation unit 25 performs averaging filter processing or the like on the first image, that is, the polarized image in a plurality of polarization directions, and generates a non-polarized image.

In step ST22, the depth map generator 25 generates a luminance image. The preprocessing unit 251 of the depth map generation unit 25 performs demosaic processing on the unpolarized three primary color images generated by the imaging unit 22 which is the second image to generate a luminance image.

In step ST23, the depth map generation unit 25 performs edge extraction processing. The preprocessing unit 251 of the depth map generation unit 25 performs edge extraction processing on the unpolarized image generated in step ST21 and the luminance image generated in step ST22 to generate a matching image. By performing the edge extraction process in this way, the depth map generation unit 25 can generate a matching image that is not affected by the brightness difference between the first image and the second image.

In step ST24, the depth map generator 25 performs the matching process. The depth map generation processing unit 255 of the depth map generation unit 25 performs matching processing using the matching image generated from the first image and the matching image generated from the second image. Further, the depth map generation unit 25 generates a depth map showing the depth value based on the matching processing result.

By performing such processing, a highly accurate depth map can be generated even when the image sensor 22 uses an image sensor having a Bayer array of pixels (R, G, B) of the three primary colors.

&lt;2-3. Second modification of the first embodiment&gt; The normal map generation unit 27 in the above-described embodiment generates a normal map using polarized images in a plurality of polarization directions. As described above, it is known that when the polarizing plate PL is rotated 180 degrees, it returns to the original polarized state, and the luminance change has a period of 180 degrees, and has a so-called 180 degree indefiniteness. Therefore, in the second modification, a case of removing the indefiniteness of 180 degrees by using a depth map will be described.

FIG. 13 illustrates the configuration of the second modification. The image processing device 20 includes imaging units 21, 22, a depth map generation unit 25, a normal map generation unit 27, and a map integration unit 29. The image pickup units 21 and 22 correspond to a stereo camera, and may be provided separately from the image processing device 20.

The imaging unit 21 corresponds to a first imaging unit having a pixel configuration including pixels having different polarization characteristics. Further, the image pickup unit 22 corresponds to a second image pickup unit composed of pixels having no polarization characteristic. The image pickup unit 21 uses, for example, an image sensor having the configuration shown in FIG. 3A or FIG. 11A. The image pickup unit 22 uses, for example, an image sensor having the configuration shown in FIG. 3 (B) or FIG. 11 (B).

The depth map generation unit 25 has a preprocessing unit 251 and a depth map generation processing unit 255. The preprocessing unit 251 generates a matching image used for the matching process as described above from each of the polarized image supplied from the imaging unit 21 and the unpolarized image supplied from the imaging unit 22. The depth map generation processing unit 255 performs matching processing using the matching image and generates a depth map. The depth map generation processing unit 255 outputs the generated depth map to the normal map generation unit 27 and the map integration unit 29.

The normal map generation unit 27 has a normal map generation processing unit 276. The normal map generation processing unit 276 performs the same processing as the normal map generation processing unit 275, and generates a normal map based on a plurality of polarized images in the polarization directions supplied from the imaging unit 21. In addition, the normal map generation processing unit 276 determines the gradient direction of the subject based on the depth map, generates a normal map, and removes the indefiniteness of 180 degrees.

FIG. 14 is a diagram for explaining the operation of the normal map generation processing unit. When the subject OB shown in FIG. 14A is imaged by the imaging unit 21 to generate a normal map, the brightness change according to the rotation in the polarization direction has a period of 180 degrees. Therefore, for example, as shown in FIG. 14 (B), the normal direction (indicated by an arrow) may be the correct direction in the upper half area GA of the subject OB, and the normal direction may be opposite in the lower half area GB. There is. Here, when the normal map generation processing unit 276 determines the gradient direction of the subject OB based on the depth map, the normal map generation processing unit 276 has a shape in which the subject OB protrudes in the direction of the imaging unit 21. Can be determined. Further, since the normal map generation processing unit 276 has a shape in which the subject OB protrudes in the direction of the imaging unit 21, the normal direction of the lower half region GB shown in FIG. 14 (B) is opposite. You can tell that. Therefore, the normal map generation processing unit 276 reverses the normal direction of the lower half area GB to remove the 180-degree indefiniteness as shown in FIG. 14 (C). You can generate a map.

The map integration unit 29 integrates the depth map and the normal map. Based on the subject surface shape shown in the normal map and the depth value shown in the depth map, the map integration unit 29 traces the subject surface shape starting from the pixel from which the depth value is obtained, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. Further, the map integration unit 29 includes the estimated depth value in the depth map supplied from the depth map generation unit 25 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 25. ..

FIG. 15 is a flowchart showing the processing operation of the second modification. In step ST31, the imaging unit 21 generates the first image. The imaging unit 21 generates polarized images in a plurality of polarization directions as a first image. Further, in step ST32, the imaging unit 22 generates a second image. The imaging unit 22 generates an unpolarized image as a second image.

In step ST33, the depth map generator 25 generates a depth map. The depth map generation unit 25 generates an unpolarized image and a matching image based on the unpolarized image. Further, the depth map generation unit 25 performs a matching process using the matching image, and generates a depth map showing the depth value based on the matching process result.

In step ST34, the normal map generation unit 27 generates a normal map. The normal map generation unit 27 generates a normal map with 180 degree indefiniteness removed based on the first image and the depth map.

In step ST35, the map integration unit 29 performs the map integration process. Based on the depth value shown in the depth map and the subject surface shape shown in the normal map, the map integration unit 29 traces the subject surface shape starting from the pixel from which the depth value is obtained to obtain the depth value. Estimate the depth value corresponding to the unobtained pixels. In addition, the map integration unit 29 includes the estimated depth value in the depth map.

In this way, according to the second modification, it is possible to remove the indefiniteness of 180 degrees and generate a correct normal map, and it is possible to correctly generate a high-precision depth map. In the second modification, the normal map generation processing unit 276 generated the correct normal map and output it to the map integration unit 29, but the map integration unit 29 is now used to remove 180-degree indeterminacy. You may. For example, the map integration unit 29 determines the gradient direction of the subject based on the depth map, and corrects the normal direction of the normal map generated by the above-mentioned normal map generation processing unit 275 to the correct direction based on the determination result. After that, the map integration process may be performed.

&lt;3. About the second embodiment&gt; Next, a second embodiment of the image processing apparatus will be described. In the second embodiment, the first image is composed of a first pixel group consisting of pixels having polarization characteristics, and pixels in a polarization direction different from that of the first pixel group or pixels having no polarization characteristics. It is assumed that the image is generated by the first imaging unit having a pixel configuration composed of two pixel groups. The second image consists of a third pixel group consisting of pixels whose polarization direction is different from that of the first image at a position corresponding to the first pixel group, and a second pixel group at a position corresponding to the second pixel group. It is assumed that the image is generated by the second imaging unit having a pixel configuration composed of a fourth pixel group composed of pixels having the same configuration as. Further, in the second embodiment, by using the image of the second pixel group in the first image and the image of the fourth pixel group in the second image, the polarization direction is the same between the images or the polarization characteristics. Matching processing is performed between images that do not have, and a depth map is generated. Further, when the second pixel group and the fourth pixel group are images having no polarization characteristics, the polarization directions of the first pixel group and the third pixel group are combined as a polarized image having three or more directions. Generate a line map.

&lt;3-1. Configuration and operation of the second embodiment&gt; FIG. 16 illustrates the configuration of the second embodiment. The image processing device 30 includes imaging units 31, 32, depth map generation unit 35, normal map generation unit 37, and map integration unit 39. The image pickup units 31 and 32 correspond to a stereo camera and may be provided separately from the image processing device 30.

The image pickup unit 31 is composed of a first pixel group composed of pixels having polarization characteristics and a second pixel group composed of pixels in a polarization direction different from the first pixel group or pixels having no polarization characteristics. It corresponds to the first imaging unit having a pixel configuration. Further, the imaging unit 32 has a third pixel group composed of pixels whose polarization direction is different from that of the first image at a position corresponding to the first pixel group, and a second pixel group at a position corresponding to the second pixel group. It corresponds to a second image pickup unit having a pixel configuration composed of a fourth pixel group composed of pixels having the same configuration as the pixel group.

FIG. 17 illustrates the pixel configuration of the image sensor that constitutes the image pickup unit. Note that FIG. 17 shows a part of the image sensor. Further, FIG. 17A shows the pixel configuration of the image sensor 310 constituting the image pickup unit 31, and FIG. 17B shows the pixel configuration of the image sensor 320 constituting the image pickup unit 32.

The image sensor 310 of the imaging unit 31 has a configuration in which unpolarized pixels and pixels provided with a plurality of polarizing filters in the polarization directions are mixed. For example, as shown in FIG. 17A, the image sensor 310 is provided with unpolarized pixel lines every other line. Further, in the line provided with the polarizing filter, pixels of two different types of polarization directions (the polarization directions are indicated by arrows) are alternately provided. Therefore, the image pickup unit 31 can obtain an image including pixels in the polarization direction in the two directions, which is the first pixel group, and unpolarized pixels, which is the second pixel group. The imaging unit 31 outputs the generated image to the depth map generation unit 35 and the normal map generation unit 37.

The image sensor 320 of the image pickup unit 32 has a configuration in which unpolarized pixels and pixels provided with a plurality of polarization filters in the polarization directions are mixed. For example, as shown in FIG. 17B, the image sensor 320 is provided with an unpolarized pixel line like the image sensor 310. Further, in the line provided with the polarizing filter, pixels of two types of polarization directions (indicated by arrows) different from the image sensor 310 of the imaging unit 31 are alternately provided. Therefore, the image pickup unit 32 can obtain an image including pixels of the third pixel group and unpolarized pixels which are the fourth pixel group, which are polarization directions in two directions different from those of the image pickup unit 31. The imaging unit 32 outputs the generated image to the depth map generation unit 35 and the normal map generation unit 37. That is, in the case of FIG. 17, a polarized image having four polarization directions is output to the normal map generation unit 37.

The depth map generation unit 35 has a preprocessing unit 351 and a depth map generation processing unit 355.

The preprocessing unit 351 extracts an image of the non-polarized portion from the images supplied from the imaging units 31 and 32, and outputs the matching image to be used for the matching process to the depth map generation processing unit 355.

The depth map generation processing unit 355 performs matching processing using the matching image and generates a depth map. The depth map generation processing unit 355 executes matching processing in the same manner as the depth map generation processing unit 255 described above, and calculates the distance (depth value) to the subject at each pixel position based on the amount of deviation of the corresponding pixel position. Further, the depth map generation processing unit 355 performs depth interpolation processing using the depth values calculated for the pixels in the unpolarized portion, and calculates the depth values of the pixels provided with the polarizing filter. The depth map generation processing unit 355 generates a depth map by associating the depth value with the pixels of the captured image. The depth map generation processing unit 355 generates a depth map by associating the calculated depth value with the pixels of the captured image. The depth map generation processing unit 355 outputs the generated depth map to the normal map generation unit 37 and the map integration unit 39.

The normal map generation unit 37 has an image phase adjustment unit 371 and a normal map generation processing unit 375.

The image phase adjustment unit 371 converts the depth map output from the depth map generation unit 35 into a disparity map. The image phase adjustment unit 371 determines the amount of parallax based on the depth value for each pixel shown in the depth map, and generates a disparity map. Since the depth map generation unit 35 calculates the deviation amount for each pixel at the pixel position by the matching process, the image phase adjustment unit 371 acquires the deviation amount for each pixel from the depth map generation unit 35 and dissipates it. A parity map may be generated.

Further, the image phase adjusting unit 371 performs image interpolation processing for lines of unpolarized images for which a polarized image has not been obtained for each of the images supplied from the imaging unit 31 and the imaging unit 32, and the polarization directions are different. A polarized image consisting of pixels in a plurality of directions is generated. FIG. 18 shows an image supplied from the imaging unit and an image after interpolation processing. Note that FIG. 18A shows the polarization direction of the image supplied from the imaging unit 31, and FIG. 18B shows the polarization direction of the image after the interpolation processing. The image phase adjustment unit 371 performs interpolation using pixels of polarized images adjacent to the upper and lower sides, and for example, unpolarizes the average value of the pixel values of the pixels adjacent to the upper side and the pixel values of the pixels adjacent to the lower side. As the pixel value of the pixel position of the image, a polarized image composed of pixels having two polarization directions is generated. The image phase adjusting unit 371 also performs image interpolation processing on the image supplied from the imaging unit 32, and obtains a polarized image consisting of pixels in two directions having different polarization directions from the image supplied from the imaging unit 31. Generate.

Further, the image phase adjusting unit 371 matches the phase of the polarized image after the interpolation processing on the image from the imaging unit 31 and the polarized image after the interpolation processing on the image from the imaging unit 32 based on the disparity map. FIG. 19 is a diagram for explaining the phase adjustment process. FIG. 19A shows a part of the polarized image after the interpolation processing of the image supplied from the imaging unit 31. FIG. 19B shows a part of the polarized image after the interpolation processing of the image supplied from the imaging unit 32. FIG. 19 (C) illustrates a part of the disparity map. The disparity value (parallax amount) of the disparity map has a one-to-one correspondence with the depth value of the depth map, and as described above, the distance "LB" between the reference positions of the two imaging units and the focal length of the imaging unit It can be easily converted in both directions from the distance "f". The parallax amount indicates the relationship between the pixels on the image supplied from the imaging unit 31 and the pixels on the image supplied from the imaging unit 32 corresponding to the same portion of the subject. For example, the pixel Pg1 (not shown) on the image supplied from the imaging unit 31 corresponding to the pixel Pg2 (not shown) on the image supplied from the imaging unit 32 refers to the parallax amount of the disparity map. Can be identified by. Here, assuming that the value of the parallax amount is "2 (pixels)", the pixel Pg1 located two pixels to the right of the pixel Pg2 corresponds to the same part of the subject. Therefore, the image phase adjusting unit 371 adjusts the phase of the image after the interpolation processing supplied from the imaging unit 32 with reference to the disparity map, and the image and the image after the interpolation processing supplied from the imaging unit 31. The phase-matched image shown in FIG. 19 (D) is generated. Here, when the disparity map has the value shown in (C) of FIG. 19, the upper half is "1 (pixel)" and the lower half is "2 (pixel)". Therefore, the image phase adjustment unit 371 shifts the upper half of the image supplied from the imaging unit 32 after the interpolation process to the right by "1 (pixel)" and the lower half to the right by "2 (pixel)". The shift process is performed to generate an image in which the phases are matched.

The image phase adjustment unit 371 performs the above processing to generate a polarized image in two directions having different polarization directions and a polarized image in two directions having the same phase as this image and different polarization directions, and a normal map. Output to the generation processing unit 375. That is, the image phase adjustment unit 371 outputs a multi-polarized image excluding the influence of the parallax of the subject to the normal map generation processing unit 375.

The normal map generation processing unit 375 generates a normal map based on a plurality of polarized images in the polarization directions supplied from the image phase adjustment unit 371, and outputs the normal map to the map integration unit 39.

The map integration unit 39 integrates the depth map and the normal map. The map integration unit 39 traces the subject surface shape starting from the pixel from which the depth value is obtained based on the subject surface shape shown in the normal map and the depth value shown in the depth map, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. In addition, the map integration unit 39 includes the estimated depth value in the depth map supplied from the depth map generation unit 35 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 35. ..

FIG. 20 is a flowchart showing the processing operation of the second embodiment. In step ST41, the imaging unit 31 generates the first image. The image pickup unit 31 is a first pixel group composed of a first pixel group composed of pixels having polarization characteristics and a second pixel group composed of pixels in a polarization direction different from that of the first pixel group or pixels having no polarization characteristics. Generate an image. Further, in step ST42, the imaging unit 32 has a third pixel group composed of pixels whose polarization direction is different from that of the first image at a position corresponding to the first pixel group, and a second pixel group at a position corresponding to the second pixel group. A second image consisting of a fourth pixel group consisting of pixels having the same configuration as the two pixel groups is generated.

In step ST43, the depth map generator 35 generates a depth map. FIG. 21 is a flowchart showing the depth map generation process. In step ST51, the depth map generation unit 35 generates a depth map using the second and fourth pixel groups. The depth map generation unit 35 performs matching processing using images of the second and fourth pixel groups, calculates the distance to the subject (depth value) for each pixel, and generates a depth map.

In step ST52, the depth map generation unit 35 performs depth interpolation processing. The depth map generation unit 35 calculates the depth value for the pixel of the first (third) pixel group by the interpolation process using the depth value calculated for the pixel of the second (fourth) pixel group. In this way, the depth map generation unit 35 generates a depth map showing the depth value for each pixel of the first (third) pixel group and the second (fourth) pixel group by performing the depth interpolation processing. ..

In step ST44 of FIG. 20, the normal map generator 37 generates a multipolarized image. FIG. 22 is a flowchart showing the generation process of the multi-polarized image. In step ST61, the normal map generator 37 generates a disparity map. The normal map generation unit 37 converts the depth map generated in step ST43 of FIG. 20 into a disparity map. The normal map generation unit 37 may use the deviation amount for each pixel position obtained by the matching process performed at the time of generating the disparity map.

In step ST62, the normal map generation unit 37 performs image interpolation processing. The normal map generation unit 37 has a pixel position of an unpolarized image for which a polarized image has not been obtained for each of the first image supplied from the imaging unit 31 and the second image supplied from the imaging unit 32. Image interpolation processing is performed on the image to generate a polarized image.

In step ST63, the normal map generator 37 adjusts the image phase. The normal map generation unit 37 moves the image based on the disparity map, and matches the phase of the image in the polarized image after the interpolation processing for the first image and the polarized image after the interpolation processing for the second image. , Generates a multi-polarized image excluding the effects of parallax.

In step ST45 of FIG. 20, the normal map generator 37 generates a normal map. The normal map generation unit 37 generates a normal map based on the multi-polarized image generated in step ST44.

In step ST46, the map integration unit 39 performs the map integration process. Based on the depth value shown in the depth map and the subject surface shape shown in the normal map, the map integration unit 39 traces the subject surface shape starting from the pixel from which the depth value is obtained, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. In addition, the map integration unit 39 includes the estimated depth value in the depth map.

As described above, according to the second embodiment, for example, even when a polarized image is generated by each of the first imaging unit and the second imaging unit, the depth map generation unit 35 generates the depth map. Is possible. Further, the normal map generation unit 37 can generate a normal map based on a multi-polarized image obtained by performing phase matching of images and excluding the influence of parallax. Therefore, by integrating the generated depth map and the normal map, a high-precision depth map having higher accuracy than the depth map generated by the depth map generation unit 35, that is, the depth value for each pixel of the subject area can be obtained. You can generate a stored depth map. In addition, it is possible to acquire an image in which the number of pixels does not decrease while generating a high-precision depth map.

Further, since the pixels of the image sensor of the imaging unit 31 and the image sensor of the imaging unit 32 are subjected to only an averaging filter having a maximum polarization direction of two directions, the influence of the averaging filter is suppressed and a highly accurate depth map is performed. Can be generated.

&lt;3-2. First variant of the second embodiment&gt; In the above-described embodiment, the configuration in which the image sensors 31 and 32 use an image sensor composed of pixels of a single color is illustrated, but an image sensor composed of pixels of a plurality of colors may be used. Next, as a second modification of the second embodiment, a case where an image sensor in which red, blue, and green pixels are arranged in a Bayer array is used in the imaging units 31 and 32 will be described. The configuration of the image processing device is the same as that shown in FIG.

FIG. 23 illustrates the pixel configuration of the image sensor constituting the imaging unit in the first modification. Note that FIG. 23 shows a part of the image sensor. Further, FIG. 23 (A) shows the pixel configuration of the image sensor 311 constituting the image pickup unit 31, and FIG. 23 (B) shows the pixel configuration of the image sensor 321 constituting the image pickup unit 32. In addition, "R" indicates a red pixel, "G" indicates a green pixel, and "B" indicates a blue pixel.

The image sensor 311 of the imaging unit 31 has a Bayer array of red, blue, and green pixels, and has a configuration in which a polarizing filter is arranged on pixels of a predetermined color, for example, green pixels. As shown in FIG. 23 (A), the polarizing filter of the image sensor 311 has two different types of polarization directions (the polarization directions are indicated by arrows). The imaging unit 31 generates a first image composed of an image of pixels in the polarization direction in two directions (green), which is the first pixel group, and unpolarized pixels (red and blue) in the second pixel group. , Output to depth map generation unit 35 and normal map generation unit 37.

The image sensor 321 of the image pickup unit 32 has a Bayer array of red, blue, and green pixels, and has a configuration in which a polarizing filter is arranged on the green pixels. For example, as shown in FIG. 23 (B), the polarizing filter of the image sensor 321 has two types of polarization directions (the polarization directions are indicated by arrows) different from those of the image sensor 311. The image pickup unit 32 is a second image composed of an image of pixels (green) which is a third pixel group in the polarization direction in two directions different from the image pickup unit 31 and unpolarized pixels (red and blue) of the fourth pixel group. An image is generated and output to the depth map generation unit 35 and the normal map generation unit 37.

The depth map generation unit 35 has a preprocessing unit 351 and a depth map generation processing unit 355. The preprocessing unit 351 generates a matching image to be used for the matching process by using the unpolarized image of the images supplied from the imaging unit 31 and the imaging unit 32. For example, image interpolation processing is performed using only red pixels or blue pixels to generate a red or blue matching image. The depth map generation processing unit 355 performs matching processing using the matching image and generates a depth map.

The normal map generation unit 37 has an image phase adjustment unit 371 and a normal map generation processing unit 375.

The image phase adjustment unit 371 converts the depth map output from the depth map generation unit 35 into a disparity map. The image phase adjustment unit 371 determines the amount of parallax based on the depth value for each pixel shown in the depth map, and generates a disparity map. Since the depth map generation unit 35 calculates the deviation amount for each pixel by the matching process, the image phase adjustment unit 371 acquires the deviation amount for each pixel from the depth map generation unit 35 and creates a disparity map. It may be generated.

The image phase adjusting unit 371 performs image interpolation processing of an unpolarized image using a polarized image on each of the images supplied from the imaging unit 31 and the imaging unit 32, and from pixels in a plurality of directions having different polarization directions. Each polarized image is generated. Further, the image phase adjusting unit 371 matches the phase of the polarized image after the interpolation processing on the image from the imaging unit 31 and the polarized image after the interpolation processing on the image from the imaging unit 31 based on the disparity map, and makes the subject Generates a multi-polarized image without the effects of parallax. The image phase adjustment unit 371 outputs the generated multi-polarized image to the normal map generation processing unit 375.

The normal map generation processing unit 375 generates a normal map based on a plurality of polarized images in the polarization directions supplied from the image phase adjustment unit 371, and outputs the normal map to the map integration unit 39.

The map integration unit 39 integrates the depth map and the normal map. The map integration unit 39 traces the subject surface shape starting from the pixel from which the depth value is obtained based on the subject surface shape shown in the normal map and the depth value shown in the depth map, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. Further, the map integration unit 39 includes the estimated depth value in the depth map supplied from the depth map generation unit 35 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 35. ..

As described above, according to the first modification, the depth map generation unit 35 can generate a depth map even when an image sensor composed of pixels of a plurality of colors is used. Further, the normal map generation unit 37 can generate a normal map based on a multi-polarized image obtained by performing phase matching of images and excluding the influence of parallax. Therefore, by integrating the generated depth map and the normal map, it is possible to generate a depth map having an accuracy higher than that of the depth map generated by the depth map generation unit 35.

&lt;3-3. Second modification of the second embodiment&gt; Next, as a second modification of the second embodiment, the first pixel group of the first image supplied from the imaging unit 31 and the third image of the second image supplied from the imaging unit 32. The polarization directions of the second pixel group of the first image and the fourth pixel group of the second image are different from those of the pixel groups, and the polarization directions of the second pixel group of the first image and the fourth pixel group of the second image are different from those of the first and second pixel groups. The case of equality will be described. The configuration of the image processing device is the same as that shown in FIG.

FIG. 24 illustrates the pixel configuration of the image sensor constituting the imaging unit in the second modification. Note that FIG. 24 shows a part of the image sensor. Further, FIG. 24 (A) shows the pixel configuration of the image sensor 312 constituting the image pickup unit 31, and FIG. 24 (B) shows the pixel configuration of the image sensor 322 constituting the image pickup unit 32.

The image sensor 312 of the image pickup unit 31 has a configuration in which a polarizing filter is arranged for each pixel. For example, as shown in FIG. 24 (A), the polarizing filter of the image sensor 312 has two different types of polarization directions (the polarization directions are indicated by arrows). Further, the pixels of the same line are configured to have the same polarization direction. The imaging unit 31 generates a first image consisting of a line of the first pixel group in one polarization direction and a line of the second pixel group in the other polarization direction, and generates a depth map generator 35 and a normal map. Output to generator 37.

The image sensor 322 of the image pickup unit 32 has a configuration in which a polarizing filter is arranged for each pixel. For example, as shown in FIG. 24 (B), the polarizing filter of the image sensor 322 has a polarization direction equal to that of the image sensor 312 (the polarization direction is indicated by an arrow) and a different polarization direction. Further, the pixels of the same line are configured to have the same polarization direction. The image pickup unit 32 generates a second image consisting of a line of a third pixel group in a polarization direction different from that of the image pickup unit 31 and a line of a fourth pixel group in the polarization direction equal to the second pixel group of the image pickup unit 31. Then, it is output to the depth map generation unit 35 and the normal map generation unit 37. Therefore, in the case of FIG. 24, a polarized image having three polarization directions is supplied to the normal map generation unit 37. Note that FIG. 24 illustrates a case where the same polarization directions are in the upper right direction in the image sensors 312 and 322.

The depth map generation unit 35 has a preprocessing unit 351 and a depth map generation processing unit 355. The preprocessing unit 351 generates a matching image to be used for the matching process using the images supplied from the imaging unit 31 and the imaging unit 32. The preprocessing unit 351 performs image interpolation processing using only images of pixels having the same polarization direction in the imaging unit 31 and the imaging unit 32 to generate a matching image. The depth map generation processing unit 355 performs matching processing using the matching image and generates a depth map.

The normal map generation unit 37 has an image phase adjustment unit 371 and a normal map generation processing unit 375.

Since the image phase adjusting unit 371 causes parallax due to the difference in position between the imaging unit 31 and the imaging unit 32, the image phase adjusting unit 371 generates a polarized image excluding the influence of the parallax. The image phase adjustment unit 371 converts the depth map output from the depth map generation unit 35 into a disparity map. The image phase adjustment unit 371 determines the amount of parallax based on the depth value for each pixel shown in the depth map, and generates a disparity map. Since the depth map generation unit 35 calculates the deviation amount for each pixel by the matching process, the image phase adjustment unit 371 acquires the deviation amount for each pixel from the depth map generation unit 35 and creates a disparity map. It may be generated.

Further, the image phase adjusting unit 371 matches the phase of the polarized image from the imaging unit 31 and the polarized image from the imaging unit 32 based on the disparity map, and creates a multi-polarized image from which the influence of the parallax of the subject is removed. Output to the normal map generation processing unit 375.

The normal map generation processing unit 375 generates a normal map based on a plurality of polarized images in the polarization directions supplied from the image phase adjustment unit 371, and outputs the normal map to the map integration unit 39.

The map integration unit 39 integrates the depth map and the normal map. The map integration unit 39 traces the subject surface shape starting from the pixel from which the depth value is obtained based on the subject surface shape shown in the normal map and the depth value shown in the depth map, so that the depth value can be obtained. Estimate the depth value corresponding to the unobtained pixels. Further, the map integration unit 39 includes the estimated depth value in the depth map supplied from the depth map generation unit 35 to generate a depth map having higher accuracy than the depth map supplied from the depth map generation unit 35. ..

As described above, according to the second modification, the image supplied from the imaging unit 31 is an image in a plurality of polarization directions, and the image supplied from the imaging unit 32 is polarized in a polarization direction equal to that of the imaging unit 31 and different from the polarization direction. It is possible to generate a depth map even if the image is in the direction. Further, the normal map generation unit 37 can generate a normal map based on a multi-polarized image obtained by performing phase matching of images and excluding the influence of parallax. Therefore, by integrating the generated depth map and the normal map, it is possible to generate a depth map having an accuracy higher than that of the depth map generated by the depth map generation unit 35.

Further, in the second embodiment as in the first embodiment, a depth map may be used to remove the indefiniteness of 180 degrees to generate a normal map. By removing the 180 degree indefiniteness, the depth map can be generated with high accuracy even in the second embodiment.

&lt;4. Third embodiment&gt; In the first and second embodiments described above, the configuration in which the image processing apparatus is provided with a plurality of image pickup units is illustrated, but the image pickup units may be separable.

FIG. 25 illustrates the appearance of the third embodiment. In FIG. 25, for example, a smartphone is used as the image processing device.

The image processing device 40 is formed in a substantially rectangular case shape, and has a signal processing unit, a communication unit, a control unit, and the like inside the outer casing 401 (not shown). Further, a display panel 402 is provided on one surface (surface) of the outer casing 401. The display panel 402 is configured by using a touch panel, and various functions are executed by operating each predetermined position of the display panel 402. An imaging unit 42 is provided on the other surface (back surface) of the outer casing 401.

The image pickup apparatus 50 has an image pickup optical system, an image pickup unit, a signal processing unit, a communication unit, a control unit, and the like, although not shown, inside the outer cylinder portion 501 formed in a cylindrical shape. An annular control ring 502 is provided at the front end of the outer cylinder 501. The image pickup apparatus 50 changes the focus position and the zoom position according to the rotation of the control ring 502. Further, a zoom button 503 and a shutter button 504 are provided on the side surface of the outer cylinder portion 501.

The image pickup apparatus 50 is provided with an attachment mechanism portion 60 for integrally attaching the image processing apparatus 40 and the image pickup apparatus 50. The mounting mechanism portion 60 is provided with a mounting member 61, and is configured to be movable in the direction of arrow FA. The user moves the mounting member 61 in the direction of the arrow FA and locks it to the outer housing 401 of the image processing device 40 to integrally fix the image pickup device 50 to, for example, the back surface side of the image processing device 40. By integrally fixing the image pickup device 50 to the image processing device 40 in this way, it becomes possible for the image pickup unit 42 and the image pickup device 50 to generate a stereo image. Further, the user moves the mounting member 61 locked to the outer casing 401 of the image processing device 40 in the direction opposite to the locking direction to separate the image processing device 40 and the image pickup device 50.

FIG. 26 illustrates the configuration of the third embodiment. The image processing device 40 includes an imaging unit 42, a communication unit 43, a depth map generation unit 45, a normal map generation unit 47, and a map integration unit 49. Further, the image pickup apparatus 50 has an image pickup unit 51 and a communication unit 53.

The image pickup unit 42 of the image processing device 40 corresponds to, for example, the image pickup section 22 of the first embodiment and the image pickup section 32 of the second embodiment. When the image pickup unit 42 corresponds to the image pickup unit 22, the image pickup unit 42 outputs the generated image to the depth map generation unit 45. When the imaging unit 42 corresponds to the imaging unit 32, the imaging unit 42 outputs the generated image to the depth map generation unit 45 and the normal map generation unit 47.

The image pickup unit 51 of the image pickup apparatus 50 corresponds to, for example, the image pickup unit 21 of the first embodiment and the image pickup unit 31 of the second embodiment. The image pickup unit 51 outputs the generated image to the communication unit 53. The communication unit 53 is configured to be capable of performing wireless communication such as NFC (Near Field Communication) communication and Wi-Fi communication. The communication unit 53 transmits the image generated by the image pickup unit 51 to the image processing device 40.

The communication unit 43 of the image processing device 40 is configured in the same manner as the communication unit 53 of the image pickup device 50. The communication unit 43 receives the radio signal transmitted from the communication unit 53 of the image pickup device 50, and outputs the image transmitted from the image pickup device 50 to the depth map generation unit 45 and the normal map generation unit 47.

The depth map generation unit 45 performs the same processing as the depth map generation unit 25 of the first embodiment and the depth map generation unit 35 of the second embodiment to generate a depth map to the map integration unit 49. Output.

The normal map generation unit 47 performs the same processing as the normal map generation unit 27 of the first embodiment and the normal map generation unit 37 of the second embodiment to generate a normal map and map it. Output to the integration unit 49.

The map integration unit 49 performs the same processing as the map integration unit 29 of the first embodiment and the map integration unit 39 of the second embodiment, and based on the depth map and the normal map, the depth map generation unit 45 Generates and outputs a depth map with accuracy higher than the depth map generated in.

According to such a third embodiment, even if the imaging unit is separable, it is possible to acquire an image in which the number of pixels does not decrease while generating a high-precision depth map. Therefore, for example, by attaching an imaging device that generates a polarized image having three or more polarization directions to an information processing device such as a smartphone to enable generation of a highly accurate depth map, the functions of the existing information processing device can be expanded. it can.

Further, the image pickup apparatus may be configured by using a part of the above-described embodiment. For example, the image pickup apparatus is generated by a first image pickup unit having a pixel configuration including pixels having different polarization characteristics, a second image pickup unit having a pixel configuration different from that of the first image pickup unit, and a first image pickup unit. The configuration includes an image processing unit that performs image processing using the image of 1 and the second image generated by the second imaging unit. Further, in the imaging device, for example, the pixel configuration of the first imaging unit includes a first pixel group composed of pixels having polarization characteristics, pixels in the polarization direction different from those of the first pixel group, or pixels having no polarization characteristics. The pixel configuration is composed of a second pixel group consisting of. Further, the pixel configuration of the second imaging unit corresponds to a third pixel group composed of pixels whose polarization direction is different from that of the first image at a position corresponding to the first pixel group, and a second pixel group. The pixel configuration is composed of a fourth pixel group composed of pixels having the same configuration as the second pixel group at the position. When the image pickup apparatus is configured in this way, it is possible to easily generate an image to be processed used for acquiring an image in which the number of pixels does not decrease while generating a high-precision depth map. If the image processing unit of the image pickup device generates the depth map and the normal map and integrates the depth map and the normal map as described above, the depth map and the number of pixels can be obtained from the image pickup device with high accuracy. It is possible to output an image that does not decrease. Further, an image processing device may be configured in the same manner as this image pickup device.

In addition, the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, the program that records the processing sequence is installed in the memory in the computer embedded in the dedicated hardware and executed. Alternatively, it is possible to install and execute the program on a general-purpose computer that can execute various processes.

For example, a program can be pre-recorded on a hard disk as a recording medium, an SSD (Solid State Drive), or a ROM (Read Only Memory). Alternatively, the program is a flexible disc, CD-ROM (Compact Disc Read Only Memory), MO (Magneto optical) disc, DVD (Digital Versatile Disc), BD (Blu-Ray Disc (registered trademark)), magnetic disc, semiconductor memory card. It can be temporarily or permanently stored (recorded) on a removable recording medium such as a CD. Such a removable recording medium can be provided as so-called package software. In addition to installing the program on the computer from a removable recording medium, the program can be installed on a computer from a download site via LAN (Local Area). It may be transferred to the computer wirelessly or by wire via a network such as Network) or the Internet. The computer can receive the program transferred in this way and install it on a recording medium such as a built-in hard disk.

In addition, the present technology should not be construed as being limited to the embodiments of the above-mentioned technology. The embodiment of this technique discloses the present technology in the form of an example, and it is obvious that a person skilled in the art can modify or substitute the embodiment without departing from the gist of the present technique. That is, in order to judge the gist of the present technology, the scope of claims should be taken into consideration.

The image processing apparatus of the present technology can also have the following configuration. (1) A first image generated by a first imaging unit having a pixel configuration including pixels having different polarization directions, and a second image generated by a second imaging unit having a pixel configuration different from that of the first imaging unit. Depth map generator that generates a depth map by matching processing using the image of A normal map generation unit that generates a normal map based on the polarization state of at least one of the polarized images of the first image or the second image generated by the depth map generation unit. An image processing device having a depth map generated by the depth map generation unit and a map integration unit that integrates the normal map generated by the normal map generation unit. (2) The image processing apparatus according to (1), wherein the normal map generation unit generates the normal map based on the brightness of a polarized image having three or more polarization directions. (3) The map integration unit estimates the depth value not shown in the depth map from the depth value shown in the depth map and the shape determined based on the normal map (1) or (2). ). The image processing apparatus according to any one of. (Four) The first image is an image generated by the first imaging unit including pixels having three or more polarization directions. The second image is an image generated by the second imaging unit composed of pixels having no polarization characteristic. The image processing apparatus according to any one of (1) to (3), wherein the normal map generation unit generates a normal map based on the first image. (5) The image processing apparatus according to (4), wherein the depth map generation unit generates an unpolarized image from the first image and performs the matching process using the unpolarized image and the second image. .. (6) The depth map generation unit performs edge extraction on each of the unpolarized image and the second image, and uses the edge extracted image of the unpolarized image and the edge extracted image of the second image. The image processing apparatus according to (5), which performs the matching process. (7) The second image is an image generated by the second image pickup unit provided with color filters of the same color for all pixels or the second image pickup unit not provided with the color filter. , The image processing apparatus according to any one of (4) to (6), wherein the depth map generation unit performs the matching process using an unpolarized image generated from the second image. (8) The first image is a first pixel group composed of pixels having polarization characteristics, and a second pixel composed of pixels in a polarization direction different from the first pixel group or pixels having no polarization characteristics. It is an image generated by the first imaging unit having a pixel configuration composed of a group. The second image corresponds to a third pixel group composed of pixels having a polarization direction different from that of the first image at a position corresponding to the first pixel group, and the second pixel group. It is an image generated by the second imaging unit having a pixel configuration composed of pixels in the polarization direction equal to the second pixel group or a fourth pixel group composed of pixels having no polarization characteristics at the position to be used. The image processing apparatus according to (3). (9) The depth map generation unit uses an image of the second pixel group in the first image and an image of the fourth pixel group in the second image, so that the image has the same polarization direction. The image processing apparatus according to (8), wherein the matching process is performed between images having no interposition or polarization characteristics. (Ten) Based on the amount of discrepancy between the first image and the second image, the phases of the image of the first pixel group in the first image and the image of the third pixel group in the second image are matched. Further, it has an image phase adjusting unit that generates a polarized image having a plurality of polarization directions. The image processing apparatus according to any one of (8) or (9), wherein the normal map generation unit generates a normal map based on the polarization state of the polarized image generated by the image phase adjustment unit. (11) When the second pixel group and the fourth pixel group are images having no polarization characteristics, the polarization directions of the first pixel group and the third pixel group are three or more in total. The image processing apparatus according to any one of (8) to (10) as an image. (12) When the second pixel group and the fourth pixel group are images having no polarization characteristic, the image phase adjusting unit performs the first pixel group by interpolation processing using the image of the first pixel group. An image of the second pixel group is generated, an image of the fourth pixel group is generated by an interpolation process using the image of the third pixel group, and the polarized image is generated using the image after interpolation. The image processing apparatus according to any one of (8) to (11). (13) The image processing according to any one of (8) to (12), wherein the first and third pixel groups are pixels of a predetermined color, and the second and fourth pixel groups are pixels of other colors. apparatus. (14) The image processing apparatus according to any one of (3) to (13), wherein the normal map generation unit generates the normal map using the depth map generated by the depth map generation unit. (15) The image processing apparatus according to any one of (1) to (14), further comprising a first image pickup unit that generates the first image and a second image pickup unit that generates the second image. (16) A communication unit that communicates with an external device provided with either one of the first image pickup unit and the second image pickup unit and acquires an image generated by the image pickup unit provided in the external device. When, The image processing apparatus according to any one of (1) to (15), further comprising another imaging unit different from the imaging unit provided in the external device.

In the image processing apparatus, the image processing method, and the imaging apparatus of this technology, the first image generated by the first imaging unit having a pixel configuration including pixels having different polarization directions, and the first imaging unit and the pixel configuration are A depth map is generated by a matching process using a second image generated by a different second imaging unit. Also, a normal map is generated based on the polarization state of at least one of the polarized images of the first or second image. Further, the generated depth map and the normal map are integrated. Therefore, it is possible to acquire an image in which the number of pixels does not decrease while generating a high-precision depth map. Therefore, it is suitable for a device or the like that acquires a three-dimensional shape of a subject.

10,20,30,40 Image processing device, 15,25,35,45 Depth map generator, 17,27,37,47 Normal map generator, 19,29,39 , 49 Map integration unit, 21,22,31,32,42,51 Imaging unit, 43,53 Communication unit, 50 Imaging device, 53 Communication unit, 60 Mounting mechanism part, 61 Mounting member, 210,220,221,310,311,312,320,321,322 Image sensor, 251,351 Preprocessing part, 255,355 Depth map generation processing part, 275,276,375 Normal map generation processing part, 371 Image phase adjustment unit, 375 Normal map generation processing unit, 401 Outer housing, 402 Display panel, 501 Outer cylinder part, 502 Control ring, 503 Zoom button, 504 Shutter button