Compound eye image pickup device

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Industrial application field]

The present invention relates to an image pickup device using an image pickup device such as a CCD (charge-coupled device) or an image pickup tube, and particularly to a compound eye image pickup device using an image pickup optical system such as a plurality of image pickup elements and a lens.

【0002】

[Conventional technology]

In recent years, for the purpose of generating a wide panoramic image or a high-definition image, it has a plurality of imaging systems including an imaging optical system and an imaging element, and by imaging a common subject, it is input from each of the imaging elements. A compound-eye image sensor that generates and outputs one composite image using the combined image signal has been proposed.

【0003】

As a method of obtaining a panoramic image, a plurality of images having different subject areas are simultaneously imaged by a plurality of imaging systems, the same subject existing in each of these images is extracted, and each is based on relative position information in the image. A composite panoramic image is obtained by connecting images.

【0004】

On the other hand, as a method of obtaining a high-definition image, the same subject existing in each image is extracted as in the panoramic image, and interpolation processing is performed based on the relative position information in the image to newly perform high-definition 1 You get one image. The compound eye imaging device based on this principle is, for example, as shown in FIG. 22, the left imaging system 1010._LAnd right imaging system 1010_RAnd prepare the left imaging system 1010_LAnd right imaging system 1010_RImage the subject 1101 with. And the left imaging system 1010_LLeft image obtained in_LAnd right imaging system 1010_RRight image I obtained in_RBy extracting the corresponding points with the CPU 1120 and combining them, one output image I with higher definition than when the subject is imaged with one imaging system._OUTTo get.

【0005】

[Problems to be Solved by the Invention]

However, the compound eye imaging device described above has the following problems. In the case of obtaining a panoramic image, since a plurality of images are connected, the maximum angle of view of one imaging system is not limited, and one panoramic image is reconstructed from a plurality of images captured at the same timing. Although it is possible to realize a flexible imaging device such as a configuration in which a plurality of imaging systems that can cover a desired subject area including an animal body are combined, each image is captured in parallel except for a part of a connection portion. Therefore, most of the image area is rather a subject without a corresponding point. In this case, the subjects corresponding to each other at the image connection portion may appear as a so-called double image that does not match on the generated image and exists in duplicate, thereby losing the continuity of the image and deteriorating the image quality. There was a problem of inviting. Here, the continuity of an image means that the subject shape (contour line, etc.), density (luminance, etc.), etc. are continuously and smoothly connected at the attention portion.

【0006】

Further, in order to obtain a high-definition image, high-precision relative position information of corresponding points (1 pixel pitch or less) is required, and a high-precision image interpolation method is also required. Therefore, if these accuracy is not sufficiently satisfied, high-frequency noise components and the like that should not originally exist may be superimposed, which may rather lead to deterioration of the image. On the other hand, regarding the time resolution, assuming that this compound eye imaging device is used for television, it is necessary to continuously capture images of about 30 frames per second, so these processes must be fairly fast. However, as simple a process as possible is desired.

【0007】

The present invention has been made in view of these problems, and the first object thereof is to generate an image that does not lose continuity from the image information in the region where the compound image exists and its vicinity. An object of the present invention is to provide a compound eye imaging device that prevents deterioration.

【0008】

A second object of the present invention is to provide a compound eye imaging device that satisfies the real time required by the imaging device and at least does not deteriorate the image quality of the composite image as compared with the original image before generation.

【0009】

[Means for solving problems]

In order to achieve the above object, the present invention has a plurality of image pickup systems including a plurality of image pickup optical systems and an image pickup element corresponding to each of the image pickup optical systems, and each of the image pickup devices by taking an image of a subject. In a compound-eye image sensor that generates and outputs one composite image using the image signals input from each of the above, the corresponding point pairs corresponding to the same part of the subject are extracted from each of the image signals, and the extracted points are extracted. For at least a part of the corresponding point pairs, the corresponding point pair has an image processing unit that uses at least one corresponding point in the corresponding point pair as the composite image generation data and does not use the other as the composite image generation data. It is characterized by that.

【0010】

Further, the composite image may be a panoramic composite image obtained by synthesizing each of the image signals in parallel. In this case, the image processing unit captures images only with one of the extracted corresponding point pairs. For the corresponding point pair belonging to the image connected to the non-double image generation region, the corresponding point pair belonging to the image having the highest ratio of being connected to the double image non-generation region in the corresponding point pair. Is used as the composite image generation data, and the corresponding point pair having a different distance from the double image non-generating region that is imaged by only one of the extracted corresponding point pairs is the corresponding point pair. The corresponding point closest to the double image non-generation region is used as the composite image generation data, or only one of the extracted corresponding point pairs is used in only one imaging system. For the corresponding point pair belonging to the image connected to the non-double image generation region that is not captured, the corresponding point in the one image is used as the composite image generation data, and the double image non-occurrence is used in two or more images. Regarding the corresponding points belonging to the image connected to the generation region, the corresponding points in any one of the two or more images are enlarged in the parallel direction of the images and used as the composite image generation data. There may be.

【0011】

Further, the composite image may be a high-definition image obtained by synthesizing the respective image signals, and in this case, the image processing unit is included in the respective corresponding point pairs among the extracted corresponding point pairs. For a correspondence point pair in which the interval between the corresponding points is smaller than a predetermined interval, one corresponding point in the corresponding point pair is used as the composite image generation data, or a corresponding point in each of the corresponding point pairs. For at least a part of the corresponding point pairs that require distortion correction before extraction, one of the corresponding points is used as the composite image generation data, or each of the extracted corresponding point pairs. Of these, at least a part of the corresponding point pair in which the degree of blurring of the image predicted from the distance from the in-focus object surface of each imaging system and the number of openings of each imaging optical system exceeds a certain threshold is the same. One of the corresponding point pairs is used as the composite image generation data, a pixel point for which a paired corresponding point cannot be obtained after extracting each corresponding point pair, and the paired corresponding point. For at least a part of the corresponding point pair that appears alternately in the region where is not obtained and above a certain spatial frequency, one corresponding point in the corresponding point pair is used as the composite image generation data. May be good.

【0012】

Further, it has a plurality of imaging systems including a plurality of imaging optical systems and imaging elements corresponding to each of the imaging optical systems, and an image signal input from each of the imaging elements by imaging a subject. In a compound eye imaging device that generates and outputs one composite image using the above, one of the image signals is used as a basic image for generating a composite image, and the other image signal is used as an auxiliary image. Corresponding point pairs corresponding to the same part of the subject are extracted from each of the image signals, and corresponding points in the basic image are combined for at least a part of the extracted corresponding point pairs. Some are characterized by having an image processing unit that is used as image generation data and does not use the corresponding points in the auxiliary image as composite image generation data.

【0013】

In this case, the composite image is a panoramic composite image obtained by synthesizing the respective image signals in parallel, and the image processing unit is a double image captured by only one imaging system of the extracted corresponding point pairs. With respect to the corresponding point pair belonging to the image connected to the non-generation region, the image signal having the highest ratio of being connected to the double image non-generation region may be used as the basic image.

【0014】

Further, the imaging system having the imaging element that outputs an image signal to be the basic image may be arranged with the optical axis of the imaging optical system directed toward the subject in the front direction of the device, and the synthesis may be performed. The image is a high-definition image obtained by synthesizing the respective image signals, and the image processing unit has the corresponding points in the basic image and the corresponding points in the auxiliary image among the extracted corresponding point pairs. For a pair of corresponding points whose interval is smaller than a predetermined interval, the corresponding points in the basic image are used as the composite image generation data, and the corresponding points in the auxiliary image are not used as the composite image generation data, or the above. The composite image is a high-definition image obtained by synthesizing the respective image signals, and the image processing unit describes at least a part of the corresponding point pairs that require distortion correction before extracting the corresponding points. Uses the corresponding points in the basic image as the composite image generation data and does not use the corresponding points in the auxiliary image as the composite image generation data, or the composite image is a height obtained by synthesizing the respective image signals. It is a fine image, and the image processing unit is an image predicted from the distance from the in-focus object surface of each imaging system and the number of openings of each imaging optical system among the extracted corresponding point pairs. For at least a part of the corresponding point pair in which the degree of blur exceeds a certain threshold, the corresponding points in the basic image are used as the composite image generation data, and the corresponding points in the auxiliary image are used as the composite image generation data. No image or the composite image is a high-definition image obtained by synthesizing the respective image signals, and the image processing unit is a pixel for which a pair of corresponding points cannot be obtained from the extracted pair of corresponding points. The corresponding points in the basic image are used as the composite image generation data for at least a part of the corresponding point pairs appearing alternately at a certain spatial frequency or higher with the region where the points and the corresponding corresponding points are not obtained. , The corresponding points in the auxiliary image may not be used as the composite image generation data.

【0015】

[Action]

In the compound-eye imaging apparatus of the present invention configured as described above, corresponding point pairs corresponding to the same part of the subject are extracted from each image signal, and at least a part of the extracted corresponding point pairs is obtained. By having an image processing unit that uses at least one corresponding point in the corresponding point pair as the composite image generation data and does not use the other as the composite image generation data, deterioration of the composite image is prevented. More specifically, in the generation of the panoramic composite image, a double image may be generated near the connection portion of the two images, but at least one correspondence point in the correspondence point pair is the composite image generation data. By using the above and not using other corresponding points as the composite image generation data, the double image is removed and good connectivity of the image with the non-double image generation region is maintained. Further, in the generation of a high-definition image, when the corresponding points in the corresponding point pair are very close to each other, at least one corresponding point in the corresponding point pair is used as the composite image generation data, and the other corresponding points are generated. By not using the corresponding points as the composite image generation data, unnecessary high-frequency components generated in response to an error in the relative pixel position of the corresponding point pair or a noise component superimposed on the pixel value can be suppressed. Furthermore, in high-definition image generation, by changing the processing in the distorted image part, the blurred image part, and the occlusion image part where the corresponding point pair cannot be obtained as described above, unnecessary processing steps are omitted or an image due to erroneous correspondence is obtained. Deterioration is suppressed.

【0016】

Further, by setting a basic image for generating a composite image, an image having at least a certain level of image quality is always output. In particular, by arranging an image pickup system having an image pickup element that outputs an image signal as a basic image so that the optical axis of the image pickup optical system faces the subject, at least an image from the front of the subject can be specially processed. It can always be output without adding it, and even during close-up shooting, the photographer can shoot without feeling an unnatural operation feeling.

【0017】

[Example]

Next, examples of the present invention will be described with reference to the drawings.

【0018】

(First Example) FIG. 1 is a schematic configuration diagram of a first embodiment of the compound eye imaging device of the present invention. The compound eye imaging device of this embodiment is the right imaging system 10._R And left imaging system 10_L One panoramic composite image is obtained by connecting two images obtained by imaging a subject using the two imaging systems of the above in parallel.

【0019】

First, the left imaging system 10_L Will be described. Left imaging system 10_L Is an imaging lens group 11 as an imaging optical system incorporated in a lens barrel (not shown)._L And this imaging lens group 11_L Color separation prism 12 mounted on the camera to separate the light from the subject into the three primary colors._L And the color separation prism 12_L Three CCD sensors each having a rectangular effective light receiving part as an image sensor provided for each light decomposed by_L (Only one is shown). Imaging lens group 11_L Is the focus motor 14_L Focused lens group driven by 15_L And zoom motor 16_L Variable magnification lens group driven by 17_L Consists of multiple lenses including and each motor 14_L、16_L Is driven in response to control signals from the entire system control unit 21 and the focus / zoom control unit 22 in the control unit 20 that controls the optical system. On the other hand, the right imaging system 10_R Left side imaging system 10_L It is configured in the same way as the right imaging system 10_R Imaging lens group 11_R Optical axis L_R Is the left imaging system 10_L Imaging lens group 11_L Optical axis L_L Is on the same plane as.

【0020】

In addition, each imaging lens group 11_L、11_REach of the lens barrels incorporating the lens barrel is driven in response to a control signal from the convergence angle control unit 23 of the control unit 20._L、18_RIt is connected to the axis of rotation of. Each convergence angle motor 18_L、18_RThe rotation axes of are both imaging lens groups 11 respectively._L、11_ROptical axis L_L, L_RExtends perpendicular to the plane containing, and each convergence angle motor 18_L、18_RBy driving, each imaging lens group 11_L、11_RIs a color separation prism 12 provided for each_L、12_RAnd CCD sensor 13_L、13_RRotated together with each imaging lens group 11_L、11_ROptical axis L_L, L_RThe angles (convergence angles) that form each other are set. In addition, each imaging system 10_L、10_RIn each focusing lens group 15_L、15_RFocus encoder 24 for detecting the position of_L、24_R, Each variable magnification lens group 17_L、17_RZoom encoder 25 for detecting the position of_L、25_R, And the congestion angle encoder for detecting the congestion angle 26_L、26_RIs provided. For these, for example, an external member such as a potentiometer may be used, or one that detects each position and angle from the signal information of the drive system itself such as a pulse motor may be used.

【0021】

On the other hand, each CCD sensor 13_L、13_RThe image output unit 40 is connected to the image output unit 40 via the image processing unit 30 which is a feature of the present invention. As shown in FIG. 2, the image processing unit 30 includes each imaging system 10._L、10_RCCD sensor 13_L、13_RLeft image memory 31 that holds video signals that are image signals from (see Fig. 1)_LAnd right image memory 31_RThe image input unit 32 is composed of an image input unit 32, an image conversion unit 38 for generating one composite image based on each of the left and right images obtained by each video signal input to the image input unit 32, and an image conversion unit 38 are combined. It has a composite image memory 39 for holding the image and outputting it to the image output unit 40. Further, the image conversion unit 38 includes a corresponding point extraction unit 33 that extracts a corresponding point pair between the two images for all the pixels of the double image generation region, which will be described later, among the two images input to the image input unit 32. A double image area division unit 34 that calculates the three-dimensional position (distance information) of each corresponding point pair from the result of the corresponding point pair extraction and divides the double image area based on the information, and two each divided into areas. The double image type determination unit 35 that determines the image with the higher proportion of the double image connected to the double image non-generation region, and the image with the higher proportion connected to the double image non-generation region. One-sided image double image batch removal unit 36 that leaves the double image belonging to the side and uses it as composite image generation data and removes the double image belonging to the other image side, and the image after the double image is removed. It is divided into a removal post-treatment unit 37, which is subjected to post-treatment.

【0022】

FIG. 3 shows the main parts of the optical system of the compound eye imaging device shown in FIG. 1 in each imaging lens group 11._L、11_ROptical axis L_L, L_RIt is a view seen from the direction perpendicular to the plane formed by. However, for the sake of brevity, each color separation prism 12_L、12_R(See Fig. 1) The part is omitted and each CCD sensor 13_L、13_RAlso, only one on each side is shown. As shown in FIG. 3, the right imaging system 10_RImaging lens group 11_RAnd CCD sensor 13_R Is the focused object surface 50_R The area that can be imaged is the CCD sensor 13_R Straight line 51 due to the effective light receiving part of_RAnd 52_RLimited to the area sandwiched between and, this focused object surface 50_R Is the end face 51_R And end face 52_R Line of intersection B_R The area from line of intersection A is the effective subject area. Left imaging system 10_L Similarly, for the focused object surface 50_L Line of intersection A to line of intersection B_L The area up to is the effective subject area. Left and right imaging system 10_L、10_RFocus motor 14_L、14_R(See Figure 1) and zoom motor 16_L、16_R(See Fig. 1) shows each in-focus object surface 50_L、50_RAnd CCD sensor 13_L、13_RThe distance to and the image magnification are controlled to be equal to each other on the left and right, and each convergence angle motor 18_L、18_R(See Fig. 1) shows each imaging system 10_L、10_RThe edges of the effective subject area of are controlled so as to coincide with each other at the line of intersection A. Each motor 14_L、14_R、16_L、16_R、18_L、18_RControl of each encoder 24_L、24_R、25_L、25_R、26_L、26_RThis is done through the control unit 20 (see Figure 1), which receives the signal from (see Figure 1). Especially the convergence angle motor 18_L、18_RIs the focus encoder 24_L、24_RAnd zoom encoder 25_L、25_RFocused object surface 50 calculated from the output signal from_L、50_RIt is controlled in conjunction with the signal of the position of and the position of the edge of the effective subject area.

【0023】

In the system configured as described above, each imaging system 10_L、10_ROf the subjects captured in, both images may include the shaded area 53 (end face 51)._L、52_RIt is a subject that has a boundary sandwiched between lines and exists in the area of the distant view from the line of intersection A). Here, the region 53 that may be included in both of these images is the region 53 of each imaging system 10._L、10_RIn consideration of the fact that it extends to infinity, each imaging lens group 11_L、11_RArea 53 is focused on the object surface 50 toward the center of the main plane on the object side of_L、50_RWhen projected onto, the area 53 is the right imaging system 10._RFor, the imaging lens group 11_R Passing through the center of the main plane on the object side of the object, and the end face 51_L Arrow C indicating a plane parallel to_R Is the focused object surface 50_R Line of intersection D_R And area 55 between line A_R Corresponds to. Similarly, the left imaging system 10_L For region 53, the imaging lens group 11_L Passing through the center of the main plane on the object side of, and the end face 52_R Arrow C indicating a plane parallel to_L Is the focused object surface 50_L Line of intersection D_L And area 55 between line A_L Corresponds to. From these things, the apparent area 55_L、55_RWhat is imaged as the upper subject may be a double image, and each of these areas 55_L、55_RIs the double image generation area. Since the other regions are regions that are imaged by only one imaging system, they are regions where double images do not occur.

【0024】

Next, the image processing procedure in the compound eye imaging device of this embodiment will be described with reference to FIG.

【0025】

Left and right imaging system 10_L、10_RWhen an object is imaged with the image processing unit 30, image information is input to the image input unit 32 of the image processing unit 30 in each imaging system 10._L、10_REach image is input as a video signal, and each image memory 31_L、31_RIs held in. Each imaging system at this time 10_L、10_RFig. 4 shows an example of the positional relationship between the object and the object, and (A) in Fig. 4 shows the position of the object when the object is viewed from the same direction as in Fig. 3, (Fig. 4). B) is a view of the effective light receiving portions of the left and right CCD sensors as viewed from the image pickup lens group side when the object at the position shown in (A) of the figure is imaged. As shown in (A) of FIG. 4, the object 60 existing in the region 53 that may be included in both images is apparently left and right imaging systems 10_L、10_RFocused object surface 50_L、50_RArea 56 above_L、56_REach CCD sensor 13 as present in_L、13_RProjected on. That is, as shown in (B) of FIG. 4, the CCD sensor 13 on the right side_R At the line of intersection D_R Area 55 between the line of intersection A and_R Inside, the image 60 corresponding to the object 60_R Is projected and the CCD sensor 13 on the left_L At the line of intersection D_L Area 55 between line of intersection A_L Inside, the image 60 corresponding to the object 60_L Is projected. Here, the left and right CCD sensors 13_L、13_RAre each point L₀, R₀The pixel positions in the horizontal and vertical directions are counted with the origin as the origin, and the image 60 in each image._L、60_RThe position of is the coordinates of the representative points such as the center of gravity of each (m)_L, n_L), (M_R, n_R).

【0026】

This will be specifically described by taking an image of a composition in which a road sign can be seen in the background of an automobile as an example. FIG. 5 is a diagram showing left and right projected images when a composition in which a road sign can be seen in the background of an automobile is imaged._L、50_R(See Figures 3 and 4) Located above. In Figure 5, region 55_L、55_R However, as described above, it is a double image generation region, and the focused object surface 50 of each imaging system is 50._L、50_RSince is on the car 61, the car 61 is smoothly connected in the left and right images, but the road sign 62 is behind the car 61, so it appears twice in both images.

【0027】

Next, the corresponding point extraction unit 33 shown in FIG. 2 extracts the corresponding point pair of each image. There is a template matching method as a typical method of the corresponding point extraction method. In this method, for example, a template surrounding a certain point in the left image is considered, and the corresponding points are determined by comparing the similarity of the template image with the image in the right image. The correlation method, which is a method of comparing similarity, takes a cross-correlation between the pixel value in the template image and the pixel value in the search image, and sets the maximum coordinate as the corresponding point. The relational expression is shown in the following equation (1).

【0028】

[Number 1]

<img file="JPH06141237A_D0001.tif" />In equation (1), R (m_R, n_R), L (m_L, n_L) Is the pixel value of the left and right images, and σ (m)_R, n_R, m_L, n_L) Indicates the degree of correlation. Also, m_R, N_R, M_L, N_LIndicates pixel coordinates. In the sum of squares or sum of products calculation, the signs before i and j are reversed in the left and right images because the pixel coordinate axes shown in (b) of Fig. 4 are defined to be symmetrical. is there. Then, the maximum value is 1 in the normalized cross-correlation of the equation 1.

【0029】

In this embodiment, first, the area 55 of the right image shown in FIG. 4_R Area 55 of the left image for each pixel point in_L Find the value of the number 1 corresponding to all the points in, and find the coordinate pair of the left and right images that is the maximum value. Next, of the obtained coordinate pairs, the value σ of the number 1 is a predetermined threshold value σ._thIn the above case, that is, σ σ_thOnly in the case of, the corresponding point pair that becomes a double image is used. This is the in-focus object surface 50 in the case of the wide image imaging system of this embodiment._L、50_RSince the region where the double image does not occur occupies a considerable part of the region projected on the image, it is the minimum necessary to prevent deterioration of the composite image due to misjudgment as a double image.

【0030】

After extracting the corresponding point pair of both the left and right images, the double image region division portion 34 shown in FIG. 2 divides the double image region. In the process of dividing the double image region, the position of each corresponding point pair in the three-dimensional space is first obtained by the triangulation method shown below.

【0031】

As shown in FIG. 6, the left and right imaging lens groups 11_L、11_R(See Fig. 3) Center point O of the main plane on the object side_L, O_RAre placed line-symmetrically on the X-axis with respect to the Z-axis, and their center point O_L, O_RIf the length of the baseline connecting between them is the baseline length b, each center point O_L, O_RThe coordinates of are represented by (-b / 2,0,0) and (b / 2,0,0), respectively. Also, one point P in the three-dimensional space is the center point O._L, O_RLeft and right CCD sensors 13 when projected toward_L、13_RThe projection points on each are P_L, P_RAnd points P, P_L, P_RCoordinates of (X, Y, Z), (X) respectively_L, Y_L, Z_L), (X_R, Y_R, Z_R). Here, points P, P in three-dimensional space_L, P_RThe plane formed by connecting the three points is called the epipolar surface, and the line of intersection between the epipolar surface and the sensor surface is called the epipolar line.

【0032】

At this time, the coordinates (X, Y, Z) of the point P are given by the following equations (2), (3), and (4), respectively.

【0033】

[Number 2]

<img file="JPH06141237A_D0002.tif" />On the other hand, the left and right imaging lens groups 11_L、11_ROptical axis L_L, L_RHowever, the center point O of the main plane on the object side, respectively._L, O_RLet θ be the angle formed by the straight line parallel to the Z axis (this is called the convergence angle), and each imaging lens group 11_L、11_RLet f be the focal length of Z_R = {X_R -(B / 2) + f sin (θ)} tan (θ) + f cos (θ) Z_L =-{X_L + (b / 2) -f sin (θ)} tan (θ) + f cos (θ) The relationship of is established, and the coordinates (X, Y, Z) of the point P can be obtained by each of the above equations.

【0034】

Then, a histogram is obtained in which the point P (X, Y, Z) is the position of each corresponding point pair of the double image and the Z coordinate value (distance in the depth direction) is used as a parameter. From this histogram, the area is divided by the area where the depth distance becomes discontinuous.

【0035】

For example, in the case of the image shown in FIG. 5, as shown in FIG. 7, a histogram divided into two regions of α and β can be obtained. That is, the region α shown in FIG. 7 corresponds to the automobile 61 shown in FIG. 5, and the region β shown in FIG. 7 corresponds to the road sign 62 shown in FIG.

【0036】

After the area division is performed, the double image type determination unit 35 shown in FIG. 2 divides each area-divided double image into the following four types. That is, in each image shown in FIG. 4 (B), (Type 1) Line of intersection D_RLine of intersection D, including the top point_LA double image that does not include the upper point. (Type 2) Line of intersection D_RLine of intersection D, not including the top point_LDouble image including the upper point. (Type 3) Line of intersection D_RLine of intersection D, not including the top point_LA double image that does not include the upper point. (Type 4) Line of intersection D_RLine of intersection D, including the top point_LDouble image including the upper point. There are four types. Then, for the double image classified into type 1, the total area S1 is obtained, and for the double image classified into type 2, the total area S2 is obtained. In this embodiment, no particular processing is performed on the double images classified into type 3 and type 4.

【0037】

Next, when the area S1 obtained by the double image type determination unit 35 is larger than or equal to the area S2 by the single image double image batch removal unit 36 shown in FIG. 2, the double image existing in the right image is present. Only the corresponding points are left, and the corresponding points in the left image are removed. On the contrary, when the area S1 is smaller than the area S2, only the double image corresponding points existing in the left image are left, and the corresponding points in the right image are removed. This is because when the area of the double image classified into type 1 is large, the double image in the right image connects more to the area where the double image does not occur, and the double image classified into type 2 is connected. When the area of the image is large, it is based on the idea that the double image in the left image is connected to the area where the double image does not occur more.

【0038】

When the corresponding point in one image is removed, the removed pixel is subjected to post-processing such as replacement with a value from a peripheral pixel by the removal post-processing unit 37 shown in FIG. After that, the left and right images are connected, held in the composite image memory 39, and output to the image output unit 40 of a monitor or the like.

【0039】

FIG. 8 shows an image in which the left and right images shown in FIG. 5 are connected by performing the above-mentioned processing. As shown in FIG. 8, the in-focus object surface 50_L、50_RThe road sign behind (see Fig. 3) is divided into the above-mentioned type 2 and the area 55 in the left image._R The double image 62'inside is removed. On the other hand, the automobile 61 is classified into type 4, so no special treatment is performed, but the in-focus object surface 50_L、50_RSince it is on the top, it is continuously connected. As a result, a natural image without the occurrence of a double image can be obtained.

【0040】

By removing one of the double images by the process as described above, it is possible to leave the double image component having more connected components in the region where the double image does not occur in the left and right images. It is possible to obtain a composite image having many parts that are smoothly connected to a region where no image is generated.

【0041】

(Second Example) Next, a second example of the compound eye imaging device of the present invention will be described. FIG. 9 is a block diagram of an image processing unit of a second embodiment of the compound eye imaging device of the present invention. This embodiment is the same as that of the first embodiment shown in FIGS. 1, 3 and 4, except that the configuration of the image processing unit is different, and the other configurations are the same. 1 The same reference numerals as those of the embodiment and the description thereof will be omitted, and the configuration of the image processing unit will be described below.

【0042】

As shown in FIG. 9, the image processing unit 130 includes each imaging system 10._L、10_RCCD sensor 13_L、13_RLeft image memory 31 that holds each video signal from (see Fig. 1)_LAnd right image memory 31_RThe image input unit 32 is composed of an image input unit 32, an image conversion unit 138 for generating one composite image based on each of the left and right images obtained by each video signal input to the image input unit 32, and an image conversion unit 138. It has a composite image memory 39 for holding the image and outputting it to the image output unit 40. The image conversion unit 138 further includes a correspondence point extraction unit 133 that extracts a correspondence point pair between both images for all pixels in the double image generation region among both images input to the image input unit 32, and a correspondence point. Based on the double image region division unit 134 that calculates the two-dimensional position of each corresponding point from the result of pair extraction and divides the double image region based on that information, and the region divided by the double image region division unit 134. It is divided into a one-sided image removing unit 136 that leaves at least one double image and uses it as composite image generation data, and a removal post-processing unit 137 that performs post-processing on the image after the double image is removed.

【0043】

Next, the image processing procedure of this embodiment will be described. In the arrangement shown in (A) of FIGS. 3 and 4, the main subject is often located in the center of the panoramic composite image and in the in-focus portion centered on the line of intersection A. Therefore, when another subject exists in the background of the main subject, the image of another subject appears on the right side of the main subject in the right image, and the image of another subject appears on the left side of the main subject in the left image. The subject is a double image. In this case, as shown in FIG. 10, if the area 53 shown in FIG. 3 is bisected to the left and right and each is defined as the left area 53L and the right area 53R, the image of another subject in the right image is the right area 53R. It is considered that the image is continuously or smoothly connected to the image from the region on the right side, and similarly, another subject in the left image is considered to be connected to the image from the left side of the left region 53L.

【0044】

Based on the above considerations, in this embodiment, image reconstruction is performed with an emphasis on image continuity. Therefore, the image of the subject existing in the right region 53R is left in the right image shown in (B) of FIG. 4, and the image of the subject existing in the left region 53L is in the left image shown in (B) of FIG. The process of leaving it in is performed. To determine whether the subject exists in the right region 53R or the left region 53L, the corresponding point pair of the double image is obtained by the corresponding point extraction method similar to that of the first embodiment, and in (B) of FIG. Image shown 60_L Representative point coordinates (m_L , n_L ) And statue 60_R Representative point coordinates (m_R, n_R) Is considered here as a corresponding point coordinate pair after the corresponding point extraction process, and as an actual process, the image 60_L、60_RThe next branching process may be performed in view of the positional relationship of. (a) m_R &gt; m_L If so, leave it in the image on the right. (b) m_R &lt;m_L If so, leave it in the left image. (cm_R = m_L If so, leave it in both the left and right images.

【0045】

Then, the removed pixels may be subjected to post-processing in which the removed pixels are replaced with, for example, the signal values of the peripheral pixels. By synthesizing the left and right images configured by the above processing so as to join them on the lines of intersection A and A', there is no discontinuity in the main subject area and discontinuity occurs in the background image area. That is, a panoramic composite image in which the generation of double images is minimized can be obtained.

【0046】

A further effect of this embodiment is the focused object surface 50._R、50_LSince the intersection lines A and A', which are the edges of the main subject, are controlled so as to always coincide with each other, the main subject image is always continuous and smooth, and the focused object surface 50 due to the change in the distance of the main subject._L、50_RThe point is that an image with a certain angle of view can always be taken even when moving.

【0047】

(Third Example) Next, a third embodiment of the compound eye imaging device of the present invention will be described. In this embodiment, the convergence angles of the left and right imaging systems are different from those in the second embodiment, which will be described below with reference to FIG. FIG. 11 is a view of the main part of the optical system of the third embodiment of the compound eye imaging device of the present invention as viewed from a direction perpendicular to the plane formed by the optical axis of each imaging lens group. In FIG. 11, the same parts as those shown in FIG. 3 are designated by the same reference numerals, and the schematic configuration of the apparatus (not shown) is the same as that of the second embodiment, and thus the description thereof will be omitted.

【0048】

As shown in FIG. 11, the left and right imaging systems 10_L、10_RConvergence angle of each imaging system 10_L、10_REnd face 51, which is one end face of the effective subject area of_LAnd 52_REach imaging system 10_L、10_RFocused object surface 50 at close-up shooting_L'And 50_RIt is controlled to match the line of intersection A'with'. In this case, each imaging lens group 11_L、11_RThe focusing distance changes due to zooming, and the end face 51, which is one end face of the effective subject area_LAnd 52_RSince the angle of view changes as the effective angle of view fluctuates due to zooming, the convergence angle control is performed in conjunction with the zoom control. Focus and zoom control are performed in the same manner as in the first embodiment. Also, each focused object surface 50_L、50_RThe position of each focus encoder 24_L、24_R(See Figure 1) and each zoom encoder 25_L、25_RCalculated from the output signal value (see Fig. 1).

【0049】

In the configuration of this embodiment, an object existing in the area 53'shown in the shaded area in FIG. 11 may be photographed as a double image. The object existing in this region 53'is apparently the focused object surface 50 of each of the left and right imaging systems, as in the first embodiment._L、50_RArea 55 above_L、55_REach CCD sensor 13 as present in_L、13_RProjected on. FIG. 12 is a view of the effective light receiving portion of each of the left and right CCD sensors shown in FIG. 11 as viewed from the image pickup lens group side. Here, the pixel coordinate values of the left and right images are each (m) as in the case of (B) in FIG._L, n_L), (M_R, n_R).

【0050】

Next, the image processing procedure of this embodiment will be described with reference to FIGS. 11 and 12. In this embodiment, the following three processes are performed in order to generate a panoramic composite image from each of the left and right images. Area 55_L、55_RExtraction of corresponding point pairs in. Area 55_L、55_RRemoval of one image based on the positional relationship of the corresponding point pair inside. Focused object surface 50_L、50_RCompositing process that continuously connects images on the line of intersection A.

【0051】

Of these, the corresponding point extraction process can be realized by the same process method as in the second embodiment.

【0052】

Next, the removal process of one of the images is shown.

【0053】

The main subject is in the center of the panoramic composite image as in the second embodiment, and the left and right imaging systems 10_L、10_RFocused object surface 50_L、50_RIn many cases, it is located near the in-focus part centered on the line of intersection A. Therefore, in this embodiment as well, image reconstruction is performed with an emphasis on image continuity, based on the same consideration as shown in the second embodiment. Therefore, the area 53'is set to the right area 53 as in the second embodiment._L'And left area 53_RDivide it into two equal parts, and based on the pixel coordinate values described above, (a) m_R &gt; m_L If so, leave it in the image on the right. (b) m_R &lt;m_L If so, leave it in the left image. (cm_R = m_L If so, leave it in both the left and right images. Perform three branch processing.

【0054】

Next, the continuous synthesis process of is shown.

【0055】

First, in FIG. 12, the area on the left side of the line of intersection A in the right image and the area on the right side of the line of intersection A in the left image are deleted. Next, by matching and synthesizing both deleted images at the line of intersection A, discontinuity does not occur in the main subject area, and discontinuity occurs in the background image area, that is, double image occurs. A minimal panoramic composite image is obtained.

【0056】

(Fourth Example) Next, a fourth embodiment of the compound eye imaging device of the present invention will be described. In this embodiment, the convergence angles of the left and right imaging systems are different from those in the second embodiment and the third embodiment, which will be described below with reference to FIG. FIG. 13 is a view of the main part of the optical system of the third embodiment of the compound eye imaging device of the present invention as viewed from a direction perpendicular to the plane formed by the optical axis of each imaging lens group. In FIG. 13, the same parts as those shown in FIG. 3 are designated by the same reference numerals, and the schematic configuration of the apparatus (not shown) is the same as that of the second embodiment, and thus the description thereof will be omitted.

【0057】

As shown in FIG. 13, each of the left and right imaging systems 10_L、10_RConvergence angle of each imaging system 10_L、10_REnd face 51, which is one end face of the effective subject area of_R、52_LAre controlled to intersect at infinity, that is, to be parallel to each other. In this case, the convergence angle control needs to be performed in conjunction with the zoom control as described in the third embodiment. Focus and zoom control are performed in the same manner as in the first embodiment.

【0058】

In the configuration of this embodiment, an object existing in the area 53'' shown by the shaded area in FIG. 13 may be photographed as a double image. When this region 53'' is projected onto the CCD sensors 13 and 13 through the imaging lens groups 11 and 11, this region 53'' extends over the entire effective light receiving portion of the CCD sensors 13 and 13. FIG. 14 is a view of the effective light receiving portion of each of the left and right CCD sensors shown in FIG. 13 as viewed from the image pickup lens group side. Here, the pixel coordinate values of the left and right images are each (m) as in the case of (B) in FIG._L, n_L), (M_R, n_R).

【0059】

Next, the image processing procedure of this embodiment will be described with reference to FIGS. 13 and 14. In this embodiment as well, as in the third embodiment, the following three processes are performed in order to generate a panoramic composite image from each of the left and right images. Extraction of corresponding point pairs over the entire image. Removal of one image based on the positional relationship of the corresponding point pair over the entire image. Focused object surface 50_L、50_RCompositing process that continuously connects images on the line of intersection A.

【0060】

Corresponding point extraction processing can be realized by the same processing method as in each of the above-described examples, but in this embodiment, it is necessary to perform the range for the entire effective light receiving portion.

【0061】

Regarding the removal process of one of the images, based on the same consideration as in the second embodiment, emphasis is placed on the continuity of the image, and for each corresponding point coordinate pair, (a) m_R &gt; m_L If so, leave it in the left image. (b) m_R &lt;m_L If so, leave it in the image on the right. (cm_R = m_L If so, leave it in both the left and right images. The branching process should be performed. In the case of this embodiment, this process is reversed from that of the second embodiment and the third embodiment.

【0062】

In FIG. 14, after deleting the area to the right of the line of intersection A in the right image and the area to the left of the line of intersection A in the left image, both the deleted images are matched by the line of intersection A. By arranging them upside down and synthesizing them in this way, the panorama has no discontinuities in the main subject area and minimizes the occurrence of discontinuities in the background image area, that is, the occurrence of double images. A composite image is obtained.

【0063】

(Fifth Example) Next, a fifth embodiment of the compound eye imaging device of the present invention will be described. FIG. 15 is a block diagram of an image processing unit of a fifth embodiment of the compound eye imaging device of the present invention. This embodiment is the same as that of the first embodiment shown in FIGS. 1, 3 and 4, except that the configuration of the image processing unit is different, and the other configurations are the same. 1 The same reference numerals as those of the embodiment and the description thereof will be omitted, and the configuration of the image processing unit will be described below.

【0064】

As shown in FIG. 15, the image processing unit 230 includes each imaging system 10._L、10_RCCD sensor 13_L、13_RLeft image memory 31 that holds each video signal from (see Fig. 1)_LAnd right image memory 31_RThe image input unit 32 is composed of an image input unit 32, an image conversion unit 238 for generating one composite image based on each of the left and right images obtained by each video signal input to the image input unit 32, and an image conversion unit 238. It has a composite image memory 39 for holding the image and outputting it to the image output unit 40. The image conversion unit 238 further includes a correspondence point extraction unit 233 that extracts a correspondence point pair between both images for all pixels in the double image generation region among both images input to the image input unit 232, and a correspondence point. The double image region division unit 234 that calculates the three-dimensional position (distance information) of each corresponding point pair from the result of pair extraction and divides the double image area according to the information, and each double image divided into regions. However, the double image generation region of both images (region 55 shown in (B) of FIG. 4)_L、55_R) Only, the double image non-generation area of one image (the area excluding the double image generation area), or the double image non-generation area of both images. The double image is a double image of both images based on the type classified by the double image type judgment unit 235 and the double image type judgment unit 235, which are classified into three types, which are also included in. Except when it is also present in the non-image generation region, the double image removal unit 236 that removes one of the double images and the removal unit when the double image is removed are replaced with each other to replace the double image. If is not removed, it is divided into a removal post-processing unit 237 that performs post-treatment that laterally scales a predetermined portion.

【0065】

Next, the image processing procedure in the compound eye imaging device of this embodiment will be described.

【0066】

First, the left and right imaging systems 10_L、10_RWhen an object is imaged with, image information is sent to the image input unit 32 of the image processing unit 230 in each imaging system 10._L、10_REach image is input as a video signal, and each image memory 31_L、31_RIs held in.

【0067】

Next, the corresponding point extraction unit 233 extracts the corresponding points of the double image generationable region by the same template matching method or the like as described in the first embodiment, and further, the double image region division unit 234 performs the corresponding point extraction. In the same manner as described in the first embodiment, the position of the corresponding point pair of each double image in the three-dimensional space is obtained by a triangulation method or the like, and a histogram using the Z coordinate value as a parameter is obtained. The area of the histogram is divided.

【0068】

After dividing the area of the double image, the double image type determination unit 235 divides each of the divided double images into the following four types. That is, in each image shown in FIG. 4 (B), (Type 1) Line of intersection D_RLine of intersection D, including the top point_LA double image that does not include the upper point. (Type 2) Line of intersection D_RLine of intersection D, not including the top point_LDouble image including the upper point. (Type 3) Line of intersection D_RLine of intersection D, not including the top point_LA double image that does not include the upper point. (Type 4) Line of intersection D_RLine of intersection D, including the top point_LDouble image including the upper point. There are four types.

【0069】

Then, the double image removing unit 236 and the post-removal processing unit 237 shown in FIG. 15 perform the following processing, which is different for each of the four types classified by the double image type determining unit 235. (Type 1 processing) The double image in the right image is left, and the double image in the left image is removed. In the left image after removal, the pixels in the removal portion are subjected to post-processing such as replacement with values from peripheral pixels. (Processing of type 2) Type 2 corresponds to the road sign 62 in the image shown in FIG. 5, and in this case, the processing of type 1 and the left-right reverse processing are performed. (Type 3 processing) One of the left and right double images is left, and the other double image is removed. The post-processing is the same as the type 1 processing. (Processing of type 4) The line of intersection D shown in (B) of Fig. 4 by magnifying the double image of either the left or right side twice in the horizontal direction._L, D_RSynthesize so that it touches. Alternatively, in order to smoothly connect to the area where the double image does not occur, conversion is performed from the corresponding point-to-coordinates of both the left and right images with the mapping shown below, and then the line of intersection D is enlarged twice in the horizontal direction._L, D_R It may be synthesized so as to be in contact with. As a mapping for that, for example X_S = (X_L D_R + X_R D_L ) / (D_R + d_L ) Y_S = (Y_L D_R + Y_R D_L ) / (D_R + d_L ) There is. Here, each region 55 shown in FIG. 4 (B)_L、55_RThe upper left point of each area 55_L、55_RWhen the origin is set to, the coordinates of the double image in the right image are (X)._R, Y_R ), The coordinates of the double image in the left image (X)_L , Y_L ), The coordinates of the composite image from the double image (X)_S , Y_S ). Also, d_R Is the coordinates of the double image in the image on the right (X)_R, Y_R) And line of intersection D_RDistance to, d_L Is the coordinates of the double image in the left image (X)_L, Y_R) And line of intersection D_RIs the distance to.

【0070】

As explained above, region 55_L、55_RIf the double image inside is included in the double image non-generation area of both images, the double image generation area is not double image generation by enlarging the double image twice in the horizontal direction. A composite image that is smoothly connected to the generation area can be obtained.

【0071】

(Sixth Example) Next, a sixth embodiment of the compound eye imaging apparatus of the present invention will be described. In this embodiment, the processing for type 4 classified in the fifth embodiment is different. Since other configurations and processes are the same as those in the fifth embodiment, the description thereof will be omitted, and the characteristic parts of the present embodiment will be described below.

【0072】

FIG. 16 is a diagram for explaining the image composition process of this embodiment, and is an image of a composition in which mountains can be seen in the background of an automobile, which is the main subject. In this case, by dividing the double image into regions in the same manner as in the fifth embodiment, as shown in FIG. 16 (A), the left and right images are divided into regions mainly including the automobile as the main subject and regions. It is divided into upper and lower parts with an area that does not include automobiles. Furthermore, the region 55, which is a region where double images can be generated._L、55_RIn the above region, the lower region is α and α', and the upper region is divided into the mountain region β and β'and the background region γ and γ'. Also, area 55_L、55_ROf the regions excluding the above, that is, the regions where the double image does not occur, the upper regions are defined as δ and δ', respectively. The pairs of β and β'and the pairs of γ and γ'in FIG. 16 (A) are double images classified into type 4 described in the fifth embodiment, respectively. In this embodiment, this is the case. Instead of magnifying only one of the double images in the horizontal direction twice, the width of one double image (shown by the diagonal line in the figure is l).₂After removing (the part), in the right image, the right end of the lower area (indicated by the diagonal line in the figure, the width is l).₁₂In the left image, the left end of the lower area (indicated by the diagonal line in the figure, the width is l) is removed.₁₁Part of. However, l₁₁= l₁₂, L₁₁+ l₁₂= l₂) Is removed and the entire image spreads to the left and right of the double image at the same ratio to the upper and lower regions, that is, the entire region δ on the right side of the regions β and γ and the entire region on the left side of the regions β'and γ' δ'is also laterally multiplied. Then, by enlarging and synthesizing this, as shown in FIG. 16 (B), a panoramic composite image smoothly connected at the center of the image can be obtained. If there is a part that is not classified into type 4, the removal process and post-treatment shown in the fifth embodiment are performed.

【0073】

In this embodiment, the edge of the non-double image generation portion is removed in the lower region, but since the main subject is usually placed in the center of the image, the edge of the image is removed. But it doesn't have much effect. Further, in this embodiment, an example in which the image is divided into two upper and lower regions is shown, but the same processing may be performed when the image is divided into three or more regions. Further, not only the horizontal scaling process but also the vertical scaling at the same ratio as the horizontal scaling process is performed, and the upper and lower ends of the image are cut to obtain a composite image, whereby an image without distortion in the aspect ratio can be obtained.

【0074】

(7th Example) Next, a 7th example of the compound eye imaging device of the present invention will be described. FIG. 17 is a block diagram of an image processing unit of a seventh embodiment of the compound eye imaging device of the present invention. In this embodiment, the right imaging system 10_R And left imaging system 10_L One high-definition image is obtained by synthesizing two images obtained by imaging a subject using the two imaging systems of the above, and the configuration is the same as that of the first embodiment shown in FIG. , Other configurations are the same except that the configuration of the image processing unit is different. Therefore, the same parts are designated by the same reference numerals as those in the first embodiment, and the description thereof is omitted. The structure of the part will be described.

【0075】

As shown in FIG. 17, the image processing unit 130 includes each imaging system 10._L、10_RCCD sensor 13_L、13_RLeft image memory 31 that holds each video signal from (see Fig. 1)_LAnd right image memory 31_RThe image input unit 32 is composed of an image input unit 32, an image conversion unit 138 for generating one composite image based on each of the left and right images obtained by each video signal input to the image input unit 32, and an image conversion unit 138. It has a composite image memory 39 for holding the image and outputting it to the image output unit 40. The image conversion unit 138 further includes a corresponding point extraction unit 133 that extracts a corresponding point pair between both images for all pixels in a region that can be a subject of both imaging systems among both images input to the image input unit 132. , The proximity correspondence point determination unit 334 that determines whether the position of each correspondence point is separated from the pixel position of the basic image by a predetermined distance from the result of the correspondence point pair extraction, and the proximity correspondence point determination unit 334. When it is determined that the position of the corresponding point is not separated by a predetermined distance from the pixel position of the basic image, the proximity corresponding point removing unit 335 that removes the corresponding point from the composite image data, The corresponding points that are not removed by the proximity corresponding point removing unit 335 are used as data for a composite image, and are divided into an image interpolation unit 336 that performs interpolation processing on the basic image.

【0076】

FIG. 18 is a view of the main part of the optical system of this embodiment as viewed from a direction perpendicular to the plane formed by the optical axis of each imaging lens group. As shown in FIG. 18, each of the left and right imaging systems 10_L、10_RConvergence angle of each optical axis L_L, L_RAre controlled to be parallel to each other. In addition, each imaging system 10_L、10_RFocus and zoom of each focused object surface 50_L、50_RAnd CCD sensor 13_L、13_RThe distance to and the image magnification are controlled to be equal to each other on the left and right, and each in-focus object surface 50_L、50_RIs formed on the same plane. Each motor for controlling focus and zoom 14_L、14_R、16_L、16_R(See FIG. 1) is the same as that of the first embodiment._L、24_R、25_L、25_RIt is driven through the control unit 20 (see Figure 1) that receives the signal from (see Figure 1). In this embodiment, the shaded area 58 is the left and right imaging system 10._L、10_RThis is a region that can be the subject of the image, and the subject existing in this region 58 is the subject of the high-definition image.

【0077】

Next, the image processing procedure in the compound eye imaging device of this embodiment will be described with reference to FIGS. 17 and 18.

【0078】

First, the left and right imaging systems 10_L、10_RWhen an object is imaged with, image information is sent to the image input unit 32 of the image processing unit 330 in each imaging system 10._L、10_REach image is input as a video signal, and each image memory 31_L、31_RIs held in.

【0079】

Next, the corresponding point extraction unit 333 extracts the corresponding point pair of the region 58 by the same template matching method or the like as described in the first embodiment. However, in this embodiment, pixel relative position accuracy at one pixel pitch or less is required. Therefore, in order to perform matching by image low frequency components with few errors, before increasing the template size or performing matching calculation. The low frequency component extraction process of the image may be performed.

【0080】

Then, based on this correspondence point pair extraction, the right imaging system 10_RLeft image system 10 on the right image of_LA high-definition image is obtained by synthesizing the left image of the above, and this process will be described with reference to FIG. FIG. 19 is a diagram showing the pixel positions of each CCD sensor in a part of the left and right images obtained by the optical system shown in FIG. The points (l, m) marked with x in the figure are the right imaging system 10 which is the basic image of the high-definition composite image._RIndicates the pixel position in the image, and the points marked with a circle indicate the pixel position (l, m)'that you want to add in the high-definition composite image. Here, l and m represent pixel coordinates. On the other hand, the point Q indicated by the square mark is the left imaging system 10 obtained by the above-mentioned corresponding point extraction._L Indicates the pixel position in the image of.

【0081】

Here, first, the proximity correspondence point determination unit 334 obtains the distance of the square mark point Q with respect to any of the × mark points, and determines whether or not the distance is 1/10 or less of the pixel pitch p. As a result, when the distance is 1/10 or less of the pixel pitch p, the square mark point Q is removed from the composite image data by the proximity corresponding point removing unit 335, and the mark point (l, m)' It is not used as a parameter to obtain the value of. On the other hand, when the distance exceeds 1/10 of the pixel pitch p, this square mark point Q is used as it is as a parameter for obtaining the value of the mark point (l, m)'. Next, the image interpolation unit 336 makes the right imaging system 10_RLeft imaging system 10 for the image of_LImage interpolation processing is performed, but regardless of whether the square mark point Q is used or not, as a complementary method for obtaining the value of the mark point, for example, coordinate conversion for a two-dimensional unequally spaced sampling point is performed. The method (IEEE Trans, Acoust, 33 (85) 1151) etc. can be used. Here, the left imaging system 10 that is not the basic image_LOf the images from, right imaging system 10_ROf course, pixel points that do not have corresponding points on the basic image side from or image points that cannot be obtained are not used as parameters for determining the value of mark points (1, m)'.

【0082】

In the present embodiment shown above, when the relative positions of the pixels in the left and right images are very close to each other, an unnecessary high frequency generated in response to an error in the relative pixel positions or a noise component on which the pixel values are superimposed is generated. It has the effect of suppressing the components. As a result, it is possible to prevent a situation in which the image quality of the composite image deteriorates more than that of the basic image.

【0083】

In this embodiment, the case where the right image is used as the basic image is shown, but of course, the reverse image composition may be performed using the left image as the basic image.

【0084】

(Eighth Example) Next, the eighth embodiment of the compound eye imaging device of the present invention will be described. This embodiment is also a compound eye imaging device for obtaining a high-definition image as in the seventh embodiment, and the convergence angles of the left and right imaging systems are different from those of the seventh embodiment. FIG. 20 is a view of the main part of the optical system of the eighth embodiment of the compound eye imaging apparatus of the present invention as viewed from a direction perpendicular to the plane formed by the optical axis of each imaging lens group, and is the same as that of the seventh embodiment. The same reference numerals are given to the parts.

【0085】

As shown in FIG. 20, each of the left and right imaging systems 10_L、10_RRight imaging system 10_RConvergence angle is fixed at zero, right imaging system 10_ROptical axis L_RTo the subject (not shown), right imaging system 10_RIs set to always face the front of the subject. On the other hand, the left imaging system 10_LConvergence angle is right imaging system 10_ROptical axis L_RLeft imaging system 10_LOptical axis L_LIs both imaging systems 10_L、10_RFocused object surface 50_L、50_RIt is controlled to intersect at the intersection of. Left imaging system 10_LFocus motor 14_L(See Figure 1) and zoom motor 16_L(See Fig. 1) shows the right imaging system 10_RFocus motor 14_R(See Figure 1) and zoom motor 16_RIn conjunction with the setting (see Fig. 1), the above-mentioned focused object surface 50_L、50_RAnd optical axis L_L, L_RIt is controlled to maintain the positional relationship of. Each motor 14_L、14_R、16_L、16_RThe control of each encoder 24 is the same as in the seventh embodiment._L、24_R、25_L、25_RThis is done through the control unit 20 (see Figure 1), which receives the signal from (see Figure 1).

【0086】

In this embodiment as well, the right imaging system 10_RA high-definition image is generated using the input image from the above as a basic image, and the method of image generation is the same as that of the seventh embodiment, so the description thereof will be omitted.

【0087】

In this embodiment, since the image from the imaging system with the fixed optical axis is used as the basic image, at least the image from the front is always output without adding any special processing even if there is a part where high-definition composition cannot be performed. it can. Further, since the image from the front of the device is used as the basic image, the photographer can take a picture without receiving an unnatural feeling of operation even at the time of close-up shooting with the compound eye imaging system.

【0088】

(Ninth Example) Next, a ninth embodiment of the compound eye imaging device of the present invention will be described. FIG. 21 is a block diagram of an image processing unit of a ninth embodiment of the compound eye imaging device of the present invention. In this embodiment, only the configuration of the image processing unit is different from that shown in the 7th and 8th embodiments, and the arrangement of the optical system and other configurations are the same. The same reference numerals as those of the first embodiment and the description thereof will be omitted, and the differences from the seventh embodiment and the eighth embodiment will be described below.

【0089】

(1) In front of the corresponding point extraction unit 433, a corresponding point non-extraction area determination unit 437 is provided to determine an area where the corresponding point pair is not extracted based on the distortion amount signal from the control unit 20, and the image distortion is caused. One image is not used as composite image data for a part that requires large and complicated correction.

【0090】

In the input image, in the peripheral part of the image, the distortion of the left and right imaging lens groups (not shown) and the distortion of the image due to the optical axes not being parallel to each other become large, so it is high before performing the corresponding point pair extraction. It is necessary to perform accurate distortion correction. On the other hand, when capturing a normal natural image or the like, the resolving power required for the peripheral portion is lower than that for the central portion of the image, so that high-definition composition at the peripheral portion of the image may not be required so much. In response to such a situation, an output image is obtained by using only one input image in the peripheral portion of the image. Distortion correction is required even in this case, but the required accuracy is lower than that in the case of compositing, so that the purpose can be achieved by using a simpler processing method.

【0091】

(2) After the corresponding point pair extraction by the corresponding point extraction unit 433, an image sharpness judgment unit 434 that estimates the degree of image blur from the F value (numerical aperture) signal from the control unit 20 is provided, and from the in-focus object surface. One of the images is not used as the composite image data for the part where the image is out of focus and the image is blurred.

【0092】

The image sharpness determination unit 434 calculates how far the corresponding point on the subject is from the in-focus object surface from the corresponding point pair extraction information performed at each point on the input image, and this distance information and each of the left and right sides. The degree of image blur is estimated from the F value (numerical aperture) of the imaging lens group. By comparing this with the pixel density of the CCD sensor (not shown), an output image is obtained using only one input image for the portion where the synthesis effect is judged to be small. As a result, some compositing processes such as interpolation can be omitted.

【0093】

(3) A certain frequency or higher from the occlusion area determination unit 435a that determines whether or not the occlusion area exists after the corresponding point pair extraction by the corresponding point extraction unit 433 and the occlusion area determined by the occlusion area determination unit 435a. A region that appears with the components of is calculated, and an occlusion generation frequency determination unit 435b is provided for that region so that one of the images is not used as composite image data.

【0094】

The occlusion area refers to an area consisting of points in one image when the points in the other image cannot be found by the corresponding point pair extraction process. In a subject in which this occlusion region appears in a complicated and complicated manner in an image, erroneous correspondence of corresponding points is likely to occur, which causes deterioration of the composite image. Therefore, only one input image is used in the complex occlusion area. As an example of a specific method, for each pixel point of each input image, the presence or absence of a corresponding point in the other image is stored in the memory, and the local spatial frequency of the change in the presence or absence of the corresponding point is constant. Calculate the region with components above the frequency. Then, in this area, output processing is performed using only one input image.

【0095】

The above-mentioned changes in the processing procedure due to image blurring or occlusion in (2) and (3) can be performed only for the corresponding area, or the ratio of that area in the image exceeds a certain limit. In some cases, the selection may be made to obtain an output image using only one input image for the entire image. In any case, one or both input images are selected in anticipation of the image quality improving effect or the degree of image quality deterioration of the composite output image.

【0096】

In addition, although three methods have been shown in this example, it is not always necessary to perform these methods at the same time, and each method has an independent effect.

【0097】

Of the examples described above, the left and right imaging systems 10_L、10_ROptical axis L_L, L_RIn each of the examples except the seventh embodiment in which the left and right imaging systems are arranged so as to be parallel to each other, the left and right imaging systems 10_L、10_RFocused object surface 50_L、50_RAre not on the same plane. Therefore, some geometric distortion occurs in the image connection portion. Further, in the case of the eighth embodiment in which one of the imaging systems is set to face the front of the subject, at least one of the corresponding points in the subject is slightly blurred. To prevent this, the left and right imaging systems 10_L、10_RFocused object surface 50_L、50_RLeft and right imaging systems 10 so that_L、10_RCCD sensor 13_L、13_RThe light receiving surface of the above can be arranged at a predetermined angle with respect to the respective optical axes L and L. For geometric distortion, distortion correction processing can also be performed on the input image.

【0098】

Further, in the first embodiment and the like, the left and right imaging systems 10_L、10_RFocused object surface 50_L、50_RAlthough the ends of these are arranged so as to coincide with each other, in order to absorb the error occurrence of the mechanism control system, these focused object surfaces 50_L、50_RIt is also possible to arrange a part of the ends of the above in an overlapping manner, perform image correspondence in the overlapping portion, and perform a composition process based on the image correspondence.

【0099】

Further, the optical arrangements of the respective examples described above are the left and right imaging systems 10 respectively._L、10_RAre arranged symmetrically, and each imaging system 10_L、10_ROptical axis L_L, L_REach CCD sensor 13 is symmetrical about the center_L、13_RHowever, the present invention is not limited to this arrangement. For example CCD sensor 13_L、13_RBy arranging the light axes L and L at a certain angle with respect to the optical axes L, the left and right focused object surfaces 50_L、50_RCan also be formed on one plane. Also, CCD sensor 13_L、13_RCan be arranged so as to be perpendicular to the optical axis.

【0100】

Further, in each of the above-described embodiments, two imaging systems 10_L、10_RAlthough the compound eye imaging system consisting of the above is shown, the number of imaging systems is not limited to this, and it is possible to use three or more imaging systems. CCD sensor 13 for the image sensor_L、13_RFor example, an image pickup device other than the above may be used, or an image pickup device in which the color separation system is removed and a mosaic-like color filter is arranged in the light receiving portion can also be used.

【0101】

[Effect of the invention]

Since the present invention is configured as described above, the effects described below are obtained.

【0102】

Corresponding point pairs corresponding to the same part of the subject are extracted from each image signal, and at least a part of the corresponding corresponding point pairs extracted is at least one corresponding point pair in the corresponding point pair. It is possible to prevent deterioration of the composite image by having an image processing unit that uses the above as the composite image generation data and does not use the others as the composite image generation data.

【0103】

In the generation of the panoramic composite image, the double image generated in the vicinity of the connection portion of the two images can be removed, and the good connectivity of the image with the region where the double image does not occur can be maintained.

【0104】

Further, in the generation of a high-definition image, when the corresponding points in the corresponding point pair are very close to each other, at least one corresponding point in the corresponding point pair is used as the composite image generation data, and the other corresponding points are generated. By not using the corresponding points as the composite image generation data, it is possible to suppress an unnecessary high-frequency component generated in response to an error in the relative pixel position of the corresponding point pair or a noise component superimposed on the pixel value. Furthermore, in the generation of high-definition images, it is expected from the corresponding point pair that requires distortion correction before extracting the corresponding point, or the distance from the in-focus object surface of each imaging system and the number of openings of each imaging optical system. Corresponding point pair in which the degree of image blur exceeds a certain threshold, pixel points where a paired corresponding point cannot be obtained, and a region where the paired corresponding point cannot be obtained, alternating at a certain spatial frequency or higher. For at least a part of the corresponding point pair appearing in, by using one corresponding point in the corresponding point pair as the composite image generation data, the effect in the distorted image part, the blurred image part, and the occlusion image part can be obtained. It is possible to omit a thin unnecessary processing procedure and suppress image deterioration due to erroneous correspondence.

【0105】

Further, by using one image signal as a basic image for generating a composite image and another image signal as an auxiliary image among each image signal, it is possible to always output an image having at least a certain level of image quality.

【0106】

In particular, by arranging an image pickup system having an image pickup element that outputs an image signal as a basic image so that the optical axis of the image pickup optical system faces the subject, at least an image from the front of the subject can be specially processed. It can always be output without adding it, and the photographer can shoot without receiving an unnatural feeling of operation even at the time of close-up shooting.

[Simple explanation of drawings]

[Figure 1]

It is a schematic block diagram of the 1st Example of the compound eye imaging apparatus of this invention.

[Figure 2]

It is a block diagram of the image processing part shown in FIG.

[Fig. 3]

It is the figure which looked at the main part of the optical system of the compound eye image pickup apparatus shown in FIG. 1 from the direction perpendicular to the plane formed by the optical axis of each image pickup lens group.

[Fig. 4]

It is a figure for demonstrating the positional relationship of the object when the object was imaged by the compound eye imaging apparatus shown in FIG. 1, and FIG. The figure showing the position, FIG. 3B is a view of the effective light receiving portion of each of the left and right CCD sensors as viewed from the image pickup lens group side.

[Fig. 5]

FIG. 5 is a diagram showing left and right images when a composition in which a road sign can be seen in the background of an automobile is imaged by the compound eye imaging device shown in FIG.

[Fig. 6]

It is a figure for demonstrating the triangulation method at the time of performing the area division by the double image area division part shown in FIG.

[Fig. 7]

It is a histogram obtained by the double image region division part shown in FIG. 2 when the composition shown in FIG. 5 was imaged.

[Fig. 8]

It is a figure which shows the composite image obtained by synthesizing the left and right projection images shown in FIG. 5 by the image processing unit shown in FIG.

[Fig. 9]

It is a block diagram of the image processing part of the 2nd Example of the compound eye imaging apparatus of this invention.

[Fig. 10]

It is an enlarged view of the area shown by the diagonal line of the main part of the optical system shown in FIG. 3 for explaining the image processing procedure of the 2nd Example of the compound eye imaging apparatus of this invention.

[Fig. 11]

It is a figure which looked at the main part of the optical system of the 3rd Example of the compound eye image pickup apparatus of this invention from the direction perpendicular to the plane formed by the optical axis of each image pickup lens group.

[Fig. 12]

It is the figure which looked at the effective light receiving part of each of the left and right CCD sensors shown in FIG. 11 from the image pickup lens group side.

[Fig. 13]

It is a figure which looked at the main part of the optical system of the 4th Example of the compound eye image pickup apparatus of this invention from the direction perpendicular to the plane formed by the optical axis of each image pickup lens group.

[Fig. 14]

It is the figure which looked at the effective light receiving part of each of the left and right CCD sensors shown in FIG. 13 from the image pickup lens group side.

[Fig. 15]

It is a block diagram of the image processing part of the 5th Example of the compound eye imaging apparatus of this invention.

[Fig. 16]

In order to explain the image composition processing in the sixth embodiment of the compound eye imaging apparatus of the present invention, it is a figure when a composition in which a mountain can be seen in the background of an automobile is imaged, and FIG. A diagram showing each of the left and right images, and the figure (B) is a diagram showing an image after image composition.

[Fig. 17]

It is a block diagram of the image processing part of the 7th Example of the compound eye imaging apparatus of this invention.

[Fig. 18]

It is a figure which looked at the main part of the optical system of 7th Example of the compound eye image pickup apparatus of this invention from the direction perpendicular to the plane formed by the optical axis of each image pickup lens group.

[Fig. 19]

It is a figure which showed the pixel position of each CCD sensor about a part in the left-right image obtained by the optical system shown in FIG.

[Fig. 20]

It is a figure which looked at the main part of the optical system of the 8th Example of the compound eye image pickup apparatus of this invention from the direction perpendicular to the plane formed by the optical axis of each image pickup lens group.

[Fig. 21]

It is a block diagram of the image processing part of the 9th Example of the compound eye imaging apparatus of this invention.

[Fig. 22]

It is a figure for demonstrating the principle of the compound eye imaging apparatus which obtains one high-definition image by synthesizing two images obtained by imaging a common subject using two imaging systems.

[Explanation of symbols]

10_L Left imaging system Ten_R Right imaging system 11_L、11_R Imaging lens group 12_L、12_R Color separation prism 13_L、13_R CCD sensor 14_L、14_R Focus motor 15_L、15_R Focusing lens group 16_L、16_R Zoom motor 17_L、17_R Variable magnification lens group 18_L、18_R Convergence angle motor 20 Control unit 21 All system control unit 22 Focus / zoom control unit 23 Congestion angle control unit twenty four_L、24_R Focus encoder twenty five_L、25_R Zoom encoder 26_L、26_R Convergence angle encoder 30, 130, 230, 330, 430 Image processing unit 31_L Left image memory 31_R Right image memory 32 Image input section 33, 133, 233, 333, 433 Corresponding point extraction unit 34, 134, 234 Double image area division 35, 235 Double image type judgment unit 36 Single-sided double image batch removal unit 37, 137, 237 Removal post-processing unit 38, 138, 238, 338 Image converter 39 Composite image memory 40 Image output section 50_L、50_R Focused object surface 50_L’、50_R'Focusing object surface at close-up shooting 53, 53', 53'', 55_L、55_R, 58 areas 53L left area 53R right area 60 objects 60_L、60_R image 136 One-sided image removal section 236 Double image remover 334 Proximity correspondence point judgment unit 335 Proximity correspondence point removal part 336, 436 Image interpolation section 434 Image sharpness judgment unit 435a Occlusion area judgment department 435b Occlusion generation frequency judgment unit 437 Corresponding point non-extracted area judgment unit