Image processing system

The present invention relates to an image processing technique for a pixel-mixed image in which pixel-mixed processing has been performed, and particularly relates to an image processing system that restores resolution deterioration due to pixel-mixing processing of a pixel-mixed image at high speed and with high accuracy.

With the development of semiconductor technology in recent years, the image sensor tends to have a large number of pixels. Although the device has been made smaller and faster, it takes a lot of time to read the data of the image sensor in the image sensor having a large number of pixels. Alternatively, it is said that it is difficult to realize due to the problem of heat generation when trying to perform high-speed reading. Therefore, there is a problem that it is difficult to shoot a moving image at a high frame rate in an image sensor having a large number of pixels. Further, when the number of pixels is increased in the same area, there is a problem that the S / N ratio deteriorates because the size of the image sensor with respect to one pixel becomes relatively small.

In order to solve the above problems, pixel mixing technology has been developed and widely known in recent years (Japanese Patent Laid-Open Nos. 2004-180284, 2004-312140, 2005-107252, Japanese Patent Laid-Open No. 2004-180284, JP-A-2004-312140, JP-A-2005-107252 Kai 2005-109968, Japanese Patent Application Laid-Open No. 2005-117192). The pixel mixing technique is a technique of mixing a plurality of pixels on an image sensor and reading them out as one pixel, and by reading out a plurality of pixels at the same time, data can be read out at high speed. Further, by reading a plurality of pixels in a mixed manner, in order to improve the reading speed of the imaging signal, less aliasing is performed than by simply performing thinning reading, and the S / N per pixel is greatly improved.

Moreover, in recent years, as a method of demosizing Bayer arrays, a method of obtaining the optimum demosizing result in the sense of least squares without using iterative operations has been proposed (HJ Trussell and Robert E. Hartwing,). "Mathematics for Demosaicking", IEEE Trans. Image Processing, Vol. 11, No. 4, April 2002).

However, since the pixel mixing technique mixes a plurality of pixels and reads them out as one pixel, the pixel mixed image after pixel mixing has a drawback that the resolution is deteriorated as compared with the image before pixel mixing. Specifically, the resolution is reduced to 1/4 by performing the 4-pixel mixing process, and the resolution is reduced to 1/9 by performing the 9-pixel mixing process. As a method for restoring the resolution lost by this pixel mixing process, an image restoration method in which the restoration process is performed by minimizing the energy between the images before and after the restoration has been proposed (Pixels using the iterative method). Restoration from Mixing Image Sensing Symposium B-8, pp75-78, Jun.12,13,2003).<patcit num="1"><text>Japanese Unexamined Patent Publication No. 2004-180284</text></patcit><patcit num="2"><text>Japanese Unexamined Patent Publication No. 2004-312140</text></patcit><patcit num="3"><text>Japanese Patent Application Laid-Open No. 2005-107252</text></patcit><patcit num="4"><text>Japanese Patent Application Laid-Open No. 2005-109968</text></patcit><patcit num="5"><text>Japanese Patent Application Laid-Open No. 2005-117192</text></patcit><nplcit num="1"><text>HJ Trussell and Robert E. Hartwing, "Mathematics for Demosaicking", IEEE Trans. Image Processing, Vol. 11, No. 4, April 2002</text></nplcit><nplcit num="2"><text>"Restoration from Pixel Mixing Using Repetition" Image Sensing Symposium B-8, pp75-78, Jun.12,13,2003</text></nplcit>

<p> However, the image restoration method by "restoration from pixel mixing using the iterative method" uses an iterative operation for the restoration process, so that the amount of calculation is large and the processing time is long. Therefore, when the restoration process of the pixel mixed image is performed, there is a demand to obtain a solution at high speed without performing the iterative calculation as in the conventional method. Therefore, in view of the above problems, the present invention uses a matrix calculation for the restoration process at high speed without using an iterative calculation when restoring the resolution of a pixel-mixed image in which a plurality of pixels on the image sensor are mixed. An object of the present invention is to provide an image processing system capable of performing pixel mixing restoration processing with high accuracy.</p>

<p> The present invention is an image processing system that restores the resolution of a pixel-mixed image represented by a mixed pixel signal acquired by mixing and reading signals of a plurality of pixels in an image pickup device, wherein the mixed pixel signal is used. A shooting status acquisition means for acquiring the shooting status at the time of acquisition, a restoration matrix generating means for generating a pixel mixed restoration matrix based on the shooting status acquired by the shooting status acquisition means, the mixed pixel signal, and the said It is characterized by having a restoration processing means for restoring the resolution of the pixel-mixed image using a pixel-mixed restoration matrix.</p><p> The present invention further includes an imaging means for generating signals of a plurality of pixels by photographing a subject, and a pixel mixing means for mixing and reading signals of the plurality of pixels obtained by the imaging means. The shooting status acquisition means obtains at least one of a situation when the subject is photographed by the imaging means and a situation when the signals of the plurality of pixels are mixed and read by the pixel mixing means. The feature is that it is acquired as a shooting situation.</p><p> In the present invention, the shooting situation acquisition means is characterized in that, as the shooting situation, the conditions relating to the shooting optical system when the subject is shot are acquired. In the present invention, the shooting status acquisition means has a PSF data generating means that generates PSF data from conditions related to a shooting optical system when the subject is shot, and is characterized in that the PSF data is acquired as the shooting status. And.</p><p> In the present invention, the PSF data generation means is characterized in that PSF data is generated by searching a LUT according to the conditions related to the photographing optical system. In the present invention, the imaging status acquisition means is characterized in that, as the imaging status, information on a pixel mixing mode when signals of a plurality of pixels in the image sensor are mixed and read out is acquired.</p><p> In the present invention, additional information indicating the shooting status is added to the mixed pixel signal, and the shooting status acquisition means acquires the shooting status by the additional information. The present invention is characterized by having a storage area in which component data of the pixel mixing restoration matrix is stored in advance.</p><p> In the present invention, the restoration matrix generation means has a storage area in which the component data of the pixel mixing restoration matrix is stored in advance according to the photographing situation, and the restoration matrix generating means is the said according to the photographing situation acquired by the photographing situation acquisition means. It is characterized in that the pixel mixed restoration matrix having the component data is generated. In the present invention, the restoration matrix generation means has acquired the basic data for constructing the component data of the pixel mixed restoration matrix by the photographing situation acquisition means. It is characterized in that a pixel mixing restoration matrix is generated by constructing the component data using the basic data according to the shooting situation.</p><p> In the present invention, the restoration processing means includes a frequency conversion means that performs frequency conversion on the pixel-mixed image, a sequence conversion means that replaces an arbitrary component on the frequency-converted pixel-mixed image, and the restoration matrix. It is characterized by having a restoration calculation means for performing a restoration calculation process on the pixel mixed image using the pixel mixing restoration matrix generated by the generation means. In the present invention, the frequency conversion means is characterized in that it performs an orthogonal matrix operation or a unitary matrix operation having a diagonalization effect on a real symmetric matrix or a Hermitian matrix.</p><p> In the present invention, the frequency conversion means is characterized in that it performs FFT arithmetic processing. In the present invention, the restoration calculation means determines the redundancy of the component data among the component data of the pixel mixed restoration matrix, and the component data determined to have redundancy are combined into one and the restoration calculation process is performed. It is characterized by performing. The present invention is characterized by having a pixel mixed recording unit that records and holds the pixel mixed image.</p>

<p> According to the present invention, when restoring the resolution of a pixel-mixed image in which a plurality of pixels on an image sensor are mixed, pixel-mixed restoration is performed at high speed and with high accuracy by using a matrix operation in the restoration process without using an iterative calculation. An image processing system capable of performing processing can be provided.</p>

Embodiments of the invention will be described with reference to the drawings. [First Embodiment] 1 to 7 relate to the image processing system of the first embodiment of the present invention, FIG. 1 is a block diagram of the image processing system of the first embodiment, and FIG. 2 is a diagram showing a configuration of a Bayer type primary color filter. FIG. 3 is a diagram showing a color difference type filter, FIG. 4 is a conceptual diagram of a 9-pixel mixing process, FIG. 5 is a conceptual diagram of a 4-pixel mixing process, and FIG. 6 is a block diagram showing a configuration of a pixel mixing image restoration processing unit in FIG. , FIG. 7 is a conceptual diagram of the image string vectorization process.

In the image processing system shown in FIG. 1, the CCD 103, which is an image pickup device, receives an optical image of an incident subject, performs photoelectric conversion, and outputs it as an image pickup signal. The image pickup signal taken through the lens system 100, the aperture 101, and the CCD 103 that constitute the image pickup means is amplified by the gain control amplifier (hereinafter abbreviated as Gain) 105, and is amplified by the A / D converter (hereinafter, A / D). (Abbreviated as) It is converted into a digital signal at 106. A color filter array (hereinafter abbreviated as CFA) 102 is provided on the front surface of the CCD 103. Further, the CCD 103 is connected to a pixel mixing processing unit 104, which is a pixel mixing means for mixing and reading out the pixels of the analog signal captured by the CCD.

The signal from the A / D 106 is transferred to the image recording unit 115 or the pixel mixed image restoration processing unit 113 which is the restoration processing means via the buffer 107. The buffer 107 is also connected to the PreWB unit 108, the photometric evaluation unit 109, and the in-focus detection unit 110. The PreWB unit 108 goes to Gain 105, the metering evaluation unit 109 goes to the aperture 101, CCD103, Gain 105 and the pixel mixing restoration matrix generation unit 114 which is the restoration matrix generation means, and the in-focus detection unit 110 goes to the AF motor 116 and the pixel mixing restoration matrix generation unit. Connected to 114.

The pixel mixing restoration matrix generation unit 114 is connected to the pixel mixing image restoration processing unit 113. The image recording unit 115 is connected to the pixel mixed image restoration processing unit 113 and the output unit 116. The pixel mixed image restoration processing unit 113 is connected to the image recording unit 115 and the output unit 116. The data supplied from the pixel mixed image restoration processing unit 113 to the image recording unit 115 and the output unit 116 is a full-color restored image described later.

The control unit 111 of a microcomputer or the like includes Gain 105, A / D 106, PreWB unit 108, photometric evaluation unit 109, focus detection unit 110, pixel mixing processing unit 104, pixel mixing image restoration processing unit 113, and pixel mixing restoration matrix generation unit. It is bidirectionally connected to 114 and the image recording unit 115. In addition, an external I / F unit 112 equipped with a power switch, a shutter button, and an interface for switching various modes during shooting is also bidirectionally connected to the control unit 111.

The functions and functions of each part in FIG. 1 will be described below. The mosaic-like image pickup signal imaged by the single-plate CCD 103, which is an image sensor with CFA 102 arranged on the front surface, is read out at high speed by mixing a plurality of pixel signals by the pixel mixing processing unit 104, which is a pixel mixing means, after imaging. After that, the pixel-mixed image restoration processing unit 113, which is a restoration processing means, performs restoration processing of the pixel-mixed image to increase the resolution (that is, the number of pixels) and make it full-color, and outputs the image to the output unit 116 or the image recording unit 115. To.

Originally, full-colorization means that for each color pixel (for example, R) obtained from an image pickup device through, for example, a Bayer array filter, for that one R pixel, the surrounding array pixels to the other two color pixels (for example, G). It refers to the process of triple the number of pixels with respect to the Bayer pixels actually received and imaged by interpolating and generating (B), which is synonymous with demosaiking to generate full-color pixels from Bayer pixels.

The present application is characterized in that a pixel mixed image obtained by mixing a plurality of pixels of an image pickup signal captured by an image sensor is subjected to restoration processing of the mixed pixels to generate an image having an increased resolution. .. When restoring pixel mixing from a pixel-mixed image of 9-pixel mixing, the number of pixels reduced to 1/9 by pixel mixing is multiplied by 9, and in the case of Bayer images, etc., each pixel is fully colored and tripled. In this case, when pixel mixing restoration is performed, the number of pixels becomes 27 times the number of pixels when 9 pixels are mixed. After setting the shooting conditions that can be specified from the outside such as the pixel mixed shooting mode via the external I / F section 112, press the shutter button halfway to enter the pre-imaging mode. The video signal taken through the lens system 100, the aperture 101, the CFA 102, and the CCD 103 is output as an analog signal.

In this embodiment, the CFA 102 is assumed to be a single-plate CCD in which a Bayer type primary color filter is arranged on the front surface of the imaging system. FIG. 2 shows the configuration of the Bayer type primary color filter. The Bayer type has 2 x 2 pixels as the basic unit, with one red (R) and one blue (B) filter and two green (Gr, Gb) filters. However, although the embodiment of CFA102 in this configuration assumes a Bayer type primary color filter as shown in FIG. 2, a configuration such as the color difference type filter shown in FIG. 3 may be used, as long as it is a color filter having periodicity. It can be anything. The green filter is the same, but in this example, it is described separately from Gr and Gb for convenience of processing.

The video signal in the buffer 107 is transferred to the PreWB unit 108, the metering evaluation unit 109, and the in-focus detection unit 110. The PreWB unit 108 calculates a simple white balance coefficient by integrating video signals within a predetermined level for each color signal. The above coefficients are transferred to Gain 105, and simple white balance processing is performed by setting different gains for each color signal. The photometric evaluation unit 109 takes into account the set ISO sensitivity, shutter speed at the camera shake limit, etc., and finds the brightness level in the video signal to obtain the proper exposure, such as the electronic shutter speed of the aperture 101 and CCD 103, and the amplification factor of Gain 105. To control. Further, the in-focus detection unit 110 detects the edge intensity in the video signal and controls the AF motor 116 so that this becomes the maximum to obtain the in-focus signal.

Next, the main shooting is performed by fully pressing the shutter button via the external I / F unit 112, and the video signal is transferred to the buffer 107 in the same manner as the pre-imaging. At this time, the control unit 111 determines the shooting mode, and when the shooting mode set in the external I / F unit 112 is the pixel mixed shooting mode, the image pickup signal shot by the CCD 103 is sent to the pixel mixed processing unit 104. Pixel mixing processing is performed and transferred to the buffer 107 through the A / D 106. Then, the pixel mixed image data generated by the pixel mixing processing unit 104 is transferred to the buffer 107 and then transferred to the pixel mixing image restoration processing unit 113, which is a restoration processing means. When the shooting mode is the pixel mixed shooting mode, the pixel mixed image data (data in which the number of pixels is reduced by mixing pixels) transferred to the buffer 107 can be recorded as it is in the pixel mixed recording unit 115. ..

When the shooting mode set in the external I / F section 112 is not the pixel mixing shooting mode, the image pickup signal shot by the CCD 103 is subjected to the A / D 106 without performing the pixel mixing processing in the pixel mixing processing section 104. It is transferred to the through buffer 107 and recorded in the image storage unit 115. The shooting status of the pixel-mixed image is determined when the pixel-mixed shooting is performed. The determined shooting status is transferred to the pixel mixing restoration matrix generation unit 114, which is a restoration matrix generation means, the pixel mixing restoration matrix is generated according to the shooting status, and the contents of the pixel mixing restoration matrix are stored in the pixel mixing image restoration processing unit 113. Transferred.

Specific examples of shooting conditions include internal parameters that are conditions related to the optical system such as the zoom amount, focus position, and aperture when shooting a pixel-mixed image, and the PSF (Point Spread Function) that can be obtained from the internal parameters at the time of shooting. ) Data, the shape of the pixel mixing kernel when performing the pixel mixing illustrated in the equations (3) and (4) described later, the sampling position information when the pixel mixing mixing process is performed, and the like.

The video signal of the pixel-mixed image passed to the pixel-mixed image restoration processing unit 113 is subjected to the pixel-mixed restoration processing. Before performing the pixel mixing restoration processing, the pixel mixing restoration matrix used for performing the pixel mixing image restoration processing is generated. The pixel mixed restoration matrix is a matrix obtained by modeling the shooting process of a pixel mixed image determined according to the shooting situation by a matrix representation and obtaining a pseudo inverse matrix of the matrix representation of the shooting process.

<img file="JP4531007B2_D0001.tif" /><img file="JP4531007B2_D0002.tif" /><img file="JP4531007B2_D0003.tif" /><img file="JP4531007B2_D0004.tif" /><img file="JP4531007B2_D0005.tif" /><img file="JP4531007B2_D0006.tif" /><img file="JP4531007B2_D0007.tif" /><img file="JP4531007B2_D0008.tif" /><img file="JP4531007B2_D0009.tif" /><img file="JP4531007B2_D0010.tif" /><img file="JP4531007B2_D0011.tif" /><img file="JP4531007B2_D0012.tif" /><img file="JP4531007B2_D0013.tif" /> As described above, the pixel mixed image data that has been pixel mixed processed by the pixel mixing processing unit 104 provided in the CCD 103 of FIG. 1 is subjected to the pixel mixing restoration processing by the pixel mixing image restoration processing unit 113. The pixel-mixed restoration image processed by the pixel-mixed image restoration processing unit 113 is output from the pixel-mixed image restoration processing unit 113 and transmitted to the output unit 116 or the image recording unit 115. The pixel mixed restoration image data output by the output unit 116 is used for image display on a display such as CRT, liquid crystal, or organic EL, image transmission using various transmission means such as USB, IEEE1394, and TCP / IP, and Compact Flash (registered trademark). ), XD picture card (registered trademark), DVDR, etc. It is used as a resource for media recording on various recording media.

[Second Embodiment] 2 to 8 relate to the image processing system of the second embodiment of the present invention, FIG. 2 is a diagram showing a configuration of a Bayer type primary color filter, FIG. 3 is a diagram showing a color difference type filter, and FIG. 4 is a 9-pixel mixture. Conceptual diagram of processing, FIG. 5 is a conceptual diagram of 4-pixel mixing processing, FIG. 6 is a block diagram showing the configuration of the pixel mixing image restoration processing unit in FIG. 1, FIG. 7 is a conceptual diagram of column sequence vectorization processing, and FIG. Is a block diagram of the image processing system of the second embodiment. Since FIGS. 2 to 7 are the same as those of the first embodiment, the description thereof will be omitted, and FIG. 8 will be mainly described.

In the image processing system shown in FIG. 8, the input unit 300 is connected to the image recording unit 303 and the pixel mixed image restoration processing unit 301. The pixel mixed image restoration processing unit 301 is bidirectionally connected to the image recording unit 303 and is connected to the output unit 306. The image recording unit 303 is connected to the output unit 306 in addition to the above connection. The input unit 300 is connected to the pixel mixing restoration matrix generation unit 302. The pixel mixing restoration matrix generation unit 302 is connected to the pixel mixing image restoration processing unit 301. The external I / F unit 305 is bidirectionally connected to the control unit 304. In addition to the external I / F unit 305, the control unit 304 is bidirectionally connected to the pixel mixing image restoration processing unit 301, the pixel mixing restoration matrix generation unit 302, and the image recording unit 303.

The functions and functions of each part in FIG. 8 will be described below. The video signal input via the input unit 300 is transferred to the image recording unit 303 or the pixel mixed image restoration processing unit 303. The video signal input via the input unit 300 is imaged by CCD through the CFA of FIG. 2 or 3 and photoelectrically converted, and further, for example, 9-pixel mixing or 4-pixel mixing is performed by the pixel mixing processing unit which is a pixel mixing means. After that, it is an A / D-converted digital video signal, and information indicating whether or not the input image is a pixel-mixed image and information on the shooting status are added as additional information. The information on the shooting status of the input image is a parameter related to the optical system such as the zoom amount, the focus amount, and the aperture at the time of CCD imaging.

When the input image is input as a video signal from the input unit 300, it is determined whether or not the input image is a pixel-mixed image, and if it is a pixel-mixed image, the video signal is transferred to the pixel-mixed image restoration processing unit 301. If it is not a pixel-mixed image, the video signal is transferred to the image recording unit 303.

In the present embodiment, even when a pixel-mixed image is input, the image is not transferred to the pixel-mixed image restoration processing unit 301 but is transferred to the image recording unit 303 in order to reduce the recording space of the image recording unit 303. And transfer mode shall be included. In this case, the image recorded by the image recording unit 303 is appropriately transferred from the image recording unit 303 to the pixel mixed image restoration processing unit 301 according to the user's specification in the external I / F unit, and the pixel mixed image is captured. It shall include a mode in which the situation is transferred to the pixel mixing restoration matrix generation unit 302.

To determine whether the input to the input unit 300 is pixel mixing, the input unit 300 determines based on the additional information included in the pixel mixed image. If the image input by the input unit 300 is a pixel-mixed image, the video signal of the input image is transferred to the pixel-mixed image restoration processing unit 301, and the shooting status included as additional information in the pixel-mixed image is stored in the pixel-mixed restoration matrix. Transfer to the generation unit 302. As the method of describing the additional information, a method of adding the shooting status as the header information of the image or the like is used.

The pixel mixing restoration matrix generation unit 302 generates a pixel mixing restoration matrix based on the shooting status of the pixel mixing image transferred to the pixel mixing restoration matrix generation unit 302.

The type of shooting situation added as additional information to the pixel-mixed image and the embodiment of the procedure for generating the pixel-mixed restoration matrix from the shooting situation are the same as the method of the first embodiment. Further, the embodiment of the recording method in the pixel mixing restoration matrix generation unit 302 of the pixel mixing restoration matrix is the same as the method of the first embodiment. The pixel mixing restoration matrix generated by the pixel mixing restoration matrix generation unit 302 is transferred to the pixel mixing image restoration processing unit 301.

Using the pixel mixed image and the pixel mixed restoration matrix transferred to the pixel mixed image restoration processing unit 301, the pixel mixed image restoration processing unit 301 performs matrix arithmetic processing to generate a pixel mixed restoration image. The embodiment of the matrix operation method for generating the pixel mixed restoration image is the same as the method of the first embodiment. The pixel mixed restoration image generated by the pixel mixed image restoration processing unit 301 is transferred to the image recording unit 303 or the output unit 306.

The pixel mixed restoration image data output by the output unit 306 is used for image display on a display such as CRT, liquid crystal, or organic EL, image transmission using various transmission means such as USB, IEEE1394, and TCP / IP, and Compact Flash (registered trademark). ), XD picture card (registered trademark), DVDR, etc. It is used as a resource for media recording on various recording media. The image recorded by the image recording unit 303 is also output and used by the output unit 306 in the same manner as described above.

The present invention can be widely applied to an image pickup device such as a digital camera equipped with a multi-pixel image pickup device and an image processing device that processes image data obtained by the image pickup device.

<figref num="1">The block diagram of the image processing system of 1st Embodiment of this invention.</figref><figref num="2">The figure which shows the structure of the Bayer type primary color filter.</figref><figref num="3">The figure which shows the color difference type filter.</figref><figref num="4">Conceptual diagram of 9-pixel mixing process.</figref><figref num="5">Conceptual diagram of 4-pixel mixing process.</figref><figref num="6">The block diagram which shows the structure of the pixel mixed image restoration processing part in FIG.</figref><figref num="7">Conceptual diagram of image sequence vectorization processing.</figref><figref num="8">The block diagram of the image processing system of the 2nd Embodiment of this invention.</figref>

Code description

102 ... Color filter array 103 ... CCD (image sensor) 108 ... PreWB section 109 ... Metering Evaluation Department 110 ... Focus detector 111,304 ... Control unit 112,305 ... External I / F section 113,301 ... Image mixed image restoration processing unit (restoration processing means) 114,302 ... Pixel mixed restoration matrix generation unit (reconstruction matrix generation means) 114a ... PSF data generation method 114b ... storage area 115,303 ... Image recording section 116,306 ... Output