Image processing system

CROSS REFERENCE TO RELATED APPLICATION

0001This application is a continuation application of PCT/JP2007/055563 filed on Mar. 19, 2007 and claims benefit of Japanese Application No. 2006-103634 filed in Japan on Apr. 4, 2006, the entire contents of which are incorporated herein by this reference.

BACKGROUND OF INVENTION

00021. Field of the Invention

0003The present invention relates to an image processing technique for a pixel-mixed image which has undergone pixel mixture processing and, more particularly, to an image processing system for restoring a resolution of a pixel-mixed image deteriorated by pixel mixture processing at high speed with high precision.

00042. Description of the Related Art

0005Owing to recent development of semiconductor technology, there has been a trend toward higher-resolution image pickup devices.

0006Although efforts to obtain devices with smaller sizes and higher speeds have been made, it takes much time to read out data from a multi-pixel image pickup device. Also, high-speed readout involves the problem of heat generation and is regarded as difficult to be realized.

0007Pixel mixture techniques recently have been developed and widely known (Japanese Patent Application Laid-Open Publication No. 2004-180284, Japanese Patent Application Laid-Open Publication No. 2004-312140, Japanese Patent Application Laid-Open Publication No. 2005-107252, Japanese Patent Application Laid-Open Publication No. 2005-109968, and Japanese Patent Application Laid-Open Publication No. 2005-117192). A pixel mixture technique is a technique for mixing a plurality of pixels on an image pickup device and reading out the plurality of pixels as one pixel. Simultaneous readout of a plurality of pixels allows high-speed data readout. Readout of a plurality of pixels after mixture increases a speed at which an image pickup signal is read out but causes less aliasing than simple thinning readout. This significantly improves a per-pixel S/N ratio.

0008As a Bayer pattern demosaicking method, a method has recently been proposed for obtaining a demosaicking result optimum in terms of least squares without using a repetitive operation (H. J Trussel and Robert E. Hartwing, “Mathematics for Demosaicking”, IEEE Trans. Image Processing, Vol. 11, No. 4, April 2002).

0009A pixel mixture technique, however, four-pixel mixture processing reduces a resolution to one-fourth, and nine-pixel mixture processing reduces a resolution to one-ninth.

0010As a method for restoring a resolution deteriorated by the pixel mixture processing, an image restoration method is proposed for performing restoration processing by minimizing energy between images before and after restoration (“Restoration from Pixel Mixture Using Iterative Method”, Symposium on Sensing via Image Information, B-8, pp. 75-78, Jun. 12-13, 2003).

0011However, since an image restoration method by “Restoration from Pixel Mixture Using Iterative Method” uses a repetitive operation for restoration processing, the method requires a large amount of calculation and much processing time. Accordingly, there is a need for obtaining a solution at high speed without performing a repetitive operation as in a conventional method when performing restoration processing on a pixel-mixed image.

SUMMARY OF THE INVENTION

0012According to the present invention, an image processing system for restoring a resolution of a pixel-mixed image represented by a mixed-pixel signal which is acquired when a plurality of pixel signals in an image pickup device are mixed and are read out, includes: a shooting situation acquiring section for acquiring a shooting situation when the mixed-pixel signal is acquired, a restoration matrix generating section for generating a pixel mixture restoration matrix based on the shooting situation acquired by the shooting situation acquiring section, and a restoration processing section for restoring the resolution of the pixel-mixed image using the mixed-pixel signal and the pixel mixture restoration matrix.

BRIEF DESCRIPTION OF THE DRAWINGS

0013<figref idref="DRAWINGS">FIG. 1</figref> is a configuration diagram of an image processing system according to a first embodiment of the present invention;

0014<figref idref="DRAWINGS">FIG. 2</figref> is a diagram showing a configuration of a Bayer primary color filter;

0015<figref idref="DRAWINGS">FIG. 3</figref> is a diagram showing a color difference filter;

0016<figref idref="DRAWINGS">FIG. 4</figref> is a conceptual diagram of nine-pixel mixture processing;

0017<figref idref="DRAWINGS">FIG. 5</figref> is a conceptual diagram of four-pixel mixture processing;

0018<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram showing a configuration of a pixel-mixed image restoration processing section in <figref idref="DRAWINGS">FIG. 1</figref>;

0019<figref idref="DRAWINGS">FIG. 7</figref> is a conceptual diagram of image-to-column vector conversion processing; and

0020<figref idref="DRAWINGS">FIG. 8</figref> is a configuration diagram of an image processing system according to a second embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

0021Embodiments of the present invention will be described with reference to the drawings.

First Embodiment

0022<figref idref="DRAWINGS">FIGS. 1 to 7</figref> relate to an image processing system according to a first embodiment of the present invention. <figref idref="DRAWINGS">FIG. 1</figref> is a configuration diagram of the image processing system according to the first embodiment of the present invention. <figref idref="DRAWINGS">FIG. 2</figref> is a diagram showing a configuration of a Bayer primary color filter. <figref idref="DRAWINGS">FIG. 3</figref> is a diagram showing a color difference filter. <figref idref="DRAWINGS">FIG. 4</figref> is a conceptual diagram of nine-pixel mixture processing. <figref idref="DRAWINGS">FIG. 5</figref> is a conceptual diagram of four-pixel mixture processing. <figref idref="DRAWINGS">FIG. 6</figref> is a block diagram showing a configuration of a pixel-mixed image restoration processing section in <figref idref="DRAWINGS">FIG. 1</figref>. <figref idref="DRAWINGS">FIG. 7</figref> is a conceptual diagram of image-to-column vector conversion processing.

0023In the image processing system shown in <figref idref="DRAWINGS">FIG. 1</figref>, a CCD <b>103</b> serving as an image pickup device receives an incident optical image of a subject, photoelectrically converts the optical image, and outputs a result of the photoelectric conversion as image pickup signals. An image pickup signal obtained through shooting via a lens system <b>100</b>, an aperture <b>101</b>, and the CCD <b>103</b> constituting an image pickup section is amplified by a gain control amplifier (hereinafter abbreviated as a Gain) <b>105</b> and is converted into a digital signal by an A/D converter (hereinafter abbreviated as an A/D) <b>106</b>.

0024A Color Filter Array (hereinafter abbreviated as a CFA) <b>102</b> is provided at a front of the CCD <b>103</b>. A pixel mixture processing section <b>104</b> serving as a pixel mixture section for subjecting pixels as analog signals obtained through shooting by the CCD to mixture processing and reading out the pixels is connected to the CCD <b>103</b>.

0025A signal from the A/D <b>106</b> is transferred to an image recording section <b>115</b> or a pixel-mixed image restoration processing section <b>113</b> serving as a restoration processing section through a buffer <b>107</b>. The buffer <b>107</b> is also connected to a PreWB section <b>108</b>, a photometric evaluation section <b>109</b>, and a focused focal point detecting section <b>110</b>.

0026The PreWB section <b>108</b> is connected to the Gain <b>105</b>. The photometric evaluation section <b>109</b> is connected to the aperture <b>101</b>, the CCD <b>103</b>, the Gain <b>105</b>, and a pixel mixture restoration matrix generating section <b>114</b> serving as a restoration matrix generating section. The focused focal point detecting section <b>110</b> is connected to an AF motor <b>117</b> and the pixel mixture restoration matrix generating section <b>114</b>.

0027The pixel mixture restoration matrix generating section <b>114</b> is connected to a pixel-mixed image restoration processing section <b>113</b>. The image recording section <b>115</b> is connected to the pixel-mixed image restoration processing section <b>113</b> and an output section <b>116</b>. The pixel-mixed image restoration processing section <b>113</b> is connected to the image recording section <b>115</b> and the output section <b>116</b>. Data supplied from the pixel-mixed image restoration processing section <b>113</b> to the image recording section <b>115</b> and the output section <b>116</b> is a restored full-color image (to be described later).

0028A control section <b>111</b> such as a microcomputer is bi-directionally connected to the Gain <b>105</b>, the A/D <b>106</b>, the PreWB section <b>108</b>, the photometric evaluation section <b>109</b>, the focused focal point detecting section <b>110</b>, the pixel mixture processing section <b>104</b>, the pixel-mixed image restoration processing section <b>113</b>, the pixel mixture restoration matrix generating section <b>114</b>, and the image recording section <b>115</b>.

0029An external I/F section <b>112</b> which includes a power switch, a shutter button, and an interface for switching among various modes at the time of shooting is also bi-directionally connected to the control section <b>111</b>.

0030Functions and operation of the sections in <figref idref="DRAWINGS">FIG. 1</figref> will be described below.

0031With respect to mosaic image pickup signals obtained through image pickup by the single plate CCD <b>103</b>, which is an image pickup device with the CFA <b>102</b> provided at the front, a plurality of pixel signals are mixed and are read out at high speed by the pixel mixture processing section <b>104</b> serving as the pixel mixture section after the image pickup. After that, the pixel-mixed image restoration processing section <b>113</b> serving as the restoration processing section performs restoration processing on a pixel-mixed image, thereby increasing a resolution (i.e., the number of pixels) of the pixel-mixed image and fully colorizing the pixel-mixed image. The pixel-mixed image is outputted to the output section <b>116</b> or the image recording section <b>115</b>.

0032The term full colorization originally refers to a process of interpolatively generating, for each color pixel (e.g., R) obtained from an image pickup device via, e.g., a Bayer pattern filter, the other two color pixels (e.g., G and B) from pixels surrounding the R pixel and thereby tripling the number of pixels of Bayer pixels actually obtained through light reception and image pickup. The term is synonymous with demosaicking, which generates full-color pixels from Bayer pixels.

0033The present application characteristically generates an image with a resolution increased by pixel mixture restoration processing from a pixel-mixed image obtained by mixing a plurality of pixels of image pickup signals obtained through image pickup by an image pickup device. When pixel mixture restoration is performed on a pixel-mixed image obtained through nine-pixel mixture, the number of pixels, which has been reduced to one-ninth by pixel mixture, is increased ninefold. In the case of a Bayer image or the like, if each pixel is fully colorized, and the number of pixels is increased threefold, pixel mixture restoration makes the number of pixels 27 times the number of pixels at the time of nine-pixel mixture.

0034The image processing system enters a pre-image-pickup mode as a result of pressing the shutter button halfway after a shooting situation which can be externally specified is set to one such as a pixel mixture shooting mode via an external I/F section <b>112</b>. A video signal obtained through shooting via the lens system <b>100</b>, aperture <b>101</b>, CFA <b>102</b>, and CCD <b>103</b> is outputted as an analog signal.

0035Note that the present embodiment contemplates use of a single plate CCD in which a Bayer primary color filter is arranged as the CFA <b>102</b> at the front in an image pickup system. <figref idref="DRAWINGS">FIG. 2</figref> shows a configuration of a Bayer primary color filter. In a Bayer primary color filter, 2×2 pixels are regarded as a base unit, with a red (R) filter and a blue (B) filter each arranged for one pixel and green (Gr, Gb) filters arranged for the other two pixels. Note that although a Bayer primary color filter as shown in <figref idref="DRAWINGS">FIG. 2</figref> is contemplated as an embodiment of the CFA <b>102</b> in the present configuration, the CFA <b>102</b> may be configured like a color difference filter shown in <figref idref="DRAWINGS">FIG. 3</figref>, and any color filter may be used as long as the color filter has periodicity. Note that although the green filters are identical to each other, the green filters will be referred to as Gr and Gb to discriminate between the green filters for processing convenience in the present example.

0036A video signal in the buffer <b>107</b> is transferred to the PreWB section <b>108</b>, a photometric evaluation section <b>109</b>, and a focused focal point detecting section <b>110</b>. The PreWB section <b>108</b> calculates a simple white balance coefficient by adding up video signals within a predetermined level range for each color signal. The above-described coefficient is transferred to the Gain <b>105</b>. Simple white balance processing is performed using different gains set for each color signal.

0037The photometric evaluation section <b>109</b> calculates a luminance level in video signals with a set ISO film speed, a shutter speed against a camera shake, and other factors in mind and controls the aperture <b>101</b>, an electronic shutter speed of the CCD <b>103</b>, an amplification factor of the Gain <b>105</b>, and the like such that correct exposure is achieved.

0038The focused focal point detecting section <b>110</b> detects an edge strength in video signals and controls the AF motor <b>117</b> to maximize the edge strength, thereby obtaining a focused signal.

0039Main shooting is performed as a result of fully pressing the shutter button via the external I/F section <b>112</b>. A video signal is transferred to the buffer <b>107</b> in a same manner as in the pre-image-pickup.

0040At the time, the control section <b>111</b> determines a shooting mode. If the shooting mode set in the external I/F section <b>112</b> is the pixel mixture shooting mode, an image pickup signal obtained through shooting by the CCD <b>103</b> is subjected to pixel mixture processing in the pixel mixture processing section <b>104</b> and is transferred to the buffer <b>107</b> via the A/D <b>106</b>. After pixel-mixed image data generated in the pixel mixture processing section <b>104</b> is transferred to the buffer <b>107</b>, the pixel-mixed image data is transferred to the pixel-mixed image restoration processing section <b>113</b> serving as the restoration processing section. Note that if the shooting mode is the pixel mixture shooting mode, pixel-mixed image data which has been transferred to the buffer <b>107</b> (the data whose pixel number is reduced after the pixel mixture) may also be directly recorded in the pixel mixture recording section <b>115</b>.

0041If the shooting mode set in the external I/F section <b>112</b> is not the pixel mixture shooting mode, an image pickup signal obtained through shooting by the CCD <b>103</b> is transferred to the buffer <b>107</b> via the A/D <b>106</b> without pixel mixture processing in the pixel mixture processing section <b>104</b> and is recorded in the image recording section <b>115</b>.

0042After pixel mixture shooting, a shooting situation for a pixel-mixed image is confirmed. The confirmed shooting situation is transferred to the pixel mixture restoration matrix generating section <b>114</b> serving as the restoration matrix generating section. A pixel mixture restoration matrix is generated according to the shooting situation. Contents of the pixel mixture restoration matrix are transferred to the pixel-mixed image restoration processing section <b>113</b>.

0043A specific example of a shooting situation includes internal parameters, such as a zoom amount, a focus position, and an aperture at the time of shooting of a pixel-mixed image, which are optical-system-related conditions, PSF (Point Spread Function) data which can be calculated from the internal parameters at the time of shooting, a form of a pixel mixture kernel when pixel mixture exemplified in Formula (3) and (4) (to be described later) is performed, and sampling position information at the time of mixture processing of pixel mixture.

0044Video signals of a pixel-mixed image passed to the pixel-mixed image restoration processing section <b>113</b> are subjected to pixel mixture restoration processing.

0045Before the pixel mixture restoration processing, a pixel mixture restoration matrix used to perform the pixel mixture restoration processing is generated.

0046A pixel mixture restoration matrix is a matrix obtained by modeling a shooting process for a pixel-mixed image determined by a shooting situation using a matrix representation and calculating a pseudo inverse matrix of the matrix representation of the shooting process.

0047As an example, a pixel mixture restoration matrix D can be composed as a following matrix:

0000<br /><i>D=T</i>(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0001.tif" /><i>I</i>₄)<i>Q</i>⁺Φ_p*  (1)

0000where the symbol ˜ represents making a frequency domain representation of a matrix below the symbol through a Fourier transform. A matrix T is given as follows.

0000<br /><i>T=</i>(<i>{tilde over (K)}</i>_ff<i>{tilde over (H)}</i>_c^T<i>{tilde over (K)}</i>₉^T)<img file="US2009016632A1_D0002.tif" />(<i>K</i>_λ)Φ_p  (2)

0048The matrix {tilde over (K)}_ffis a result of performing a DFT transform (Discrete Fourier Transform) on a spatial autocorrelation matrix of an original image.

0049The matrix {tilde over (H)}_c^Tis a result of performing a DFT transform on a PSF filtering matrix and transposing the filtering matrix. A form of the matrix varies depending on a PSF value of a shooting situation.

0050The matrix {tilde over (K)}₉^Tis a result of performing a DFT transform on a nine-pixel mixture kernel filtering matrix and transposing the filtering matrix. A form of the matrix varies depending on a pixel mixture method. A form of a nine-pixel mixture kernel expressed within the nine-pixel mixture kernel filtering matrix is given by:

0000<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mo>[</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd></mtr></mtable><mo>]</mo></mrow></mtd><mtd><mrow><mo>(</mo><mn>3</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0003.tif" />

0000when nine-pixel mixture sampling as in <figref idref="DRAWINGS">FIG. 4</figref> is to be performed.

0051When four-pixel mixture sampling as in <figref idref="DRAWINGS">FIG. 5</figref> is to be performed, a form of a four-pixel mixture kernel expressed within a four-pixel mixture kernel filtering matrix is given as follows.

0000<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mtable><mtr><mtd><mrow><mo>[</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>1</mn></mtd><mtd><mn>0</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd><mtd><mn>0</mn></mtd></mtr></mtable><mo>]</mo></mrow></mtd><mtd><mrow><mo>(</mo><mn>4</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0004.tif" />

0052The matrix K_λ is a color autocorrelation matrix of the original image and is a 4×4 matrix with elements, each of which is a value of correlation between colors in the original image.

0053The matrix Φ is a phase matrix for describing a phase between colors in pixel mixture sampling and is a matrix which is a representation of a result of performing a DFT transform on a spatial phase between color data sampled at the time of pixel mixture. A form of the matrix Φ varies depending on a pixel mixture method. The matrix Φ_pis a result of performing permutation conversion on the matrix Φ.

0054The matrix Φ_pis represented by:

0000<br />Φ_p=P^TΦP  (5)

0000using a permutation conversion matrix P^Tand an inverse permutation conversion matrix P. The matrix Φ_prepresents a sampling spatial phase between colors shown in <figref idref="DRAWINGS">FIG. 4</figref> or <figref idref="DRAWINGS">FIG. 5</figref> depending on whether the matrix Φ_pis intended for nine-pixel mixture sampling or four-pixel mixture sampling. A form of the matrix Φ_pthus varies depending on a pixel mixture method.

0055The matrix Q⁺ can be represented as follows.

0000<br /><i>Q</i>⁺=[(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0005.tif" /><i>I</i>₄)<i>E</i>(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0006.tif" /><i>I</i>₄)]⁺  (6)

0000Note that E is a block diagonal matrix composed of 4×4 matrices and is given as follows.

0000<br /><i>E=Φ</i>_p*(<i>{tilde over (K)}</i>₉<i>{tilde over (H)}</i>_c<i>{tilde over (K)}</i>_ff<i>{tilde over (H)}</i>_c^T<i>{tilde over (K)}</i>₉^T)<img file="US2009016632A1_D0007.tif" />(<i>K</i>_λ)Φ_p  (7)

0056The matrix {tilde over (C)}₀₀is given by:

0000<br /><i>{tilde over (C)}</i>₀₀<i>=W</i>_N_{<sup2>2</sup2>}_×N_{<sup2>2</sup2>}(diag(1,0,0,0,0,0, . . . , 1,0,0,0,0,0)<img file="US2009016632A1_D0008.tif" />diag(1,0,0,0,0,0, . . . , 1,0,0,0,0,0))<i>W</i>_N_{<sup2>2</sup2>}_×N_{<sup2>2</sup2>}⁻¹  (8)

0000when nine-pixel mixture sampling as in <figref idref="DRAWINGS">FIG. 4</figref> is to be performed and is given by:

0000<br /><i>{tilde over (C)}</i>₀₀<i>=W</i>_N_{<sup2>2</sup2>}_×N_{<sup2>2</sup2>}(diag(1,0,0,0, . . . , 1,0,0,0)<img file="US2009016632A1_D0009.tif" />diag(1,0,0,0, . . . , 1,0,0,0))<i>W</i>_N_{<sup2>2</sup2>}_×N_{<sup2>2</sup2>}⁻¹  (9)

0000when four-pixel mixture sampling as in <figref idref="DRAWINGS">FIG. 5</figref> is to be performed. A form of the matrix {tilde over (C)}₀₀varies depending on a pixel-mixed image sampling method. diag( . . . ) represents a block diagonal matrix with elements on a main diagonal.

0057A method for generating the above-described pixel mixture restoration matrix D will be described below.

0058Since a pixel mixture restoration matrix is obtained by modeling, as a matrix, a shooting process for a pixel-mixed image determined by a shooting situation and calculating a pseudo inverse matrix of the matrix, a shooting situation needs to be acquired in advance to obtain the pixel mixture restoration matrix D.

0059A shooting situation acquiring section will be described.

0060In a shooting situation, a zoom amount, a focus position, and the like are acquired from an in-focus position detection value in the focused focal point detecting section <b>110</b> in <figref idref="DRAWINGS">FIG. 1</figref>. A value of an aperture or the like is acquired from a value of an f number in the photometric evaluation section <b>109</b>. The pixel mixture restoration matrix generating section <b>114</b> has a mechanism for calculating a PSF from internal parameters such as a zoom amount, a focus position, and an aperture when the internal parameters are supplied. The pixel mixture restoration matrix generating section <b>114</b> includes a PSF data generating section <b>114</b><i>a </i>capable of uniquely determining PSF data from a combination of values of internal parameters such as a zoom amount, a focus position, and an aperture by reference to a LUT (Look Up Table) if, for example, the internal parameters are set.

0061In a shooting situation, pieces of information such as a form of a pixel mixture kernel for pixel mixture and a pixel mixture sampling method are determined by the pixel mixture shooting mode (e.g., nine-pixel mixture or four-pixel mixture) specified via the external I/F section <b>112</b>.

0062A noise-amount-related shooting situation such as a result of modeling, as a shooting process, the amount of noise included in an image is also conceivable. In the case, the amount of noise generated is modeled within the pixel mixture restoration matrix using a predetermined ISO film speed with a gain width in the Gain <b>105</b> and the like in mind.

0063The pixel mixture restoration matrix generating section will be described.

0064The pixel mixture restoration matrix is composed and generated using the above-described pieces of information acquired as a shooting situation. More specifically, pieces of elemental data of the pixel mixture restoration matrix are composed using pieces of data (basic data) of the matrix {tilde over (H)}_c^T, the matrix {tilde over (K)}₉^T, the matrix {tilde over (K)}₉, the matrix Φ_p, the matrix {tilde over (C)}₀₀, the matrix {tilde over (K)}_ff, and the matrix K_λ, which are matrices constituting the pieces of elemental data of the pixel mixture restoration matrix, thereby generating the pixel mixture restoration matrix.

0065The matrix {tilde over (H)}_c^Tis composed using PSF data of the shooting situation.

0066The matrix {tilde over (K)}₉^Tmatrix {tilde over (K)}₉are composed from a form of a pixel mixture kernel of the shooting situation.

0067The matrix Φ_pand matrix {tilde over (C)}₀₀are composed from a pixel mixture sampling method of the shooting situation.

0068The matrix {tilde over (K)}_ffand matrix K_λ are each a piece of information on autocorrelation of an original image, and a form of each matrix is not uniquely determined at the time of shooting. Several matrices describing a characteristic of the image are held as candidates for each of the matrix {tilde over (K)}_ffand matrix K_λ in a storage region <b>114</b><i>b </i>of the pixel mixture restoration matrix generating section <b>114</b>, on the basis of prior information on shot images. Any one of the candidates is selected and used.

0069The matrices (pieces of basic data) constituting the pixel mixture restoration matrix D are obtained from pieces of information of the shooting situation as described above, and the pixel mixture restoration matrix D is generated from the obtained matrices (the pieces of basic data). In the present embodiment, a plurality of pieces of basic data (matrices) are stored in advance in the storage region <b>114</b><i>b </i>in the pixel mixture restoration matrix generating section <b>114</b> for each of shooting situations, and pieces of basic data corresponding to a specific shooting situation are used in combination. With the operation, the pixel mixture restoration matrix D is generated. Note that the present embodiment may be configured to generate the pixel mixture restoration matrix D by holding a plurality of pixel mixture restoration matrices D for respective shooting situations in the storage region <b>114</b><i>b </i>and selecting one of the pixel mixture restoration matrices D corresponding to one of the shooting situations, instead of storing, in advance, pieces of basic data (matrices) for each of shooting situations in the storage region <b>114</b><i>b. </i>

0070In the case, each pixel mixture restoration matrix D may also be stored in the storage region <b>114</b><i>b </i>after moving redundant elemental data. Redundancy in the pixel mixture restoration matrix D will be described later.

0071The pixel mixture restoration matrix D generated by the pixel mixture restoration matrix generating section <b>114</b> is used for pixel mixture restoration processing in the pixel-mixed image restoration processing section <b>113</b>.

0072A procedure for pixel mixture reconstruction processing in the pixel-mixed image restoration processing section <b>113</b> serving as a section for restoring a pixel-mixed image transmitted from the buffer <b>107</b> or an image recording section <b>115</b> is described below.

0073<figref idref="DRAWINGS">FIG. 6</figref> is a block diagram showing a processing flow of the pixel-mixed image restoration processing section <b>113</b> in <figref idref="DRAWINGS">FIG. 1</figref>. A description will be given below in the context of nine-pixel mixture processing shown in <figref idref="DRAWINGS">FIG. 4</figref>.

0074A pixel-mixed image which has undergone pixel mixture processing in the pixel mixture processing section <b>104</b> serving as the pixel mixture section connected to the CCD <b>103</b> is transmitted to the pixel-mixed image restoration processing section <b>113</b> and is converted into a column vector in an image-to-column vector conversion processing section <b>201</b>. The image-to-column vector conversion processing section <b>201</b> performs a process of sequentially placing columns of data of the image in a line on a column vector from a left side of the image.

0075<figref idref="DRAWINGS">FIG. 7</figref> shows a processing conceptual diagram of image-to-column vector conversion processing in the image-to-column vector conversion processing section <b>201</b>, which performs conversion into a one-dimensional data string. A blank region in <figref idref="DRAWINGS">FIG. 7</figref> represents a zero component and, in other words, indicates that the region is a region which is unrelated to an operation (for which no operation is performed). If an image is transmitted as one-dimensional data at the time of image transmission, the processing can be omitted by regarding the one-dimensional data as column vector data.

0076The image data which has undergone the column vector conversion in the image-to-column vector conversion processing section <b>201</b> is subjected to a two-dimensional DFT operation in a two-dimensional DFT processing section <b>202</b> serving as a frequency conversion section. The DFT (Discrete Fourier Transform) operation in the two-dimensional DFT processing section <b>202</b> is performed by multiplying the image which has undergone the column vector conversion in the image-to-column vector conversion processing section <b>201</b> by a matrix as given by Formula (10) below. W₂refers to a matrix for performing a two-dimensional DFT. In the matrix, N represents both a vertical size and a horizontal size of an original image, and W_N_{<sup2>2</sup2>}_×N_{<sup2>2 </sup2>}is a DFT matrix for performing a DFT operation on an N×N image. Note that a blank region in Formula (10) represents a zero component. As the DFT operation, a process of performing a FFT (Fast Fourier Transform) operation on the image column vector from the image-to-column vector conversion processing section <b>201</b> may be performed instead of multiplication by a DFT matrix.

0000<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>W</mi><mn>2</mn></msub><mo>=</mo><mrow><mo>[</mo><mtable><mtr><mtd><msub><mi>W</mi><mrow><msup><mi>N</mi><mn>2</mn></msup><mo>×</mo><msup><mi>N</mi><mn>2</mn></msup></mrow></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>W</mi><mrow><msup><mi>N</mi><mn>2</mn></msup><mo>×</mo><msup><mi>N</mi><mn>2</mn></msup></mrow></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>W</mi><mrow><msup><mi>N</mi><mn>2</mn></msup><mo>×</mo><msup><mi>N</mi><mn>2</mn></msup></mrow></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>W</mi><mrow><msup><mi>N</mi><mn>2</mn></msup><mo>×</mo><msup><mi>N</mi><mn>2</mn></msup></mrow></msub></mtd></mtr></mtable><mo>]</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>10</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0010.tif" />

0077Since the matrix with the above-described composition is a diagonal matrix, elements are sparse, and a high-speed operation can be performed.

0078Note that although a DFT operation has been described as an embodiment in the present embodiment, the present invention is not limited to a DFT operation. Any matrix operation may be applied to the present configuration as long as the matrix operation is an orthogonal matrix operation or unitary matrix operation which has a diagonalization effect on a real symmetric matrix or an Hermitian matrix.

0079The column vector image data which has undergone the two-dimensional DFT operation in the two-dimensional DFT processing section <b>202</b> is subjected to data replacement in a permutation conversion processing section <b>203</b> serving as a permutation conversion section. More specifically, a product of the permutation conversion matrix P^Tand the column vector data is calculated. The permutation conversion matrix P^Tand inverse permutation conversion matrix P are each a matrix for replacement, which interchanges right and left elements of a Kronecker product of a 4×4 matrix and a N²×N²matrix. Since the permutation conversion processing section <b>203</b> does not actually perform a data operation but performs a process of rearranging pieces of data, change of order of operations at the time of operations in the pixel mixture restoration matrix data operation section <b>204</b> makes it possible to omit processing in the permutation conversion processing section <b>203</b>.

0080The column vector image data which has undergone the data permutation conversion processing in the permutation conversion processing section <b>203</b> is subjected to pixel mixture restoration processing in the pixel mixture restoration matrix data operation section <b>204</b> serving as a restoration operation section. The pixel mixture restoration matrix data operation section <b>204</b> performs a matrix operation using the pixel mixture restoration matrix D generated in the pixel mixture restoration matrix generating section <b>114</b> in <figref idref="DRAWINGS">FIG. 1</figref> and the column vector image data which has undergone the data permutation conversion processing in the permutation conversion processing section <b>203</b>, thereby performing the restoration processing.

0081Pixel mixture-restored data may be generated by simply performing, as a matrix operation, a multiplication of the pixel mixture restoration matrix D and the column vector image data. In the present embodiment, however, a matrix operation is performed at high speed using redundancy between a structure of the pixel mixture restoration matrix D and a data structure of the column vector image data. Note that the pixel mixture restoration matrix D is given by Formula (1) described above.

0082An operation method which achieves speedup with a focus on redundancy will be described below.

0083A result of performing operations in the two-dimensional DFT processing section <b>202</b>, the permutation conversion processing section <b>203</b>, and the pixel mixture restoration matrix data operation section <b>204</b> on image data y which has undergone column vector conversion in the image-to-column vector conversion processing section <b>201</b> is represented as follows.

0000<br />T({tilde over (C)}₀₀<img file="US2009016632A1_D0011.tif" />I₄)Q⁺Φ_p*P^TW₂y  (11)

0084Redundancy of Formula (11) will be described.

0085First, redundancy of data in Formula (11) will be discussed.

0086Letting y₁be image data which is a result of processing from a right end to the phase matrix Φ_p* of Formula (11), a following formula:

0000<br /><i>y</i>₁=Φ_p<i>*P</i>^T<i>W</i>₂<i>y</i>  (12)

0000holds. A structure of the image data y₁can be transformed as follows.

0000<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mtable><mtr><mtd><mtable><mtr><mtd><mrow><msub><mi>y</mi><mn>1</mn></msub><mo>=</mo><mrow><msubsup><mi>Φ</mi><mi>p</mi><mo>*</mo></msubsup><mo></mo><msup><mi>P</mi><mi>T</mi></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><mi>y</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><msubsup><mi>Φ</mi><mi>p</mi><mo>*</mo></msubsup><mo></mo><msup><mi>P</mi><mi>T</mi></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msub><mi>CH</mi><mn>9</mn></msub><mo></mo><msub><mi>H</mi><mi>fc</mi></msub><mo></mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><msup><mi>P</mi><mi>T</mi></msup><mo></mo><msup><mi>Φ</mi><mo>*</mo></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msubsup><mi>CW</mi><mn>2</mn><mrow><mo>-</mo><mn>1</mn></mrow></msubsup><mo></mo><msup><mi>ΦΦ</mi><mo>*</mo></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msub><mi>H</mi><mn>9</mn></msub><mo></mo><msub><mi>H</mi><mi>fc</mi></msub><mo></mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><mrow><msup><mi>P</mi><mi>T</mi></msup><mo></mo><mrow><mo>(</mo><mrow><msub><mi>I</mi><mn>4</mn></msub><mo>⊗</mo><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub></mrow><mo>)</mo></mrow></mrow><mo></mo><msup><mi>Φ</mi><mo>*</mo></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msub><mi>H</mi><mn>9</mn></msub><mo></mo><msub><mi>H</mi><mi>fc</mi></msub><mo></mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><mrow><msup><mi>P</mi><mi>T</mi></msup><mo></mo><mrow><mo>(</mo><mrow><msub><mi>I</mi><mn>4</mn></msub><mo>⊗</mo><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub></mrow><mo>)</mo></mrow></mrow><mo></mo><msup><mi>PP</mi><mi>T</mi></msup><mo></mo><msup><mi>Φ</mi><mo>*</mo></msup><mo></mo><msup><mi>PP</mi><mi>T</mi></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msub><mi>H</mi><mn>9</mn></msub><mo></mo><msub><mi>H</mi><mi>fc</mi></msub><mo></mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><mrow><mo>(</mo><mrow><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub><mo>⊗</mo><msub><mi>I</mi><mn>4</mn></msub></mrow><mo>)</mo></mrow><mo></mo><msubsup><mi>Φ</mi><mi>p</mi><mo>*</mo></msubsup><mo></mo><msup><mi>P</mi><mi>T</mi></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><msub><mi>H</mi><mn>9</mn></msub><mo></mo><msub><mi>H</mi><mi>fc</mi></msub><mo></mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><mrow><mo>(</mo><mrow><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub><mo>⊗</mo><msub><mi>I</mi><mn>4</mn></msub></mrow><mo>)</mo></mrow><mo></mo><msup><mi>y</mi><mi>′</mi></msup></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mo>=</mo><mrow><mrow><mo>(</mo><mrow><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub><mo>⊗</mo><msub><mi>I</mi><mn>4</mn></msub></mrow><mo>)</mo></mrow><mo></mo><msubsup><mi>Φ</mi><mi>p</mi><mo>*</mo></msubsup><mo></mo><msup><mi>P</mi><mi>T</mi></msup><mo></mo><msub><mi>W</mi><mn>2</mn></msub><mo></mo><mi>y</mi></mrow></mrow></mtd></mtr></mtable></mtd><mtd><mrow><mo>(</mo><mn>13</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0012.tif" />

0000In the transform, a following formula:

0000<br />(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0013.tif" /><i>I</i>₄)=(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0014.tif" /><i>I</i>₄)(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0015.tif" /><i>I</i>₄)  (14)

0000is used.

0087The fact that a shooting process of obtaining the pixel-mixed image data y from an original image x by shooting can be modeled as follows:

0000<br />y=CH₉H_fcx  (15)

0000is also used. In Formula (15), following formulae hold.

0000<maths id="MATH-US-00005" num="00005"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>H</mi><mi>fc</mi></msub><mo>=</mo><mrow><mo>[</mo><mtable><mtr><mtd><msub><mi>H</mi><mi>c</mi></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>H</mi><mi>c</mi></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>H</mi><mi>c</mi></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>H</mi><mi>c</mi></msub></mtd></mtr></mtable><mo>]</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>16</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><msub><mi>H</mi><mn>9</mn></msub><mo>=</mo><mrow><mo>[</mo><mtable><mtr><mtd><msub><mi>K</mi><mn>9</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>K</mi><mn>9</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>K</mi><mn>9</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>K</mi><mn>9</mn></msub></mtd></mtr></mtable><mo>]</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>17</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mi>C</mi><mo>=</mo><mrow><mo>[</mo><mtable><mtr><mtd><msub><mi>C</mi><mn>00</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>C</mi><mn>03</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>C</mi><mn>30</mn></msub></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd><mtd><msub><mi>C</mi><mn>33</mn></msub></mtd></mtr></mtable><mo>]</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>18</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0016.tif" />

0000In Formula (18), following formulae hold.

0000<br /><i>C</i>^kl<i>=C</i>^k<img file="US2009016632A1_D0017.tif" /><i>C</i>^l  (19)

0000<br /><i>C</i>⁰=diag(1,0,0,0,0,0, . . . , 1,0,0,0,0,0)  (20)

0000<br /><i>C</i>³=diag(0,0,0,1,0,0, . . . , 0,0,0,1,0,0)  (21)

0088As can be seen from Formula (13), as for the image data y₁, at a row index:

0000<maths id="MATH-US-00006" num="00006"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>y</mi><mn>1</mn></msub><mo></mo><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><mi>m</mi></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><mi>n</mi></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><mi>m</mi><mo>,</mo><mi>n</mi></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><mi>m</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>n</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>22</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0018.tif" />

0000data at an index position is same for each of 36 possible combinations of (m,n), and the image data y₁has redundancy. Since the redundancy can be determined according to {tilde over (C)}₀₀<img file="US2009016632A1_D0019.tif" />I₄representing a sampling style in Formula (13) described above, the redundancy can be determined from a pixel mixture sampling style. Elimination of sameness (=redundancy) among the pieces of data at the 36 index positions makes it possible to save a data storage space required to hold data.

0089The matrix T is a block diagonal matrix with diagonally arranged N²4×4 matrices. As for T({tilde over (C)}₀₀<img file="US2009016632A1_D0020.tif" />I₄), at a column index:

0000<maths id="MATH-US-00007" num="00007"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mrow><mi>T</mi><mo></mo><mrow><mo>(</mo><mrow><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub><mo>⊗</mo><msub><mi>I</mi><mn>4</mn></msub></mrow><mo>)</mo></mrow></mrow><mo></mo><mrow><mo>[</mo><mrow><mo>·</mo><mrow><mo>,</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow></mrow></mrow><mo>]</mo></mrow></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><mi>m</mi><mo>,</mo><mi>n</mi></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>23</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0021.tif" />

0000indices for 6²(=36) possible combinations of (m₂,n₂) are same, and T({tilde over (C)}₀₀<img file="US2009016632A1_D0022.tif" />I₄) has redundancy. In Formula (23), • in a row index indicates that the row index is an arbitrary row index.

0090Since the redundancy can be determined according to {tilde over (C)}₀₀<img file="US2009016632A1_D0023.tif" />I₄representing a sampling style in T({tilde over (C)}₀₀<img file="US2009016632A1_D0024.tif" />I₄) in Formula (23) described above, the redundancy can be determined from a pixel mixture sampling style. Elimination of sameness (=redundancy) among pieces of data at the 36 index positions makes it possible to save a memory space required to hold elemental data of a pixel mixture restoration matrix in the pixel mixture restoration matrix generating section <b>114</b>.

0091As for the matrix Q⁺, at a index:

0000<maths id="MATH-US-00008" num="00008"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msup><mi>Q</mi><mo>+</mo></msup><mo></mo><mrow><mo>(</mo><mtable><mtr><mtd><mrow><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>1</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>1</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow><mo>,</mo></mrow></mtd></mtr><mtr><mtd><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow></mtd></mtr></mtable><mo>)</mo></mrow></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><msub><mi>m</mi><mn>1</mn></msub><mo>,</mo><msub><mi>m</mi><mn>2</mn></msub><mo>,</mo><msub><mi>n</mi><mn>1</mn></msub><mo>,</mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>1</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>1</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>24</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0025.tif" />

0000pieces of data for 6⁴(=1,296; 36×36) possible combinations of (m₁,m₂,n₁,n₂) are same, and the matrix Q⁺ has redundancy.

0092Since the redundancy can be determined according to {tilde over (C)}₀₀<img file="US2009016632A1_D0026.tif" />I₄representing a sampling style in Q⁺ in Formula (24) described above, the redundancy can be determined from a pixel mixture sampling style. Elimination of sameness (=redundancy) among the pieces of data at the 1,296 index positions makes it possible to save a memory space required to hold elemental data of a pixel mixture restoration matrix in the pixel mixture restoration matrix generating section <b>114</b>.

0093Redundancy in the operation (product-sum operation) in Formula (11) will be discussed on the basis of the above-described data redundancies.

0094As for the product of Q⁺ and y₁, Q⁺×y₁, if a row index for y₁is given by:

0000<maths id="MATH-US-00009" num="00009"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>y</mi><mn>1</mn></msub><mo></mo><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><mi>m</mi></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><mi>n</mi></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><mi>m</mi><mo>,</mo><mi>n</mi></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><mi>m</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>n</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>25</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0027.tif" />

0000there is redundancy among 36 possible combinations of (m,n).

0095If an index for Q+ is given by:

0000<maths id="MATH-US-00010" num="00010"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mo>(</mo><mtable><mtr><mtd><mrow><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>1</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>1</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow><mo>,</mo></mrow></mtd></mtr><mtr><mtd><mrow><mo>[</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow><mo>]</mo></mrow></mtd></mtr></mtable><mo>)</mo></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><msub><mi>m</mi><mn>1</mn></msub><mo>,</mo><msub><mi>m</mi><mn>2</mn></msub><mo>,</mo><msub><mi>n</mi><mn>1</mn></msub><mo>,</mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>1</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>1</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>26</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0028.tif" />

0000there is redundancy among 6⁴(=1,296) possible combinations of (m₁,m₂,n₁,n₂). Accordingly, as for the product of Q⁺ and y₁, Q⁺×y₁, 6⁴(=1,296) possible operations are redundant in a product-sum operation.

0096Assume that y₂=Q⁺y₁. In the case, data y₂has same redundancy as y₁. Because of the fact that an index for y₂has same redundancy as y₁and that if an index for T({tilde over (C)}₀₀<img file="US2009016632A1_D0029.tif" />I₄) is given by:

0000<maths id="MATH-US-00011" num="00011"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mrow><mi>T</mi><mo></mo><mrow><mo>(</mo><mrow><msub><mover><mi>C</mi><mo>~</mo></mover><mn>00</mn></msub><mo>⊗</mo><msub><mi>I</mi><mn>4</mn></msub></mrow><mo>)</mo></mrow></mrow><mo></mo><mrow><mo>[</mo><mrow><mo>·</mo><mrow><mo>,</mo><mrow><mrow><mn>4</mn><mo></mo><mrow><mo>(</mo><mrow><mrow><mi>N</mi><mo></mo><mrow><mo>(</mo><mrow><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>m</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>i</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mfrac><mi>N</mi><mn>6</mn></mfrac><mo></mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>+</mo><mi>j</mi></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mi>k</mi></mrow></mrow></mrow><mo>]</mo></mrow></mrow><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mi>where</mi><mo></mo><mstyle><mtext></mtext></mstyle><mo></mo><mtable><mtr><mtd><mrow><mrow><mi>i</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr><mtr><mtd><mrow><mrow><mi>j</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mi>…</mi><mo></mo><mstyle><mspace width="0.6em" height="0.6ex" /></mstyle><mo>,</mo><mfrac><mi>N</mi><mn>6</mn></mfrac></mrow></mtd><mtd><mrow><mrow><mo>(</mo><mrow><msub><mi>m</mi><mn>2</mn></msub><mo>,</mo><msub><mi>n</mi><mn>2</mn></msub></mrow><mo>)</mo></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mrow><msub><mi>m</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>n</mi><mn>2</mn></msub><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn><mo>,</mo><mn>4</mn><mo>,</mo><mn>5</mn></mrow></mtd></mtr></mtable></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mi>k</mi><mo>=</mo><mn>0</mn></mrow><mo>,</mo><mn>1</mn><mo>,</mo><mn>2</mn><mo>,</mo><mn>3</mn></mrow></mtd><mtd><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle></mtd></mtr></mtable></mrow></mtd><mtd><mrow><mo>(</mo><mn>27</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US2009016632A1_D0030.tif" />

0000data at a single row is redundant among 6²(=36) possible combinations of (m₂,n₂), and 36 pieces of data are present for the single row, 36 possible operations are redundant in a product-sum operation on a product of T({tilde over (C)}₀₀<img file="US2009016632A1_D0031.tif" />I₄) and y₂, T({tilde over (C)}₀₀<img file="US2009016632A1_D0032.tif" />I₄)×y₂.

0097With the above-described operations, calculation amount of operations in the matrix operation for Q⁺ and y₁is reduced to one-1,296th. Calculation amount of operations in the matrix operation for T({tilde over (C)}₀₀<img file="US2009016632A1_D0033.tif" />I₄) and y₂is reduced to one-36th.

0098A column vector of pixel mixture-restored image data obtained in the above-described manner is subjected to inverse permutation conversion in an inverse permutation conversion processing section <b>205</b> by being multiplied by the inverse permutation conversion matrix. The inverse permutation conversion matrix will be denoted by P if the permutation conversion matrix used in the permutation conversion processing section <b>203</b> is denoted by P^T.

0099The pixel mixture-restored image data which has undergone the inverse permutation conversion processing in the inverse permutation conversion processing section <b>205</b> is subjected to two-dimensional IDFT processing using an IDFT matrix in a two-dimensional IDFT processing section <b>206</b>. Although an IDFT operation can be performed by multiplying the column vector of the pixel mixture-restored image data by the IDFT matrix, similar processing may be performed in a FFT operation. The IDFT matrix is an inverse matrix of the DFT matrix W₂described above and is denoted by W₂⁻¹. IDFT processing has been described as an embodiment in the present embodiment. If the frequency conversion section adapted to the two-dimensional DFT processing section <b>202</b> described above is an orthogonal matrix operation or unitary matrix operation which has a diagonalization effect on a real symmetric matrix or an Hermitian matrix, the processing may be performed by any operation method as long as an operation defining an inverse matrix of the orthogonal matrix or unitary matrix is used.

0100From the above description, a pixel mixture restoration processing matrix Dpm in a part denoted by reference numerals <b>202</b> to <b>206</b> in <figref idref="DRAWINGS">FIG. 6</figref> can be represented as follows.

0000<br /><i>Dpm=W</i>₂⁻¹<i>P T</i>(<i>{tilde over (C)}</i>₀₀<img file="US2009016632A1_D0034.tif" /><i>I</i>₄)<i>Q</i>⁺Φ_p<i>*P</i>^T<i>W</i>₂  (28)

0101By calculating a product of the pixel mixture restoration processing matrix Dpm and the pixel-mixed image y which has undergone the column vector conversion in the image-to-column vector conversion processing section <b>201</b>, a pixel mixture-restored image f which is a fully colorized column vector can be obtained.

0000<br /><i>f=Dpm·y</i>  (29)

0102Column vector data of the pixel mixture-restored image data which has undergone the two-dimensional IDFT processing in the two-dimensional IDFT processing section <b>206</b> is converted into an image in a column vector-to-image conversion processing section <b>207</b> and is outputted as two-dimensional data.

0103In the above-described manner, the pixel-mixed image data which has undergone the pixel mixture processing in the pixel mixture processing section <b>104</b> of the CCD <b>103</b> in <figref idref="DRAWINGS">FIG. 1</figref> is subjected to the pixel mixture restoration processing in the pixel-mixed image restoration processing section <b>113</b>.

0104A pixel mixture-restored image which has undergone the pixel mixture restoration processing in the pixel-mixed image restoration processing section <b>113</b> is outputted from the pixel-mixed image restoration processing section <b>113</b> and is transmitted to the output section <b>116</b> or the image recording section <b>115</b>. Data of the pixel mixture-restored image outputted from the output section <b>116</b> is used as resources for image display by, e.g., a CRT, liquid crystal, or organic EL display, image transmission using various transmission section such as USB, IEEE1394, and TCP/IP, and media recording on various recording media such as a CompactFlash (registered trademark), xD-Picture Card (registered trademark), and DVDR.

0105According to the first embodiment of the present invention, it is possible to provide an image processing system capable of performing pixel mixture restoration processing at high speed with high precision, not using a repetitive operation but using a matrix operation for restoration processing, when restoring a resolution of a pixel-mixed image obtained by mixing a plurality of pixels on an image pickup device.

Second Embodiment

0106<figref idref="DRAWINGS">FIGS. 2 to 8</figref> relate to an image processing system according to a second embodiment of the present invention. <figref idref="DRAWINGS">FIG. 2</figref> is a diagram showing a configuration of a Bayer primary color filter. <figref idref="DRAWINGS">FIG. 3</figref> is a diagram showing a color difference filter. <figref idref="DRAWINGS">FIG. 4</figref> is a conceptual diagram of nine-pixel mixture processing. <figref idref="DRAWINGS">FIG. 5</figref> is a conceptual diagram of four-pixel mixture processing. <figref idref="DRAWINGS">FIG. 6</figref> is a block diagram showing a configuration of a pixel-mixed image restoration processing section in <figref idref="DRAWINGS">FIG. 1</figref>. <figref idref="DRAWINGS">FIG. 7</figref> is a conceptual diagram of image-to-column vector conversion processing. <figref idref="DRAWINGS">FIG. 8</figref> is a configuration diagram of an image processing system according to the second embodiment of the present invention. <figref idref="DRAWINGS">FIGS. 2 to 7</figref> also relate to the first embodiment, and a description with reference to <figref idref="DRAWINGS">FIGS. 2 to 7</figref> will be omitted. A description will be given with a main focus on <figref idref="DRAWINGS">FIG. 8</figref>.

0107In an image processing system shown in <figref idref="DRAWINGS">FIG. 8</figref>, an input section <b>300</b> is connected to an image recording section <b>303</b> and a pixel-mixed image restoration processing section <b>301</b>. The pixel-mixed image restoration processing section <b>301</b> is bi-directionally connected to the image recording section <b>303</b> and is connected to an output section <b>306</b>. The image recording section <b>303</b> is connected to the output section <b>306</b>, in addition to the above-described connection destination. The input section <b>300</b> is connected to a pixel mixture restoration matrix generating section <b>302</b>.

0108The pixel mixture restoration matrix generating section <b>302</b> is connected to the pixel-mixed image restoration processing section <b>301</b>. An external I/F section <b>305</b> is bi-directionally connected to a control section <b>304</b>. The control section <b>304</b> is bi-directionally connected to the pixel-mixed image restoration processing section <b>301</b>, pixel mixture restoration matrix generating section <b>302</b>, and image recording section <b>303</b>, in addition to the external I/F section <b>305</b>.

0109Functions and operation of the sections in <figref idref="DRAWINGS">FIG. 8</figref> will be described below.

0110Video signals inputted via the input section <b>300</b> are transferred to the image recording section <b>303</b> or pixel-mixed image restoration processing section <b>301</b>. The video signals inputted via the input section <b>300</b> are digital video signals which are obtained by A/D conversion after an image is picked up by a CCD via a CFA in <figref idref="DRAWINGS">FIG. 2</figref> or <figref idref="DRAWINGS">FIG. 3</figref>, is photoelectrically converted, and is subjected to, e.g., nine-pixel mixture or four-pixel mixture in a pixel mixture processing section serving as a pixel mixture section. Information indicating whether the input image is a pixel-mixed image and shooting situation-related information are added to the video signals as additional information. Note that the shooting situation-related information for the input image includes optical system-related parameters such as a zoom amount, a focus amount, and an aperture at the time of CCD image pickup.

0111When the input image is inputted as the video signals from the input section <b>300</b>, a determination as to whether the input image is a pixel-mixed image is made. If the input image is a pixel-mixed image, the video signals are transferred to the pixel-mixed image restoration processing section <b>301</b>. On the other hand, if the input image is not a pixel-mixed image, the video signals are transferred to the image recording section <b>303</b>.

0112Note that the present embodiment includes a mode of transferring a pixel-mixed image to the image recording section <b>303</b> instead of transferring the pixel-mixed image to the pixel-mixed image restoration processing section <b>301</b> to reduce a recording space in the image recording section <b>303</b> even if the pixel-mixed image is inputted. In the case, the present embodiment includes a mode of appropriately transferring the image recorded on the image recording section <b>303</b>, i.e., the pixel-mixed image from the image recording section <b>303</b> to the pixel-mixed image restoration processing section <b>301</b> by a user's specification via an external I/F section and transferring a shooting situation for the pixel-mixed image to the pixel mixture restoration matrix generating section <b>302</b>.

0113The input section <b>300</b> makes the determination as to whether an input to the input section <b>300</b> is a pixel-mixed input, on the basis of additional information included in a pixel-mixed image. If the image inputted via the input section <b>300</b> is a pixel-mixed image, video signals of the input image are transferred to the pixel-mixed image restoration processing section <b>301</b>, and a shooting situation included as additional information in the pixel-mixed image is transferred to the pixel mixture restoration matrix generating section <b>302</b>.

0114A method for adding a shooting situation for an image as header information of the image or the like is used as an additional information description method.

0115The pixel mixture restoration matrix generating section <b>302</b> generates a pixel mixture restoration matrix on the basis of the shooting situation for the pixel-mixed image transferred to the pixel mixture restoration matrix generating section <b>302</b>.

0116Embodiments of the type of a shooting situation added as additional information to a pixel-mixed image and a procedure for generating a pixel mixture restoration matrix from the shooting situation are the same as embodiments in the method according to the first embodiment.

0117An embodiment of a method for recording a pixel mixture restoration matrix in the pixel mixture restoration matrix generating section <b>302</b> is the same as an embodiment in the method according to the first embodiment.

0118The pixel mixture restoration matrix generated in the pixel mixture restoration matrix generating section <b>302</b> is transferred to the pixel-mixed image restoration processing section <b>301</b>.

0119The pixel-mixed image restoration processing section <b>301</b> performs a matrix operation using the pixel-mixed image and pixel mixture restoration matrix transferred to the pixel-mixed image restoration processing section <b>301</b> and generates a pixel mixture-restored image. An embodiment of a matrix operation method for generating a pixel mixture-restored image is the same as an embodiment in the method according to the first embodiment.

0120The pixel mixture-restored image generated in the pixel-mixed image restoration processing section <b>301</b> is transferred to the image recording section <b>303</b> or output section <b>306</b>.

0121Data of the pixel mixture-restored image outputted from the output section <b>306</b> is used as resources for image display by, e.g., a CRT, liquid crystal, or organic EL display, image transmission using various transmission section such as USB, IEEE1394, and TCP/IP, and media recording on various recording media such as a CompactFlash (registered trademark), xD-Picture Card (registered trademark), and DVDR. Note that an image recorded in the image recording section <b>303</b> is also outputted from the output section <b>306</b> and utilized in the above-described manner.

0122According to the second embodiment of the present invention, it is possible to provide an image processing system capable of performing pixel mixture restoration processing at high speed with high precision, not using a repetitive operation but using a matrix operation for restoration processing, when restoring a resolution of a pixel-mixed image obtained by mixing a plurality of pixels on an image pickup device.

0123The present invention can be widely applied to an image pickup apparatus such as a digital camera including a multi-pixel image pickup device and an image processing apparatus which processes image data obtained by an image pickup apparatus.

0124The present invention is not limited to the above-described embodiments, and various changes, modifications, and the like may be made without departing from spirit and scope of the present invention.