EP2175656A1

Image processing device, image processing method, program, and imaging device

Abstract

An image processing device includes: a coordinate conversion unit (142) which calculates a corresponding sampling coordinate on a color mosaic image corresponding to a pixel position of a color image when a deformation process is performed, according to the pixel position of the color image; a sampling unit (143); a sampling unit (143) which interpolates-generates a pixel value in a sampling coordinate for each of color planes obtained by decomposing the color mosaic image; and a color generation unit (144) which generates a color image by synthesizing interpolation values of the respective color planes. Each pixel value of a color image subjected to a deformation process is obtained as a pixel value of the sampling coordinate from the color mosaic image by interpolation calculation, thereby realizing the color interpolation process for generating a color image from the color mosaic image and a deformation process of the color image by one interpolation calculation.

EP2175656A1, drawing sheet 1 — Sheet 1 of 66

Term

Projected expiry 24 June 2028.

26 claims: 3 independent, 23 dependent

1
An image processing apparatus that generates a color image containing multiple color luminance information of each pixel using a color mosaic image that contains single color luminance information of each pixel obtained from single-chip imaging elements as a plurality of photoelectrical converted pixels, comprising - a color plane separator unit that separates the above noted mosaic image into multiple color planes having the same color and- a coordinates conversion unit that calculates the sampling coordinates for the above noted color mosaic image from the pixel position generated from the color mosaic image using the coefficient of the expression of the image processing of the color image during execution of the color image process for the color image and- a sampling unit that interpolative-generates the color luminance values of the same color pixels in the above noted color planes as calculated by the above noted coordinates conversion unit from the pixel values of the above noted sampling coordinates and- a color image generation unit that synthesizes the interpolated values of each of the color planes provided by the above noted sampling unit .
12
A color image processing method that generates a color image having multiple color information of each pixel from differing multiple photo electrically converted colored light pixels from a single-chip imaging elements using a color mosaic image containing all single color luminance information of each pixel, with a first step to separate the above noted color mosaic image into multiple color planes having only the same color and in a second step calculating the sampling coordinates for the above noted color mosaic image from the pixel position generated from the color mosaic image using the coefficient of the expression of the image processing of the color image, during execution of the color image process for the color image and in said color plane separation step, for each of the plurality of color planes, the sampling coordinates calculated in the above noted coordinates conversion step are subjected, for the pixel values within the color plane of the same color, an interpolation generation sampling step and a color generation step that generates the above noted color image through synthesis of the color plane interpolative value of the above noted sampling step.
15
A color plane separation method that takes the color mosaic image having all of each pixels single color luminance information obtained from the multiple color photo electrically converted pixels of the imaging elements and separates these into color planes, each plane having only the exact color, the above noted color mosaic image sampling coordinates conversion method using the coefficients that express the color image generated from the above noted color mosaic image, from the above noted color image pixel position when image processing of the pixel position is executed, an interpolative generation sampling method of all the same color value pixels within the above noted color plane by the above coordinates conversion method for each of the multiple color planes separated by the above noted color plane separation method and through synthesis of each color plane interpolative value with the above noted sampling method a color generation method for the color image containing the respective color value for one pixel, including a program that can be read by computer for computer function operation.

Independent claims3

195 paragraphs, as filed

The invention is an image processing device, imaging processing method, program and imaging device.

Field of Invention

The present invention relates to an image processing apparatus, image processing method, program and image-taking device, especially one with single-chip color imaging elements, that executes a demosaic process that generates a color image with interpolation of all pixel multiple color luminance of the color mosaic image from the single-chip color imaging elements.

Background of the Invention

In recent years the popularity of digital cameras among consumers has been making assured progress. One of the reasons that can be cited for this is that the cost of digital cameras has continued to decrease and their price has reached levels within the reach of most consumers. In order to keep the cost down most digital cameras use what is called a single-chip image pickup device. In a single-chip digital camera only one imaging elements device is used to get the color information of each of the pixels within the color image.

With single-chip imaging elements each pixel image only contains the color information of a single color. However, the color image is expressed by combining 3 separate single-color images. In other words, in order to display the color image, the red, green and blue (RGB) values of each pixel is necessary. Because of this, in single-chip digital cameras, demosaic processing (also called color interpolation processing) of each of the pixels, using the color mosaic image, having one of the RGB constituents is performed (for example, see Patent References 1 and 2). The demosaic process is a process that generates a color image with each pixel having all the RGB constituents with the use of interpolation calculation on the lacking color luminance information gathered from the surrounding pixels of the corresponding color mosaic image pixels.

Also, as an example of correction processing, chromatic aberration correction is sometimes performed. The refraction index of lenses used for the imaging optics in digital camera differs in accordance with the wavelength of the image light. Because of this, the image magnification ratio for each of the RGB colors becomes different. Due to this, the size of the image formed on the imaging elements is different for each color. It is known, as can be seen in <figref idref="f0020">Figure 20</figref>, that the image formed on the imaging elements can be misaligned for each color component. This is called the lens magnification chromatic aberration (horizontal distortion). If magnification chromatic aberration is present when photographing under a white light source, the area near both edges of the frame especially look like they become more iridescent and seem to lengthen in the radial ray direction. Also, along with the image misalignment, color shifting at the edges of the frame appears and there is a problem with damage to the quality of the image.

An imaging pickup device capable of suppressing this kind of color shifting is provided such that when there is a color shifting occurrence, in accordance with the image pickup obtained color image signal, the color aberration amount from the color image base position is detected and, in accordance with the detected amount of chromatic aberration, distortion correction is added to the color image signal.

There are also image pickup devices that detect the chromatic aberration amount by detecting the effective edge within the color image and the distance from the base position.

Furthermore, there are also proposals for digital cameras that output a synthesized RGB image signal that has been corrected for chromatic aberration. This is achieved by first taking the RGB signal output from the optical lens image pickup and correcting the chromatic aberration of each (G is base and magnification or reduction correction is done for RB) with the use of the optical lens specific focal point distance value (for example, refer to Patent Reference 3). <dl id="dl0001" compact="compact"><dt>Patent Reference 1:</dt><dd><patcit id="pcit0001" dnum="JPTOKUHYO2004534429B"><text>Tokuhyo 2004-534429</text></patcit></dd><dt>Patent Reference 2:</dt><dd><patcit id="pcit0002" dnum="JPTOKUKAI2000270294B"><text>Tokukai 2000-270294</text></patcit></dd><dt>Patent Reference 3:</dt><dd><patcit id="pcit0003" dnum="JPTOKUKAIHEI6113309B"><text>Tokukai Hei 6-113309</text></patcit></dd></dl>

Summary of the Invention

However, with the traditional art, when generating a color image from a color mosaic image, in addition to the interpolation processing, afterwards a separate interpolation is required for distortion of the color image. In other words, pixel interpolation is being conducted twice. Because of this there is a problem that the processing load became greater. Also, for the image generated by the interpolation, as another interpolation is performed, there is a problem of image deterioration of the generated image becoming greater.

In addition, with the above traditional art, when detecting the chromatic aberration amount at the edge position with the color image, the proper RGB relation cannot be taken and sometimes the edge position cannot be detected.

In other words, in the color mosaic image having magnification chromatic aberration, as can be seen in <figref idref="f0021">Figure 21</figref>, the luminance information values are different for the sampling position due to the RGB constituents, the edges of each color constituent do not match and high resolution cannot be obtained.

Because of this there is sometimes a problem in that the traditional edge position is not recognized as the edge and demosaicing that uses the edge position cannot be performed correctly.

Moreover, the digital camera cited in Patent Reference 3 is not a single-chip, it's a digital camera that uses the so called triple-chip. A triple-chip digital camera is equipped with a single imaging device for each of the RGB colors and uses a synthesis of the RGB output signals from the imaging elements to obtain the color image. In the triple-chip case, the number of the RGB pixels all match the number of pixels in the output image so, with a relatively simple image synthesis, a color image can be obtained. In short, in comparison to single-chip image sensor camera where the RGB pixel number is always less than the image output pixels, a triple-chip digital camera does not have the need of a demosaic process (color interpolation processing).

As opposed to this, in the case of a single-chip digital camera, in addition to the interpolation processing when the color image is generated from the color mosaic image, a separate interpolation process is required to correct the magnification chromatic aberration. In other words, pixel interpolation is being conducted twice. Because of this there is a problem that the processing load became greater. Also, for the image generated by the interpolation, as another interpolation is performed, there is a problem of image deterioration of the generated image becoming greater.

This invention is configured so as to resolve these kinds of problems and its purpose is to enable a lightening of the processing load and a lessening of the image deterioration when the distortion processed color image is generated from the color mosaic image.

In order to resolve these issues, this invention is equipped with a color plane decomposition unit that separates multiple color planes that include just the same chromatic pixel value as that of the color mosaic image, a coordinate conversion unit that calculates sampling coordinates of the corresponding color mosaic image from the pixel position of the color image, a sampling unit that interpolates the pixels within the sampling coordinates of the multiple of color planes and a color generation unit that generates a color image from the interpolation synthesis of each color plane.

The calculations for the sampling coordinates corresponding to the color mosaic image pixel position obtained after distortion processing of the color image is done in the coordinates conversion unit.

With the invention configured in this manner, as the prior step for the generation of the deformed color image from the color mosaic image, the sampling coordinates of the color mosaic image that correspond to the output deformed color image pixel position are calculated. Next, the deformed color image pixel value is generated using interpolation calculation of the color mosaic image pixel value in the aforementioned sampling coordinates. By doing so, from the color mosaic image, the sampling coordinates as pixel values can be sought from the color image that has been distortion processed.

In doing this, the color interpolation processing generating the color image from the color mosaic image and the aforementioned color image distortion processing can be realized at the same time with just one interpolation calculation. Due to this, in addition to being able to lessen the burden of the processing when generating the deformed color image from the color mosaic image the deterioration of image quality caused by the double interpolation required with the traditional method can be suppressed.

Also, in other embodiments of this invention, the invention is equipped with a color plane decomposition unit that separates multiple color planes that include just the same chromatic pixel value as that of the color mosaic image, a coordinate conversion unit that calculates sampling coordinates of the corresponding color mosaic image from the pixel position of the color image, a sampling unit that interpolates the pixels within the sampling coordinates of the multiple of color planes and a color generation unit that generates a color image from the interpolation synthesis of each color plane. In the coordinates conversion unit, for each of the plurality of color planes, using the different value coefficients from each of the color planes, the different sampling coordinate of each color plane is calculated from the color image pixel position.

With the invention configured in this manner, as the prior step for the generation of the deformed color image from the color mosaic image, the pixel position in each color plane adjusted for each color constituent misalignment due to magnification chromatic aberration is calculated using the sampling coordinates of the color mosaic image that correspond to the output deformed color image pixel position. Next, the deformed color image pixel value is generated using interpolation calculation of the color mosaic image pixel value in the aforementioned sampling coordinates. By doing so, from each of the magnification chromatic aberration corrected color image pixel position, the sampling coordinates as pixel values can be sought from the color mosaic image.

Also, in other embodiments of this invention, the coordinates conversion unit, in addition to the color aberration coefficient value that differs according to the color plane, uses at least one of the image distortion coefficient that expresses distortion of the color images, the imaging apparatus hand blur correction coefficient or the distortion aberration coefficient determined by the set focal point distance and distance to photographed object of the imaging optics that introduces the optical image to calculate the differing color plane sampling coordinate from the color image pixel position.

In doing this, the color interpolation processing (demosaic processing) generating the color image from the color mosaic image and the aforementioned imaging optics magnification chromatic aberration correction can be realized at the same time with just one interpolation calculation. Due to this, in addition to being able to lessen the burden of the processing when generating the deformed color image from the color mosaic image the deterioration of image quality caused by the double interpolation required with the traditional method can be suppressed.

With the invention configured in this manner, as the prior step for the generation of the deformed color image from the color mosaic image, in addition to that of the correction of color shifting due to magnification chromatic aberration, the image distortion, hand blur correction, lens state skew correction values of the corresponding color image pixel position on the color mosaic image are used for sampling coordinates calculation.

In doing this, the color interpolation processing generating the color image from the color mosaic image and the aforementioned imaging optics magnification chromatic aberration correction as well as hand blur correction, skew correction and other processes can be realized at the same time with just one interpolation calculation. Due to this, in addition to being able to lessen the burden of the processing when generating the deformed color image from the color mosaic image the deterioration of image quality caused by required multiple interpolations can be suppressed.

Also, other embodiments of this invention are equipped with an aberration coefficient unit that calculates the above noted chromatic aberration coefficient through the taking of a specified image and, using this aberration coefficient calculates each of the differing color plane sampling coordinates from the color image pixel position.

With the invention configured in this manner, as it is equipped with an aberration coefficient unit that calculates the color aberration coefficient for the specified image photographed, this calculated color aberration coefficient is used for realization of the color interpolation processing (demosaic processing), generating the color image from the color mosaic image and the aforementioned imaging optics magnification chromatic aberration correction all with just one interpolation calculation.

Also, in other embodiments of this invention, as it is equipped with an image output unit that is connected to an external device to receive that output and a coefficient input method for the aforementioned color aberration coefficient, using that input color aberration coefficient the color plane differing sample coordinates can be calculated from the color image pixel position.

With the invention configured in this manner, as it is equipped with an image output unit that is connected to an external device to receive that output and a coefficient input method for the aforementioned color aberration coefficient, using that color aberration coefficient, the color interpolation processing (demosaic processing), generating the color image from the color mosaic image and the aforementioned imaging optics magnification chromatic aberration correction all can be realized with just one interpolation calculation.

Brief Explanation of the Figures

<ul id="ul0001" list-style="none"><li><figref idref="f0001">Figure 1</figref> is a diagram showing a schematic configuration of the color imaging apparatus of the 1st embodiment of this invention's image processing device.</li><li><figref idref="f0002">Figure 2</figref> is a diagram indicating the Bayer array of primary color filters for the 1st ∼ 5th embodiments of this invention.</li><li><figref idref="f0003">Figure 3</figref> is a block diagram showing a schematic configuration of the demosaicing unit of the 1st embodiment.</li><li><figref idref="f0004">Figure 4</figref> is a flow diagram illustrating an image processing operation example of demosaicing of the 1st embodiment.</li><li><figref idref="f0005">Figure 5</figref> is an image diagram that concretely illustrates and explains an image processing example of the demosaicing unit of the 1st embodiment.</li><li><figref idref="f0006">Figure 6</figref> is an image diagram that explains the bilinear interpolation of the 1st embodiment.</li><li><figref idref="f0007">Figure 7</figref> is an image diagram that concretely illustrates and explains another image processing example of the demosaicing unit of the 1st embodiment.</li><li><figref idref="f0008">Figure 8</figref> is a diagram showing a schematic configuration of the color imaging apparatus of the 2nd and 3rd embodiments of this invention's image processing device.</li><li><figref idref="f0009">Figure 9</figref> is a block diagram showing a schematic configuration of the demosaic unit of the 2nd and 3rd embodiments.</li><li><figref idref="f0010">Figure 10</figref> is a flow diagram illustrating an image processing operation example of demosaicing of the 2nd ∼ 5th embodiments.</li><li><figref idref="f0011">Figure 11</figref> is an image diagram that concretely illustrates and explains an image processing example of the demosaic unit of the 2nd and 3rd embodiments.</li><li><figref idref="f0012">Figure 12</figref> is an image diagram that concretely illustrates and explains another image processing example of the demosaic unit of the 2nd ∼ 5th embodiments.</li><li><figref idref="f0013">Figure 13</figref> is a diagram showing a schematic configuration of the color image apparatus of the 4th embodiment of this invention's image processing apparatus.</li><li><figref idref="f0014">Figure 14</figref> is an illustration showing an aberration coefficient table example of the 4th embodiment.</li><li><figref idref="f0015">Figure 15</figref> is a block diagram showing a schematic configuration of the demosaic unit of the 4th embodiment.</li><li><figref idref="f0016 f0017">Figure 16</figref> is a schematic configuration of the color image apparatus of the 5th embodiment.</li><li><figref idref="f0018">Figure 17</figref> is a diagram that explains the chromatic aberration amount detection of the 5th embodiment, with <figref idref="f0018">Figure 17 (a)</figref> showing it in chart form and <figref idref="f0018">Figure 17 (b)</figref> showing a chart array for the imaging elements.</li><li><figref idref="f0019">Figure 18</figref> is a diagram that explains the chromatic aberration amount detection of the 5th embodiment, with <figref idref="f0019">Figure 18 (a) (b)</figref> explaining intersection detection and <figref idref="f0019">Figure 18 (c)</figref> explaining edge detection at each intersection. <figref idref="f0019">Figure 18 (d)</figref> shows the setting for the pixel rows.</li><li><figref idref="f0020">Figure 19</figref> is a diagram showing an explanation of edge detection in the 5th embodiment.</li><li><figref idref="f0020">Figure 20</figref> is a diagram explaining the lens's magnification chromatic aberration.</li><li><figref idref="f0021">Figure 21</figref> is a diagram explaining the color shifting from magnification chromatic aberration.</li><li><figref idref="f0021">Figure 22</figref> shows the formula used when converting the RGB color (taking the suppression of false color into consideration) information into YUV color information.</li><li><figref idref="f0021">Figure 23</figref> is a diagram showing the relational expression for the XY coordinates and the UV coordinates.</li><li><figref idref="f0022">Figure 24</figref> shows the formula used for blur correction.</li><li><figref idref="f0022">Figure 25</figref> shows the formula used when converting the coordinates between color planes.</li><li><figref idref="f0022">Figure 26</figref> shows the relational expression formula for the pixel coordinates of the color mosaic image for the XY coordinates of each color plane.</li><li><figref idref="f0022">Figure 27</figref> shows the formula used for finding the chromatic aberration matrix.</li></ul>

Best Embodiment for Implementation of the Invention

Embodiment)

The 1st Embodiment of this present invention will be described in detail in accordance with the accompanying drawings.

Figure lis a diagram illustrating a schematic configuration of a color image apparatus (100) according to a 1st embodiment of the present invention. The color image apparatus (100) of the embodiment is comprised of imaging optics (110), single-chip color image sensor (120), A/D conversion unit (130), demosaic unit (140), visual correction unit (150), compression unit (160) and a recording unit (170). Of these, the demosaic unit (140) corresponds to the image processing apparatus of the present invention.

The single-chip color image sensor (120) is equipped with imaging elements (122) which generates an image signal, the image signal having been passed through the imaging optics (110) and output to the color filter array (121) where it is split into the specified color constituents and, after passing through the color filter array (121) and undergoing photoelectric conversion. The imaging elements (122) are comprised of, for example, either a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor.

The role of the imaging optics (110) is to introduce the optical image of an object onto the single-chip color image sensor (120). For example, the imaging optics (110) can be comprised including an optical low pass filter, a photographic lens and infrared removal filter and other such things. Moreover, the infrared removal filter is for the purpose of removing infrared light from reaching the single-chip color image sensor (120) and is affixed before the optical low pass filter and is configured of a single layer glass block.

The color filter array (121) of the single-chip color image sensor (120) is arrayed on top of the imaging elements (122) pixel light receptor surface in the specified regular pattern and has a role of filtering the received image light into the specified color components. In this embodiment, for the color components, a three color (R, G & B) primary color Bayer color filter array (121).

The primary color Bayer array, as indicated in <figref idref="f0002">Figure 2</figref>, has the R color filter and B color filter alternating rows with each other in addition to the G color filter being arrayed in a checker board pattern. Moreover, for the G color filter, there are filters named Gr and Gb color filters with the Gr color filters being between R color filters in the horizontal direction and the Gb color filters being between B color filters in the horizontal direction.

The roles of the imaging elements (122) is to convert, in a photoelectrical manner, the received optical image into pixel information, storing it as an electric charge and outputting this electric charge to the A/D conversion unit (130). The imaging elements (122) has a plurality of pixel (photo diodes) arrayed in the specified pattern and on top of each of the pixel's light receptor surface the above mentioned primary color Bayer color filter array (121) is overlaid in a regular pattern.

In the above configuration, the light of the photographed subject passes through the imaging optics (110) and forms an image on the imagining elements (122) of the single-chip color image sensor (120). At that time, due to the various inherent aberrations of the imaging optics (110), the image formed of the photographed object suffers deterioration. For example, the straight lines on the photographed object become curved lines on the image due to distortion aberration.

The single-chip color image sensor (120) converts the photographed image on the imaging elements (122) into a color mosaic image as an analog electrical signal.

In other words, the color filter array (121) is, as shown in <figref idref="f0002">Figure 2</figref>, a primary color Bayer array and the RGB (each) color filter is lined up over each pixel of the imaging elements (122). Therefore, the image light of the photographed subject is passed to each pixel as a color mosaic image with only the color component for that pixel being passed to the imaging elements (122). The imaging elements (122) then converts, in a photoelectrical fashion, the received light and outputs it as a color mosaic image electric signal to the A/D conversion unit (130).

The A/D conversion unit (130) processes the photo optically converted analog color mosaic signal from the imaging elements (122) into a digital output signal. Moreover, at the A/D conversion unit (130), directly after conversion, this color mosaic data is also called the RAW DATA. The demosaic unit (140) converts the color mosaic image into a color image. In this present embodiment, by performing image distortion processing at this time, simultaneously, the above mentioned image deterioration from the imaging optics (110) distortion aberration is corrected. The imaging processing method of this demosaic unit (140) is explained in detail later.

The visual correction unit (150) processes the color mosaic image generated by the demosaic unit (140) mainly to improve the appearance of the image. For example, such image correction processing as tone curve (gamma) correction, color saturation enhancement and edge enhancement is done by the visual correction unit (150). The compression unit (160) takes the visual correction unit (150) correct image and compresses it using such method as the JPEG (Joint Photographic Experts Group) method to reduce its size at recording time. The recording unit (170) records the compressed digital image signal to flash memory or other recordable media (no Figure illustration).

Furthermore, each of the constituent components from the demosaic unit (140) to the recording unit (170) may be configured as a separate device or as a single microprocessor device. If the latter case, the single microprocessor executes all the processing of the constituent configuration from the demosaic unit (140) through the recording unit (170).

<figref idref="f0003">Figure 3</figref> is a block diagram illustration of a demosaic unit (140) functional configuration example. <figref idref="f0004">Figure 4</figref> is a flow chart illustration of an example of the image processing operation executed by the demosaic unit (140). <figref idref="f0005">Figure 5</figref> is an image diagram to concretely explain the content of the image processing executed by the demosaic unit (140). As can be seen in <figref idref="f0003">Figure 3</figref>, the demosaic unit (140) is equipped with, as functional constituents, the color plane decomposition unit (141), the coordinate conversion unit (142), the sampling unit (143) and the color generation unit (144).

The color plane decomposition unit (141) separates the color mosaic image output from the A/D conversion unit (130) into multiple color planes having the same chromatic light pixel value (<figref idref="f0004">Figure 4</figref> Step S1). In this embodiment, as indicated in <figref idref="f0005">Figure 5</figref>, the color planes are separated into a R plane with only the R component pixels, the G plane with only the G component pixels and the B plane with only the B component pixels for a total of 3 color planes. Each of the separated color planes is used for the processing by the sampling unit (143).

For example, in order to correct color image distortion due to imaging optics (110) aberration, a non-linear coordinates conversion of the deformed color image generation of the color mosaic image is performed. This coordinates conversion method is known art and the pixel position on the deformed color image and the position of that on the color mosaic image (sampling coordinates) can be sought by calculations.

Below is a detailed explanation of the calculation procedure for the sampling coordinates. First, for the xy coordinate system, the origin point 1 is the image center, the maximum image height (maximum distance from the origin point) with the plus x coordinates being in the right direction from the origin point and the plus y coordinates being in the down direction of the screen from the origin point. In this case, for a 640×480 pixel square color image, if the uv coordinates system pixel coordinates (ud, vd), are assigned as seen in <figref idref="f0002">Figure 2</figref> start from the upper left of the screen and go in the right direction as (0,0), (1,0), (2,0)···, and continue to the next row as (1,0),(1,1),(2,1). . . then the uv coordinates system pixel coordinates (319.5, 239.5 ) are the origin point of the xy coordinates system. Also, for the pixel coordinates system's deflection angle half- length 400 = (6402+4802) 1/2/ 2 the corresponding xy coordinates system maximum height, xy coordinates (xd, yd) must be as in the relational expression shown below for handling pixel coordinates (ud, vd). <maths id="math0001"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">u</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">319.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0001.tif" /></maths><maths id="math0002"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">v</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">239.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0002.tif" /></maths>

The formula listed below is used for the xy coordinates (xd, yd) corresponding coordinates conversion with consideration of correction of the imaging optics (110) distortion aberration. <maths id="math0003"><math display="block"><mi mathvariant="normal">x</mi><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced></math><img file="EP2175656A1_D0003.tif" /></maths><maths id="math0004"><math display="block"><mi mathvariant="normal">y</mi><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced></math><img file="EP2175656A1_D0004.tif" /></maths>

(However, r2 = xd2 + yd2)

Moreover, {k 1,k2} is the coefficient indicating the imaging optics (110) distortion aberration with k1 being the 3rd aberration coefficient and k2 the 5th aberration coefficient. As stated above, the conversion method of non-linear coordinates of color images with distortion aberration is known art, and it is possible to use such things as simulations to seek the distortion aberration coefficient {k1, k2}. This distortion aberration coefficient {k1, k2} is equivalent to this invention's image distortion coefficient.

The above noted calculations results, as the pixel coordinates (us, vs) are not limited to an integer value, are generally non-integral. These color mosaic image pixel coordinates (us, vs) are sampling coordinates. In <figref idref="f0005">Figure 5</figref> one of these sampling coordinates is shown by the 200 code. In the manner as noted above, the color mosaic image is separated into 3 color planes. In <figref idref="f0005">Figure 5</figref> the sampling coordinates 200 on each of the color planes is illustrated. As noted above, since the sampling coordinates value is a non-integer, the sampling coordinates 200 position is offset from the pixel center.

The sampling unit (143), for each of the plurality of color planes separated by the color plane decomposition unit (141), generates an interpolation from the pixel value (sampling value) of the same color within the color plane for the sampling coordinates 200 calculated by the coordinate conversion unit (142) (<figref idref="f0004">Figure 4</figref> Step S3). In other words, the sampling unit (143) outputs the sampling coordinates 200 pixel value of each of the R plane, G plane and B plane from interpolation calculation.

As noted above, as the value for the sampling coordinates 200 (us, vs) is not always necessarily an integer value, linear interpolation is performed from the 4 pixel content (the same color light pixel value that each color plane originally had) surrounding the aforementioned sampling coordinates 200.

This linear interpolation is preferably performed with bilinear interpolation.

As indicated in <figref idref="f0005">Figure 5</figref>, as both the R and B planes have a crisscross lattice point shape for pixel content, the position of the four pixels surrounding the aforementioned sampling contents 200 is at the apex of length of each side of the 2 square. For example, if the sampling coordinates 200 is (us, vs) = (100.8, 101.4) then, in the R plane the four pixels (ud, vd) = (100, 100), (100, 102), (102, 100), (102, 102) that surround it is the R plane pixel content.

When each pixel value of the aforementioned pixel content is expressed by R (100, 100), R (100, 102), R(102, 100), R (102, 102), then the sampling coordinate 200 interpolation pixel content value R (100.8, 101.4) on the R plane, generated by bilinear interpolation as in <figref idref="f0006">Figure 6</figref>, is expressed by the formula shown below. <maths id="math0007"><math display="block"><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100.8</mn><mn mathvariant="normal">101.4</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.6</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.6</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">102</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.4</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.4</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">102</mn></mfenced></math><img file="EP2175656A1_D0007.tif" /></maths>

Also, in the B plane, the position of the four pixels (ud, uv) = (99, 101), (99, 103), (101, 101), (101, 103) surrounding the sampling coordinates 200 position (us, vs) = (100.8, 101.4) is the B plane pixel content. When each pixel value of the aforementioned pixel content is expressed by B (99, 101), B (99, 103), B (101, 101), B (101, 103), then the sampling coordinate 200 interpolation pixel content value B (100.8, 101.4) on the B plane is expressed by the formula shown below. <maths id="math0008"><math display="block"><mi mathvariant="normal">B</mi><mfenced><mn mathvariant="normal">100.8</mn><mn mathvariant="normal">101.4</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.1</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.8</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">B</mi><mfenced><mn mathvariant="normal">99</mn><mn mathvariant="normal">101</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.1</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.2</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">B</mi><mfenced><mn mathvariant="normal">99</mn><mn mathvariant="normal">103</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.9</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.8</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">B</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">101</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.9</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.2</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">B</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">103</mn></mfenced></math><img file="EP2175656A1_D0008.tif" /></maths>

On the other hand, as the G plane has a checkerboard pattern for pixel content, the position of the four pixels surrounding the aforementioned sampling contents 200 is at the apex of length of each side of the square inclined at the of √2 of 45°. In this case the sampling coordinates 200 position in the G plane is (us, vs) = (100.8, 101.4) then the four pixels (ud, vd) = (100, 100), (101, 100), (102, 101), (102, 101) that surround it is the G plane pixel content.

When each pixel value of the aforementioned pixel content is expressed by G (100, 101), G (101, 100), G (101, 102), G (102, 101), then the sampling coordinate 200 interpolation pixel content value G (100.8, 101.4) on the G plane generated by interpolation is expressed by the formula shown below. <maths id="math0009"><math display="block"><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">100.8</mn><mn mathvariant="normal">101.4</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">101</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">102</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">101</mn></mfenced></math><img file="EP2175656A1_D0009.tif" /></maths>

The color generation unit (144) with the synthesis of the interpolation generation color plane pixel value for each color plane of the sampling unit (143) generates a color image having the luminance information of each of the plurality of the pixel values (<figref idref="f0004">Figure 4</figref> Step S4). For example, if for 1 sampling coordinates (us, vs) the interpolation value sought for each color plane by the sampling unit (143) is, each R (us, vs) G (us, vs) B (us, vs) then the color information of the sampling coordinates (us, vs) is generated from a synthesis of these three interpolations by the color generation unit (144).

Furthermore, the color generation unit (144) converts the RGB color information found in this manner into YUV color information (U is luminance level information and U, V are color information) and puts U. V through a low pass filter. Moreover, it is possible to use public knowledge processes for the RGB to YUV conversion process and the low pass filter process for U, V.

The color generation unit (144) does the above processing for all pixels of deformed color image (all sampling coordinates) and outputs the obtained from the deformed color image to the visual correction unit (150). The processing after the visual correction unit (150) is performed exactly as noted above.

As explained in detail above, in the 1st embodiment, as the prior step for the generation of the deformed color image from the color mosaic image, the sampling coordinates 200 of the color mosaic image that corresponds to the output deformed color image pixel position is calculated. Next, the deformed color image pixel value is generated using interpolation calculation of the color mosaic image pixel value in the aforementioned sampling coordinates 200.

By doing so, the color interpolation processing generating the color image from the color mosaic image and the aforementioned color image distortion processing can be realized at the same time with just one interpolation calculation. Due to this, in addition to being able to lessen the burden of the processing when generating the deformed color image from the color mosaic image the deterioration of image quality caused by the double interpolation required with the traditional method can be suppressed.

Also, in the 1st embodiment, as the color mosaic image is split into R, G and B color planes and the sampling coordinates 200 interpolation pixel value is sought for each color plane, a synthesized value for the luminance of the 3 colors in one pixel is included and the color information generated. By doing it in this manner, the sampling coordinates 200 interpolation pixel value can be sought with a simple linear interpolation from the same color luminance in each color plane and the processing burden lessened.

Furthermore, in the above noted 1st embodiment, although we explained with examples of the color plane decomposition unit (141) separating the color mosaic image into three color (R, G and B) planes it is not limited to just this.

For example, as shown in <figref idref="f0007">Figure 7</figref>, it is possible to separate two kinds of green pixels (Gr and Gb) to different color planes. In other words, the color plane decomposition unit (141) takes the color mosaic image output from the A/D conversion unit (130) and separates it into four different color planes, the R plane with only the R component pixels, the Gr plane with only the Gr component pixels, the Gb plane with only the Gb component pixels and the B plane with only the B component pixels.

In this case, the color generation unit (144) adds the sampling coordinates 200 pixel value in the Gr and Gb planes (sampling value) generated by the sampling unit (143) and generates the color information including individual R,G and B component luminance information within 1 pixel. For example, the R component or B component sampling value is used as is while the G component value is the average value of Gr and Gb.

Also, when the sampling coordinates 200 pixel value is calculated after separating the G component into the Gr plane and the Gb plane, and the demosaic unit (140) calculates the difference in the sampling unit (143) generated interpolation value of the Gr and Gb plane, it is possible to add a pseudo-color judgment unit and, in accordance with the aforementioned interpolation pixel value difference judge the presence or lack of pseudo-color.

The Bayer array single-chip color image sensor (120) has a problem with red and blue pseudo-color generation for black and white stripe patterns near the Nyquist frequency. By subtracting the difference between the sampling coordinates 200 interpolation pixel value of the Gr plane and the sampling coordinates 200 interpolation pixel value of the Gb plane, the presence of pseudo-color on the stripe pattern can be detected and if pseudo-color is detected it can be suppressed.

In other words, as the filter for both Gr and Gb is normally the same G color filer, the interpolation pixel value of the Gr plane and the Gb plane should both be the same value.

However, if pseudo-color is occurring there is a difference in the found interpolation pixel values for the Gr and Gb planes. Therefore, by looking at the interpolation pixel value difference a pseudo-color occurrence can be detected. In cases where pseudo-color presence is determined the color generation unit (144) performs a conversion, at the RGB color information to YUV color information conversion processing, with suppression of the pseudo-color using the formula as shown in <figref idref="f0021">Figure 22</figref>.

As explained above, in the 1st embodiment, as an example of color image distortion processing, skew correction was explained in detail but it is not limited to just this. For example, in place of the above noted skew correction for image distortion, or in addition to it, when at least one of magnification, reduction or rotation is included in the color image processing it is possible to use the demosaic unit (140) of this embodiment. In this case, it is possible for the enlargement, reduction or rotation to be expressed by affine transformation and be used for the image distortion coefficient.

Embodiment)

The 2nd embodiment of this present invention will be described in accordance with the accompanying drawings. <figref idref="f0008">Figure 8</figref> is a diagram illustrating a schematic configuration of a color image apparatus (100) according to the 2nd embodiment of the present invention. The color image apparatus (100) of the embodiment is comprised of imaging optics (110), single-chip color image sensor (120), A/D conversion unit (130), demosaic unit (140), visual correction unit (150), compression unit (160) and a recording unit (170). Of these, the demosaic unit (140) corresponds to the image processing apparatus of the present invention. Moreover, the code symbols that have the same code as those in <figref idref="f0001">Figure 1</figref> indicates the same function so we will not repeat their explanation here.

In the 2nd embodiment, the photographed object image goes through the imaging optics (110) and forms an image on the imaging elements (122) of the single-chip color image sensor (120).

At this time, due to the magnification chromatic aberration of the imaging optics (110), for each RGB color component image shifting (color shifting) on the imaging elements (122) occurs. The demosaic unit (140) converts the color mosaic image into a color image. In this embodiment, through the processing of the magnification chromatic aberration correction at the same time, the color shifting of the magnification chromatic aberration of the above noted imaging optics (110) is corrected.

<figref idref="f0009">Figure 9</figref> is a block diagram illustration of a demosaic unit (140) functional configuration example. <figref idref="f0010">Figure 10</figref> is a flow chart illustration of an example of the image processing operation executed by the demosaic unit (140). <figref idref="f0011">Figure 11</figref> is an image diagram to concretely explain the content of the image processing executed by the demosaic unit (140).

As can be seen in <figref idref="f0009">Figure 9</figref>, the demosaic unit (140) is equipped with, as functional constituents, the color plane decomposition unit (141), the coordinate conversion unit (142), the sampling unit (143) and the color generation unit (144). Moreover, the code symbols in <figref idref="f0009">Figure 9</figref> that have the same code as those in <figref idref="f0003">Figure 3</figref> have the same functions so we will concentrate on explaining the different parts.

The color plane decomposition unit (141) separates, as shown in <figref idref="f0011">Figure 11</figref>, the color mosaic image output from the A/D conversion unit (130) into multiple color planes having the same chromatic light pixel value - R plane with only R pixels, G plane with only G pixels and B plane with only B pixels (<figref idref="f0010">Figure 10</figref> Step S11).

The coordinate conversion unit (142), using the coefficient of the expression of the image distortion of the color image, from the pixel position generated from the color mosaic image, calculates the color mosaic image sampling coordinates (<figref idref="f0010">Figure 10</figref> Step S12) when distortion processing (hereinafter "deformed color image") is executed for the color image. In this embodiment especially, the coordinates unit (142), for each of the plurality of color planes separated by the color plane decomposition unit (141), using a different chromatic aberration coefficient for the values of each color plane, calculates the differing color planes sampling coordinates. In other words, although there is an effect from the magnification chromatic aberration of the imaging optics (110), in order to not have that effect expressed in the color image, the sampling coordinates are given different values for the color plane corresponding to the differing color light.

Below is a detailed explanation of the 2nd embodiment's calculation procedure for the sampling coordinates. Moreover, the xy coordinates system considered here are the same as those of the 1st embodiment and the xy coordinates (xd, yd) corresponding to the pixel coordinates (ud, vd) are expressed by the relational expression shown below. <maths id="math0010"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">u</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">319.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0010.tif" /></maths><maths id="math0011"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">v</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">239.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0011.tif" /></maths>

The imaging optics (110) magnification chromatic aberration corrected coordinates conversion for these xy coordinates (xd, yd) is performed using the formula shown below. Here the coordinates conversion is done separately for each of the three color planes, the R plane, G plane and B plane. <maths id="math0012"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">,</mo><mspace width="1em" /><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">R</mi></msub><mspace width="1em" /></math><img file="EP2175656A1_D0012.tif" /></maths><maths id="math0013"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><mn>1</mn><mo mathvariant="normal">,</mo><mspace width="1em" /><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><mn>1</mn><mspace width="1em" /></math><img file="EP2175656A1_D0013.tif" /></maths><maths id="math0014"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">,</mo><mspace width="1em" /><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">B</mi></msub><mspace width="1em" /></math><img file="EP2175656A1_D0014.tif" /></maths>

Moreover, {kR, kB} is the coefficient indicating the imaging optics (110) magnification chromatic aberration, kR being the R component magnification ratio of the G component and kB being the B component magnification ratio for the G component. It is possible for these chromatic aberration coefficients {kR, kB} to be calculated by optical simulation or other means.

On the other hand, if the color mosaic image is made of a 1600×1200 pixel square and if the pixel coordinates (us, vs) of the uv coordinates system is assigned in the same manner as that noted for the above color image, the uv coordinates system's pixel coordinates (799.5, 599.5) are the origin point for the xy coordinates and the coordinates deflection angle half-length is 1000 = (16002 + 12002) 1/2/ 2, corresponding to the xy coordinates system maximum image height. Due to this, the color mosaic image pixel coordinates (usR, vsR) (usG, vsG) (usB, vsB) corresponding to the corrected chromatic aberration color image xy coordinates (xR, yR), (xG, yG) and (xB, yB) on each color plane corresponding to the color image pixel coordinates are as shown below. <maths id="math0015"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sR</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0015.tif" /></maths><maths id="math0016"><math display="block"><msub><mi mathvariant="normal">v</mi><mi>sR</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0016.tif" /></maths><maths id="math0017"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sG</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0017.tif" /></maths><maths id="math0018"><math display="block"><msub><mi mathvariant="normal">v</mi><mi>sG</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0018.tif" /></maths><maths id="math0019"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sB</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0019.tif" /></maths><maths id="math0020"><math display="block"><msub><mi mathvariant="normal">v</mi><mi>sB</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0020.tif" /></maths>

The above noted calculations results, as the pixel coordinates (usR , vsR), (usG ,vsG) and (usB ,vsB) are not limited to an integer value, are generally non-integral. These color mosaic image pixel coordinates (usR, vsR), (usG,vsG) and (usB,vsB) are sampling coordinates for each color plane. In <figref idref="f0011">Figure 11</figref> the R plane sampling coordinates is shown by the 301 code, the G plane sampling coordinates is shown by the 302 code and the B plane sampling coordinates is shown by the 303 code. As noted above, since the sampling coordinates value is a non-integer, the sampling coordinates 301, 302 and 303 position location is offset from the pixel center in each color plane.

The sampling unit (143), for each of the plurality of color planes separated by the color plane decomposition unit (141), generates an interpolation from the pixel value (sampling value) of the same color within the color plane for the sampling coordinates 301,302 and 303 calculated by the coordinate conversion unit (142) (<figref idref="f0010">Figure 10</figref> Step S13). In other words, the sampling unit (143) outputs the sampling coordinates 301, 302 and 303 pixel value of each of the R plane, G plane and B plane from interpolation calculation.

As noted above, as the value for the sampling coordinates 301, 302 and 303 (usR, vsR), (usG, vsG) and (usB, vsB) is not always necessarily an integer value, linear interpolation is performed from the 4 pixel content (the same color light pixel value that each color plane originally had) surrounding the aforementioned sampling coordinates 301, 302 and 303.

As indicated in <figref idref="f0011">Figure 11</figref>, as both the R and B planes have a crisscross lattice point shape for pixel content, the position of the four pixels surrounding the aforementioned sampling contents 301, 303 is at the apex of length of each side of the 2 square. For example, if the sampling coordinates 301, 303 is (usR, vsR) = (100.8, 101.4) then, in the R plane the four pixels (ud, vd) = (100, 100), (100, 102), (102, 100), (102, 102) that surround it is the R plane pixel content

When each pixel value of the aforementioned pixel content is expressed by R (100, 100), R (100, 102), R(102, 100), R (102, 102), then the sampling coordinate 301 interpolation pixel content value R (100.8, 101.4) on the R plane, generated by bilinear interpolation as in <figref idref="f0006">Figure 6</figref>, is expressed by the formula shown below. <maths id="math0021"><math display="block"><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100.8</mn><mn mathvariant="normal">101.4</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.6</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.6</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">102</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.4</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.4</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">R</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">102</mn></mfenced></math><img file="EP2175656A1_D0021.tif" /></maths>

On the other hand, as the G plane has a checkerboard pattern for pixel content, the position of the four pixels surrounding the aforementioned sampling contents 302 is at the apex of length of each side of the square inclined at the of √2 of 45°. In this case the sampling coordinates 302 position in the G plane is (usG, vsG) = (101.0, 101.4) then the four pixels (ud, vd) = (100, 101), (101, 100), (101, 102), (102, 101) that surround it is the G plane pixel content.

When each pixel value of the aforementioned pixel content is expressed by G (100, 101), G (101, 100), G (101, 102), G (102, 101), then the sampling coordinates 302 interpolation pixel content value G (101.0, 101.4) on the G plane generated by interpolation is expressed by the formula shown below. <maths id="math0022"><math display="block"><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">101.0</mn><mn mathvariant="normal">101.4</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">100</mn><mn mathvariant="normal">101</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0</mn><mspace width="1em" /><mn mathvariant="normal">3</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">100</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">101</mn><mn mathvariant="normal">102</mn></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">0.3</mn><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.7</mn><mo mathvariant="normal">*</mo><mi mathvariant="normal">G</mi><mfenced><mn mathvariant="normal">102</mn><mn mathvariant="normal">101</mn></mfenced></math><img file="EP2175656A1_D0022.tif" /></maths>

The color generation unit (144) with the synthesis of the interpolation generation color plane pixel value for each color plane of the sampling unit (143) generates a color image having the luminance information of each of the plurality of the pixel values (<figref idref="f0010">Figure 10</figref> Step S14). Furthermore, the RGB color information found by the color generation unit (144) is converted into YUV color information (Y is luminance information and U, V is color information) and the U, V information goes through a low pass filter. Moreover, it is possible to accomplish this RGB to YUV conversion and low pass filter processing of U, V with public knowledge art. The color generation unit (144) performs the above processing for all pixel (all sampling coordinates) of the color image and the obtained deformed color image results are output to the visual correction unit (150). Processing from the visual correction unit (150) stage is exactly the same as noted above.

The sampling coordinates 301,302 and 303 R (usR, vsR), G (usG, vsG) and B (usB, vsB) interpolation pixel values use in processing at the color generation unit (144) have magnification chromatic aberration color shifting considered and indicated the same part one the photographed object. Due to this, in the part where there is a black and white boundary, the R, G and B changes at the same time. In this manner a signal with a brilliant and sharp RGB synthesized color image can be obtained. In other words, a color image equivalent to one that was photographed with optics not having any magnification chromatic aberration can be obtained.

As explained in detail above, in the 2nd embodiment, as the prior step for the generation of the deformed color image from the color mosaic image, the sampling coordinates 301 302 and 303 of the color mosaic image that correspond to the output deformed color image pixel position are calculated. Next, the deformed color image pixel value is generated using interpolation calculation of the color mosaic image pixel value in the aforementioned sampling coordinates 301 302 and 303.

Also, in the 2nd embodiment, as the color mosaic image is split into R, G and B color planes and the sampling coordinates 301 302 and 303 interpolation pixel value is sought for each color plane, a synthesized value for the luminance of the 3 colors in one pixel is included and the color information generated. By doing it in this manner, the sampling coordinates 301 302 and 303 interpolation pixel value can be sought with a simple linear interpolation from the same color luminance in each color plane and the processing burden lessened.

Furthermore, in the above noted 2nd embodiment, although we explained with examples of the color plane decomposition unit (141) separating the color mosaic image into three color (R, G and B) planes it is not limited to just this.

For example, as shown in <figref idref="f0012">Figure 12</figref>, it is possible to separate two kinds of green pixels (Gr and Gb) to different color planes. In other words, the color plane decomposition unit (141) takes the color mosaic image output from the A/D conversion unit (130) and separates it into four different color planes, the R plane with only the R component pixels, the Gr plane with only the Gr component pixels, the Gb plane with only the Gb component pixels and the B plane with only the B component pixels.

In this case, the color generation unit (144) adds the sampling coordinates 302 pixel value in the Gr and Gb planes (sampling value) generated by the sampling unit (143) and generates the color information including individual R,G and B component luminance information within 1 pixel. For example, the R component or B component sampling value is used as is while the G component value is the average value of Gr and Gb.

Also, when the sampling coordinates 302 pixel value is calculated after separating the G component into the Gr plane and the Gb plane, and the demosaic unit (140) calculates the difference in the sampling unit (143) generated interpolation value of the Gr and Gb plane, it is possible to add a pseudo-color judgment unit and, in accordance with the aforementioned interpolation pixel value difference judge the presence or lack of pseudo-color.

The Bayer array single-chip color image sensor (120) has a problem with red and blue pseudo-color generation for black and white stripe patterns near the Nyquist frequency. By subtracting the difference between the sampling coordinates 302 interpolation pixel value of the Gr plane and the sampling coordinates 302 interpolation pixel value of the Gb plane, the presence of pseudo-color on the stripe pattern can be detected and if pseudo-color is detected it can be suppressed.

In other words, as the filter for both Gr and Gb is normally the same G color filer, the interpolation pixel value for the Gr plane and the Gb plane should both be the same value.

Embodiment)

The 3rd embodiment of this present invention will be described in accordance with the accompanying drawings. <figref idref="f0008">Figure 8</figref> is a diagram illustrating a schematic configuration of a color image apparatus (100) according to the 3rd embodiment of the present invention. Also, the functional configuration of the demosaic unit (140) is the same as that shown in <figref idref="f0009">Figure 9</figref>. However, the processing content performed by the coordinate conversion unit (142) that the demosaic unit (140) is equipped with is different than that of the 2nd embodiment. Below, we will concentrate on explaining the parts that are different from those of the 2nd embodiment.

In the 3rd embodiment, the coordinate conversion unit (142) uses each of the differing chromatic aberration coefficients of the plurality of color planes that were separated by the color plane decomposition unit (141) and the image distortion coefficient for the image distortion of the color image and, from the color image pixel position, calculates the different sampling coordinates of each color plane for the sampling coordinates of the color mosaic image corresponding to the color image pixel position after chromatic aberration correction and image distortion is executed.

Here we will use an image distortion example to explain the distortion processing performed to correct distortion aberration of the imaging optics (110). If the imaging optics (110) has any distortion aberration, when the light of the photographed subject passes through the imaging optics (110) and forms an image on the imagining elements (122) of the single-chip color image sensor (120) there is image deterioration. In other words, for example, the straight lines on the photographed object become curved lines on the image due to distortion aberration. In this 3rd embodiment, in addition to the correction of image deterioration from the magnification chromatic aberration, correction of distortion aberration can also be accomplished at the same time.

Due to this, from the color image pixel position generated from the color mosaic image, the coordinate conversion unit (142) calculates the sampling coordinates on each color plane on the color mosaic image that has had the magnification chromatic aberration and distortion aberration corrected. In more concrete detail, the coordinate conversion unit (142) uses the formulas shown below with the xy coordinates (xd + yd) that have the corresponding imaging optics (110) chromatic aberration and distortion aberration correction (However, r2 = x d2 + yd2). <maths id="math0023"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">g</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0023.tif" /></maths><maths id="math0024"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">g</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0024.tif" /></maths><maths id="math0025"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><mn>1</mn><mo>+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0025.tif" /></maths><maths id="math0026"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><mn>1</mn><mo>+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0026.tif" /></maths><maths id="math0027"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0027.tif" /></maths><maths id="math0028"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">=</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><mfenced><msub><mi mathvariant="normal">k</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">d</mi></msub><mo>⁢</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0028.tif" /></maths>

Furthermore, in order to correct color image distortion due to imaging optics (110) aberration, a non-linear coordinates conversion of the deformed color image generation of the color mosaic image can be performed. As stated above, the conversion method of non-linear coordinates of color images with distortion aberration is known art, and it is possible to use such things as simulations to seek the above mentioned distortion aberration coefficients {k1, k2}. This distortion aberration coefficient {k1, k2} is equivalent to this invention's image distortion coefficient.

In seeking the color mosaic image pixel coordinates (usR, vsR) (usG, vsG) (usB, vsB) of these xy coordinates (xR, yR) (xG, yG) (xB, yB) on the color mosaic image, the calculation procedure is the same as in the 2nd embodiment for finding the color mosaic image pixel coordinates (us, vs) from the xy coordinates (x, y) and is found using the formulas shown below. <maths id="math0029"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sR</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0029.tif" /></maths><maths id="math0030"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sR</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">R</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0030.tif" /></maths><maths id="math0031"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sG</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0031.tif" /></maths><maths id="math0032"><math display="block"><msub><mi mathvariant="normal">v</mi><mi>sG</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0032.tif" /></maths><maths id="math0033"><math display="block"><msub><mi mathvariant="normal">u</mi><mi>sB</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">799.5</mn></math><img file="EP2175656A1_D0033.tif" /></maths><maths id="math0034"><math display="block"><msub><mi mathvariant="normal">v</mi><mi>sB</mi></msub><mo mathvariant="normal">=</mo><mn mathvariant="normal">1000</mn><mo mathvariant="normal">*</mo><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">B</mi></msub><mo mathvariant="normal">+</mo><mn mathvariant="normal">599.5</mn></math><img file="EP2175656A1_D0034.tif" /></maths>

Next with the sampling unit (143) set as above, the pixels within the color plane having the same color luminance value (sampling value) calculated by the coordinate conversion unit (142) is generated with interpolation. Also, the color generation unit (144), through the synthesis of the interpolation generated pixel positions of each color plane with the sampling unit (143), generates, a color image with multicolor luminance information containing pixels.

As explained in detail above, in the 3rd embodiment, the color interpolation processing of the color image from the color mosaic image and the correction of the magnification chromatic aberration of the imaging optics (110) correction (image distortion processing) can be realized at the same time with just one interpolation calculation. Due to this, the magnification chromatic aberration is corrected and in addition to being able to lessen the burden of the processing when generating the deformed color image from the color mosaic image the deterioration of image quality caused by multiple interpolation processing can be suppressed.

Furthermore, in the above noted 3rd embodiment, although we explained in detail with examples of the distortion processing for the correction of the distortion aberration of the imaging optics (110) it is not limited to just this. For example, as regards image distortion, it is also possible for color image rotation due to digital zoom magnification or reduction processing or hand blur correction processing. In these cases, it is possible for the enlargement, reduction or rotation to be expressed by affine transformation and the coefficient expressed by the affine transformation used as the image distortion coefficient.

Embodiment)

Next, the 4th embodiment of this present invention will be described in accordance with the accompanying drawings. <figref idref="f0013">Figure 13</figref> is a diagram illustrating a schematic configuration of a color image apparatus (100) according to the 4th embodiment of the present invention. Moreover, as the items with the same code numbers as those in <figref idref="f0008">Figure 8</figref> have the exact same function we will not repeat the explanation for them here.

As indicated in <figref idref="f0013">Figure 13</figref>, the 4th embodiment's color image apparatus (100) is comprised from imaging optics (110), single-chip color image sensor (120), A/D conversion unit (130), demosaic unit (140), visual correction unit (150),compression unit (160), recording unit (170), blur detection unit (180), aberration coefficient setting unit (190), calibration unit (200) and aberration coefficient table and recording unit (210). Of these, the demosaic unit (140) and the aberration coefficient setting unit (190) correspond to the image processing apparatus of this invention.

In <figref idref="f0013">Figure 13</figref> the light of the photographed subject goes through the imaging optics (110) and forms an image on the imaging elements (122) of the single-chip color image sensor (120). At this time, the various types of distortion aberration of the imaging optics (110) cause a deterioration of the picked up image of the photographed object. For example, due to distortion aberration, straight lines of the subject become curved lines and, due to magnification chromatic aberration, the image formed on imaging elements (122) is subjected to image (color) shifting for each color constituent. Moreover, in the 4th embodiment, for the imaging optics (110), it is possible to change the lens state of such things as the focal point distance (zoom) or the distance to the photographed subject (focus).

The blur detection unit (180) detects shaking of the color image apparatus (100) and has a blur correction coefficient {z, θ, dx, dy} that is set in the demosaic unit (140). There are methods such as gyro sensor use or measurement of feature-point matching variation when taking multiple photographs that are used for blur detection but this invention does not have any restriction as to the method used. Here, z is the blur size correction value for a photographed object in the color image apparatus (100) longitudinal direction, θis the rotational correction value for roll axis blur of the object, dx is the correction value for blur in the right-left direction or yaw of the object and dy is the correction value for blur in the up-down direction or pitch of the object. Moreover, it is also permissible for z to include the digital zoom magnification ratio

The aberration coefficient setting unit (190) corresponds to the coefficient setting part of the present invention. The aberration coefficient setting unit (190) detects the lens state and reads the aberration coefficient table recording unit (210) for the aberration coefficient {k1, k2, p1, p2, kR, kB} appropriate to the lens state and then sets the coefficient for the demosaic unit (140). Here, {k1, k2, p1, p2} are coefficients indicating the imaging optics (110) distortion aberration, {k1, k2} indicates distortion in the radial ray direction and {p 1, p2} indicate the linear direction distortion. {kR,kB} are the coefficients indication the imaging optics (110) magnification chromatic aberration, kR is he R component magnification ratio for the G component and kB is the B component magnification ratio for the G component.

Moreover, in cases where there is a change in the lens state, the aberration coefficient setting unit (190) detects the change and reads the aberration coefficient table recording unit (210) for the aberration coefficient appropriate to the changed lens state and then sets the coefficient for the demosaic unit (140). The lens state detection can be performed, for example, by receiving the lens state setting information of the imaging optics (110) from the controller (no Figure illustrated) which controls color image apparatus (100).

By the way, it is not practical to record every possible aberration coefficient for the lens state to the table. Therefore, in this embodiment, only the aberration coefficients for a limited number of lens states are recorded to the aberration coefficient table recording unit (210). For example, for focal point distance and photographed subject distance, each has 3 patterns of lens states recorded for a total of 9 lens states and also their corresponding aberration coefficient values are recorded. To further explain, <figref idref="f0014">Figure 14</figref> shows part related to the aberration coefficient k1 of this aberration coefficient table.

In <figref idref="f0014">Figure 14</figref>, for example, if the lens state set in the imaging optics (110) is distance to photographed subject = 2.0m, focal point distance = 3.5mm, there is no suitable aberration coefficient value in the <figref idref="f0014">Figure 14</figref> aberration coefficient table. At that point, the aberration coefficient setting unit (190) reads in the aberration coefficient 0.08 for photographed subject distance = Mid: 1.0 m, focal point distance = Wide: 28mm aberration coefficient 0.08; the aberration coefficient 0.05 for photographed subject distance = Far: Inf, focal point distance = Wide: 28mm; the aberration coefficient 0.02 for photographed subject distance = Mid:1.0m, focal point distance = Mid: 50mm; and the aberration coefficient 0.00 for photographed subject distance = Far: Inf, focal point distance = Mid: 5 0mm from the aberration coefficient table and then does an interpolation on these 4 aberration coefficients.

Here, it is best to do an interpolation for the focal point distance and the distance to the photographed subject. <maths id="math0035"><math display="block"><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo mathvariant="normal">=</mo><mfenced><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">2.0</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mi>lnf</mi></mfenced><mo>⁢</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">35</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">50</mn></mfenced><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.08</mn><mo mathvariant="normal">+</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">1.0</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">2.0</mn></mfenced><mo>⁢</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">35</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">50</mn></mfenced><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.05</mn><mo mathvariant="normal">+</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">2.0</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mi>lnf</mi></mfenced><mo>⁢</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">28</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">35</mn></mfenced><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.02</mn><mo mathvariant="normal">+</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">1.0</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">2.0</mn></mfenced><mo>⁢</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">28</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">35</mn></mfenced><mo mathvariant="normal">*</mo><mn mathvariant="normal">0.00</mn></mfenced><mo mathvariant="normal">/</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">1.0</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mi>lnf</mi></mfenced><mo mathvariant="normal">/</mo><mfenced><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">28</mn><mo mathvariant="normal">-</mo><mn mathvariant="normal">1</mn><mo mathvariant="normal">/</mo><mn mathvariant="normal">50</mn></mfenced><mo mathvariant="normal">=</mo><mn mathvariant="normal">0.04</mn></math><img file="EP2175656A1_D0035.tif" /></maths>

As shown above, this kind of interpolative calculated value is set to the demosaic unit (140). The other distortion coefficients, {k2, p1, p2, kR, kB}, are set to the demosaic unit (140) in the same manner after their interpolative values are calculated.

The calibration unit (200) generates the values for recording to the aberration coefficient table recording unit (210) in accordance with the digital image signal output form the A/D conversion unit (130). The calibration unit (200) seeks the aberration values of all coefficients {k1, k2, p1, p2, kR, kB} for the plurality of lens states that are to be recorded to the table and records these to the aberration coefficient table recording unit (210).

Moreover, although the calibration unit (200) can be established as an internal part of the color image apparatus (100), when it is configured as a standalone unit separate from the color image apparatus (100) it must be equipped with a communications means so that the color image apparatus (100) can output the photographed image (the A/D conversion unit (130) digital image signal output) to the calibration unit or input the aberration coefficient received from the calibration unit.

While the demosaic unit (140) is performing correction in accordance with the blur correction coefficients {z, θ, dx, dy} as set by the blur detection unit (180) and the aberration coefficients {k1, k2, p1, p2, kR, kB} as set the aberration coefficient unit (190) the conversion from the color mosaic image to the color image is performed.

<figref idref="f0015">Figure 15</figref> is a block diagram showing a functional schematic configuration example of the demosaic unit (140). As is indicated in <figref idref="f0015">Figure 15</figref>, the demosaic unit (140) functional configuration is comprised together with the color plane decomposition unit (141), coordinate conversion unit (142), sampling unit (143) and the color generation unit (144).

The color plane decomposition unit (141) takes the color mosaic image output from the A/D conversion unit (130) and separates it into multiple color planes having only the same value pixel color. For example, the color plane decomposition unit (141) as indicated in <figref idref="f0012">Figure 12</figref>, separates it into 4 color planes, a R plane which has only R pixels, a Gr plane which has only Gr pixels, a Gb plane that has only Gb pixels and a B plane that has only B pixels.

The coordinates conversion unit (142) using the above mentioned blur correction coefficients {z, θ, dx, dy} and the aberration coefficients {k1, k2, p1, p2, kR, kB}, calculates the sampling coordinates on the color mosaic image for the corresponding color image pixel position when chromatic aberration correction, distortion aberration and had blur correction is performed using the color image pixel position generation of the color mosaic image. Below we will explain the sampling coordinates calculation procedure in detail.

First, as the xy coordinates system, the origin point is the center of the image and 1 is the maximum image height (maximum distance from the origin point), with the plus x coordinates in the right direction from the origin point of the screen and the plus y coordinates in the down direction from the origin point of the screen. Also, the output color image is 1600×1200 pixel square. In this case, the pixel coordinates (ud, vd) for the uv coordinate system of the color image are, as indicated in <figref idref="f0002">Figure 2</figref>, from the upper left of the screen going to right (0,0), (1,0), (2,0).... with the next row being (1,0), (1,1), (2,1) ... then the xy coordinates (xd, yd) corresponding to the pixel coordinates (ud, vd) are as expressed by the relational expression shown in <figref idref="f0021">Figure 23</figref>.

The coordinates conversion unit (142) first applies the above mention blur correction coefficients {z, θ, dx, dy} to these xy coordinates (xd, yd) and applies the formula shown in <figref idref="f0022">Figure 24</figref> to obtain the post blur correction xy coordinates (x', y').

Furthermore, the coordinate conversion unit (142) applies the formulas shown below to the distortion aberration coefficients {k1, k2, p1, p2} and see the xy coordinates (xG + yG) in the Gr plane and the Gb plane. <maths id="math0036"><math display="block"><msub><mi mathvariant="normal">x</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><mi mathvariant="normal">xʹ</mi><mo>⁢</mo><mfenced><mn>1</mn><mo>+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><mi mathvariant="normal">xʹyʹ</mi><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">rʹ</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><mi mathvariant="normal">xʹ</mi><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0036.tif" /></maths><maths id="math0037"><math display="block"><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">G</mi></msub><mo mathvariant="normal">=</mo><mi mathvariant="normal">yʹ</mi><mo>⁢</mo><mfenced><mn>1</mn><mo>+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">k</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">4</mn></msup></mfenced><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mspace width="1em" /><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">2</mn></msub><mo>⁢</mo><mi mathvariant="normal">xʹyʹ</mi><mo mathvariant="normal">+</mo><msub><mi mathvariant="normal">p</mi><mn mathvariant="normal">1</mn></msub><mo>⁢</mo><mfenced><msup><mi mathvariant="normal">r</mi><mn mathvariant="normal">2</mn></msup><mo mathvariant="normal">+</mo><mn mathvariant="normal">2</mn><mo>⁢</mo><msup><msub><mi mathvariant="normal">y</mi><mi mathvariant="normal">d</mi></msub><mn mathvariant="normal">2</mn></msup></mfenced></math><img file="EP2175656A1_D0037.tif" /></maths>

(However, r'2 = x'2 + y'2)

In addition, the coordinate conversion unit (142) by doing the coordinate conversion between planes considering the imaging optics (110) magnification chromatic aberration coefficients {kB, kB} using the formula as shown in <figref idref="f0022">Figure 25</figref>, finds each of the xy coordinates (xR, yR) and (xB, yB) for the R pane and the B plane. Here, with the G plane as the locator, dRx, dRy are the coefficients that represent the B plane parallel misalignment.

On the other hand, with the color mosaic image as a 1600×1200 pixels square and the uv coordinate system pixel coordinates (uS, vS) assigned in the same manner as noted above for the color image, the color mosaic pixel coordinates (xsR. ysR), (xsG, ysG), and (xsB, ysB) corresponding to the xy coordinates (xR. yR), (xG, yG), and (xB, yB) of each color plane are as indicated by the formula in <figref idref="f0022">Figure 26</figref>.

The sampling unit (143), for each of the plurality of color planes separated by the color plane decomposition unit (141), generates an interpolation from the pixel value (sampling value) of the same color within the color plane for the sampling coordinates calculated by the coordinate conversion unit (142). The color generation unit (144), by synthesis of each of the color plane interpolative pixel value of the interpolation generation of the sampling unit (143), generates a color image having each of the multiple color luminance information for all the pixels. The processing content of this sampling unit (143) and the color generation unit (144) is the same as that of the 2nd embodiment or 3rd embodiment.

As explained in detail above, in the 4th embodiment, the color interpolation processing of the color image from the color mosaic image, the correction of the magnification chromatic aberration of the imaging optics (110) correction (image distortion processing), aberration correction according to the lens state of the imaging optics (110) and image stabilization process correction of the color image apparatus (100) can be realized at the same time with just one interpolation calculation. Due to this, the magnification chromatic aberration is corrected and the processing burden is lightened when generating the deformed color image having distortion aberration correction and image stabilization process correction from the color mosaic image. In addition, the deterioration of image quality caused by multiple interpolation processing can be suppressed.

Furthermore, in the above noted 4th embodiment, although we explained in detail with examples using the blur correction coefficients {z, θ, dx, dy} and the aberration coefficients {k1, k2, p1, p2, kR, kB} when the coordination conversion unit (142) calculates the sampling coordinates, they do not all need to be used. In other words, only the aberration coefficient {kR, kB} is absolutely necessary with the remainder being able to be used in a combination.

Also, in the above noted 4th embodiment, although we explained in detail with examples about establishing the calibration unit (200) and making the aberration coefficient values variable, it is not limited to just this. For example, it is also possible not to establish a calibration unit (200) and fix the values of the aberration coefficient table.

Embodiment)

Next, the 5th embodiment of this present invention will be described in accordance with the accompanying drawings. <figref idref="f0016 f0017">Figure 16</figref> is a diagram illustrating a schematic configuration of a color image apparatus (100A) according to the 5th embodiment of the present invention. Moreover, as the items in <figref idref="f0016 f0017">Figure 16</figref> with the same code numbers as those in <figref idref="f0008">Figure 8</figref> have the exact same function we will not repeat the explanation for them here.

The 5th embodiment is equipped with a chromatic aberration amount detection apparatus (200A) that detects the color aberration coefficient from the specified CH Chart photographed image and the chromatic aberration coefficients detected by the chromatic aberration amount detection apparatus (200A) are stored in the coordinates conversion unit (142) of the 3rd embodiment or the aberration coefficient setting unit (190) of 4th embodiment and, in the same manner as for the 2nd - 4th embodiments , in the demosaic unit (140), configured so as it calculates the sampling coordinates for each color plane of chromatic aberration corrected color image pixel position.

Moreover, this CH Chart is equivalent to the specified image of 23 and 24 in this inventions Scope of Claims.

Below is an explanation of the configuration and operation of the chromatic aberration amount detection apparatus (200A) following the diagrams in <figref idref="f0016 f0017 f0018 f0019 f0020">Figure 16 - Figure 19</figref>. <figref idref="f0018">Figure 17</figref> is an explanatory figure, of the same embodiment, of the chromatic aberration amount detection with <figref idref="f0018">Figure 17 (a)</figref> being the expression in a chart format while <figref idref="f0018">Figure 17 (b)</figref> is a figure showing the chart array for the imaging elements (122).

Also, <figref idref="f0019">Figure 18</figref> is a diagrammatic explanation, in the same embodiment, of chromatic aberration amount detection with <figref idref="f0019">Figure 18 (a) and (b)</figref> being a figure explaining when intersection is detected with <figref idref="f0019">Figure 18 (c)</figref> being a figure explaining when there is edge detection for each intersection and <figref idref="f0019">Figure 18 (d)</figref> is a figure explaining the sampling pixel array setting for the intersection with <figref idref="f0020">Figure 19</figref> being a figure that explains edge detection within the same embodiment.

First, as expressed in <figref idref="f0016 f0017">Figure 16</figref>, the chromatic aberration amount detection apparatus (200A) of this embodiment, using the digital image signal obtained from the pickup image of the CH Chart, calculates the color aberration coefficient of the imaging optics (110).

The CH Chart, as expressed in <figref idref="f0018">Figure 17 (a) and (b)</figref>, shows, for the pixel array of the imaging elements (122) of the single-chip color image sensor (120), the 1st image pattern array P1 and the 2nd image pattern array P2 with just the inclination as that of the inclination angle α. Also, in this embodiment, the area of one image pattern that is read in by the imaging elements (122) is approximately equivalent to 100 pixels.

Next, the chromatic aberration amount detection apparatus (200A), as expressed in <figref idref="f0016 f0017">Figure 16</figref>, takes the digital image signal (what is called the pixel signal that expresses the pixel luminance) input from the A/D conversion unit (130) and following the recorded pixel signal for each RGB color recorded to the field memory (221) and field memory (222),with the use of a program stored to ROM (231) by the CPU (230) controls all the processing executed by the aforementioned chromatic aberration detection apparatus (200A) with the following, the intersection point detection processing unit (225) that detects the image pattern 1st intersection point P1 and the image pattern 2nd intersection point P2, the RGB edge detection processing unit (226) that detects the RGB edge position of the 1st image pattern P1 and the 2nd image pattern P2 around the intersection detected by the intersection detection processing unit (225), the edge position recording unit (228) that records the edge position intersections as detected by the RGB edge detection processing unit (226), color aberration coefficient calculation unit (229) that uses color aberration coefficients from the edge positions recorded to the edge position recording unit (228), CPU (central processing unit) (230) and ROM (231).

The field memory (221) with the associated Bayer array, is configured from the R field memory (222) which records the red (R) pixel signal, the G field memory (223) which records the green (G) pixel signal and the B field memory (224) which records the blue (B) pixel signal.

The intersection detection processing unit (225) as expressed in <figref idref="f0019">Figure 18 (a) and (b)</figref>, calculates the luminance slope using the pixel value within the specified extent with the attention pixel at the center and detects the attention pixel position having the greatest luminance slope as the intersection point Int. Here, as expressed in <figref idref="f0019">Figure 18 (b)</figref>, with the attention pixel at the center 5 pixels are set vertically and horizontally and a weighted value is attached corresponding to pixel position and the intersection position is detected. In short, the up, down, left, right pixel values with the attention pixel at the center are multiplied by the <figref idref="f0019">Figure 18 (b)</figref> coefficient and the results totaled. The absolute value of the totaled results is used as the attention pixel evaluation value and in cases where the evaluation value coefficient exceeds the specified threshold value that attention pixel posit is set as intersection Int as, as expressed by <figref idref="f0019">Figure 18 (a)</figref>, the multiple intersection point Int as detected as a matrix.

Also, in this embodiment, the 1st image pattern P1 and the 2nd image pattern P2 are arrayed so that the intersection point Int is expressed in equal interval matrix.

The RGB edge detection processing unit (226), as expressed in <figref idref="f0019">Figure 18 (c</figref>), with a set sampling line length, scans the multiple up, down, left and right pixel rows Hs, Vs that go through the RGB intersection point Int and in addition to consecutively acquiring the pixel values, detects the sampling position that has the greatest pixel value change for the surrounding samplings locations as the edge.

In detail, as expressed by curved line In in <figref idref="f0020">Figure 19</figref>, each pixel's luminance (pixel value) is sought at each sampling and, as expressed by the curved line SL, in accordance with the pixel values sought by the sampling, the pixel value change amount (slope SL) is calculated and the position EP where there is the greatest change amount (slope SL) is detected as the edge.

Also, when seeking the edge EP, as expressed in <figref idref="f0019">Figure 18 (c)</figref>, in the up, down pixel extent going through the intersection point Int, multiple row sampling (Hs) for each is executed and the edge of each row is detected and then the average of the upper part detected edge position and the lower part detected edge position is calculated and set as the left, right direction edge position within the intersection point Int.

Moreover, also for the left, right pixel extent that goes through intersection point Int, sampling (Vs) of each of the multiple rows is executed and the edge of each row detected and then the average of the left part detected edge position and the right part detected edge position is calculated and set as the up down direction edge position within the intersection point Int.

Also, sampling is done on each of the same color pixels, and when executing Hs sampling along the left-right direction, as expressed in <figref idref="f0018">Figure 17 (b)</figref>, the left-right sampling length SL (11) and the sampling number SN (4) which expresses the sampling up-down direction row number is set beforehand as required by the detection precision. Moreover, also when the up, down sampling Vs is executed the up, down direction sampling length and sampling number is set beforehand.

Also, as expressed in <figref idref="f0019">Figure 18 (d)</figref>, when the left, right direction edge position of the edge extending upwards through the intersection point Int is detected, if the sampling Hs1 position is too close to the intersection point Int, as the edge EH is inclined, an effect form the image patterns P1-2 to the left and right of the intersection point Int is felt and edge detection is difficult. Therefore, it is best that an appropriate interval S be established between the sampling line Hs1 and the intersection point Int.

The interval S, for example as expressed in <figref idref="f0019">Figure 18 (d)</figref>, can be sought geometrically. In other words, if the interval S from the intersection point Int is the edge blur amount E, inclination angle α, sampling line length are given the formula L= SL/2, S = (W+L) x tan α can be used to seek the interval S. In short, appropriately, the sampling Hs1 start position is never inside the P1-2 image pattern of <figref idref="f0019">Figure 18 (d</figref>) and the interval S is sought separated from only the edge blur amount E from P1-2.

Next, the edge position recording unit (228) in the intersection point Intj (j is the serial number 1, 2, .... assigned to each intersection point) records the G (green), R (red), B (blue) edge positions as, respectively, left, right direction (u direction) and up, down direction (v direction) for the corresponding (uGj, vGj), (uRj, vRj) and (uBj, vBj) in accordance to the RGB edge detection processing unit (226) detected edge position for each color.

Next the aberration coefficient calculation unit (229), using the edge positions (uGi, vGj), (uRj, vRj) and (uBj, vBj) recorded to the edge position recording unit (228), calculates the color aberration coefficients kR, kB.

In detail, first, in the same manner as that of the 2nd embodiment, as the xy coordinates system, the origin point is the center of the image and 1 is the maximum image height (maximum distance from the origin point), with the plus x coordinates in the right direction from the origin point of the screen and the plus y coordinates in the down direction from the origin point of the screen. Then, in the same manner as the 2nd embodiment, for the color image formed by the 640 x 480 pixel square, the pixel coordinates (ud, vd) for the uv coordinate system of the color image are, as indicated in <figref idref="f0002">Figure 2</figref>, from the upper left of the screen going to right (0,0), (1,0), (2,0).... with the next row being (1,0), (1,1), (2,1) ... and with the uv coordinate system pixel coordinates (319.5, 239.5) as the origin point for the xy coordinate system, the xy coordinates (xd, yd) corresponding to the pixel coordinates (ud, vd) are calculated for each color.

Here, in the arithmetic expression xd = (ud - 319.5) / 400, ya = (vd - 239.5) /400 of the 2nd embodiment, in addition to replacing xd and yd with ,xGj and yGj, xRj and yRj, xBj and yBj, ud and vd are replaced with uGj and vGj, uRj and vRj, uBj and vBj and each RGB xy coordinates are calculated using the arithmetic expressions below. <maths id="math0038"><math display="block"><msub><mi mathvariant="normal">x</mi><mi>Gj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">u</mi><mi>Gj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">319.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0038.tif" /></maths><maths id="math0039"><math display="block"><msub><mi mathvariant="normal">y</mi><mi>Gj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">v</mi><mi>Gj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">239.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0039.tif" /></maths><maths id="math0040"><math display="block"><msub><mi mathvariant="normal">x</mi><mi>Rj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">u</mi><mi>Rj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">319.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0040.tif" /></maths><maths id="math0041"><math display="block"><msub><mi mathvariant="normal">y</mi><mi>Rj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">v</mi><mi>Rj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">239.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0041.tif" /></maths><maths id="math0042"><math display="block"><msub><mi mathvariant="normal">x</mi><mi>Bj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">u</mi><mi>Bj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">319.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0042.tif" /></maths><maths id="math0043"><math display="block"><msub><mi mathvariant="normal">y</mi><mi>Bj</mi></msub><mo mathvariant="normal">=</mo><mfenced><msub><mi mathvariant="normal">v</mi><mi>Bj</mi></msub><mo mathvariant="normal">-</mo><mn mathvariant="normal">239.5</mn></mfenced><mo mathvariant="normal">/</mo><mn mathvariant="normal">400</mn></math><img file="EP2175656A1_D0043.tif" /></maths>

Next the R color aberration coefficient kR is calculated using the kR = Σj (xRj2 + yRj2) / Σj (xRj xGj + yRj yGj) arithmetic expression and the B color aberration coefficient kB is calculated using the kB = Σj (xBj2 + yBj2) / Σj (xBj xGj + yBj yGj) arithmetic expression.

Next, the color aberration coefficients kR and kB calculated here are stored in the demosaic unit (140') (140") of the coordinate conversion unit (142') (142") and in the same manner as in the 2nd - 4th embodiments, the chromatic aberration correct sampling coordinates for the pickup image are calculated in the coordinate conversion unit (142') (142").

Also, as with the formula indicated in <figref idref="f0022">Figure 27</figref>, the chromatic aberration rows can be sought by inserting the xy coordinates and aberration coefficient as calculated by the aberration coefficient calculation unit (229) into the formula indicated in <figref idref="f0022">Figure 25</figref>. Moreover, in the <figref idref="f0022">Figure 27</figref> formula, the matrix at the farthest right side position indicates the pseudo inverse matrix.

As is seen in the above, the 5th embodiment's color image apparatus (100A) detects each color edge position with matrix intersection position mapping and, in accordance with the detected color edge position the color aberration coefficient (kR, kB) can be calculated and, using this detected color aberration coefficient, the color interpolation processing (demosaic process) that generates the color image from the color mosaic image and the imaging optics magnification chromatic aberration process can be realized with just one interpolation calculation.

In the above we have explained about an embodiment of this invention but this invention is not limited to an embodiment described above and can take many different embodiments.

For example, in the above 1st - 5th embodiments, the explanation of the color filter array (121) was based on the use of a primary color Bayer array but it is not limited to this. For example, a complimentary color filter array could also be used.

Also, in the 5th embodiment, the image apparatus (100A) is equipped with the chromatic aberration amount detection apparatus (200) but the chromatic aberration amount detection apparatus (200A) need not be configured as integral with the image apparatus (100A), it can be equipped as an external device. And it can be equipped with a part for pickup image output and color aberration coefficient input method, so that the input color aberration coefficient can be stored in the coordinate conversion unit (142'), (142") or aberration coefficient setting unit (190).

Also, the demosaic process method of the above 1st - 5th embodiments can also be realized through either of a hardware configuration, DSP (Digital Signal Processor), or a software method. For example, if software is used for the realization, the demosaic unit (140, 140', 140") for the above noted 1st - 5th embodiments would be equipped with a computer CPU or MPU, RAM or ROM and it would operate from a program recorded to RAM or ROM.

In addition, all of the above 1st - 5th embodiments of this invention are simply just substantive examples of the execution of this invention and cannot be interpreted to not limit the technical extent of this invention in anyways whatsoever. In other words, many different forms of this invention can be implemented that do not deviate from the spirit or major characteristics of the invention.

Industrial Applicability

The present invention is suitable for use in an image processing apparatus, image processing device that incorporates execution of a demosaic process that generates a color image with interpolation of all pixel multiple color luminance of the color mosaic image from the single-chip color imaging elements.

Every citation, both waysCites: the store holds 0 of 1

Citations
Document	Relation	Office	Cited during
EP2800380A4	Cited by	European Patent Office (EPO)	Search report
US9325957B2	Cited by	United States of America	Applicant
US9230297B2	Cited by	United States of America	Applicant
WO2013169384A1	Cited by	World Intellectual Property Organization (WIPO)	Applicant
EP2800380A1	Cited by	European Patent Office (EPO)	Search report
US9066001B2	Cited by	United States of America	Applicant
EP2847998A4	Cited by	European Patent Office (EPO)	Search report
US9883095B2	Cited by	United States of America	Applicant

14 members in 7 offices

This recordMember, by publication datePriority claim

Priority claims19

Priority claims
Document	Office	Kind	Date
2007166106	Japan	A	2007-06-25
2007166106	Japan	A	2007-06-25
2007166106	Japan	–	2007-06-25
2007166190	Japan	A	2007-06-25
2007166190	Japan	A	2007-06-25
2007166190	Japan	–	2007-06-25
2007335065	Japan	A	2007-12-26
2007335065	Japan	A	2007-12-26
2007335065	Japan	–	2007-12-26
2008061806	Japan	W	2008-06-24
2008061806	Japan	W	2008-06-24
2007166106	–	–	–
2007166190	–	–	–
2007335065	–	–	–
JP20070166106	–	–	–
JP20070166190	–	–	–
JP20070335065	–	–	–
JP2008061806	–	–	–
WO2008JP61806	–	–	–

Members14

Family members, oldest first
Document	Office	Kind
CN101335900A	China	A	2008-12-31
WO2009001956A1	World Intellectual Property Organization (WIPO)	A1	2008-12-31
TW200901747A	Taiwan Province of China	A	2009-01-01
JP2009005232A	Japan	A	2009-01-08
JP2009033699A	Japan	A	2009-02-12
EP2175656A1This record	European Patent Office (EPO)	A1	2010-04-14
KR20100051056A	Republic of Korea	A	2010-05-14
US2010208106A1	United States of America	A1	2010-08-19
JP5036421B2	Japan	B2	2012-09-26
JP5036524B2	Japan	B2	2012-09-26
US8494260B2	United States of America	B2	2013-07-23
US2013278801A1	United States of America	A1	2013-10-24
KR101356286B1	Republic of Korea	B1	2014-01-28
US8934713B2	United States of America	B2	2015-01-13

8 legal events, as the office reported them to INPADOC

Over the term

Point at a mark for the event

Events

Legal events, newest first
	Event	Code
2016-06-08	Application deemed to be withdrawnWithdrawn18D	18D
2016-05-06	Information on the status of an ep patent application or granted ep patentGrantedSTAA	STAA
2014-03-19	Party data changed (applicant data changed or rights of an application transferred)RAP1	RAP1
2010-09-29	Request for extension of the european patent (deleted)DAX	DAX
2010-04-14	Request for examination filed17P	17P
2010-04-14	Designated contracting statesAK	AK
2010-04-14	Request for extension of the european patentAX	AX
2010-03-12	Public reference made under article 153(3) epc to a published international application that has entered the european phasePUAI	PUAI

Image processing device, image processing method, program, and imaging device

Abstract

Term

Priority claims19

Members14

Over the term

Events

Numbers

Titles3

Classification

Designated states38