Image processing device and image processing method

CROSS REFERENCE TO THE RELATED APPLICATIONS

0001This application is a continuation application of International Application PCT/JP2008/000146, filed Feb. 5, 2008, designating the U.S., and claims the benefit of priority from Japanese Patent Application No. 2007-028218, filed on Feb. 7, 2007, the entire contents of which are incorporated herein by reference.

BACKGROUND

00021. Field

0003The present application relates to an image processing device and an image processing method for processing image data obtained by a two-dimensional image sensor mounted in a digital camera or the like at high speed.

00042. Description of the Related Art

0005A two-dimensional image sensor is mounted in a digital camera, a color image scanner, or the like. Color filters of three colors corresponding to three primary colors of light are arranged in conformity with the arrangement of pixels of the image sensor. Electrical signals reflecting the light intensities of light components passing through these color filters are obtained at the respective pixels.

0006In a general two-dimensional image sensor, square pixels are arranged two-dimensionally, color filters corresponding to respective R, G, and B components are arranged according to the Bayer array. In this Bayer array, filters for passing therethrough light whose wavelength corresponds to the G component reflecting a brightness component are continuously arranged in an oblique direction. Furthermore, filters for passing light whose wavelengths correspond to the R component and the B component are alternately arranged on lines and columns.

0007In the image sensor having the color filters arranged in the Bayer array as described above, the pixel data of each pixel represents only the intensity of the component corresponding to the penetration wavelength of the corresponding color filter. Accordingly, the color at the position of each pixel of an image formed on the image sensor is obtained by interpolation with surrounding pixel data for each of the R, G, and B components.

0008As described above, the interpolation processing of obtaining image data having three color elements of R, G, and B at each of the respective pixels from a Bayer image obtained by the image sensor is called as deBayer processing.

0009When a Bayer image is obtained by a two-dimensional image sensor, the deBayer processing as described above is first executed to obtain a lineup of R, G, and B components for the respective pixels, and then various image processes such as a distortion correcting process, etc. are executed in many cases (see Patent Document 1: Japanese Patent No. 3549413).

0010In the image after the deBayer processing described above, all the pixels have the pixel data of the R, G, and B components. Therefore, in order to execute the distortion correcting process and a magnification converting process, it is required to execute a linear interpolation process on the pixel data of each of the R, G, and B components of one frame. This processing quantity is tremendous.

0011Furthermore, it is possible to execute the distortion correcting process and the magnification converting process on the image based on the Bayer array and then execute the deBayer processing to obtain an image for which distortion is corrected or image data on which the magnification converting process is executed.

0012However, when the distortion correcting process and also the magnification converting process are executed on the Bayer image itself, the following problems occur. The following problems are caused by the difference in feature between the arrangement of the R and B pixels on which the filters for passing light of the wavelengths corresponding to the R, B components are arranged and the arrangement of the G pixels on which the filters for passing light of the wavelength corresponding to the G component are arranged in the Bayer image.

0013One of the problems occurs in the case that the high pixel density of the G pixels in the Bayer image cannot be actively used, that is, when interpolation processing for correcting distortion is executed on the pixel arrangement data in which the G component in consideration of a coordinate system which has coordinate axes corresponding to the line and column directions of the pixel arrangement in the two-dimensional image sensor and whose unit corresponds to the arrangement pitch of the R pixels and the B pixels.

0014According to the above method, the interpolation calculation is performed in a grid formed by G pixels arranged at the same arrangement pitch as the R and B pixels as a unit. At this time, the pixel data of the G pixel located at the center of this grid is not reflected by the interpolation calculation for determining the pixel data at any position in this grid although the G pixel concerned is located at the center of the grid.

0015Furthermore, it may be considered that a grid formed by four G pixels arranged so as to surround an R pixel or B pixel is used and the interpolation processing of the pixel data of the G component at any position in the grid is executed in the Bayer image. According to this method, the pixel density of the G pixels can be actively used, however, the position calculation processing to correct distortion is complicated.

SUMMARY

0016It is a proposition of the present embodiment to provide an image processing device and an image processing method that can increase the processing speed of position calculation processing for distortion correction, etc. by utilizing the features of the arrangements of the respective color components in a Bayer image.

0017An image processing device according to one aspect of the present embodiment has the following construction. In an image processing device for executing processing concerning a pixel value of each pixel corresponding to one of a plurality of wavelength components including a first, a second, and a third wavelength components contained in image data obtained by a two-dimensional image sensor in which each of filters corresponding to one of the plurality of wavelength components including the first, the second, and the third wavelength components is disposed in connection with each pixel according to a rule concerning an arrangement of the filters corresponding to each of the wavelength components, a coordinate transforming unit transforms a basic coordinate system having coordinate axes corresponding to arrangement directions of the pixel in the two-dimensional image sensor to a virtual coordinate system corresponding to an arrangement rule of the filters of each of the wavelength components. A position calculating unit executes position calculation processing for image processing containing an aberration correction in the virtual coordinate system corresponding to each of the wavelength components to determine coordinates in the virtual coordinate system. An inverse transforming unit transforms the coordinates in the virtual coordinate system obtained by the position calculating unit to coordinates in the basic coordinate system.

0018An image processing device according to another aspect of the present embodiment has the following construction.

0019In the image processing device which is first described above, a Bayer image obtained by the two-dimensional image sensor in which square pixels are arranged in a two-dimensional arrangement and filters corresponding to the first wavelength component are arranged continuously in an oblique direction is a processing target image, and the coordinate transforming unit has a first transforming unit which performs a transformation from the basic coordinate system to a virtual coordinate system having coordinate axes corresponding to arrangement directions of the filters corresponding to the first wavelength component.

0020An image processing device according to another aspect of the present embodiment has the following construction.

0021In the first transforming unit provided to the image processing device which is secondly described above, a coordinate rotating unit performs a rotation of a coordinate system around a predetermined pixel position on the two-dimensional image sensor as a center of the rotation. A first translating unit translates a position of origin of the virtual coordinate system corresponding to the first wavelength component while a size of pixel forming the two-dimensional image sensor is set as a unit. A first scale converting unit performs a scale conversion for setting a length different from the size of pixel forming the two-dimensional image sensor as a unit in the virtual coordinate system corresponding to the first wavelength component.

0022An image processing device according to another aspect of the present embodiment has the following construction.

0023In the coordinate transforming unit provided to the image processing device which is secondly described above, a second translating unit translates each position of origin of the virtual coordinate system corresponding to the second wavelength component and the third wavelength component while a size of pixels forming the two-dimensional image sensor is set as a unit. A second scale converting unit performs a scale conversion for setting a length different from the size of pixel forming the two-dimensional image sensor as a unit in the virtual coordinate system corresponding to each of the second wavelength component and the third wavelength component.

0024An image processing device according to another aspect of the present embodiment has the following construction.

0025In the image processing device which is thirdly or fourthly described above, a translation amount determining unit independently determines the position of origin of the virtual coordinate system corresponding to each of the wavelength components, the position of origin being translated by the first translating unit or the second translating unit. A unit determining unit independently determines a unit of a scale concerning the virtual coordinate system corresponding to each of the wavelength components, the scale being converted by the first scale converting unit or the second scale converting unit.

0026An image processing method according to another aspect of the present embodiment has the following construction.

0027In an image processing method for executing processing concerning a pixel value of each pixel corresponding to one of a plurality of wavelength components including a first, a second, and a third wavelength components contained in image data obtained by a two-dimensional image sensor in which each of filters corresponding to one of the plurality of wavelength components including at least the first, the second, and the third wavelength components is disposed in connection with each pixel according to a rule concerning an arrangement of the filters corresponding to each of the wavelength components, a coordinate transforming step transforms a basic coordinate system having coordinate axes corresponding to arrangement directions of the pixel in the two-dimensional image sensor to a virtual coordinate system corresponding to an arrangement rule of the filters of each of the wavelength components. A position calculating step executes position calculation processing for image processing containing an aberration correction in the virtual coordinate system corresponding to each of the wavelength components to determine coordinates in the virtual coordinate system. An inverse transforming step transforms the coordinates in the virtual coordinate system obtained by the position calculating unit to coordinates in the basic coordinate system.

0028According to the image processing device and the image processing method of the present embodiment, with respect to image data in which pixel data of R, G, and B components are obtained at different densities like a Bayer image, for example, the processing speed of position calculating processing for distortion correction, etc. can be rapidly increased.

BRIEF DESCRIPTION OF THE DRAWINGS

0029<figref idref="DRAWINGS">FIG. 1</figref> is a diagram representing an embodiment of an image processing device.

0030<figref idref="DRAWINGS">FIG. 2</figref> is a diagram representing a Bayer array.

0031<figref idref="DRAWINGS">FIG. 3</figref> is a diagram representing an example of virtual coordinate systems for R and B pixels.

0032<figref idref="DRAWINGS">FIG. 4</figref> is a diagram representing an example of a virtual coordinate system for G pixels.

0033<figref idref="DRAWINGS">FIGS. 5A</figref> and B are a diagram representing neighboring pixels.

0034<figref idref="DRAWINGS">FIG. 6</figref> is a diagram representing another embodiment of an image processing device.

DETAILED DESCRIPTION OF THE EMBODIMENTS

0035Preferred embodiments according to the present application will be described hereunder with reference to the accompanying drawings.

Embodiment 1

0036<figref idref="DRAWINGS">FIG. 1</figref> represents an embodiment of an image processing device according to the present application.

0037In a digital camera represented in <figref idref="DRAWINGS">FIG. 1</figref>, when an image is picked up, an optical image formed onto an imaging device <b>22</b> by an imaging optical system <b>21</b> is converted to an electrical signal corresponding to the intensity of the light by the imaging device <b>22</b>. This electrical signal is converted to digital data by an analog/digital (A/D) converter <b>23</b>, and then held in a memory <b>24</b>.

0038The memory <b>24</b> represented in <figref idref="DRAWINGS">FIG. 1</figref> is connected to an image processor <b>25</b>, a recording processing part <b>26</b> and an imaging controller <b>28</b> through a bus. The image processor <b>25</b> executes image processing containing coding processing on the digital data stored in the memory <b>24</b>. Compressed image data obtained as an image processing result are delivered to the recording processing part <b>26</b> through the bus, and recorded in a recording medium <b>27</b>. The operation of the above parts, an auto focus controller and a strobe driving circuit for driving a strobe light source, both of which are not represented, are controlled by the imaging controller <b>28</b> according to an instruction which is input through an operation panel (not represented) by a user.

0039As the imaging device <b>22</b> represented in <figref idref="DRAWINGS">FIG. 1</figref> is provided with a two-dimensional CCD image sensor in which color filters are arranged according to a Bayer array as represented in <figref idref="DRAWINGS">FIG. 2</figref>. In this case, Bayer image data are obtained by the analog/digital converter <b>23</b>, and subjected to the processing of the image processor <b>25</b> through the memory <b>24</b>. In <figref idref="DRAWINGS">FIG. 2</figref>, the arrangements of the respective color filters corresponding to the R, G, and B components are illustrated with different types of hatching.

0040In this image processor <b>25</b>, prior to deBayer processing of an interpolating part <b>32</b>, an aberration correcting part <b>31</b> executes the processing of correcting distortion and a magnification converting process following the distortion correcting processing. Thereafter, the interpolating part <b>32</b> executes the processing of generating image data containing a lineup of R, G, and B components for the respective pixels constituting the two-dimensional image sensor from the Bayer image. Furthermore, the image data obtained by the interpolating part <b>32</b> are compressed and encoded by a coding processing part <b>33</b>, and then delivered to the recording processing part <b>26</b>.

0041For example when distortion is corrected by the aberration correcting part <b>31</b> represented in <figref idref="DRAWINGS">FIG. 1</figref>, a coordinate representing a position needed for the distortion correction is determined at a position calculating part <b>35</b>.

0042At this time, the position calculating part <b>35</b> calculates the coordinate value for a coordinate (x₀, y₀) on a distortion-corrected image by using that on the arrangement of pixels in the two-dimensional CCD image sensor in the following formula (1). The formula (1) represents the relationship between the coordinate value (x_CCD, y_CCD) in a coordinate system corresponding to the pixel arrangement in the two-dimensional CCD image sensor and the coordinate (x_o, y_o) of the distortion-corrected image. In <figref idref="DRAWINGS">FIG. 2</figref>, the coordinate system corresponding to the pixel arrangement is represented a coordinate axes x_CCDand y_CCD. The position calculating part <b>35</b> calculates the coordinate (x_CCD, y_CCD) satisfying the formula (1) represented by using the coordinate value (x_c, y_c) representing the center Sc of the two-dimensional image sensor, the distortion amount D(r) at a normalized image point distance of r and a magnification converting rate D_α for trimming the distortion-corrected image.

0043<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>CCD</mi></msub><mo>=</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>o</mi></msub><mo>-</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><msub><mi>x</mi><mi>c</mi></msub></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>CCD</mi></msub><mo>=</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>o</mi></msub><mo>-</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>1</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US8089535B2_D0001.tif" />

0044A method for increasing the processing speed of determining the pixel data at the position represented by the coordinate (x_CCD, y_CCD) through the interpolation processing by using coordinate transformation will be described.

0045In the aberration correcting part <b>31</b> represented in <figref idref="DRAWINGS">FIG. 1</figref>, with respect to the calculation result of the position calculating part <b>35</b>, the coordinate transforming part <b>34</b> executes the processing of transforming the coordinate system representing the respective positions of the R, G, and B pixels constituting the Bayer image according to the pixel arrangement in the two-dimensional CCD image sensor described above to the coordinate system suited to the features of the respective arrangements.

0046The coordinate transforming part <b>34</b> calculates the coordinates representing the positions of the respective R pixel by using, as a unit, the arrangement pitch of the R pixels in the two-dimensional CCD image sensor in a virtual coordinate system for the R pixels as described below. For example, as represented in <figref idref="DRAWINGS">FIG. 3</figref>, with respect to the R pixels, an origin of the virtual coordinate system may be located at the center position of the R pixel nearest to the center Sc of the two-dimensional CCD image sensor, which is the imaging device <b>22</b>. Also, coordinate axes X_VR, Y_VRof the virtual coordinate system can be set parallel to the above coordinate axes X_CCD, Y_CCD.

0047Likewise, the coordinate transforming part <b>34</b> calculates the coordinates representing the positions of the respective B pixels by using, as a unit, the arrangement pitch of the B pixels in the two-dimensional CCD image sensor in a virtual coordinate system for the B pixels as described below. For the B pixels, as represented in <figref idref="DRAWINGS">FIG. 3</figref>, an origin of the virtual coordinate system may be located at the center position of the B pixel nearest to the center Sc of the two-dimensional image sensor. Also, coordinate axes X_VB, Y_VBof the virtual coordinate system can be set parallel to the above coordinate axes X_CCD, Y_CCDas represented in <figref idref="DRAWINGS">FIG. 3</figref>, for example.

0048Furthermore, the coordinate transforming part <b>34</b> calculates the coordinates representing the positions of the respective G pixels by using, as a unit, the arrangement pitch in an oblique direction of the G pixels in the two-dimensional CCD image sensor in a virtual coordinate system for the G pixels as described below. For the G pixels, as represented in <figref idref="DRAWINGS">FIG. 4</figref>, an origin of the virtual coordinate system may be located at the center position of any one of G pixels nearest to the center Sc of the two-dimensional image sensor. Also, coordinate axes X_VG, X_YGof the virtual coordinate system can be obtained by counterclockwise rotating the above coordinate axes X_CCD, Y_CCDby π/4.

0049With respect to the R, G, and B pixels, the coordinate values (x_i, y_i)(i=R, G, B) in the virtual coordinate systems corresponding to the respective color components which are obtained by the above coordinate transformations are represented according to the following formulas (2), (3), and (4). In the formulas (2), (3), and (4), the coordinate value (x_CCD, y_CCD) represents the position in the coordinate system represented by the coordinate axes X_CCD, Y_CCD, and the coordinate value (x_c, y_c) represents the center Sc of the two-dimensional image sensor in this coordinate system.

0050<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mo>(</mo><mtable><mtr><mtd><msub><mi>x</mi><mi>R</mi></msub></mtd></mtr><mtr><mtd><msub><mi>y</mi><mi>R</mi></msub></mtd></mtr></mtable><mo>)</mo></mrow><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mtable><mtr><mtd><mrow><mrow><mn>2</mn><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mn>1</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mn>1</mn></mrow></mtd></mtr></mtable><mo>)</mo></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>2</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mo>(</mo><mtable><mtr><mtd><msub><mi>x</mi><mi>G</mi></msub></mtd></mtr><mtr><mtd><msub><mi>y</mi><mi>G</mi></msub></mtd></mtr></mtable><mo>)</mo></mrow><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mrow><mo>(</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd><mrow><mo>-</mo><mn>1</mn></mrow></mtd></mtr><mtr><mtd><mrow><mo>-</mo><mn>1</mn></mrow></mtd><mtd><mrow><mo>-</mo><mn>1</mn></mrow></mtd></mtr></mtable><mo>)</mo></mrow><mo>⁢</mo><mrow><mo>(</mo><mtable><mtr><mtd><mrow><msub><mi>x</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>x</mi><mi>c</mi></msub></mrow></mtd></mtr><mtr><mtd><mrow><msub><mi>y</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mtd></mtr></mtable><mo>)</mo></mrow></mrow><mo>-</mo><mrow><mo>(</mo><mtable><mtr><mtd><mn>1</mn></mtd></mtr><mtr><mtd><mn>0</mn></mtd></mtr></mtable><mo>)</mo></mrow></mrow><mo>}</mo></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>3</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mo>(</mo><mtable><mtr><mtd><msub><mi>x</mi><mi>B</mi></msub></mtd></mtr><mtr><mtd><msub><mi>y</mi><mi>B</mi></msub></mtd></mtr></mtable><mo>)</mo></mrow><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mtable><mtr><mtd><mrow><mrow><mn>2</mn><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mn>1</mn></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>y</mi><mi>CCD</mi></msub><mo>-</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mn>1</mn></mrow></mtd></mtr></mtable><mo>)</mo></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>4</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US8089535B2_D0002.tif" />

0051The calculation processing for the coordinate transformations represented by these formulas (2), (3), and (4) can be implemented by addition, subtraction and bit shift operation, and thus it can be performed at an extremely high speed.

0052Furthermore, by substituting the formula (1) into these formulas (2), (3), and (4), the formulas (5), (6), and (7) are obtained. The formulas (5), (6), and (7) represent the coordinate transformation results corresponding to the coordinates in the image after the correcting processing, in which coordinates are determined for each R pixel, each G pixel, and each B pixel by the position calculating part <b>35</b>.

0053<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>R</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>2</mn><mo>⁢</mo><mi>x</mi></mrow><mo>-</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mn>1</mn></mrow><mo>}</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>R</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mi>y</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mn>1</mn></mrow><mo>}</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>5</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>G</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>2</mn><mo>⁢</mo><mi>x</mi></mrow><mo>-</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>-</mo><mrow><mn>2</mn><mo>⁢</mo><mi>y</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mn>2</mn></mrow><mo>}</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>G</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mi>x</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>-</mo><mrow><mn>2</mn><mo>⁢</mo><mi>y</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>6</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>B</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>2</mn><mo>⁢</mo><mi>x</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mn>1</mn></mrow><mo>}</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>B</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>4</mn></mfrac><mo>⁢</mo><mrow><mo>{</mo><mrow><mrow><mfrac><mrow><mn>100</mn><mo>+</mo><mrow><mi>D</mi><mo>⁡</mo><mrow><mo>(</mo><mi>r</mi><mo>)</mo></mrow></mrow></mrow><mrow><mn>100</mn><mo>+</mo><msub><mi>D</mi><mi>α</mi></msub></mrow></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mi>y</mi></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow></mrow><mo>)</mo></mrow></mrow><mo>-</mo><mn>1</mn></mrow><mo>}</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>7</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US8089535B2_D0003.tif" />

0054The shift processing part <b>37</b> executes proper bit shift processing on the coordinate values (x_i, y_i)(i=R, G, B) obtained according to these formulas (5), (6), and (7). Accordingly, when pixel data at the position represented by the coordinate value (x_i, y_i) in the virtual coordinate system for each color component is determined by the interpolation processing, information for specifying a combination of neighboring pixels to provide pixel data to be considered and information representing a weighting to be applied with respect to the pixel data corresponding to these pixels are obtained.

0055For example, the shift processing part <b>37</b> executes the proper bit shift processing on the coordinate value (x_R, y_R) obtained by the transformation to the virtual coordinate system represented by the coordinate axes X_VR, Y_VRin the coordinate transforming part <b>34</b>, thereby obtaining the following two kinds of information. One kind of information is the coordinate (x_RS, y_RS) representing a reference for specifying the minimum grid formed by R pixels surrounding the coordinate value (x_R, y_R) as indicated by a heavy two-dotted chain line in <figref idref="DRAWINGS">FIG. 5A</figref> (in this case, the reference located at the left lower side of the grid). The other kind of information is the distance ratio from the center coordinates of each of the R pixels forming the grid to the coordinate value (x_R, y_R).

0056Likewise, the shift processing part <b>37</b> executes proper bit shift processing on the coordinate value (x_B, y_B) obtained by the transformation to the virtual coordinate system represented by the coordinate axes X_VB, Y_VBin the coordinate transforming part <b>34</b>, thereby obtaining the following two kinds of information. One kind of information resides in the coordinate (x_BS, y_BS) representing a reference for specifying the minimum grid formed by B pixels surrounding the coordinate value (x_B, y_B) as indicated by a heavy dashed line in <figref idref="DRAWINGS">FIG. 5A</figref> (in this case, the reference located at the left lower side of the grid). The other kind of information resides in the distance ratio from the center coordinate of each of the B pixels forming the grid to the coordinate value (x_B, y_B).

0057Likewise, the shift processing part <b>37</b> executes proper bit shift processing on the coordinate value (x_G, y_G) obtained by the transformation to the virtual coordinate system represented by the coordinate axes X_VG, Y_VGin the coordinate transforming part <b>34</b>, thereby obtaining the following two kinds of information. One kind of information resides in the coordinate (x_GS, y_GS) representing a reference for specifying the minimum grid formed by G pixels surrounding the coordinate value (x_G, y_G) as indicated by a heavy dashed line in <figref idref="DRAWINGS">FIG. 5B</figref> (in this case, the reference located at the left lower side of the grid). The other kind of information resides in the distance ratio from the center coordinate of each of the G pixels forming the grid to the coordinate value (x_G, y_G).

0058At an inverse transforming part <b>36</b> represented in <figref idref="DRAWINGS">FIG. 1</figref>, the coordinate values (x_CCDiS, y_CCDiS)(i=R, G, B) in the coordinate system represented by the coordinate axes X_CCD, Y_CCDrepresented in <figref idref="DRAWINGS">FIG. 2</figref> or <b>3</b> can be obtained after substituting the coordinates (x_iS, y_iS_{<sub2>—</sub2>})(i=R, G, B) representing the thus-obtained references for inverse transforming equations represented by the following formulas (8), (9), and (10).

0059<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>CCDRS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>4</mn><mo>⁢</mo><msub><mi>x</mi><mi>RS</mi></msub></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>+</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>CCDRS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>4</mn></mrow><mo>⁢</mo><msub><mi>y</mi><mi>RS</mi></msub></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>+</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>8</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>CCDGS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>2</mn><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>GS</mi></msub><mo>-</mo><msub><mi>y</mi><mi>GS</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>+</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>CCDGS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>2</mn></mrow><mo>⁢</mo><mrow><mo>(</mo><mrow><msub><mi>x</mi><mi>GS</mi></msub><mo>+</mo><msub><mi>y</mi><mi>GS</mi></msub></mrow><mo>)</mo></mrow></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>-</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>9</mn><mo>)</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><msub><mi>x</mi><mi>CCDBS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mn>4</mn><mo>⁢</mo><msub><mi>x</mi><mi>BS</mi></msub></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>x</mi><mi>c</mi></msub></mrow><mo>-</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow><mo>⁢</mo><mstyle><mtext></mtext></mstyle><mo>⁢</mo><mrow><msub><mi>y</mi><mi>CCDBS</mi></msub><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><mo>(</mo><mrow><mrow><mrow><mo>-</mo><mn>4</mn></mrow><mo>⁢</mo><msub><mi>y</mi><mi>BS</mi></msub></mrow><mo>+</mo><mrow><mn>2</mn><mo>⁢</mo><msub><mi>y</mi><mi>c</mi></msub></mrow><mo>-</mo><mn>1</mn></mrow><mo>)</mo></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>10</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US8089535B2_D0004.tif" />

0060The coordinate value obtained by the inverse transforming part <b>36</b> as described above corresponds to the address in the two-dimensional CCD image sensor of the pixel at the reference position of the grid formed by the pixels for the color component being noted.

0061Accordingly, based on this coordinate value, a pixel data calculating part <b>38</b> represented in <figref idref="DRAWINGS">FIG. 1</figref> first determines addresses on the two-dimensional CCD image sensor of four pixels corresponding to the apexes of a grid surrounding a position at which pixel data should be determined. Subsequently, by performing the interpolation processing, in which the distance ratio received from the shift processing part <b>37</b> had been applied, with the pixel data read out from the memory <b>24</b> according to these addresses at the pixel data calculating part <b>38</b>, whereby the pixel data which the distortion correction is applied for noted component can be obtained.

0062As described above, according to the embodiment of the present application represented in <figref idref="DRAWINGS">FIG. 1</figref>, by applying the coordinate transformation using the features of the pixel arrangements of the respective color components in the Bayer image, the calculation processing, which used to be calculate by performing complicated case classification, can be implemented only by using addition, subtraction, and the bit shift processing. Accordingly, the processing of specifying the four pixels corresponding to the apexes of the grid surrounding the pixel and the processing of specifying the contribution degree of the pixel data of these pixels for the purpose of the distortion correction can be dramatically enhanced in processing speed.

0063Particularly, in the distortion correcting process concerning the G pixels, in order to actively use the high arrangement density of the G pixels in the Bayer array, the case classification is particularly complicated for the position calculation in the coordinate system corresponding to the address of the two-dimensional CCD image sensor in conventional art. The processing speed of the processing performed with this complicated procedure can be increased by applying the combination processing of the coordinate transformations and the bit shift processing according to this application, and thus an obtained advantage is very big.

0064Furthermore, the calculation amount can be further reduced by using the fact that distortion appears in optical axis symmetry.

0065For example, at the case for the G pixels, when each coordinate value (x_G, y_G) in the coordinate system represented by the coordinate axes X_VG, Y_VGis obtained within a range in which the coordinate value y is equal to or more than a numerical value “0” (an upper half area of CCD), the coordinate value (x_Gt, y_Gt) corresponding to a G pixel which is located within a lower side area of CCD and is centrosymmetric with respect to the G pixel being noted can be represented by using the determined coordinate value (x_G, y_G) according to the following formula (11).

0066<maths id="MATH-US-00005" num="00005"><math overflow="scroll"><mtable><mtr><mtd><mrow><mrow><mo>(</mo><mtable><mtr><mtd><msub><mi>x</mi><mi>Gt</mi></msub></mtd></mtr><mtr><mtd><msub><mi>y</mi><mi>Gt</mi></msub></mtd></mtr></mtable><mo>)</mo></mrow><mo>=</mo><mrow><mo>(</mo><mtable><mtr><mtd><mrow><mrow><mo>-</mo><msub><mi>x</mi><mi>G</mi></msub></mrow><mo>-</mo><mn>1</mn></mrow></mtd></mtr><mtr><mtd><mrow><mo>-</mo><msub><mi>y</mi><mi>G</mi></msub></mrow></mtd></mtr></mtable><mo>)</mo></mrow></mrow></mtd><mtd><mrow><mo>(</mo><mn>11</mn><mo>)</mo></mrow></mtd></mtr></mtable></math></maths><img file="US8089535B2_D0005.tif" />

0067As described above, the distortion correcting process concerning the pixel data of each color component constituting a Bayer image can be executed at the stage of the Bayer image. The Bayer image after the distortion correction is subjected to the deBayer processing by the interpolating part <b>32</b>, thereby obtaining distortion-corrected image data.

0068Furthermore, in addition to the above distortion, chromatic aberration of magnification can be corrected for the R pixels and the B pixels constituting the Bayer image. Accordingly, false color in the neighborhood of an edge can be removed, and thus the image quality can be further enhanced.

Embodiment 2

0069<figref idref="DRAWINGS">FIG. 6</figref> represents another embodiment of the image processing device according to the present application.

0070With respect to the constituent elements represented in <figref idref="DRAWINGS">FIG. 6</figref>, the same parts as represented in <figref idref="DRAWINGS">FIG. 1</figref> are represented by the same reference numerals as represented in <figref idref="DRAWINGS">FIG. 1</figref>, and the description thereof is omitted.

0071In <figref idref="DRAWINGS">FIG. 6</figref>, the image processing device according to this application is implemented by software which implements the functions of the aberration correcting part <b>31</b> and the interpolating part <b>32</b> in CPU <b>43</b> equipped in a personal computer (PC) <b>41</b>.

0072An image input device <b>40</b> represented in <figref idref="DRAWINGS">FIG. 6</figref> is a digital camera or color scanner which has, for example, a two-dimensional CCD image sensor having color filters arranged according to the Bayer array and has a function of outputting an obtained Bayer image. The Bayer image delivered from this image input device <b>40</b> is stored in a memory <b>42</b> of the personal computer <b>41</b>.

0073The aberration correcting part <b>31</b> represented in <figref idref="DRAWINGS">FIG. 6</figref> executes the aberration correction processing based on the information concerning distortion and chromatic aberration of magnification delivered from the image input device <b>40</b> as described with respect to the Embodiment 1 described above. At this time, the Bayer image stored in the memory <b>42</b> is rewritten by using the pixel data of each color component constituting the Bayer image calculated by the pixel data calculating part <b>38</b>, thereby obtaining a Bayer image for which the distortion and the chromatic aberration of magnification are corrected.

0074The distortion-corrected Bayer image is read out from the memory <b>42</b> by the interpolating part <b>32</b> represented in <figref idref="DRAWINGS">FIG. 6</figref>. The Bayer image is subjected to the deBayer processing by the interpolating part <b>32</b>, whereby image data containing a lineup of R, G, B components are obtained for all the pixels of the two-dimensional CCD image sensor provided to the image input device <b>40</b>.

0075The image data obtained as described above are used for display processing of a display processing part <b>44</b> and a display unit <b>45</b> and also used for various image processing.

0076As described above, the image processing device according to this application can implement high-precision aberration correction by combining the Bayer-image outputting function equipped to a digital camera or the like and the high processing capability of a personal computer, for example by using detailed information on the distortion.

0077By installing the image processing device of this application into an image input device such as a digital camera, a color scanner or the like, the processing speed of the image processing containing aberration correction can be increased as the stage of processing the Bayer image. Furthermore, the image processing device is executed by a personal computer or the like as a part of image processing software, whereby high-speed aberration correcting processing can be implemented.

0078Accordingly, this application is remarkably effective in the image input device field and also in an image processing device field in which various processing is executed on image data obtained by an image inputting device.

0079The many features and advantages of the embodiments are apparent from the detailed specification and, thus, it is intended by the appended claims to cover all such features and advantages of the embodiments that fall within the true spirit and scope thereof. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the inventive embodiments to the exact construction and operation illustrated and described, and accordingly all suitable modifications and equivalents may be resorted to, falling within the scope thereof.