Image interpolation method

The present invention relates to a video interpolation method, and more particularly to a video interpolation method that enables the image captured by a portable terminal having a camera function to be interpolated to the optimum image quality.

Recently, with the technological development of mobile terminals, mobile terminals have come to have many functions in their mobile phones in addition to the basic functions of providing voice calls. The most popular function among the functions provided in such a conventional portable terminal is the camera function.

The user can operate the camera function provided in the portable terminal to save the subject to be photographed anytime and anywhere in the image. At this time, the image acquisition is performed by CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor), which are elements that convert the optical signal reflected from the subject into an electrical signal. In order to obtain a color image using such a CCD and CMOS element, a color filter is required at the front end of the CCD and CMOS.

Most cameras installed in portable terminals use a color filter called CFA (Color Filter Array) in consideration of the size and price of the equipment. This CFA is attached to the front end of CCD and CMOS in the form of a microlens, and has a regular arrangement structure in which only light indicating one color per pixel is passed through.

FIG. 1 is an exemplary diagram showing a bayer pattern for the conventional G, FIG. 2 is an exemplary diagram showing the spatial frequency characteristics of FIG. 1, and FIGS. 3A to 3C show the frequency characteristics of the conventional video interpolation method. It is an example figure which shows. CFA has various forms depending on the arrangement structure of color elements, but the most widely used pattern is the bayer pattern. This bayer pattern is as shown in FIG. 1, and the spatial frequency characteristic of the bayer pattern for such G (green) is as shown in FIG. In FIG. 1, the output of the pixel located at the upper left corner has only the G component, and R and B are alternately arranged on different lines. Μ and v in FIG. 2 show the horizontal and vertical frequency components for the G component, respectively, and show how many samples are in the unit space (mm). Figure 3A shows such two-dimensional frequency characteristics as a horizontal component, that is, one-dimensionally.

After that, in the Bayer pattern of FIG. 1, the process of calculating three types of color information for each pixel is executed by using the color information of the surrounding pixels for image interpolation at the R, G, and B positions. The simplest and most common method among such conventional video interpolation methods is closest pixel interpolation using surrounding pixel values having the same color information as the color to be calculated with a given pixel. There are a method (Nearest neighbor replication), a bilinear interpolation method (Bilinear Interpolation), an intermediate value interpolation method (Median interpolation), and the like.

The bilinear interpolation method in such a conventional video interpolation method will be described as follows. FIG. 4 is an example diagram showing an example of a bayer pattern for explaining the conventional bilinear interpolation method. With reference to FIG. 4, first, the interpolation of the G component at the B position and the R position will be described as follows. For example, when trying to interpolate the G component at the B8 position, the following equation (1) is used.

<maths num="1"><img file="JP2005295555A_D0001.tif" /></maths>

Where B_8GMeans the G component at the B8 position, G₃, G₇, G₉, G₁₃Means the G component at the G3, G7, G9, and G13 positions. At this time, the G component interpolation at the B position and the R position can be obtained by the same method.

After that, in order to interpolate the R component at the G position, the arithmetic mean value of two adjacent R position values is obtained, and to interpolate the B component at the G position, the two adjacent B position values are obtained. Find the arithmetic mean. That is, this is as shown in the following equation (2). Here, the following equation (2) is an example of an equation relating to interpolation of B component and R component at the G7 position.

<maths num="2"><img file="JP2005295555A_D0002.tif" /></maths>

Where G_7BMeans the B component at the G7 position, B₆, B₈Means the B component at the B6 and B8 positions, G_7RMeans the R component at the G7 position, R₂, R₁₂Means the R component at the R2 and R12 positions.

After that, in order to interpolate the R component at the B position, the arithmetic mean value of four adjacent diagonal pixels is obtained. That is, this is as shown in the following equation (3). Here, the following number (3) is an example showing the interpolation of the R component at the B8 position and the interpolation of the B component at the R12 position.

<maths num="3"><img file="JP2005295555A_D0003.tif" /></maths>

Where B_8RMeans the R component at the B8 position, R₂, R₄, R₁₂, R₁₄Means the R component at the R2, R4, R12, R14 positions, and R_12BMeans the B component at the R12 position, B₆, B₈, B₁₆, B₁₈Means the B component at the B6, B8, B16, B18 positions.

In the conventional image interpolation method described above, the color information of the pixels adjacent to the top, bottom, left, right, and diagonal lines is arithmetically averaged and interpolated in order to interpolate the three types of color images of each pixel with the Bayer pattern.

The frequency characteristics of the arithmetic mean through such a conventional interpolation method show a low frequency pass filter as shown in FIG. 3B. Therefore, the frequency characteristics after performing the interpolation for the G component in each pixel through the above process will have the frequency characteristics as shown in FIG. 3C, which is the multiplication of FIGS. 3A and 3B. Here, in FIG. 3C, the triangular straight line of FIG. 3A is changed to a curve by interpolation, and by multiplying (a) of FIG. 3A and FIG. 3B, it has a portion as shown in (b).

Further, in the conventional video interpolation method, the resolution is lowered because the peripheral pixel values of four adjacent pixels are simply arithmetically averaged without considering the distribution of the peripheral pixel values. That is, since this has the same result as taking a low pass filter, there is a problem that the resolution is lowered. Further, the conventional video interpolation method has a problem that the resolution is further lowered as the number of output pixels corresponding to one input pixel is large. This causes a decrease in resolution due to the aliasing phenomenon in which the sampling frequency is low and the high frequency component is confused with the low frequency component.

<p> The present invention has been made to solve the problems of the prior art as described above, and an object of the present invention is to provide an image interpolation method capable of interpolating an image to be captured to the optimum image quality. is there.</p>

<p> In order to achieve the above object, the present invention is a method of interpolating a color image, in which a process of extracting a bayer pattern from an optical signal of a subject to be photographed and a top and bottom adjacent to the G position of the extracted bayer pattern. Alternatively, the process of arithmetically averaging the left and right R and B components and interpolating the R and B components at the G position, and after interpolating the R and B components at the G position, G at the G position adjacent to the R and B positions. It has already been set as the absolute value of the result value obtained by subtracting the vertical difference from the horizontal difference of the G, B, and R components at the G position and the process of executing the calculation of the horizontal difference and the vertical difference of the B and R components. The magnitude of the threshold value is compared, and if the result of the judgment is larger than the threshold value, the G, B, and R components of the adjacent horizontal or vertical G positions where the difference value is small in the horizontal difference and the vertical difference The process of interpolating the G, B, R components at the R and B positions through the arithmetic mean and, as a result of the comparison, if it is smaller than the threshold value, the G, B, R components are interpolated at the R and B positions through the median technique. It is characterized by including the process of interpolating.</p><p> Further, the present invention for achieving the above object is a method of interpolating a color image, which is adjacent to a process of extracting a bayer pattern from an optical signal of a subject to be photographed and a G position of the extracted bayer pattern. The process of interpolating the R and B components at the G position by arithmetically averaging the upper and lower or left and right R and B components, and after interpolating the R and B components at the G position, they are adjacent to each other around the R and B positions. The process of interpolating the G component at R and B positions by analyzing the pattern of the discrete cosine transform coefficient obtained by discrete cosine transform (DCT) for the G component at G position, and the G position adjacent to the R and B positions. Compare the absolute value of the result value of the B and R components of B and R and the magnitude of the preset threshold value, and if the comparison result is larger than the threshold value, the lateral difference is used. The process of interpolating the B and R components at the R and B positions through the arithmetic average of the B and R components at the adjacent horizontal or vertical G positions where the difference value is small in the vertical difference, and as a result of the comparison, from the above threshold value. If it is small, it is characterized by including the process of interpolating the B and R components at the R and B positions through the median technique.</p>

<p> According to the present invention, it is possible to obtain an image with improved resolution without aliasing by the Bayer signal interpolation method.</p>

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following description, for the purpose of clarifying only the gist of the present invention, specific description of related known functions or configurations will be omitted. Further, in the following description, the specific details of the bayer pattern are shown for the purpose of providing a more general understanding of the present invention, and are not limited thereto.

FIG. 5 is a diagram showing a configuration of a portable terminal according to an embodiment of the present invention, and FIG. 6 is an exemplary diagram showing a bayer pattern for explaining video interpolation of the present invention. Referring to FIG. 5, the RF unit 21 executes the communication of the portable terminal. The RF unit 21 includes an RF transmitter (not shown) that ascends and amplifies the frequency of the transmitted signal, and an RF receiver (not shown) that amplifies the received signal with low noise and lowers the frequency. Etc. are included.

The modem 23 includes a transmitter (not shown) that encodes and modulates the transmitted signal and a receiver (not shown) that demodulates and decodes the received signal.

The audio processing unit 25 has a function of receiving the received audio signal output from the modem 23 from the control unit 10 and playing it, or transmitting the transmitted audio signal generated from the microphone (MIC) to the modem 23 through the control unit 10. To execute. Further, the audio processing unit 25 converts the audio data among the data received from the modem 23 into audible sound through the speaker (SPK) and outputs the data, converts the audio signal input from the microphone (MIC) into data, and converts the modem into data. Output to 23. Here, the audio processing unit 25 can also be included in the control unit 10.

The key input unit 27 includes a key for inputting number and character information and a function key for setting various functions.

The memory 29 can be composed of a program memory and a data memory. A program for controlling the general operation of the portable terminal is stored in the program memory. Further, the program memory stores a general program that enables image interpolation for each pixel related to the image of the subject captured in the camera mode according to the embodiment of the present invention. Further, the program memory is compared with the absolute value of the difference value between the R, G or B components between adjacent pixels for the interpolation of the RGB components at the R and B positions according to the embodiment of the present invention. The threshold value is already saved. At this time, the threshold value becomes a value when the image quality is evaluated according to the change of the threshold value and the highest image quality is obtained. In addition, the data memory temporarily stores the data generated during the execution of the program. In addition, video data captured in the camera mode is stored in the data memory.

The control unit 10 executes a function of controlling the general operation of the portable terminal, and may also include a modem 23. Further, the control unit 10 controls the general operation so as to execute the image interpolation method according to the present invention.

The camera 50 includes a camera sensor that captures video data and converts the captured optical signal into an electrical signal. Here, the camera sensor is assumed to be a CCD sensor. At this time, the CCD sensor adopts CFA at its front end and has a regularly arranged structure in order to pass only an optical signal indicating one color for each pixel through the camera 50. As a result, the pixel extracts the pixel value for only one color among many colors.

The signal processing unit 60 converts the video signal output from the camera 50 into an image signal. Here, the signal processing unit 60 can be embodied by a DSP (Digital Signal Processor), and at this time, the signal processing unit 60 converts the video signal into an image signal according to the sampling period. As a result, the information of the lost pixels for each pixel is converted into an electrical signal through a Bayer pattern inferred using the information about the adjacent pixels, and the video signal is output.

The video processing unit 70 executes a function of generating screen data for displaying the video signal output from the signal processing unit 60. The video processing unit 70 transmits a video signal received under the control of the control unit 10 or video data captured through the camera 50 in accordance with the standard of the display unit 80, and compresses and decompresses the video data.

The display unit 80 displays a message generated during the execution of the program under the control of the control unit 10. Further, the display unit 80 displays the video signal output from the video processing unit 70 after video interpolation in the camera shooting mode on the screen, and displays the user data output from the control unit 10. Here, the display unit 80 can use an LCD, and in this case, the display unit 80 can include an LCD controller, a memory capable of storing video data, an LCD display element, and the like. When the LCD is embodied in a touch screen system, the key input unit 27 and the LCD can be included in the input unit.

Explaining the operation of the portable terminal with reference to FIG. 5, when making a call, if the user sets the call mode after executing the dialing operation through the key input unit 27, the control unit 10 will perform this. After sensing and processing the dial information received through the modem 23, it is converted into an RF signal through the RF unit 21 and output. After that, if the other subscriber generates a response signal, it is detected through the RF unit 21 and the modem 23. After that, the user executes a communication function by forming a voice communication path through the audio processing unit 25. Further, in the incoming mode, the control unit 10 detects whether or not the incoming mode is in the incoming mode through the modem 23, and generates a ring signal through the audio processing unit 25. After that, when the user responds, the control unit 10 senses this, and after all, a voice communication path is formed through the audio processing unit 25 to execute the communication function. Although voice communication is described as an example in the outgoing and incoming modes, it is also possible to execute a data communication function for communicating packet data and video data in addition to voice communication. Further, when executing the standby mode or character communication, the control unit 10 displays the character data processed through the modem 23 on the display unit 80.

In addition, the portable terminal according to the present invention can perform an operation of photographing a person or the surrounding environment and displaying or transmitting the image on a video screen. First, the camera 50 can be attached to a mobile phone or connected to a predetermined position on the outside. That is, the camera 50 can be formed by an external storage type or a built-in type camera. The camera 50 can use a CCD sensor. The image captured by the camera 50 is converted into an electrical signal by the internal CCD sensor and then applied to the signal processing unit 60. After that, the signal processing unit 60 converts the color video (RGB) of the adjacent pixels into interpolated digital video data using the data obtained by sampling the electrical signal, and outputs the data to the video processing unit 70.

Explaining the operation of the portable terminal that processes the video interpolation related to the video signal acquired by the camera 50 when shooting the above-mentioned person or the surrounding environment, first, the control unit 10 inputs through the key input unit 27. When the key data generated by the key is in the camera mode, the display unit 80 indicates that the key data is in the camera mode. After that, the control unit 10 converts the optical signal reflected from the subject (person, surrounding environment) through the camera 50 into an electrical video signal. The converted electrical video signal has a regularly arranged bayer pattern through which only an optical signal indicating one color for each pixel is passed through the CFA. The control unit 10 controls the signal processing unit 60 in order to obtain all the color information (RGB) at the R, G, and B positions of the bayer pattern. The signal processing unit 60 obtains three types of color information in each pixel by using the color information of the surrounding pixels and outputs a video signal.

That is, as shown in FIG. 6, it has a general bayer pattern, and in the CFA of the bayer pattern, G indicating brightness is distributed at a ratio of 50%, and color components R and B are distributed at a ratio of 25% each. At this time, R and B are arranged alternately on different lines, and G is located between R and B. Here, each pixel of each bayer pattern can acquire three types of color information. The specific explanation will be described later.

7A and 7B are flow charts showing the video interpolation method of the present invention, and FIGS. 8A to 8D are exemplary diagrams showing Bayer patterns interpolated through the video interpolation method of the present invention. With reference to FIGS. 7A and 7B, the control unit 10 determines whether or not the camera mode is in the standby state (step 711, step 713). At this time, if the camera mode is determined, the control unit 10 executes the camera mode and applies a control signal according to the camera mode to each module associated with the camera mode (step 715).

Thereby, the control unit 10 can obtain the bayer pattern shown in FIG. 6 from the camera 50 (step 717). After that, the camera 50 executes the following video interpolation process using the bayer pattern under the control of the control unit 10.

That is, for the interpolation of the R and B components at the G position shown in FIG. 6, as shown in FIG. 8A, the R and B components are used by using the arithmetic mean of R or B above and below or to the left and right of the G position. Interpolate (step 719). This will be explained in more detail with reference to FIG.

9A to 9C are illustrations showing an example of a bayer pattern for explaining R and B image interpolation at the G position according to the embodiment of the present invention. Referring to FIG. 9A, first, in the bayer pattern of FIG. 6, the interpolation of the B component at the G5 position is obtained by adding B2 and B8 located at the upper and lower positions at the G5 position and then dividing the result into two. Interpolate the B component, which is the resulting value. Here, numbers are added to the components of each pixel to help understand the interpolation of the R and B components at the G position.

In the interpolation of the R component at the G5 position, after adding R4 and R6 located on the left and right at the G5 position, the result is divided into two and the R component, which is the result value obtained, is interpolated. That is, this can be expressed as the following equation (4).<maths num="4"><img file="JP2005295555A_D0004.tif" /></maths>

Here, (a) means the B component interpolated at the G5 position, and (b) means the R component interpolated at the G5 position, and B₂, B₈Means the B component at the B2 and B8 positions, and R₄, R₆Means the R component at the R4 and R6 positions.

After that, the control unit 10 determines whether or not the interpolation of the R and B components at the G position is completed, and repeatedly executes the process until the interpolation of the R and B components at the G position is completed (step 721). When the R and B components are interpolated for all G positions of the bayer pattern by repeating the above process, the result is as shown in FIG. 8B.

As a result of the determination, when the interpolation of the R and B components is completed at the G position, the interpolation of the G and B components is executed at the R position, and the interpolation of the G and R components is executed at the B position. At this time, the order of executing the interpolation at the R position and the video interpolation at the B position is performed at the same time point. That is, the priority of the procedure for executing the interpolation at the R position and the video interpolation at the B position has not been determined. Therefore, it does not matter what kind of thing (video interpolation at R and B positions) is executed first at the time of video interpolation at R position and B position. In the present invention, in order to help the understanding of the video interpolation method, the procedure of interpolating the G component for the R and B positions first, interpolating the B component at the R position, and interpolating the R component at the B position is described.

First, the lateral difference (ΔX) and vertical difference (ΔY) of the G component between the G positions adjacent to the R and B positions are obtained (step 723). That is, the control unit 10 executes the calculation related to the lateral difference (ΔX) and the vertical difference (ΔY) by using the following equation (5).<maths num="5"><img file="JP2005295555A_D0005.tif" /></maths>

For example, assuming that the G component is interpolated at the R14 position as shown in Fig. 9B, the lateral difference (ΔX) is G.₁₃-G₁₅Is the absolute value of. And the vertical difference ( Y) is G₁₁-G₁₇Is the absolute value of. That is, lateral difference ( X) = | G₁₃-G₁₅|, Vertical difference ( Y) = | G₁₁-G₁₇| Become.

Also, in Fig. 9C, assuming that the G component is interpolated at the B23 position, the lateral difference (ΔX) is G.₂₂-G₂₄Is the absolute value of. And the vertical difference ( Y) is G₂₀-G₂₆Is the absolute value of. That is, lateral difference ( X) = | G₂₂-G₂₄|, Vertical difference ( Y) = | G₂₀-G₂₆| Become.

After that, the control unit 10 determines whether or not the absolute value of the result value obtained by subtracting the vertical difference (ΔY) from the lateral difference (ΔX) obtained in the above process is larger than the threshold value (T) (step). 724). That is, it is determined whether or not | ΔX-ΔY |&gt; T. As a result of the determination, if it is larger than the threshold value (T), the control unit 10 determines whether the vertical difference (ΔY) is larger than the result value obtained by adding the lateral difference (ΔX) and the threshold value (T). Determine if (step 726). That is, it is determined whether or not ΔY&gt; ΔX + T.

As a result of the determination, if the vertical difference (ΔY) is larger than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 determines the G component at the G position adjacent to the side. After adding each other, the result is divided into two and the value is interpolated to the G component at the R position (step 728). That is, the G component at the R position can be obtained by the following equation (6).<maths num="6"><img file="JP2005295555A_D0006.tif" /></maths>

Where R_14GMeans the G component interpolated to the R14 position, G₁₃, G₁₅Means the G component at the G13 and G15 positions.

However, as a result of the determination, if the vertical difference (ΔY) is smaller than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 is set to G at the G position vertically adjacent to each other. After adding the components to each other, the result is divided into two and the value is interpolated to the G component at the R position (step 730). That is, the G component at the R position can be obtained by the following equation (7).

<maths num="7"><img file="JP2005295555A_D0007.tif" /></maths>

Where R_14GMeans the G component interpolated to the R14 position, G₁₁, G₁₇Means the G component at the G11 and G17 positions.

On the other hand, as a result of the judgment, if the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the maximum value is used by using the median technique. After adding the G component at the G position, which is the intermediate value excluding the minimum value, the value divided into 2 is interpolated to the G component at the R position (step 732).

If the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the control unit 10 determines the G component at the G position that is horizontally and vertically adjacent. It is also possible to interpolate the value obtained by dividing the result into 4 into the G component at the R position after the addition. That is, the G component at the R position can be obtained by the following equation (8).

<maths num="8"><img file="JP2005295555A_D0008.tif" /></maths>

Where R_14GMeans the G component interpolated to the R14 position, G₁₁, G₁₃, G₁₅, G₁₇Means the G component at the G11, G13, G15, and G17 positions.

On the other hand, the method of interpolating the G component at the B position is also performed in the same manner as the above-mentioned process. That is, in the control unit 10, the absolute value of the result of subtracting the vertical difference (ΔY) from the lateral difference (ΔX) performed for the interpolation of the G component at the B position described above is from the threshold value (T1). Determine if it is large (step 724). That is, it is determined whether or not | ΔX-ΔY |&gt; T.

As a result of the determination, if it is larger than the threshold value (T), the control unit 10 determines whether the vertical difference (ΔY) is larger than the result value obtained by adding the lateral difference (ΔX) and the threshold value (T). Determine if (step 726). That is, it is determined whether or not ΔY&gt; ΔX + T. As a result of the determination, if the vertical difference (ΔY) is larger than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 determines the G component at the G position adjacent to the side. After adding each other, the result is divided into two and the value is interpolated to the G component at the B position (step 728). That is, the G component at the B position can be obtained by the following equation (9).

<maths num="9"><img file="JP2005295555A_D0009.tif" /></maths>

Where B_23GMeans the G component interpolated to the B23 position, G₂₂, G₂₄Means the G component at the G22 and G24 positions.

However, as a result of the determination, if the vertical difference (ΔY) is smaller than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 is set to G at the G position vertically adjacent to each other. After adding the components to each other, the result is divided into two and the value is interpolated to the G component at the B position (step 730). That is, the G component at the B position can be obtained by the following equation (10).

<maths num="10"><img file="JP2005295555A_D0010.tif" /></maths>

Where B_23GMeans the G component interpolated to the B23 position, G₂₀, G₂₆Means the G component at the G20 and G26 positions.

On the other hand, as a result of the determination, when the absolute value of the result of subtracting the vertical difference (ΔY) from the lateral difference (ΔX) is smaller than the threshold value (T), the control unit 10 uses the median technique. After adding the G component at the G position, which is the intermediate value excluding the maximum and minimum values, the value divided into 2 is interpolated to the G component at the B position (step 732).

If the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the control unit 10 determines the G component at the G position that is horizontally and vertically adjacent. It is also possible to interpolate the value obtained by dividing the result into 4 into the G component at the R position after the addition. That is, the G component at the R position can be obtained by the following equation (11).

<maths num="11"><img file="JP2005295555A_D0011.tif" /></maths>

Where B_23GMeans the G component interpolated to the B23 position, G₂₀, G₂₂, G₂₄, G₂₆Means the G component at the G20, G22, G24, and G26 positions.

After that, the control unit 10 determines whether or not the interpolation of the G component at all R and B positions of the bayer is completed, and repeats the above process until the interpolation of the G component at all R and B positions is completed. Repeat (step 734). When the above process is iteratively executed and the G component is interpolated for all the R and B positions of the bayer pattern, the result is as shown in FIG. 8C.

Next, when the interpolation of the G component at the R and B positions is completed, the control unit 10 interpolates the B component at the R position and interpolates the R component at the B position. At this time, the interpolation process of the B component at the R position and the interpolation process of the R component at the B position are performed at the same time. Also at this time, the order of performing the interpolation at the R position and the image interpolation at the B position is performed at the same time point, and in the present invention, the B component with respect to the R position is interpolated first in order to help the understanding of the image interpolation method. After that, describe in the procedure for interpolating the R component at the B position.

The process of interpolating the B component at the R position is as follows. First, the horizontal difference (ΔX) and vertical difference (ΔY) operations for the B component at the G position adjacent to the R position are executed (step 736). That is, the control unit 10 calculates the lateral difference (ΔX) and the vertical difference (ΔY) by using the following equation (12) in order to interpolate the B component at the R position.

<maths num="12"><img file="JP2005295555A_D0012.tif" /></maths>

For example, assuming that the B component is interpolated at the R14 position as shown in Fig. 9B, the lateral difference (ΔX) is the absolute value of the result of subtracting the B component interpolated to G15 from the B component interpolated to G13. Become a value. Then, the vertical difference (ΔY) becomes the absolute value of the result value obtained by subtracting the B component interpolated by G17 from the B component interpolated by G11. That is, lateral difference ( X) = | G_13B-G_15B|, Vertical difference ( Y) = | G_11B-G_17B| Become. Where G_11B, G_13BAnd G_15BAnd G_17BMeans the B component interpolated at each G11, G13, G15, G17 position.

Assuming that the R component is interpolated at the B23 position, the lateral difference (ΔX) is the absolute value of the result value obtained by subtracting the R component interpolated by G24 from the R component interpolated by G22. Then, the vertical difference (ΔY) becomes the absolute value of the result value obtained by subtracting the R component interpolated by G26 from the R component interpolated by G20. That is, lateral difference ( X) = | G_22R-G_24R|, Vertical difference ( Y) = | G_20R-G_26R| Become. Where G_20R, G_22RAnd G_24RAnd G_26RMeans the R component interpolated at each G20, G22, G24, G26 position.

After that, the control unit 10 determines whether or not the absolute value of the result value obtained by subtracting the vertical difference (ΔY) from the lateral difference (ΔX) obtained in the above process is larger than the threshold value (T) (step). 738). That is, it is determined whether or not | ΔX-ΔY |&gt; T. As a result of the determination, if it is larger than the threshold value (T), the control unit 10 determines whether the vertical difference (ΔY) is larger than the result value obtained by adding the lateral difference (ΔX) and the threshold value (T). Determine if (step 740). That is, it is determined whether or not ΔY&gt; ΔX + T.

As a result of the determination, if the vertical difference (ΔY) is larger than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 determines the B component at the G position adjacent to the side. After adding each other, the result is divided into two and the value is interpolated to the B component at the R position (step 742). That is, the B component at the R position can be obtained by the following equation (13).

<maths num="13"><img file="JP2005295555A_D0013.tif" /></maths>

Where R_14BMeans the B component interpolated to the R14 position, G_13B, G_15BMeans the B component at the G13 and G15 positions.

However, as a result of the determination, if the vertical difference (ΔY) is smaller than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 is set to B at the G position vertically adjacent to each other. After adding the components to each other, the result is divided into two and the value is interpolated to the B component at the R position (step 744). That is, the B component at the R position can be obtained by the following equation (14).

<maths num="14"><img file="JP2005295555A_D0014.tif" /></maths>

Where R_14BMeans the B component interpolated to the R14 position, G_11B, G_17BMeans the B component at the G11 and G17 positions.

On the other hand, as a result of the judgment, when the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the maximum value and the minimum value are used by using the median technique. After adding the B component at the G position, which is the intermediate value excluding, the value divided into 2 is interpolated to the B component at the R position (step 746).

If the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the control unit 10 is set to B at the G position horizontally and vertically adjacent to each other. It is also possible to interpolate the value obtained by dividing the result into 4 into the B component at the R position after adding the components. That is, the G component at the R position can be obtained by the following equation (15).

<maths num="15"><img file="JP2005295555A_D0015.tif" /></maths>

Where R_14BMeans the B component interpolated to the R14 position, G_11B, G_13B, G_15B, G_17BMeans the B component at the G11, G13, G15, and G17 positions.

On the other hand, the method of interpolating the R component at the B position is also performed in the same manner as the above-mentioned process. First, the horizontal difference (ΔX) and vertical difference (ΔY) operations for the R component at the G position adjacent to the B position are executed (step 736). That is, the control unit 10 calculates the lateral difference (ΔX) and the vertical difference (ΔY) by using the following equation (16) in order to interpolate the R component at the B position.

<maths num="16"><img file="JP2005295555A_D0016.tif" /></maths>

For example, assuming that the R component is interpolated at the B23 position as shown in Fig. 9C, the lateral difference (ΔX) is the absolute value of the result of subtracting the R component interpolated to G24 from the R component interpolated to G22. Become a value. Then, the vertical difference (ΔY) becomes the absolute value of the result value obtained by subtracting the R component stored in G26 from the R component interpolated in G20. That is, lateral difference ( X) = | G_22R-G_24R|, Vertical difference ( Y) = | G_20R-G_26R| Become. Where G_20R, G_22RAnd G_24RAnd G_26RMeans the R component interpolated at each G20, G22, G24, G26 position.

After that, the control unit 10 determines whether or not the absolute value of the result value obtained by subtracting the vertical difference (ΔY) from the lateral difference (ΔX) obtained in the above process is larger than the threshold value (T) (step). 738). That is, it is determined whether or not | ΔX-ΔY |&gt; T. As a result of the determination, if it is larger than the threshold value (T), the control unit 10 determines whether the vertical difference (ΔY) is larger than the result value obtained by adding the lateral difference (ΔX) and the threshold value (T). Determine if (step 740). That is, it is determined whether or not ΔY&gt; ΔX + T.

As a result of the determination, if the vertical difference (ΔY) is larger than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 determines the R component at the G position adjacent to the side. After adding each other, the result is divided into two and the value is interpolated to the R component at the B position (step 742). That is, the R component at the B position can be obtained by the following equation (17).

<maths num="17"><img file="JP2005295555A_D0017.tif" /></maths>

Where B_23RMeans the R component interpolated to the B23 position, G_22R, G_24RMeans the R component at the G22 and G24 positions.

However, as a result of the determination, if the vertical difference (ΔY) is smaller than the result value of adding the lateral difference (ΔX) and the threshold value (T), the control unit 10 is set to R at the G position vertically adjacent to each other. After adding the components to each other, the result is divided into two and the value is interpolated to the R component at the B position (step 744). That is, the R component at the B position can be obtained by the following equation (18).

<maths num="18"><img file="JP2005295555A_D0018.tif" /></maths>

Where B_23RMeans the R component interpolated to the B23 position, G_20R, G_26RMeans the R component at the G20 and G26 positions.

On the other hand, as a result of the determination, when the absolute value of the result of subtracting the vertical difference (ΔY) from the lateral difference (ΔX) is smaller than the threshold value (T), the control unit 10 uses the median technique. After adding the R component at the G position, which is the intermediate value excluding the maximum and minimum values, the value divided into 2 is interpolated to the R component at the B position (step 746).

If the absolute value of the result of subtracting the vertical difference ( Y) from the horizontal difference ( X) is smaller than the threshold value (T), the control unit 10 will perform R at the G position that is horizontally and vertically adjacent. It is also possible to interpolate the value obtained by adding the components and then dividing the result into 4 into the R component at the B position. That is, the R component at the B position can be obtained by the following equation (19).

<maths num="19"><img file="JP2005295555A_D0019.tif" /></maths>

Where B_23RMeans the R component interpolated to the B23 position, G_20R, G_22R, G_24R, G_26RMeans the R component at the G20, G22, G24, and G26 positions.

After that, the control unit 10 determines whether or not the interpolation of the B and R components at the R and B positions is completed, and repeats the process until the interpolation of the B and R components at the R and B positions is completed. Exit (step 748). When the above process is iteratively executed and all the lost components are interpolated for all R, G, and B positions of the bayer pattern, the result is as shown in FIG.

Therefore, the video interpolation method of the present invention interpolates the video color information lost for each pixel through the process described above. At this time, since the pixels located at the edge of the Bayer pattern do not have surrounding pixels to be compared, the adjacent components that can be considered are used as they are.

10A and 10B are illustrations showing a Bayer pattern of the G component in consideration of the directionality of the surrounding components by the image interpolation method of the present invention, and FIGS. 11A to 11C show the frequency characteristics of the image interpolation method according to the present invention. It is an example figure which shows. When the G component is interpolated in consideration of the directionality of the surrounding components as shown in FIG. 10A by the interpolation method of the present invention, if this is expressed only vertically, the interpolation can be performed as shown in FIG. 10B. I understand. In FIG. 10A, the frequency characteristics related to the odd horizontal component are as shown in FIG. 11A, and the frequency characteristics related to the even horizontal component are displayed in different phases as shown in FIG. 11B.

Therefore, the frequency characteristics of the interpolated pattern as shown in FIG. 10B are such that the spectra of the odd and even lines have opposite phases at odd multiple sampling frequencies, and the opposite phase portions are canceled out as shown in FIG. 11C. become. Therefore, the interpolation method proposed in consideration of the components related to the mutual opposite phases has the spatial frequency characteristics as shown in FIG. 11C.

12A and 12B are flow charts showing a video interpolation method according to another embodiment of the present invention, and FIGS. 13A to 13C are diagrams for explaining a video interpolation method according to another embodiment of the present invention. Is. With reference to FIGS. 12A and 12B and FIGS. 13A to 13C, steps 711 to 721 are methods for interpolating the R and B components at the G position described above, and steps 734 to 748 are the B components at the R position. As the interpolation method of the above and the interpolation method of the R component at the B position, since they are executed through the same methods as those in FIGS. 7A and 7B, detailed description thereof will be omitted. That is, since FIGS. 12A and 12B are executed in place of FIG. 7A, the process of FIG. 7B is executed after the process of FIG. 12B is executed.

The control unit 10 interpolates the R and B components at the G position, and then extracts the G components adjacent to the R and B positions in order to interpolate the G components at the R and B positions (step 823). After that, the control unit 10 converts the extracted G component into a 2 × 2 pixel form (step 825). At this time, the control unit 10 positions the G component at the G position located on the left side of the R and B positions at which the G component is to be interpolated in the first column of the first row, and G located at the upper side. The G component at the position is located in the second column of the first row, the G component at the G position located on the right side is located at the first column of the second row, and the G at the G position located at the lower side. Place the component in the second column of the second row. This is shown in Figures 13A and 13B.

After the above process, the control unit 10 transforms the pixels into 2 × 2 pixels using the DCT coefficient through the Discrete Cosine Transform (hereinafter referred to as DCT) (step 827). That is, by converting the video data defined in the two-dimensional matrix from the spatial domain to the frequency domain, as shown in FIG. 13C, the C1 to C4 coefficients are respectively shown in the order of the DCT coefficients of C1, C2, C3, and C4. Coefficients corresponding to low frequency (DC), horizontal frequency, vertical frequency, and high frequency will be located.

After that, the control unit 10 determines whether or not the low frequency coefficient is the largest among the DCT coefficients, and if it is the largest, sets the G component of the G position with respect to the four adjacent R and B positions. The result value obtained by arithmetic mean is interpolated into the G component at the R and B positions (step 829, step 831). Here, the fact that the low frequency coefficient is the largest among the DCT coefficients means that the pattern is uniform as a whole. Therefore, the G components at adjacent G positions are arithmetically averaged and the resulting value is interpolated. That is, the G components at the R and B positions can be interpolated through the following equation (20).

<maths num="20"><img file="JP2005295555A_D0020.tif" /></maths>

Here, G1 to G4 indicate the G component of the G position adjacent to the R position and the B position.

Also, without interpolating the G components at the R and B positions through equation (20), select one of the four G components at the G positions adjacent to the R and B positions and select the G components at the R and B positions. Can also be interpolated (step 831). That is, one of G1, G2, G3, and G4 can be selected to interpolate the G component at the R and B positions.

On the other hand, as a result of the determination, if the low frequency coefficient is not the largest among the DCT coefficients, the control unit 10 determines whether or not the horizontal frequency coefficient is the largest among the DCT coefficients (step 833). ), In the case of the largest, among the G components converted to the 2 × 2 pixel form, the G components at the vertically adjacent G positions are arithmetically averaged and the G components at the R and B positions are interpolated (step). 835). Here, the fact that the horizontal frequency coefficient is the largest among the DCT coefficients means that there is a large difference in the horizontal pattern in the 2 × 2 pixel form. Therefore, the vertical pattern must be used in the 2 × 2 pixel form. That is, the G components at the R and B positions can be interpolated through the following equation (21).

<maths num="21"><img file="JP2005295555A_D0021.tif" /></maths>

Here, R indicates the G component at the R position, B indicates the G component at the B position, and G1 and G3 and G2 and G4 are the vertical G components converted into 2 × 2 pixels. The G component at the G position adjacent to is shown.

However, as a result of the determination, if the horizontal frequency coefficient is not the largest among the DCT coefficients, the control unit 10 determines whether or not the vertical frequency coefficient is the largest among the DCT coefficients (step 837). ), In the case of the largest, among the G components converted to the 2 × 2 pixel form, the G components at the horizontally adjacent G positions are arithmetically averaged and the G components at the R and B positions are interpolated (step). 839). Here, the fact that the coefficient of the vertical frequency is the largest among the DCT coefficients means that there is a large difference in the vertical pattern in the 2 × 2 pixel form. Therefore, the horizontal pattern must be used in the form of 2 × 2 pixels. That is, the G component at the R and B positions can be interpolated through the following equation (22).

<maths num="22"><img file="JP2005295555A_D0022.tif" /></maths>

Here, R indicates the G component at the R position, B indicates the G component at the B position, and G1 and G2, G3 and G4 are horizontal among the G components converted into 2 × 2 pixels. The G component at the G position adjacent to is shown.

However, as a result of the determination, if the vertical frequency coefficient is not the largest among the DCT coefficients, the control unit 10 determines that the high frequency coefficient is the largest among the DCT coefficients, and has a 2 × 2 pixel form. Among the G components converted to, the G components at the G positions adjacent to the diagonal line are arithmetically averaged, and the G components at the R and B positions are interpolated (step 841).

Here, the fact that the high frequency coefficient is the largest among the DCT coefficients means that there is a large difference between the horizontal and vertical patterns in the 2 × 2 pixel form. Therefore, the diagonal pattern must be used in the 2 × 2 pixel form. That is, the G components at the R and B positions can be interpolated through the following equation (23).

<maths num="23"><img file="JP2005295555A_D0023.tif" /></maths>

Here, R indicates the G component at the R position, B indicates the G component at the B position, and G1 and G4, G2 and G3 are diagonal lines in the G component converted into 2 × 2 pixels. The G component at the G position adjacent to is shown.

Hereinafter, the interpolation of the B component at the R position and the interpolation of the R component at the B position are performed through the same method as described above, and thus detailed description thereof will be omitted.

Although the present invention has been described in detail with reference to specific embodiments, the scope of the present invention should not be limited by the above-described embodiments, and the scope of claims and the scope equivalent thereto should not be limited. It is clear to those with ordinary knowledge in the art that various modifications are possible within.

<figref num="1">It is an example figure which shows the bayer pattern about the conventional G.</figref><figref num="2">It is an example figure which shows the spatial frequency characteristic of FIG.</figref><figref num="3A">It is explanatory drawing which shows the frequency characteristic of the conventional video interpolation method.</figref><figref num="3B">It is explanatory drawing which shows the frequency characteristic of the conventional video interpolation method.</figref><figref num="3C">It is explanatory drawing which shows the frequency characteristic of the conventional video interpolation method.</figref><figref num="4">It is explanatory drawing which shows an example of the bayer pattern for demonstrating the conventional bilinear interpolation method.</figref><figref num="5">It is a figure which showed the structure of the portable terminal by embodiment of this invention.</figref><figref num="6">It is explanatory drawing which shows the bayer pattern for demonstrating the image interpolation of this invention.</figref><figref num="7A">It is a flow chart which shows the image interpolation method of this invention.</figref><figref num="7B">It is a flow chart which shows the image interpolation method of this invention.</figref><figref num="8A">It is explanatory drawing which shows the Bayer pattern interpolated through the image interpolation method of this invention.</figref><figref num="8B">It is explanatory drawing which shows the Bayer pattern interpolated through the image interpolation method of this invention.</figref><figref num="8C">It is explanatory drawing which shows the Bayer pattern interpolated through the image interpolation method of this invention.</figref><figref num="8D">It is explanatory drawing which shows the Bayer pattern interpolated through the image interpolation method of this invention.</figref><figref num="9A">It is explanatory drawing which shows an example of the bayer pattern for demonstrating R, B image interpolation at the G position by embodiment of this invention.</figref><figref num="9B">It is explanatory drawing which shows an example of the bayer pattern for demonstrating R, B image interpolation at the G position by embodiment of this invention.</figref><figref num="9C">It is explanatory drawing which shows an example of the bayer pattern for demonstrating R, B image interpolation at the G position by embodiment of this invention.</figref><figref num="10A">It is an example figure which shows the bayer pattern of the G component which considered the directionality of the surrounding component by the image interpolation method of this invention.</figref><figref num="10B">It is an example figure which shows the bayer pattern of the G component which considered the directionality of the surrounding component by the image interpolation method of this invention.</figref><figref num="11A">It is explanatory drawing which shows the frequency characteristic of the image interpolation method of this invention.</figref><figref num="11B">It is explanatory drawing which shows the frequency characteristic of the image interpolation method of this invention.</figref><figref num="11C">It is explanatory drawing which shows the frequency characteristic of the image interpolation method of this invention.</figref><figref num="12A">It is a flow chart which shows the image interpolation method by another Embodiment of this invention.</figref><figref num="12B">It is a flow chart which shows the image interpolation method by another Embodiment of this invention.</figref><figref num="13A">It is a figure for demonstrating the image interpolation method by another Embodiment of this invention.</figref><figref num="13B">It is a figure for demonstrating the image interpolation method by another Embodiment of this invention.</figref><figref num="13C">It is a figure for demonstrating the image interpolation method by another Embodiment of this invention.</figref>

Code description

10 Control 21 RF 23 Modem 25 Audio processing 27 Key input 29 Memory 50 Camera 60 Signal processing 70 Video processing 80 Display