Image processing device and method, and image display device and method

The present invention relates to an image processing device and method, and an image display device and method. The present invention particularly relates to a frame interpolation process for inserting a new interpolation frame between frames of an image.

A hold-type display such as a liquid crystal display continues to display the same image for one frame period, and when the image element in the image (for example, the image element corresponding to the object to be imaged) moves, the moving image element While the tracking of the human eye moves continuously, there is a problem that the edge portion looks blurry because the movement of the image element is discontinuous movement in units of one frame. On the other hand, it is conceivable to increase the number of display frames by interpolating the frames to smooth the movement of the image elements.

For materials such as movies in which film images are converted into TV signals, two or three frames become image signals made from the same frame due to the difference in frame frequency between the two (film image and TV signal). However, if it is displayed as it is, there is a problem that the movement is blurred or judder with jerky movement occurs.

Similarly, with respect to a material obtained by converting a computer-processed image into a television signal, two frames are image signals created from the same frame, and there is a problem that judder occurs when the image is displayed as it is.

Conventional image processing devices and methods are either a zero-order hold method that interpolates the interpolation frame with the same image as the frame one frame before, or an average image of the image one frame before and the image one frame after the interpolation frame. There is an average value interpolation method that interpolates with, but the zero-order hold method does not move smoothly with respect to an image that moves in a certain direction, so the problem of blurring of the hold type display is still not solved. In addition, the mean value interpolation method has a problem that a moving image becomes a double image.

As a remedy for this, interpolation is performed from the pixel having the largest correlation between the pixel on the timely previous frame and the pixel on the timely later frame at a position symmetrical with respect to the interpolated pixel of the interpolation frame. Some generate interpolated pixels for frames (see, for example, Patent Document 1). In this method, since the correlation is detected on a pixel-by-pixel basis, it may be detected that the correlation between the pixels is large even though the contents of the images are different, and the interpolation frame may not be generated correctly.

<p><patcit num="1"><text>Japanese Unexamined Patent Publication No. 2006-129181 (Page 8, Fig. 3)</text></patcit></p>

<p> The conventional frame interpolation process is configured as described above, and has a problem that the movement is blurred or judder with jerky movement is generated. In addition, the method of detecting the correlation in pixel units has a problem that the interpolation frame cannot be generated correctly because the correlation cannot be detected correctly. Further, when the image element is hidden or appears before and after the frame to be interpolated, the interpolated frame is remarkably disturbed.</p>

<p> The image processing apparatus according to one aspect of the present invention is In an image processing device that inserts a new interpolation frame between the current frame of an image and the data of the frame one frame before the current frame. From the first delay frame to the current frame with reference to the data of the current frame, the data of the first delay frame, and the data of the second delay frame which is a frame two frames before the current frame. Motion vector detector that calculates the first motion vector of A motion vector conversion unit that converts the first motion vector into a second motion vector from the first delay frame to the interpolation frame and a third motion vector from the current frame to the interpolation frame. Based on the second motion vector and the third motion vector, from the region in which the pixel corresponding to the pixel existing in the first delay frame or the current frame does not exist in the interpolation frame and the region around it. Occlusion area estimation unit that estimates the occlusion area, modifies the second motion vector and the third motion vector based on the area information indicating the occlusion area, and outputs the fourth motion vector and the fifth motion vector. When, The data of the interpolated frame is generated from the fourth motion vector, the fifth motion vector, the data of the first delay frame, and the data of the current frame, and the generated data of the interpolated frame is used in the current frame. It is provided with an interpolation frame generator that outputs image data inserted between the data and the data of the first delay frame. The motion vector detection unit A test interpolation unit that generates a plurality of test interpolation data from at least one of the data of the second delay frame and the data of the current frame, and An interpolation data evaluation unit that evaluates the plurality of test interpolation data based on the data of the first delay frame, and an interpolation data evaluation unit. It is provided with a motion vector determining unit that generates a first motion vector based on the plurality of evaluation data. It is characterized by that. The image processing method of another aspect of the present invention is In an image processing method that inserts a new interpolation frame between the current frame of an image and the data of the frame one frame before the current frame. From the first delay frame to the current frame with reference to the data of the current frame, the data of the first delay frame, and the data of the second delay frame which is a frame two frames before the current frame. Motion vector detection step to calculate the first motion vector of A motion vector conversion step of converting the first motion vector into a second motion vector from the first delay frame to the interpolated frame and a third motion vector from the current frame to the interpolated frame. Based on the second motion vector and the third motion vector, from the region in which the pixel corresponding to the pixel existing in the first delay frame or the current frame does not exist in the interpolation frame and the region around it. Occlusion region estimation step that estimates the occlusion region, modifies the second motion vector and the third motion vector based on the region information indicating the occlusion region, and outputs the fourth motion vector and the fifth motion vector. When, The data of the interpolated frame is generated from the fourth motion vector, the fifth motion vector, the data of the first delay frame, and the data of the current frame, and the generated data of the interpolated frame is used in the current frame. It includes an interpolation frame generation step that outputs image data inserted between the data and the data of the first delay frame. The motion vector detection step A test interpolation step that generates a plurality of test interpolation data from at least one of the data of the second delay frame and the data of the current frame. An interpolation data evaluation step that evaluates the plurality of test interpolation data based on the data of the first delay frame, and It includes a motion vector determination step that generates a first motion vector based on the plurality of evaluation data. It is characterized by that.</p>

<p> According to the present invention, since the result of the test interpolation is evaluated and the motion vector is determined based on the evaluation result, the motion vector can be calculated with high accuracy, and the frame interpolation can be performed without image distortion. Further, even in the occlusion region, frame interpolation can be performed without image distortion.</p>

<figref num="1">It is a block diagram which shows the structure of the image display apparatus which concerns on Embodiment 1 of this invention.</figref><figref num="2">It is a block diagram which shows the specific example of the test interpolation part 7, the interpolation data evaluation part 8 and the motion vector determination part of the motion vector detection part 2 of FIG.</figref><figref num="3">It is a figure for demonstrating the operation of the motion vector detection part 2 of FIG.</figref><figref num="4">It is a figure for demonstrating the operation of the motion vector conversion part 3 of FIG.</figref><figref num="5">It is a figure for demonstrating the operation of the occlusion area estimation part 4 and the interpolation frame generation part 5 of FIG.</figref><figref num="6">It is a figure for demonstrating the correspondence between the image data and the data of the current frame, the data of the 1st delay frame, and the data of a 2nd frame used for the specific example of the motion vector detection unit 2 of FIG.</figref><figref num="7">(a) to (d) are diagrams for explaining a specific example of the operation of the motion vector detection unit 2 of FIG.</figref><figref num="8">(a) and (b) are diagrams for explaining a specific example of the operation of the motion vector conversion unit 3 of FIG.</figref><figref num="9">(a) to (c) are diagrams for explaining a specific example of the operation of the occlusion area estimation unit 4 of FIG.</figref><figref num="10">It is a figure for demonstrating a specific example of the operation of the interpolation frame generation part 5 of FIG.</figref><figref num="11">It is a flowchart which shows the processing process of the image display apparatus which concerns on this embodiment.</figref>

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The image processing apparatus according to the present invention inserts new interpolation frame IF data between the current frame F0 data of the image and the first delay frame which is the data of the frame F1 one frame before the current frame F0. The image display device according to the present invention includes an image display unit that displays image data output from the above-mentioned image processing device.

Embodiment 1. FIG. 1 is a diagram showing a configuration of an image display device according to a first embodiment of the present invention. The image display device according to the first embodiment includes a frame memory 1, a motion vector detection unit 2, a motion vector conversion unit 3, an occlusion area estimation unit 4, an interpolation frame generation unit 5, and an image display unit 6. ..

The image data F0 is input to the frame memory 1, the motion vector detection unit 2, and the interpolation frame generation unit 5. The frame memory 1 stores the image data F0 for two frames, and outputs the image data F1 delayed by one frame with respect to the image data F0 and the image data F2 delayed by two frames with respect to the image data F0. From this, the image data F0 is referred to as the data of the current frame, the image data F1 is referred to as the data of the first delay frame, and the image data F2 is referred to as the data of the second delay frame. Further, the current frame, the first delay frame, and the second delay frame are represented by the same codes F0, F1, and F2 as the respective frame data.

The data of the first delay frame F1 is input to the motion vector detection unit 2 and the interpolation frame generation unit 5, and the data of the second delay frame F2 is input to the motion vector detection unit 2.

The motion vector detection unit 2 refers to the data of the current frame F0, the data of the first delay frame F1, and the data of the second delay frame F2, and each block (of the frame) on the first delay frame F1. The first motion vector MV1 from the first delay frame F1 to the current frame F0 is calculated and output to the motion vector conversion unit 3 for (a part of which is composed of a plurality of pixels).

The motion vector conversion unit 3 uses the first motion vector MV1, the second motion vector MV2 from the first delay frame F1 to the interpolation frame IF, and the third motion vector MV3 from the current frame F0 to the interpolation frame IF. Is converted to and output to the interpolation area estimation unit 4.

The occlusion region estimation unit 4 estimates the occlusion region (OC) from the second motion vector MV2 and the third motion vector MV3, and based on the estimated occlusion region information, the second motion vector MV2 and the third motion vector MV2 and the third motion vector MV3. The motion vector MV3 of is modified and the fourth motion vector MV4 and the fifth motion vector MV5 are output to the interpolation frame generator 5.

The occlusion area is a block corresponding to each frame, for example, a block existing in a frame before or after the interpolation frame (hence, the pixels constituting the block) (hence, the pixels constituting the block) are present in the interpolation frame. It means the area not to be interpolated and the area around it. For example, when one image element in an image (for example, an image element corresponding to an object to be imaged) is hidden or appears behind another image element, the boundary portion between those image elements is regarded as an occlusion area. Become.

For example, the occlusion area estimation unit 4 estimates a region in which the second motion vector MV2 or the third motion vector MV3 changes abruptly and a region around the region as an occlusion region.

Here, whether or not the motion vector changes abruptly is determined by, for example, whether or not the change of the motion vector between adjacent blocks is equal to or greater than a predetermined value. That is, as described in the present embodiment, when the motion vector is obtained for each block composed of a plurality of pixels, the difference between the motion vector for each block and the motion vector for adjacent blocks is a predetermined value. If the above is the case, it is determined that an edge exists between these two blocks.

The interpolation frame generator 5 is connected between the data of the first delay frame F1, the data of the current frame F0, the fourth motion vector MV4 and the fifth motion vector MV5, between the current frame F0 and the first delay frame F1. The data of the positioned interpolation frame IF is generated, and the image data DO inserted between the data of the current frame F0 and the data of the first delay frame F1 is output to the image display unit 6. The image display unit 6 displays the image data DO.

Next, the configuration of the motion vector detection unit 2 will be described in detail. The motion vector detection unit 2 includes the current frame block cutout unit 10, the first delay frame block cutout unit 11, the second delay frame block cutout unit 12, the test interpolation unit 7, the interpolation data evaluation unit 8, and the motion vector determination unit 9. Be prepared.

The current frame block cutting section 10, the first delay frame block cutting section 11, and the second delay frame block cutting section 12 each cut out a block forming a part of the screen, and the data (pixel value) of the pixels in the block is obtained. Output the set as block data. Each block consists of, for example, a rectangular area having the size of X pixels in the horizontal direction and Y pixels in the vertical direction (Y lines). That is, the block cut out from the current frame F0, the block cut out from the first delay frame F1, and the block cut out from the second delay frame F2 have a vertical size (number of pixels) and a horizontal size (pixels). Number to number of lines) are equal to each other.

The current frame block cutting section 10 cuts out a block from the current frame F0, the first delay frame block cutting section 11 cuts out a block from the first delay frame F1, and the second delay frame block cutting section 12 is the second. Cut out a block from the delay frame F2.

The processing for generating one block in the interpolation frame IF by interpolation will be described below. For this processing, one block in the first delay frame F1 corresponding to the block to be interpolated in the interpolation frame IF, multiple blocks in the current frame F0, and in the second delay frame F2. Multiple blocks of are cut out. The block cut out from the current frame F0 and the block cut out from the second delay frame F2 are in point-symmetrical positions with respect to the block in the first delay frame F1 (strictly speaking, its center position). And these are used as a pair. That is, the current frame block cutout portion 10 and the second delay frame block cutout portion 12 are centered on the block in the first delay frame F1, one is located in the current frame F0, and the other is the second delay frame F2. Cut out multiple pairs of blocks located inside.

The pair of blocks from the current frame F0 and the second delay frame F2 corresponds to the motion vector candidates detected by the motion vector detection unit 2, for example, all the blocks within the search range of the motion vector are cut out. Is done. For example, when searching for a range of ± HS pixels in the horizontal direction and ± VS pixels (± VS line) in the vertical direction centered on the center position of one block in the first delay frame F1, the second delay frame (2HS + 1) × (2VS + 1) blocks are cut out from F2 and the current frame F0, respectively.

If it is not necessary to evaluate all the blocks in the search range, for example, if the range of the direction of movement can be predicted in advance or by other information, only the blocks within the predicted range of the above search range are available. It may be cut out. Further, the blocks in the search range may be thinned out (for example, every other pixel in the horizontal direction and the vertical direction).

In the following, the number of blocks cut out from the current frame F0 and the second delay frame F2 is M, and the blocks cut out from the current frame F0 are cut out from the first to M blocks F0B1 to F0BM and the second delay frame F2. The blocks to be used are called the first to M blocks F2B1 to F2BM. In addition, the data of each block is represented by the same code as the block.

The mth block F2Bm (m = 1 ~ M) in the second delay frame F2 and the mth block F0Bm in the current frame F0 are the block F1B1 (strictly speaking, its center) in the second delay frame F1. Since it is in a point-symmetrical position with respect to the center of the block F2Bm, the horizontal deviation of the block F2Bm with respect to the block F1B1 is h (h = -HS ~ + HS), and the vertical deviation is v (v = -VS ~ + VS). ), The horizontal deviation of the block F0Bm with respect to the block F1B1 is -h, and the vertical deviation is -v.

The current frame block cutting unit 10 cuts out a plurality of blocks, that is, the first to Mth blocks in the current frame, and outputs the first to Mth block data F0B1 to F0BM. The first delay frame block cutting unit 11 cuts out the block F1B1 in the first delay frame. This block F1B1 corresponds to the block to be interpolated in the interpolation frame IF. The second delay frame block cutting unit 12 cuts out a plurality of blocks, that is, the first to M blocks in the second delay frame, and outputs the first to M block data F2B1 to F2BM.

The block data of the current frame F0 and the block data of the second delay frame F2 are input to the test interpolation unit 7. Of the block data of the second delay frame F2 and the block data of the current frame F0, the test interpolation unit 7 is the first delay located at a point-symmetrical position with respect to the block F1B1 in the first delay frame F1. Test interpolation data is generated based on the data of the block pair consisting of the blocks in the frame F1 and the blocks in the current frame F0. Multiple test interpolation data are generated based on multiple block pairs. This test interpolation is performed on the assumption that the center position of the point symmetry, that is, the data of the block F1B1 in the first frame F1 is unknown, and the more accurate the interpolation, the more the test interpolation data. Is highly correlated with the data in block F1B1.

The interpolation data evaluation unit 8 evaluates a plurality of test interpolation data with reference to the block data of the first delay frame F1, and outputs the evaluation data ED to the motion vector determination unit 9. In this evaluation, the correlation between the test interpolation data and the block data of the first delay frame F1 is obtained, and the higher the correlation, the higher the evaluation.

The motion vector determination unit 9 generates and outputs the first motion vector MV1 based on the evaluation data ED.

Next, specific examples of the test interpolation unit 7, the interpolation data evaluation unit 8, and the motion vector determination unit 9 of the motion vector detection unit 2 will be described in more detail with reference to FIG.

The test interpolation unit 7 includes a plurality of, that is, the first to Mth test interpolation data generation units 7-1 to 7-M, and the interpolation data evaluation unit 8 has a plurality of, that is, the first to Mth difference absolute values. It is illustrated as having a sum calculation unit 8-1 to 8-M.

The test interpolation data generators 7-1 to 7-M average the data F0B1 to F0BM of the block of the current frame F0 and the data F2B1 to F2BM of the block of the second delay frame F2 paired with these for each pixel. The data consisting of the average values obtained by the above is calculated as the test interpolation data TD1 to TDM. In FIG. 1, a set of test interpolation data TD1 to TDM is represented by the code TD.

Hereinafter, it will be described in more detail. The first block data F0B1 of the current frame F0 and the first block data F2B1 of the second delay frame F2 are input to the test interpolation data generation unit 7-1.

The test interpolation data generation unit 7-1 differs the average value of the first block data F0B1 of the current frame F0 and the first block data F2B1 of the second delay frame F2 for each pixel as the first test interpolation data TD1. Output to the absolute value sum calculation unit 8-1. Here, the average value for each pixel is the pixel value of each pixel in the block in the current frame F0 and the pixel value of the pixel at the corresponding position in the block in the second delay frame (for example, the reference position of each block, for example, the upper left corner). Means the average value of (pixel values of pixels represented by the same coordinate values) with

Similarly, the second block data F0B2 of the current frame F0 and the second block data F2B2 of the second delay frame F2 are input to the test interpolation data generation unit 7-2. The test interpolation data generation unit 7-2 differs the average value of the second block data F0B2 of the current frame F0 and the second block data F2B2 of the second delay frame F2 for each pixel as the second test interpolation data TD2. Output to the absolute value sum calculation unit 8-2.

Similarly, the test interpolation data generators 7-3 to 7-M also have the third block data F0B3 to M of the current frame F0 and the third block data F2B3 to M of the second delay frame F2. The third test interpolation data TD3 to the third test interpolation data TDM are generated based on the block data F2BM, and are output to the difference absolute value sum calculation unit 8-3 to 8-M.

Generally speaking, the test interpolation data generator 7-m is based on the mth block data F0Bm (m = 1 to M) of the current frame F0 and the mth block data F2Bm of the second delay frame F2. The mth test interpolation data TDm is generated and output to the difference absolute value sum calculation unit 8-m.

The block data F1B1 of the first delay frame F1 is input to the difference absolute value sum calculation unit 8-1 to 8-M in the interpolation data evaluation unit 8.

The difference absolute value sum calculation unit 8-1 to 8-M calculates the difference absolute value sum of the test interpolation data TD1 to TDM output from the test interpolation data generation unit 7 and the block data F1B1 of the first delay frame F1, respectively. Then, it is output as evaluation data ED1 to EDM.

The difference absolute value sum calculation unit 8-1 calculates and evaluates the sum of the difference absolute values of the data of each pixel of the first test interpolation data TD1 and the data of each pixel of the block data F1B1 of the first delay frame F1. It is output to the motion vector determination unit 9 as data ED1. The sum of the absolute values of the differences is expressed by Eq. (1).

<maths num="1"><img file="JP2012089986A_D0001.tif" /></maths>

The smaller the value of the absolute difference sum given in Eq. (1), the higher the correlation. When the absolute difference sum SAD is used as the evaluation data, the smaller the value, the higher the evaluation. Represent.

Here, if BK1 and BK2 are the data of each pixel in the block, BK1 is the data of each pixel constituting the test interpolation data TD1, and BK2 is the data of each pixel in the block F1B1, the equation (1) is first. The difference absolute value sum is the sum of the data of each pixel of the test interpolation data TD1 and the data of each pixel of the block data F1B1 of the first delay frame F1, and the difference absolute value sum SAD is the evaluation data ED1 and the difference absolute value sum calculation unit 8 Output from -1.

Similarly, the difference absolute value sum calculation unit 8-2 to 8-M calculates the difference absolute value sum of the second test interpolation data TD2 to the M test interpolation data TDM and the block data F1B1 of the first delay frame F1. Then, the evaluation data ED2 to EDM are output to the motion vector determination unit 9.

The motion vector determination unit 9 is a block of the current frame F0 and a block of the second delay frame F2 that form a block pair corresponding to the highest evaluation data (smallest difference absolute value sum) among the evaluation data ED1 to EDM. 1/2 of the position difference (the relative position of the block in the current frame F0 with respect to the block in the second delay frame F2) is output as the motion vector MV1.

FIG. 3 is a diagram for explaining the operation of the motion vector detection unit 2. The case of M = 2 will be described. A part of the area of the first delay frame F1 is cut out as the block data F1B1 of the first delay frame F1.

The area corresponding to the position shifted by vector -V1 with respect to the block data F1B1 of the first delay frame F1 is set as the first block data F2B1 of the second delay frame F2 and cut out, and the position shifted by vector + V. The area corresponding to is set as the first block data F0B1 of the current frame F0 and is cut out.

In addition, the area corresponding to the position shifted by vector -V2 with respect to the block data F1B1 of the first delay frame F1 is set as the second block data F2B2 of the second delay frame F2 and cut out, and the vector + V2 is shifted. The area corresponding to the position is set as the second block data F0B2 of the current frame F0 and is cut out.

The test interpolation data generation unit 7-1 generates the test interpolation data TD1 by averaging the first block data F0B1 of the current frame F0 and the first block data F2B1 of the second delay frame F2 for each pixel. Similarly, the test interpolation data generation unit 7-2 also generates the test interpolation data TD2 by averaging the second block data F0B2 of the current frame F0 and the second block data F2B2 of the second delay frame F2 for each pixel. ..

The difference absolute value sum calculation unit 8-1 calculates the difference absolute value sum SAD from the test interpolation data TD1 and the block data F1B1 of the first delay frame F1 using the equation (1), and outputs it as the evaluation data ED1.

Similarly, the difference absolute value sum calculation unit 8-2 calculates the difference absolute value sum SAD from the test interpolation data TD2 and the block data F1B1 of the first delay frame F1 using equation (1) and outputs it as the evaluation data ED2. To do.

The motion vector determination unit 9 outputs the above deviation (+ V1 or + V2) of the blocks constituting the block pair that caused the smaller value of the evaluation data ED1 and ED2 as the motion vector MV1. For example, if the evaluation data ED1 is smaller than the evaluation data ED2, the vector V1 is output as the motion vector MV1.

Although the method of determining the first motion vector from the two vectors has been described with reference to FIG. 3, the configuration of the embodiment according to the present invention is not limited to M = 2. That is, three or more vectors as candidates may be set. For example, with respect to the block F1B1 in the first delay frame, all the blocks in the second delay frame located in the search range corresponding to the predetermined movement amount and in the current frame at a point symmetric position with the blocks F1B1. You may also perform test interpolation for the block.

Since the motion vector detection unit 2 determines the motion vector by evaluating the vector candidates using the data in the second delay frame F1 which is the actual data as described above, the motion vector detection unit 2 is the first from the current frame F0. The motion vector to the delay frame F1 can be calculated accurately.

The interpolation data evaluation unit 8 calculates the evaluation data using the sum of the absolute values of the differences, but there are many other functions for obtaining the correlation, such as the sum of squared errors, which can be replaced.

Next, the operation of the motion vector conversion unit 3 will be described in more detail with reference to FIG. The motion vector conversion unit 3 performs the motion vector MV1 from the first delay frame F1 to the current frame F0, the second motion vector MV2 from the first delay frame F1 to the interpolation frame IF, and the interpolation frame from the current frame F0. Convert to the third motion vector MV3 to IF.

As shown in Fig. 4, when the time interval of the input frame is t1 and the time interval from the first delay frame F1 to the interpolation frame IF is t2, the motion vectors MV2 and MV3 are calculated by equations (2A) and (2B). calculate. For example, when converting a 60 Hz input image signal to a 120 Hz image signal, t1 takes 1/60 seconds and t2 takes 1/120 seconds.

MV2 = MV1 × t2 / t1 ... (2A) MV3 = -MV1 × (t1-t2) / t1 ... (2B)

Such a conversion takes the vector MV1 from the first delay frame F1 to the current frame F0 according to the time interval (t2, t1-t2) between the first delay frame F1, the interpolation frame IF and the current frame F0. It can be said that this is a conversion using the process of interpolation.

Next, the operations of the occlusion region estimation unit 4 and the interpolation frame generation unit 5 will be described in more detail with reference to FIGS. 5A to 5C.

The occlusion area estimation unit 4 detects an edge in which the motion vectors MV2 and MV3 change rapidly on the interpolation frame. At the edge of the motion vector (the boundary between one region of continuous blocks with small motion vector differences and another region of consecutive blocks with small motion vector differences), before and after the interpolation frame. Image elements are hidden or appear, but as shown in Fig. 5 (b), in the occlusion area estimation unit 4, the area including the periphery of the edge is defined as the occlusion area OC. Here, the range of the occlusion region OC depends on the magnitude of the motion vector, and the larger the motion vector, the larger the range of the occlusion region OC.

Next, in the obtained occlusion area OC, it is determined whether the data should be acquired from the frame before the time or the frame after the time and used for interpolation. If the motion vectors of two adjacent regions across the edge of the motion vector are in the direction of approaching each other, one image element is hidden and data is taken from the past frame (the data of the past frame is taken from the interpolated frame). Use as data). That is, the motion vector MV2 is output as the fourth vector MV4, and the fifth vector MV5 is output as "no correspondence" (information indicating that the vector corresponding to the fifth vector MV5 does not exist is output).

On the other hand, if the motion vectors of two regions adjacent to each other across the edge of the motion vector (that is, located on both sides of the edge) are in the direction away from each other, hidden image elements will appear, so data from the future frame will appear. (Use the data of the future frame as the data of the interpolation frame). That is, the fourth vector MV4 is output as "no correspondence" (information indicating that the vector corresponding to the fourth vector MV4 does not exist is output), and the motion vector MV3 is output as the fifth vector MV5.

When the fourth and fifth vectors MV4 and MV5 are obtained in this way, the vector from the interpolation frame IF-the data of the first delay frame F1 at the position of MV4 (data in the block F1B1) and the vector from the interpolation frame IF- The average of the data of the current frame F0 (data in the block F0B1) at the position of MV5 is calculated as the data of the interpolation frame IF. At this time, the block data F1B1 of the first delay frame F1 used for interpolation and the block data F0B1 of the current frame F0 are positioned symmetrically with respect to each other with the position of the data obtained by interpolation in the interpolation frame IF as the center. is there. However, if one of the fourth and fifth vectors MV4 and MV5 is "no corresponding block", the data of only the other of the fourth and fifth vectors MV4 and MV5 is calculated as it is as the data of the interpolation frame IF. ..

The data of the interpolation frame IF consisting of the generated data is interpolated between the data of the first delay frame F1 and the data of the current frame F0 and output.

6 (a) and 6 (b) are for explaining the correspondence between the data of the current frame F0, the data of the first delay frame F1, and the data of the second frame F2 used in the specific example of the motion vector detection unit 2. It is a figure of. The operation of the first embodiment when a signal representing an image as shown in FIGS. 6A and 6B is input will be described. In the images shown in FIGS. 6 (a) and 6 (b), the English characters as image elements are moving from left to right, and the black square as a static image element (in the figure, "black" is crossed). English characters are hidden by (indicated by hatching).

7 (a) to 7 (d) are diagrams for explaining a specific example of the operation of the motion vector detection unit 2. FIG. 7A is a diagram for explaining a specific example of the image data input to the motion vector detection unit 2. 7 (b) to 7 (d) are diagrams for explaining the operation of the test interpolation unit 7 and the interpolation data evaluation unit 8.

The operation of the motion vector detection unit 2 when the data of the second delay frame F2, the data of the first delay frame F1, and the data of the current frame F0 are input as shown in FIG. 7A will be described.

The test interpolation unit 7 generates test interpolation data for each motion vector. As shown in Fig. 7 (b), a part of the area of the first delay frame F1 is defined as the block data F1B1 of the first delay frame F1, and the position shifted by -V1 from the block data F1B1 of the first delay frame F1. The block data of the second delay frame F2 is referred to as the first block data F2B1, and the block data of the current frame F0 at a position shifted by V1 from the block data F1B1 of the first delay frame F1 is referred to as block data F0B1. Further, the block data of the second delay frame F2 at the position shifted by -V2 from the block data F1B1 of the first delay frame F1 is set as the block data F2B2, and the block data of the first delay frame F1 is shifted by V2 from the block data F1B1. Let the block data of the current frame F0 be the block data F0B2.

As shown in FIG. 7 (c), the average for each pixel is generated as the test interpolation data TD1 from the first block data F2B1 and the block data F0B1. FIG. 7 (c) shows that the image represented by the test interpolation data TD1 contains the character F. Similarly, the test interpolation data TD2 is generated as shown in FIG. 7 (d).

The interpolation data evaluation unit 8 calculates the sum of the difference absolute values between each of the test interpolation data TD1 to TD2 and the block data F1B1 as shown in FIGS. 7 (c) to 7 (d), and outputs the evaluation data ED1 to ED2. .. In the specific examples of FIGS. 7 (a) to 7 (d), the image represented by the test interpolation data TD1 contains the character "F", and the sum of the absolute values of the differences with the blocks F1B1 is the smallest, so the evaluation data ED1 is one. It becomes the smallest.

In the motion vector determination unit 9, the motion vector V1 corresponding to the smallest evaluation data ED1 among the evaluation data ED1 to ED2 is output as the motion vector of the block data F1B1.

By setting blocks without exception (without gaps) for the first delay frame F1 and calculating the motion vector, the first motion vector MV1 is generated for all parts of the first delay frame F1. That is, the first delay frame F1 is divided into a plurality of blocks having the same size, for example, and the above processing is performed for each block to generate the first motion vector MV1 for the block. The second and third motion vectors may be obtained by performing conversion using the first motion vector for the blocks in the interpolation frame at the positions corresponding to the blocks, for example, the same positions.

Further, by performing the above processing on a block having a predetermined size centered on each pixel in the first delay frame, the first motion vector MV1 for the pixel is generated, and the block corresponds to the pixel. The second and third motion vectors may be obtained by performing conversion using the first motion vector for the pixels in the interpolated frame at the position, for example, the same position.

8 (a) and 8 (b) are diagrams for explaining a specific example of the operation of the motion vector conversion unit 3. FIG. 8 (a) shows the input of the motion vector conversion unit 3, and FIG. 8 (b) shows the output of the motion vector conversion unit 3.

The motion vector conversion unit 3 uses equations (2A) and (2B) for the motion vector MV1 from the first delay frame F1 to the current frame F0 shown in FIG. 8A, and further t2. With = t1 / 2, the second motion vector MV2 from the first delay frame F1 to the interpolated frame IF and the third motion vector MV3 from the current frame F0 to the interpolated frame IF, shown in Fig. 8 (b). Convert. In the specific examples shown in FIGS. 8 (a) and 8 (b), MV1 is V1, t1 is 1/60 second, and t2 is 1/120 second. Therefore, from equations (2A) and (2B), MV2 is V1 / 2. MV3 becomes -V1 / 2.

9 (a) to 9 (c) are diagrams for explaining a specific example of the operation of the occlusion area estimation unit 4. FIG. 9A shows the value of the third motion vector MV3 from the current frame F0 to the interpolated frame IF in each of the two regions Ra and Rb in the image. In the illustrated example, the third motion vector MV3 is -V1 / 2 in the region Ra, and the motion vector MV3 is 0 in the region Rb. FIG. 9 (b) shows the estimated occlusion region OC, and FIG. 9 (c) shows the fourth motion vector MV4 from the first delay frame F1 to the interpolation frame IF and the fourth motion vector MV4 from the current frame F0 to the interpolation frame IF. The five motion vectors MV5 are shown.

The occlusion region estimation unit 4 starts with the third motion vector MV3 having a value as shown in FIG. 9 (a), and the edge of the motion vector as shown in FIG. 9 (b) (the motion vector is -V1 / 2). The boundary between the region Ra, which is the region Ra, and the region Rb where the vector is 0) and the region around it are estimated as the occlusion region OC. Moreover, it is judged that the motion vector -V1 / 2 and "0" in the region Ra are far from each other.

As mentioned earlier, edge detection is based on abrupt changes in motion vectors. In the estimated occlusion region OC, the fourth motion vector MV4 from the first delay frame F1 to the interpolation frame IF and the fifth motion vector MV5 from the current frame F0 to the interpolation frame IF are shown in FIG. 9 (c). It becomes as shown in. That is, the fourth motion vector MV4 has the same value as MV2 = -MV3 = V1 / 2, and the fifth motion vector MV5 is "no correspondence".

FIG. 10 is a diagram for explaining a specific example of the operation of the interpolation frame generation unit 5. As shown in FIG. 10, the average of the data of the first delay frame F1 at the position moved by -MV4 and the data of the current frame F0 at the position moved by -MV5 from the interpolation frame IF is calculated as the data of the interpolation frame IF. However, as shown by the arrow in FIG. 10, when one motion vector (MV5) does not correspond, the data corresponding to the other motion vector (MV4) is used as the interpolation frame IF. The generated interpolated frame IF is interpolated between the first delay frame F1 and the current frame F0 and output.

Here, for comparison, consider the case where the occlusion area estimation unit 4 is not provided. In this case, the second motion vector MV2 and the third motion vector MV3 are input to the interpolation frame generator 5 (instead of the fourth motion vector MV4 and the fifth motion vector MV5). Then, the area where the alphabetic character "F" is in the interpolation frame IF is black (indicated by cross-hatching in the figure) (data of the current frame F0) and alphabetic character "F" (data of the first delay frame F1). The image will be averaged, the edges will be blurred, and the image quality will deteriorate. On the other hand, in the present invention, the above averaging is not performed, and if the data of the interpolated frame is determined based on one of the motion vectors, the sharpness of the edge can be maintained and the image distortion can be suppressed. Can be done.

From the above, in the present embodiment, the motion vector can be detected with high accuracy by evaluating the motion vector, and the interpolated frame can be generated without image distortion. Furthermore, even when image elements are hidden or appear before and after the frame to be interpolated, frame interpolation can be performed without image distortion.

FIG. 11 is a flowchart showing a processing process of the image display device according to the present embodiment described above.

First, in the motion vector detection step ST1, the first image data F0, the image data F1 delayed by 1 frame with respect to the image data F0, and the image data F2 delayed by 2 frames with respect to the image data F0 are referred to. Generate the first motion vector MV1 from the delayed frame F1 to the current frame F0. This operation is equivalent to the motion vector detection unit 2.

In the motion vector conversion step ST2, the first motion vector MV1 is transferred from the first delay frame F1 to the interpolation frame IF (inserted between the current frame F0 and the first delay frame F1). Convert to MV2 and the third motion vector MV3 from the current frame F0 to the interpolated frame IF. This operation is equivalent to the motion vector conversion unit 3.

In the occlusion region estimation step ST3, the occlusion region estimation unit 4 estimates the occlusion region OC from the second motion vector MV2 and the third motion vector MV3, and the second motion is based on the estimated occlusion region information. The vector MV2 and the third motion vector MV3 are modified to generate the fourth motion vector MV4 and the fifth motion vector MV5. This operation is equivalent to the occlusion area estimation unit 4.

In the interpolation frame generation step ST4, the data of the interpolation frame IF is generated from the data of the first delay frame F1, the data of the current frame F0, the fourth motion vector MV4 and the fifth motion vector MV5, and the generated interpolation frame IF Is inserted between the data of the current frame F0 and the data of the first delay frame F1 to generate the image data DO. This operation is equivalent to the interpolation frame generation unit 5.

As can be understood from the explanation given with reference to FIG. 11, a part of the image processing apparatus of FIG. 1, particularly the motion vector detection unit 2, the motion vector conversion unit 3, the occlusion area estimation unit 4, and the interpolation frame generation unit 5 Can be realized in part or in whole by software, i.e. by a programmed computer.

1 frame memory, 2 motion vector detection unit, 3 motion vector conversion unit, 4 occlusion area estimation unit, 5 interpolation frame generation unit, 6 image display unit, 7 test interpolation unit, 8 interpolation data evaluation unit, 9 motion vector determination unit, 10 Current frame block cutout part, 11 1st delay frame block cutout part, 12 2nd delay frame block cutout part.