Improved interpolation of video compression frames

Improved video compression frame interpolation

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a partial continuation of the US application serial number 09/904203 filed on July 11, 2001, and claims priority to the US CIP application serial number 10/187395 filed on June 28, 2002.

Technical field

The present invention relates to video compression, and more particularly to improved video compression frame interpolation in MPEG-type encoding and decoding systems.

Background technique

MPEG video compression MPEG-2 and MPEG-4 are international video compression standards. Their respective video grammars define an efficient method to represent image sequences with more compact coded data. The language of the coded bits is the so-called "grammar". For example, several markers can represent the entire block of samples (for example, 64 samples for MPEG-2). Both MPEG standards describe the decoding (reconstruction) process, which transforms the encoded bits from a compact representation into an approximation of the original format of the image sequence. For example, a flag in the coded bitstream may indicate whether to use a prediction algorithm to process these bits before using a discrete cosine transform (DCT) algorithm to decode subsequent bits. The algorithm including the decoding process is specified by the semantics defined by these MPEG standards. This grammar can be used to develop common video features, such as spatial redundancy, temporal redundancy, uniform motion, spatial masking, and so on. The MPEG decoder must be able to parse and decode the input data stream, but as long as the data stream conforms to the corresponding MPEG syntax, a variety of possible data structures and compression techniques can be used (although this deviates from the standard technically, because of the semantics Inconsistent). It is also possible to carry the required semantics within a certain substitution grammar.

These MPEG standards use various compression methods, including intra-frame methods and inter-frame methods. In most video scenes, the background remains relatively stable and the action occurs in the foreground. The background may move, but usually many scenes are redundant. These MPEG standards start compression by creating reference frames called "intra" frames or "I frames." The I frame is compressed without referring to other frames, so the I frame contains a complete frame of video information. The I frame provides an entry point for the data bit stream for random access, but it can only be moderately compressed. Typically, data representing I frames is placed every 12 to 15 frames in the bitstream (although in some cases it may be useful to use wider intervals between I frames). According to this, since only a small part of the frames that fall between the reference I frames is different from the bracketing I frames, only the image difference is captured, compressed, and stored. Two types of frames are used for this difference-predicted frames (P frames) and bidirectionally predicted (or interpolated) frames (B frames).

P-frames are generally coded based on past frames (I-frames or previous P-frames), and are usually used as references for subsequent P-frames. P-frames are subject to considerable compression. B-frames provide the highest amount of compression, but in order to be encoded, both past reference frames and future reference frames are required. P-frames and I-frames are "referenceable frames" because they can be referenced by P-frames or B-frames.

A macroblock is an image pixel area. For MPEG-2, a macroblock is a 16×16 pixel group of 4 8×8 DCT blocks, one motion vector is added for P frames, and one or two motion vectors are added for B frames. The macroblocks within the P frame can be individually coded using intra-frame or inter-frame (predicted) coding. It is possible to use intra-frame coding, forward predictive coding, backward predictive coding, or simultaneous use of forward and backward (i.e., two-way interpolation) predictive coding to separately code the macroblocks in the B frame. The structure used in MPEG-4 video coding is slightly different, but similar.

After encoding, the MPEG data bit stream includes a sequence of I-frames, P-frames, and B-frames. The sequence can consist of almost any pattern of I-frames, P-frames, and B-frames (there are several minor semantic restrictions regarding their positions). However, it is common in industry practice to have a fixed frame mode (for example, IBBPBBPBBPBBPBB).

Motion vector prediction In MPEG-2 and MPEG-4 (and similar standards, such as H.263), the use of B-type (bidirectionally predictive) frames has proven to be beneficial to compression efficiency. The motion vector of each macroblock of such a frame can be predicted by any one of the following three methods: Mode 1: Forward prediction based on the previous I frame or P frame (ie, non-bidirectionally predicted frame).

Mode 2: Backward prediction based on subsequent I frames or P frames.

Mode 3: Bidirectional prediction based on subsequent and previous I frames or P frames.

Mode 1 is the same as the forward prediction method used for P frames. Mode 2 is the same concept except that it works backward based on subsequent frames. Mode 3 is an interpolation mode, which combines information from previous and subsequent frames.

In addition to these three modes, MPEG-4 also supports another interpolation motion vector prediction mode for B frames: direct mode prediction, which uses the motion vector from the subsequent P frame plus an increment (δ) value (if The motion vectors from the co-located P macroblocks are divided into 8×8 patterns-for 16×16 macroblocks 4 motion vectors are generated-then the increment is applied to all 4 independent motion vectors in the B frame) . The motion vector of the subsequent P frame points to the previous P frame or I frame. A ratio is used to weight the motion vectors of subsequent P frames. The ratio is the relative time position of the current B frame relative to the subsequent P frame and the previous P (or I) frame.

Figure 1 is a timeline of frames and MPEG-4 direct mode motion vectors according to the prior art. The concept of MPEG-4 direct mode (mode 4) means that the motion of the macroblock in each intervening B frame may be close to the motion used to encode the same position in the subsequent P frame. The increment is used to make small corrections to the proportional motion vector derived from the corresponding motion vector (MV) 103 of the subsequent P frame. Figure 1 shows the proportional weighting given to the motion vectors 101 and 102 of each intermediate B frame 104a, 104b as a function of the "time distance" between the previous P frame or I frame 105 and the next P frame 106. The motion vectors 101 and 102 assigned to the corresponding intermediate B frames 104a and 104b are equal to the assigned weight values (1/3 and 2/3, respectively) multiplied by the motion vector 103 of the next P frame, plus the increment value.

Regarding MPEG-2, all prediction modes of B-frames are tested in encoding and compared to find the best prediction for each macroblock. If there is no good prediction, then the macroblocks are individually coded into "I" ("intra") macroblocks. The coding mode is selected as the best mode among forward (mode 1), backward (mode 2) and bidirectional (mode 3) coding, or is selected as internal coding. Regarding MPEG-4, internal coding is not allowed. On the contrary, the direct mode becomes the fourth option. Similarly, the optimal coding mode is selected based on a certain best matching criterion. In software encoders based on MPEG-2 and MPEG-4, DC matching (sum of absolute difference, or "SAD") is used to determine the best match.

The number of consecutive B frames in the coded data bit stream is determined by the parameter value "M" in MPEG. M minus 1 is the number of B frames between each P frame and the next P frame (or I frame). Therefore, if M=3, there are 2 B frames between each P frame (or I frame), as shown in Figure 1. In terms of the limitation of the M value (and thus the number of consecutive B frames), the main limitation is that the amount of motion change between P frames (or I frames) becomes larger. More B frames means longer time between P frames (or I frames). Therefore, the efficiency of the motion vector and the limitation of the coding range constitute the limit of the number of intermediate B frames.

It is also important to note that P frames carry "variation energy" and move forward with the moving image stream, because each decoded P frame is used as the starting point for predicting the next subsequent P frame. However, B frames are discarded after use. Therefore, any bits used to create a B-frame are only used for that frame, and unlike P-frames, they do not provide corrections to aid the decoding of subsequent frames.

Summary of the invention

The present invention relates to a method, system and computer program for improving the image quality of one or more predicted frames in a video image compression system, wherein each frame includes a plurality of pixels.

In one aspect, the present invention includes determining the value of each pixel of the bidirectional predictive frame as a weighted ratio of the corresponding pixel value in a non-bidirectional predictive frame surrounding a bidirectional predictive frame sequence. In one embodiment, the weighting ratio is a function of the distance between the bracketed non-bidirectionally predicted frames. In another embodiment, the weighting ratio is a mixed function of the distance between the bracketed non-bidirectionally predicted frames and the equal average value of the bracketed non-bidirectionally predicted frames.

In another aspect of the present invention, the interpolated representation of the pixel value is expressed in a linear space, or in another optimized non-linear space different from the original non-linear representation.

Other aspects of the present invention include a system, a computer program, and a method, which include: A video image compression system having a sequence of referenceable frames including image regions, wherein at least one image region of at least one predicted frame refers to two or more referenceable frames The frame is encoded.

A video image compression system with a sequence of referenceable frames including image regions, wherein at least one image region of at least one predicted frame is coded with reference to one or more referenceable frames in display order, wherein at least one such referenceable frame is not in The referenceable frame closest to the front of at least one predicted frame in the display order.

A video image compression system with a sequence of referenceable frames including macroblocks, where at least one macroblock within at least one prediction frame is coded by interpolation from two or more referenceable frames.

A video image compression system with reference frames including image regions and a sequence of bidirectional predictive frames, wherein at least one image region of at least one bidirectional predictive frame is coded to include at least two or more motion vectors, and each such motion vector refers to Corresponding image area in at least one referenceable frame.

A video image compression system with a sequence of referenceable frames including image regions, in which at least one image region of at least one prediction frame is coded to include at least two motion vectors, and each such motion vector refers to a corresponding reference in the referenceable frame The image area of such at least one predicted frame is coded by interpolation from two or more referenceable frames.

A video image compression system with a sequence of reference frames including image regions and bidirectional predictive frames, in which at least one image region of at least one bidirectional predictive frame is coded as a non-compression of selected image regions from two or more referenceable frames Equally weighted.

A video image compression system with a sequence of reference frames and bidirectional predictive frames including image regions, where at least one image region of at least one bidirectional predictive frame is encoded by interpolation from two or more referenceable frames, two of which are Or at least one of the plurality of referenceable frames is separated by at least one intervening referenceable frame and the bidirectional predictive frame in the display order, and wherein such at least one image area is encoded as such at least two or more referenceable frames The unequal weighting of the selected image area of the frame.

A video image compression system with a sequence of reference frames and bidirectional predictive frames including image regions, where at least one image region of at least one bidirectional predictive frame is encoded by interpolation from two or more referenceable frames, two of which are Or at least one of the plurality of referenceable frames is separated from the bidirectional predictive frame by at least one intervening subsequent referenceable frame in the display order.

A video image compression system with a sequence of reference frames including image regions and bidirectional predictive frames, in which at least one image region of at least one bidirectional predictive frame is coded as a non-compression of selected image regions from two or more referenceable frames Equally weighted.

A video image compression system with a sequence of prediction frames and bidirectional prediction frames, each frame includes pixel values arranged in macroblocks, where at least one macroblock in the bidirectional prediction frame is based on two or more prediction frames The direct mode prediction of the motion vector is determined.

A video image compression system with reference frames and a sequence of bidirectional predictive frames, each frame includes pixel values arranged in macroblocks, where at least one macroblock in the bidirectional predictive frame is based on one or more The direct mode prediction of the motion vector of the two prediction frames is determined, wherein at least one of such one or more prediction frames is before the bidirectional prediction frame in the display order.

A video image compression system with reference frames and bidirectional predictive frame sequences, each frame includes pixel values arranged in macroblocks, where at least one macroblock in the bidirectional predictive frame is based on data from one or more predictive frames The direct mode prediction of the motion vector is determined, wherein at least one of such one or more prediction frames is after the bidirectional prediction frame in the display order and is separated from the bidirectional prediction frame by at least one intervening reference frame.

A video image compression system with a sequence of frames, each frame includes a plurality of image regions with a DC value, each pixel included in each such image region has an AC pixel value, wherein at least one of the frames has a DC value of at least one image region At least one of the value and the AC pixel value is determined as a weighted interpolation of the corresponding respective DC value and AC pixel value from at least one other frame.

A video image compression system with a sequence of referenceable frames, each frame includes a plurality of image regions with a DC value, each pixel included in each such image region has an AC pixel value, and at least one image of at least one predicted frame At least one of the DC value and the AC pixel value of the region is obtained by interpolation from the corresponding respective DC values and AC pixel values of two or more referenceable frames.

Improve the image quality of a sequence of two or more bidirectional predictive intermediate frames in a video image compression system, each frame includes a plurality of image areas with a DC value, and each such image area includes an AC pixel for each pixel The value includes at least one of the following: determining the AC pixel value of each image region of the bidirectional prediction intermediate frame is the first weighted ratio of the corresponding AC pixel value in the referenceable frame surrounding the bidirectional prediction intermediate frame sequence; and determining this The DC value of each image area of the bidirectional prediction intermediate frame is a second weighted ratio of the corresponding DC value in the referenceable frame surrounding the bidirectional prediction intermediate frame sequence. A video image compression system having a sequence of frames, the frame comprising a plurality of pixels having an initial representation, wherein the pixels of at least one frame are interpolated from corresponding pixels of at least two other frames, wherein the pixels of the at least two other frames The corresponding pixels are interpolated when transformed to a different representation, and cause the interpolated pixels to be transformed back to the original representation.

In a video image compression system with a reference frame and a bidirectional predictive frame sequence, dynamically determining the coding mode of such a frame with a variable number of bidirectional predictive frames includes: selecting an initial sequence that starts from the referenceable frame , Has at least one subsequent bidirectional prediction frame and ends with a reference frame; adds the reference frame to the end of the initial sequence to create a test sequence; evaluates the test sequence against the selected evaluation criteria; for evaluating the For each satisfying step of the test sequence, insert a bidirectional frame before the added reference frame and repeat the evaluation step; and if the evaluation of the test sequence is not satisfied, then the previous test sequence is accepted as the current encoding mode.

A video image compression system with a sequence of reference frames, the reference frames are separated by at least one bidirectional predictive frame, where the number of such bidirectional predictive frames varies in this sequence, and at least one such bidirectional predictive frame At least one image area of is determined using unequal weights corresponding to pixel values of at least two referenceable frames.

A video image compression system with a sequence of frames that are encoded by an encoder and decoded by a decoder, where at least one image area of at least one frame is based on a weighted interpolation of two or more other frames, and this weighted interpolation is based on At least one set of weights available to the encoder and the decoder, wherein a designation of one of the selected at least one set of weights is transmitted from the encoder to the decoder to select one or more currently valid weights.

A video image compression system with a sequence of frames that are encoded by an encoder and decoded by a decoder, where at least one image area of at least one frame is based on a weighted interpolation of two or more other frames, and this weighted interpolation is based on At least one set of weights, of which at least one set of weights is downloaded to the decoder, and then the designation of one of the selected at least one set of weights is transmitted from the encoder to the decoder to select one or more currently valid The weight of.

A video image compression system with a sequence of reference frames, the reference frames are encoded by the encoder and decoded by the decoder, wherein the predicted frames in the sequence of reference frames are transmitted from the encoder to the decoder, and the transmission order is different from this The display order of the predicted frame after decoding.

A video image compression system with a sequence of referenceable frames. The referenceable frame includes pixels arranged in an image area, where at least one image area of at least one predicted frame is coded with reference to two or more referenceable frames, where each One such image area is determined using unequal weights corresponding to the pixel values of such two or more referenceable frames.

A video image compression system with predicted frames, bi-directional predicted frames and intra-frame sequences, each frame includes an image area, wherein at least one filter selected from the sharpening and softening filter group is in the predicted frame or bi-directional predicted frame The motion vector compensation prediction process is used for at least one image area of such a prediction frame or a bidirectional prediction frame.

The drawings and the following description illustrate the details of one or more embodiments of the invention. According to the specification, drawings and claims, other features, objectives and advantages of the present invention are obvious.

Description of the drawings

Figure 1 is a timeline of frames and MPEG-4 direct mode motion vectors according to the prior art.

Figure 2 is a timeline and proportional pixel weighted values of a frame according to this aspect of the invention.

Figure 3 is a timeline of a frame and mixed, proportional and equal pixel weights according to this aspect of the invention.

Fig. 4 is a flowchart showing an illustrative embodiment of the present invention as a method that can be implemented by a computer.

Fig. 5 shows an example of multiple previous references in which the current P frame refers to two previous P frames and one previous I frame.

Fig. 6A is a typical prior art MPEG-2 coding mode diagram, showing a constant number of B frames between I frames and/or P frames to be surrounded.

Fig. 6B is a theoretically possible prior art MPEG-4 video coding mode diagram, showing the number of B frames with varying numbers of I frames and/or P frames to be surrounded, and the varying distances between I frames.

Figure 7 is a diagram of an encoding mode.

Fig. 8 is a flowchart showing an embodiment of the interpolation method. DC interpolation is different from AC interpolation.

FIG. 9 is a flowchart showing an embodiment of the interpolation method using the brightness pixels of the replacement representation.

Fig. 10 is a flowchart showing an embodiment of an interpolation method of chrominance pixels using alternate representations.

Fig. 11 is a diagram showing the unique motion vector area size of each of two P frames.

Figure 12 is a diagram showing a sequence of P and B frames, with the interpolation weight of the B frame, which is determined as a function of the distance from a 2-away subsequent P frame with reference to a 1-away ( 1-away) Subsequent P frames.

Figure 13 is a diagram showing a sequence of P and B frames, with the interpolation weight of the B frame, which is determined as a function of the distance from a 1 away from the subsequent P frame, which refers to a 1 away from the previous P frame.

Fig. 14 is a diagram showing a sequence of P frames and B frames, in which subsequent P frames have multiple motion vectors that refer to previous P frames.

FIG. 15 is a diagram showing a sequence of P frames and B frames, where the nearest subsequent P frame has a motion vector that refers to a previous P frame, and the next nearest subsequent P frame has multiple motion vectors that refer to the previous P frame.

Fig. 16 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame.

Fig. 17 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has two motion vectors referring to the previous P frame.

Fig. 18 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame.

Figure 19 is a frame sequence showing three P frames P1, P2, P3, where P3 uses an interpolation reference with two motion vectors, one for each of P1 and P2.

Figure 20 shows a frame sequence of four P frames P1, P2, P3, and P4, where P4 uses an interpolation reference with three motion vectors, and P1, P2, and P3 each have a motion vector.

Figure 21 is a diagram showing a sequence of P frames and B frames, where different P frames have one or more motion vectors that refer to different previous P frames, and show the respective forward and backward references assigned to a specific B frame. Different weights.

Fig. 22 is a diagram showing a sequence of P frames and B frames, where the order of the bit stream of the P frames is different from the display order.

Fig. 23 is a diagram showing a sequence of P frames and B frames with assigned weights.

Fig. 24 is a time chart of the position of the object in the frame.

The same reference symbols in different drawings denote the same elements.

detailed description

Overview One aspect of the present invention is based on the recognition that a common practice is to use an M value of 3, which provides two B frames between each P (or I) frame. But M=2 and M=4 or higher are both useful. It is particularly important to note that the value of M (the number of B frames plus 1) also has a natural relationship with the frame rate. At 24 frames per second (fps), which is the playback rate of a movie, a time distance of 1/24 second between frames can cause substantial image changes from frame to frame. But at frame rates of 60fps, 72fps or higher, the time distance between adjacent frames is correspondingly reduced. The result is that as the frame rate increases, the larger the number of B frames (that is, the larger the value of M) becomes useful and beneficial in compression efficiency.

Another aspect of the present invention is based on the recognition that both MPEG-2 and MPEG-4 video compression use too simple interpolation methods. For example, for mode 3, the bidirectional prediction of each macroblock of the frame is the equal average of the subsequent and previous frame macroblocks, which is replaced by two corresponding motion vectors. This equal average is suitable for M=2 (that is, a single intermediate B frame), because the time from the B frame to the previous P (or I) frame and the subsequent P (or I) frame is equal. However, for a larger value of M, using equal weighting can only perform the best prediction on the centrally symmetrical B frame (that is, the intermediate frame if M=4, 6, 8, etc.). Similarly, in MPEG-4 Direct Mode 4, even if the motion vector is proportionally weighted, the predicted pixel value of each intermediate B frame is an equal proportion of the corresponding pixels of the previous P (or I) frame and the subsequent P frame.

Therefore, for M>2, it is an improvement to apply the appropriate proportional weight to the predicted pixel value of each B frame. The proportional weight of each pixel of the current B frame corresponds to the relative position of the current B frame with respect to the previous and subsequent P (or I) frames. Therefore, if M=3, the first B frame uses 2/3 of the corresponding pixel value (adjusted motion vector) of the previous frame and 1/3 of the corresponding pixel value (adjusted motion vector) of the subsequent frame.

Figure 2 is a timeline and proportional pixel weighted values of a frame according to this aspect of the invention. The pixel values in each macroblock of each intermediate B frame 201a, 201b are weighted as a function of the "distance" between the previous P or I frame A and the next P or I frame B, the farther away from the P or I frame the "distance" is Closer, the greater the weighting value. That is, each pixel value of the bidirectionally predicted B frame is a weighted combination of the corresponding pixel values of the non-bidirectionally predicted frames A and B that are surrounded. In this example, for M=3, the weight of the first B frame 201a is equal to 2/3A+1/3B, and the weight of the second B frame 201b is equal to 1/3A+2/3B. Also shown is the equal average weights assigned under the conventional MPEG system; the MPEG-1, 2 and MPEG-4 weights of each B frame 201a, 201b are equal to (A+B)/2.

Application of Extended Dynamic Range and Contrast Range If M is greater than 2, the proportional weighting of the pixel values in the middle B frame will improve bidirectional (mode 3) and direct (effectiveness of MPEG-4 mode 4) coding in many cases. Examples include common film and video editing effects, such as fade and cross fade. These types of video effects are problematic coding examples of MPEG-2 and MPEG-4. This is due to the use of a simple DC matching algorithm, and the usual M=3 (ie, two intermediate B frames), resulting in equal B frames proportion. The coding in these cases is improved by using proportional B-frame interpolation according to the present invention.

Proportional B-frame interpolation also has a direct application for improving the coding efficiency of the extended dynamic and contrast range. A common event in image coding is a change in brightness, which occurs when an object gradually moves into (or out of) the shadow (soft shadow boundary). If a logarithmic encoding representation is used for brightness (for example, the brightness specifically represented by logarithmic brightness Y), then the lighting brightness change will be a DC offset change. If the lighting brightness is halved, then the pixel value will be reduced by the same amount. Therefore, in order to encode this change, an AC match should be found, and the encoded DC difference should be applied to this area. This DC difference encoded into the P frame should also be applied proportionally in each intervening B frame (see co-pending U.S. Patent Application No. 09/905039, titled "Method and System for Improving Compressed Image Chroma Information ", which is assigned to the assignee of the present invention, and is therefore incorporated herein by reference for additional information regarding the logarithmic code representation).

In addition to changes in brightness, proportional B-frame interpolation is also beneficial to changes in contrast. For example, as the airplane moves toward the audience, and moves out of clouds or mist, its contrast gradually increases. This contrast increase will be expressed as the magnitude of the increase in the AC coefficients of the DCT in the corresponding P-frame coded macroblock. Likewise, the contrast change in the intervening B-frames will be most closely approximated by proportional interpolation, thus improving the coding efficiency.

As the frame rate becomes larger and as the value of M increases, it becomes increasingly important to use proportional B-frame interpolation to improve dynamic range and contrast coding efficiency.

Applying a high value of M to temporal layering. The embodiments of the present invention allow the value of M to be increased, thereby increasing the number of B-frames between the P-frames and/or I-frames that are bracketed, while maintaining or increasing the coding efficiency. . This usage is beneficial for many applications, including temporal stratification. For example, in U.S. Patent No. 5,988,863 entitled "Temporal and Resolution Layering for Advanced Television" (which is assigned to the assignee of the present invention and is hereby incorporated by reference), it is noted that the B frame is the time for layering. The appropriate mechanism for the (frame) rate. The flexibility of this rate is related to the number of consecutive B-frames available. For example, a single B frame (M=2) can support a 36 fps decoded time layer in a 72 fps stream or a 30 fps decoded time layer in a 60 fps stream. Three B frames (M=4) can support the time layer decoded at 36fps and 18fps in the 72fps stream, and the time layer decoded at 30fps and 15fps in the 60fps stream. Using M=10 in a 120fps stream can support 12fps, 24fps, and 60fps decoding time layers. M=4 can also be used for a 144fps stream to provide a time layer decoded at 72fps and 36fps.

As an improvement to each Nth frame, multiple frames of 120fps or 72fps can be decoded and mixed proportionally, as described in the co-pending U.S. Patent Application No. 09/545233, which is entitled "Enhancements to Temporal and Resolution Layering" (which is assigned to the assignee of the present invention and incorporated herein by reference) to improve the motion blur characteristics of the 24fps result.

Even higher frame rates can be synthesized using the method described in the co-pending U.S. Patent Application No. 09/435277, which is entitled "System and Method for Motion Compensaion and Frame Rate Conversion" (which is assigned to the present invention The assignee of is hereby incorporated by reference). For example, through motion-compensated frame rate conversion, 72 fps camera film (original) can be used to create an effective frame rate of 288 frames per second. Using M=12, you can get frame rates of 48fps and 24fps and other useful rates, such as 144fps, 96fps, and 32fps (of course, the film is 72fps). The frame rate conversion using this method does not need to be an integral multiple. For example, an effective rate of 120fps can be created from a source of 72fps, and then used as a source of 60fps and 24fps (using M=10).

Therefore, time layering is beneficial to optimize the performance of B-frame interpolation. The proportional B-frame interpolation described above makes the function of a larger number of consecutive B-frames more efficient, so these benefits can be realized.

The mixed B-frame interpolation ratio in the traditional system uses equal-average weighting as the motion compensation mode predictor of the B-frame pixel value is that the P (or I) frame before or after the specific B frame may be noisy, so Indicates an undesirable match. Equal mixing in the interpolated motion compensation block will optimize the noise reduction. There is a difference residual that is coded using a quantized DCT function. Of course, the better the match from the motion compensation ratio, the fewer differential residual bits required, and the higher the final image quality.

In the case of objects moving in and out of shadows or haze, the true ratio when M>2 provides a better prediction. However, when the light and contrast have not changed, equal weighting can prove to be a better predictor, because the error of moving the macroblock forward along the motion vector will be averaged with the error from the backward shifted block, so each The error is halved. Even so, it is more likely that the B-frame macroblock of the closer P (or I) frame is more related to that frame than the farther P (or I) frame.

Therefore, in some cases, such as changes in the contrast or brightness of the area, it is ideal to use the true ratio (for brightness and color) of the B-frame macroblock pixel weighting, as described above. In other cases, it may be better to use equal proportions, as in MPEG-2 and MPEG-4.

Another aspect of the present invention uses a mixture of these two ratio techniques (equal average and frame-distance ratio) for B-frame pixel interpolation. For example, for the case of M=3, 3/4 of the ratio of 1/3 and 2/3 can be mixed with the equal average of 1/4, resulting in the two ratios being 3/8 and 5/8. This technique can be summarized using the "mixing factor" F.

Weight=F·(frame distance proportional weight)+(1-F)·Equal-average weight The useful range of the mixing factor F is from 1 to 0, 1 means complete proportional interpolation, 0 means complete equal average (the opposite assignment may also be used ).

Figure 3 is a timeline of a frame and mixed, proportional and equal pixel weights according to this aspect of the invention. The pixel value of each macroblock of each intermediate B frame 301a and 301b is weighted as a function of the "time distance" between the previous P or I frame A and the next P or I frame B, and the values of A and B, etc. Average function. In this example, for M=3 and blending factor F=3/4, the blending weight of the first B frame 301a is equal to 5/8A+3/8B (that is, the proportional weighting 2/3A+1/3B is 3/ 4 plus 1/4 of the equal average weight (A+B)/2). Similarly, the weight of the second B frame 301b is equal to 3/8A+5/8B.

The value of the blending factor F can be set for all encodings as a whole, or set for each group of pictures (GOP), a range of B frames, each B frame, or each area within a B frame (including, for example, refinement). To each macroblock, or for MPEG-4 direct mode using P vectors in 8×8 mode, or even a single 8×8 motion block).

In order to save bits and reflect the fact that the mixing ratio is usually not important enough to be transmitted with each macroblock, the optimal use of mixing should be related to the type of image to be compressed. For example, for an image that is fading out or fading, or an image whose overall lighting or contrast is gradually changing, the blending factor F is close to or equal to 1 (ie, proportional interpolation) is generally optimal. For continuous images without such lighting or contrast changes, a lower blending factor value such as 2/3, 1/2 or 1/3 may form the optimal choice, thereby retaining some of the benefits of proportional interpolation and equal average interpolation Some of the benefits. All blending factor values in the range of 0 to 1 are generally useful, because for any given B frame, there is a specific value in this range that proves to be optimized.

For images with a wide dynamic range and a wide contrast range, the blending factor can be determined depending on the characteristic area of the local area. However, it is generally recommended that a wide range of lighting and contrast ratios supports full proportional interpolation instead of equal average interpolation.

The optimized blending factor is usually determined by experience, although experience for a specific type of scene can be used to create a blending factor table by scenario type. For example, the determination of image change characteristics can be used to select a blending ratio for a frame or region. Alternatively, a number of candidate blending factors (for the entire frame or region) can be used to encode B-frames, and each blending factor can be evaluated to optimize image quality (for example, determined by the highest signal-to-noise ratio (SNR)) and the lowest number of bits . The evaluation of these candidates can then be used to select the best value for the mixing ratio. A combination of image change characteristics and encoding quality/efficiency can also be used.

The B frame near the middle of the B frame sequence or the B frame from the low value M will not be too much affected by the proportional interpolation, because the calculated ratio is already close to the equal average. However, for a higher value of M, the extreme value of the B frame position can be significantly affected by the choice of the blending factor. Note that the mixing factors can be different for these extreme positions. These extreme positions use the average more than the more central positions (which gain little or no benefit from deviating from the average) because they already have adjacent P (or I) frames A high percentage of people. For example, if M=5, the first and fourth B-frames may use a blending factor F that blends more equal averages, but the second and third B-frames may use strict 2/5 and 3/ The equal average ratio of 5. If the proportional-to-average blend factor (propotion-to-average blend factor) changes as a function of the position of the B-frame in the sequence, then the changed value of the blend factor can be transmitted in the compressed bitstream or as auxiliary information to the decoder.

If a static general mixing factor is required (due to the lack of a way to transmit the value), the value 2/3 is usually close to optimal and can be selected as a static value for B-frame interpolation in the encoder and decoder. For example, using a blending factor of F=2/3, for consecutive frames with M=3, the ratio will be 7/18 (7/18=2/3*1/3+1/3*1/2) and 11/ 18 (11/18=2/3*2/3+1/3*1/2).

Linear interpolation video frame pixel values are generally stored in a specific representation, which maps the original image information to digital values. This mapping may result in linear or non-linear representations. For example, the brightness value used in compression is non-linear. The use of various forms of non-linear representation includes logarithms, exponents (various powers), and black-corrected exponents (usually used for video signals).

In a narrow dynamic range or for interpolation in neighboring regions, non-linear representation is acceptable because these neighboring interpolations represent piecewise linear interpolation. Therefore, small changes in brightness can be reasonably approximated by linear interpolation. However, for wide variations in brightness, such as those occurring in images with a wide dynamic range and wide contrast range, it will be inaccurate to process non-linear signals as linear ones. Even for images in the normal contrast range, linear fading and cross-fade can be reduced by linear interpolation. Some fade and cross fades use nonlinear fade and fade rates, adding further complexity.

An additional improvement to using proportional blending, or even simple proportional or equal-average interpolation, is to perform this interpolation on the pixel values represented in linear space, or in other optimized non-linear spaces that are different from the original non-linear brightness representation Perform this interpolation.

This can be done, for example, by first converting the two non-linear luminance signals (from the previous and subsequent P (or I) frames) into a linear representation, or a different non-linear representation. Proportional mixing is then applied, and after applying the inverse transform, the result of the mixing is produced in the original non-linear brightness representation of the image. However, the scaling function has been implemented on a better representation of the luminance signal.

When colors are fading or becoming more saturated, as occurs in contrast changes associated with changes in mist and cloudy skies, in addition to brightness, this linear or non-linear transformation is advantageously applied to the color (chromaticity ) Value is also useful.

Exemplary embodiment Fig. 4 is a flowchart showing an exemplary embodiment of the present invention as a method that can be implemented by a computer: Step 400: In a video image compression system, for calculating the direct sum interpolation of B frames Mode, using (1) the frame distance ratio, or (2) a mixture of equal weight and frame distance ratio to determine the interpolation value to be applied to each pixel of the input sequence of two or more bidirectional prediction intermediate frames. The interpolation value is derived from Enclose at least two non-bidirectionally predicted frames of such a sequence input from a source (for example, a video image stream).

Step 401: Optimize the interpolation with respect to image units (eg, group of pictures (GOP), frame sequence, scene, frame, region within the frame, macro block, DCT block, or similar useful grouping or pixel selection). The interpolation value can be set statically for the entire encoding period, or dynamically set for each image unit.

Step 402: Further optimize the interpolation value with respect to the scene type or coding simplicity. For example, the interpolation value can be set: statically (such as 2/3 ratio and 1/3 equal average); proportionally, it is a frame close to the equal average, but mixed with equal average near the adjacent P (or I) frame; Dynamically, based on overall scene characteristics, such as fade and cross-fade; dynamically (and locally), based on local image area characteristics, such as local contrast and local dynamic range; or dynamically (and locally), based on coding Performance (e.g. highest coded SNR) and least coded bits generated.

Step 403: Transmit an appropriate proportional amount to the decoder, if it is not statically determined.

Step 404: Optionally, convert the brightness information to a linear or alternative non-linear representation for each frame, and transmit this replacement to the decoder, if it is not statically determined.

Step 405: Use the determined interpolation value to determine the proportional pixel value.

Step 406: If necessary (due to step 404), convert to the original representation.

The extended P frame reference is as described above. In the prior art MPEG-1, 2 and 4 compression methods, the P frame refers to the previous P or I frame, and the B frame refers to the nearest previous or subsequent P and/or I frame . The same technology is used in the DCT compression standard for H.261 and H.263 motion compensation, which includes low bit rate compression technology.

In the H.263++ and H.26L standards under development, the B frame reference is extended to point to P or I frames that do not directly enclose the current frame. That is, the macro block in the B frame may point to a P or I frame before the previous P frame, or point to a P or I frame after the subsequent P frame. Since each macro block has one or more bits, skipping the previous or subsequent P frames can be simply represented. Conceptually, using the previous P frame for reference in B requires only storage space. For H.263++ or H.26L low bit rate encoding use, this is a small amount of additional storage. For subsequent P frame references, the P frame encoding sequence must be modified relative to the B frame encoding, so that the future P frame (or possibly I frame) must be decoded before the intervening B frame. Therefore, the coding sequence is also a problem for subsequent P frame references.

The main difference between P frame and B frame type is: (1) B frame can be bidirectionally referenced (up to two motion vectors per macro block); (2) B frame is discarded after use (this also means that They can be skipped during the decoding process to provide temporal layering); and (3) P frames are used as "springboards", one to the next, because each P frame must be decoded to be used as a reference for each subsequent P frame .

As another aspect of the present invention, a P frame (as opposed to a B frame) is decoded according to one or more previous P or I frames (excluding the case where each P frame only refers to the nearest previous P frame or I frame). Therefore, for example, two or more motion vectors of each macroblock can be used for the current P frame, and all motion vectors point to the back in time (that is, point to one or more previously decoded frames). Such P-frames still maintain the "springboard" characteristics. FIG. 5 is an example showing multiple previous references, two previous P frames 502, 504 and previous I frame 506 are referenced from the current P frame 500.

In addition, the concept of macroblock interpolation can be applied in such a P frame reference, as described above. Therefore, in addition to representing a single reference to more than one previous P or I frame, a motion vector can also be used for each such frame reference to mix the proportions of multiple previous P or I frames. For example, the technique described above using the B frame interpolation mode with two frame references can be applied to allow any macroblock in a P frame to use two motion vectors to refer to the previous two P frames or one previous P frame and one The previous I frame. This technique interpolates between two motion vectors, but it is not bidirectional in time (as in the case of B-frame interpolation), because both motion vectors point to the back in time. The storage overhead is reduced to a point at which it is quite practical to store the previous multiple P or I frames in the memory for this concurrent reference.

When applying this kind of P frame interpolation, it is definitely necessary to select various useful ratios of the previous two or more P frames (and optionally one previous I frame) and notify the decoder. In particular, the equal mixing of frames is one of the useful mixing ratios. For example, with the first two P frames as references, the equal 1/2 of each P frame can be mixed. For the first three P frames, an equal mix of 1/3 can be used.

Another useful blend of two P frames is 2/3 of the nearest front frame and 1/3 of the farthest front frame. For the first 3 P frames, another useful mix is 1/2 of the nearest front frame, 1/3 of the middle front frame, and 1/6 of the farthest front frame.

In any case, a simple set of useful mixes of multiple previous P frames (and optionally an I frame) can be used and they can be simply notified from the encoder to the decoder. Whenever it is useful to optimize the coding efficiency of a picture unit, the specific mixing ratio used can be selected. A small number of bits can be used to select many mixing ratios, and these bits can be transmitted to the decoder as long as they are suitable for the desired image unit.

As another aspect of the present invention, it is also useful to select a single P frame to transition from the nearest previous P (or I) frame to a more "far" previous P (or I) frame. In this way, P-frames will use a single motion vector for each macro block (or alternatively, each 8×8 block in MPEG-4 mode encoding), but will use one or more bits to represent the Reference refers to a single specific previous frame. The P frame macroblock in this mode is not interpolated, but will refer to the selected previous frame, which is selected from the possible two, three or more previous P (or I) frames for reference. For example, 2-bit encoding can designate up to 4 of the previous frames as a single selection frame. This 2-bit encoding can be changed in any convenient image unit.

The number of adaptive B frames typically uses fixed patterns of I, P, and B frame types in MPEG encoding. The number of B frames between P frames is typically constant. For example, it is typical to use two B frames between P (or I) frames in MPEG encoding. FIG. 6A is a typical prior art MPEG-2 encoding mode diagram, showing a fixed number of B frames (ie, two) between the I frame 600 and/or P frame 602 to be surrounded.

The MPEG-4 video coding standard conceptually allows a variable number of B frames and varying distances between I frames and/or P frames to be surrounded. Fig. 6B is a theoretically possible prior art MPEG-4 video coding mode diagram, showing a variable number of B frames between the enclosed I frame 600 and/or P frame 602, and the change between I frame 600 distance.

This flexible coding structure can theoretically be used to improve coding efficiency by matching the most effective B-frame and P-frame coding types to moving image frames. Although this flexibility has been specifically allowed, it is rarely studied, and the mechanism for actually determining the positions of B and P frames in this flexible structure is also unknown.

Another aspect of the present invention applies the concepts described here to this flexible coding structure and simple fixed coding mode in normal use. Therefore, the above-mentioned method can be used to interpolate the B frame, and the P frame can refer to more than one previous P frame or I frame and be interpolated according to this specification.

In particular, the macroblocks in the B-frames can use a ratio mix suitable for a flexible coding structure that is as effective as a fixed structure. When a B frame refers to a P frame or an I frame that is farther than the closest P frame or I frame for bracketing, proportional mixing can also be used.

Similarly, in this flexible coding structure, P-frames can refer to more than one previous P-frame or I-frame, which is as effective as the fixed-mode structure. In addition, when these P frames refer to more than one previous P frame (optionally plus an I frame), the mixing ratio can be applied to the macroblocks in such P frames.

(A) Determine the position in the flexible coding mode The following method allows the encoder to optimize the frame coding mode and utilize the efficiency of the mixing ratio. For a selected range of frames, many candidate encoding modes can be tried to determine the optimal or near optimal (relative to the specified standard) mode. Figure 7 is a diagram of an encoding mode that can be checked. The initial sequence 700 ending with a P or I frame is arbitrarily selected and used as the basis for additional P and/or B frames to be added, and then the additional P and/or B frames are evaluated (as described below). In one embodiment, a P frame is added to the initial sequence 700 to create the first test sequence 702 for evaluation. If the evaluation is satisfied, an intervening B frame is inserted to create a second test sequence 704. For each satisfied assessment, additional B-frames are inserted to create gradually longer test sequences 706-712 until the assessment criteria become unsatisfied. At that point, the previous coding sequence is accepted. Then repeat this process, using the last P frame of the previously accepted coding sequence as the starting point for adding a new P frame, and then inserting a new B frame.

The optimal or near optimal encoding mode can be selected based on various evaluation criteria, which usually involves the trade-off of various encoding features, such as the image quality of the encoding and the number of encoding bits required. Commonly used evaluation criteria include the minimum number of bits used (in a fixed quantization parameter test), or the best signal-to-noise ratio (in a fixed bit rate test), or a combination of the two.

It is also common to minimize a sum of absolute differences (SAD), which forms a measure of DC matching. As described in the co-pending US Patent Application No. 09/904192 entitled "Motion Estimation for Video Compression System" (which is assigned to the assignee of the present invention and therefore incorporated by reference), the AC matching standard is also specific A useful measure of the quality of candidate matches (this patent application also describes other useful optimizations). Therefore, the AC and DC matching criteria accumulated on the best matching of all macroblocks can be checked to determine the overall matching quality of each candidate coding mode. When this AC/DC matching technique is used with the estimation of the number of coded bits for each frame mode type, it can improve or replace the signal-to-noise ratio (SNR) and use the least bit test. Typically, the quantization parameter value (QP) used for encoding B-frame macroblocks is higher than that used for P-frames, which affects the number and quality of bits used in various candidate encoding modes (usually measured as signal-to-noise ratio) .

(B) Optimization of the mixing ratio in the flexible coding mode. Optionally, for each candidate mode determined according to the above method, the suitability of the mixing ratio with respect to one or more standards (such as optimal or close to optimal) can be tested. The mixing ratio). This can be done, for example, by testing for best quality (lowest SNR) and/or efficiency (least bits used). The use of one or more previous references for each macroblock in the P frame can also be determined in the same way. The reference mode and mixing ratio of each candidate are tested to determine one or more sets of suitable references.

Once the coding mode is selected for the next step (step 700 in FIG. 7), the subsequent steps (steps 702-712) can be tested for various candidate coding modes. In this method, more efficient coding of moving image sequences can be determined. Therefore, efficiency can be optimized/improved as described in subsection (A) above; hybrid optimization can be applied in the coding step of each test.

DC to AC interpolation In many cases of image encoding, for example, when the logarithmic representation of an image frame is used, the above-mentioned interpolation of frame pixel values will optimally encode changes in brightness. However, in alternative video "gamma curves", linear and other representations, applying a different interpolation blending factor to the DC value of the pixel instead of the AC value often proves to be useful. FIG. 8 is a flowchart showing an embodiment of an interpolation method with DC interpolation that is significantly different from AC interpolation. For selected image regions (usually DCT blocks or macroblocks) 802, 802' from the first and second input frames, the average pixel value 804, 804' of each such region is subtracted, so that the selected region Separate the DC value (that is, the average value of the entire selected area) 806, 806' and the AC value (that is, the reserved signed pixel value) 808, 808'. Then the respective DC values 806, 806' can be multiplied by the interpolation weights 810, 810', which are different from the 814, 814' used for multiplying the AC (signed) pixel values 808, 808'. Then, the newly interpolated DC value 812 and the newly interpolated AC value 816 can be merged into 818, resulting in a new prediction 820 for the selected region.

Like other interpolation values in the present invention, the decoder of each image unit can be notified of the appropriate weight. It is possible to choose a small number of bits among many coded values, as well as to choose the independent interpolation of the AC to DC aspects of the pixel value.

Linear & Nonlinear Interpolation Interpolation is a linear weighted average. Since the interpolation operation is linear, and since the pixel value in each image frame is usually expressed in a non-linear form (for example, video gamma representation or logarithmic representation), further optimization of the interpolation process becomes possible. For example, the interpolation of pixels in a specific frame sequence, and the interpolation of DC values separated from AC values, sometimes expressed by linear pixels will be optimal or close to optimal. However, for other frame sequences, if the pixels are expressed as logarithmic values or other pixel representations, then this interpolation will be optimal or close to optimal. In addition, the optimal representation or near optimal representation for interpolating U and V (chrominance) signal components may be different from the optimal representation or near optimal representation of Y (luminance) signal components. Therefore, a useful aspect of the present invention is to convert the pixel representation into an alternative representation as part of the interpolation process.

Fig. 9 is a flow chart showing an embodiment of a method of luminance pixel interpolation using alternate representations. Starting from a region or block of luminance (Y) pixels in the initial representation (e.g., video gamma or logarithmic) (step 900), the pixel data is transformed into an alternative representation that is different from the initial representation (e.g., linear, logarithmic). , Video gamma) (step 902). Then, as described above, the transformed pixel area or block is interpolated (step 906) and transformed back to the original representation (step 906). As a result, the interpolated pixel brightness value is obtained (step 908).

Fig. 10 is a flowchart showing an embodiment of a method of chrominance pixel interpolation using replacement representation. Starting from a region or block of chrominance (U, V) pixels in the initial representation (such as video gamma or logarithm) (step 1000), the pixel data is transformed into an alternative representation (such as linear, Logarithmic, video gamma) (step 1002). Then, as described above, the transformed pixel area or block is interpolated (step 1006), and transformed back to the original representation (step 1006). As a result, the interpolated pixel chromaticity value is obtained (step 1008).

The conversion between the various representations can be made in accordance with the explanation of US Patent Application No. 09/905039 entitled "Method and System for Improving Compressed Image Chroma Information", which is assigned to the assignee of the present invention and is therefore incorporated by reference. Note that a simple lookup table is usually used for the representation transformation and its inverse transformation of the replacement.

As a variation of this aspect of the invention, the replacement (linear or non-linear) representation space used for AC interpolation may be different from the replacement representation space used for DC interpolation.

Like the interpolation weights, which alternative interpolation representation is selected for each luminance (Y) and chrominance (U and V) pixel representation, a small number of bits per selected image unit can be used to inform the decoder.

The number of motion vectors per macro block is in MPEG-2, and each 16×16 macro block in the P frame allows one motion vector. In B-frames, MPEG-2 allows up to two motion vectors per 16×16 macroblock, corresponding to a bidirectional interpolation mode. In MPEG-4 video coding, each 16×16 macroblock in a P frame allows up to 4 motion vectors, corresponding to one motion vector for each 8×8 DCT block. In MPEG-4 B-frames, when the interpolation mode is used, each 16×16 macroblock allows up to two motion vectors. If the subsequent corresponding P frame macroblock is set to an 8×8 mode with 4 motion vectors, a single motion vector increment in MPEG-4 direct mode can result in 4 independent "implicit" motion vectors. This is after converting the time distance (the B frame is closer in time to the previous P frame or I frame reference of the P frame), by adding a motion vector increment carried in the 16×16 B frame macroblock to This is achieved by each of the corresponding 4 independent motion vectors in the P frame macroblock.

One aspect of the invention includes the option of increasing the number of motion vectors per image region (e.g., macroblock). For example, it sometimes proves beneficial to have more than two motion vectors per B-frame macroblock. These can be applied by referring to additional P-frames or I-frames and having 3 or more interpolation terms in the weighted sum. Additional motion vectors can also be applied to allow independent vectors of 8×8 DCT blocks of B-frame macroblocks. Also, 4 independent increments can be used to extend the concept of direct mode by applying independent increments to each of the 4 8×8 area motion vectors of the subsequent P frame.

In addition, the B frame interpolation technique can be used to modify the P frame to refer to more than one previous frame in the interpolation mode, using the two interpolation term technique of the B frame described above. This technique can be easily extended to more than two previous P-frames or I-frames, and the resulting interpolation has 3 or more terms in the weighted sum.

As with other aspects of the present invention (such as pixel representation and DC-to-AC interpolation methods), a small number of bits per image unit can be used to pass a specific weighted sum to the decoder.

When applying this aspect of the present invention, the correspondence between the 8×8 pixel DCT block and the motion vector domain does not need to be as strict as in MPEG-2 and MPEG-4. For example, it may be useful to use replacement area sizes other than 16×16, 16×8 (used only in interlaced scanning in MPEG-4), and 8×8 for motion vectors. This replacement may include any number of useful area sizes, such as 4×8, 8×12, 8×16, 6×12, 2×8, 4×8, 24×8, 32×32, 24×24, 24 ×16, 8×24, 32×8, 32×4, etc. Using a small amount of this useful size, a few bits can inform the decoder of the correspondence between the motion vector area size and the DCT block size. In a system using traditional 8×8 DCT blocks, a set of simple correspondences to the motion vector domain is useful to simplify the processing in motion compensation. In a system where the DCT block size is not 8×8, then greater flexibility can be achieved in the specified motion vector domain, as described in the title "Enhanced Temporal and Resolution Layering in Advanced Television" co-pending US Patent Application No. 09/545233, which is assigned to the assignee of the present invention, and therefore incorporated by reference. Note that the motion vector area boundary does not need to correspond to the DCT area Boundary. In fact, it is often useful to define the motion vector area in such a way that the boundary of the motion vector area falls within the DCT block (and not on its boundary).

The concept of extending the flexibility of the motion vector field is also applicable to the interpolation aspect of the present invention. As long as the correspondence between each pixel and one or more motion vectors pointing to one or more reference frames is specified, all the versatility of the present invention can be used to apply the above interpolation method to the full flexibility of useful motion vectors. Even when using P frames, for each previous frame reference, the size of the area corresponding to each motion vector can be different, and when using B frames, for each previous and future frame reference, corresponding to each motion vector The area size can be different. If the area size of the motion vector is different when applying the improved interpolation method of the present invention, the interpolation reflects the overlapping common area. The overlapping common area of the motion vector reference can be used as the area on which the DC term is determined when the DC and AC pixel values are respectively interpolated.

Fig. 11 shows the respective unique motion vector area sizes 1100 and 1102 of the two P frames 1104 and 1106. When calculating the interpolation according to this invention, the joint 1108 of the size of the motion vector area is determined. Union 1108 defines all areas that are considered to have allocated motion vectors.

Therefore, for example, when the 4×4 DCT area of the B frame 1112 is interpolated back to the previous P frame 1104, the 4×4 area 1110 in the union 1108 will use the motion vector corresponding to the 8×16 area 1114 in the previous P frame . If forward prediction, the 1110 area in the union 1108 will use the motion vector corresponding to the 4×16 area 1115 in the next P frame. Similarly, the backward interpolation of the region 116 in the union 1108 will use the motion vector corresponding to the 8×16 region 1114, and the same region in the forward prediction will use the motion vector corresponding to the 12×16 region 1117.

In one embodiment of the present invention, two steps are used to realize the interpolation of universal (ie, non-uniform size) motion vectors. The first step is to determine the common area of the motion vectors, as described in relation to FIG. 11. This establishes the correspondence between pixels and motion vectors for each previous frame or subsequent frame reference (that is, the number of motion vectors for each specified pixel area size). The second step is to use appropriate interpolation methods and effective interpolation factors for each area of the pixel. The task of the encoder is to ensure that the optimal or near optimal motion vector area and interpolation method are specified, and all pixels have their vectors and interpolation methods fully specified. For the fixed mode of the motion vector (for example, specify a motion vector for each 32×8 block for the entire frame), there is a single specified interpolation method (for example, the fixed ratio mixing of each distance from the reference frame to the reference frame specified for the entire frame) The situation is very simple. If the size of the motion vector area is changed regionally, and the difference in area size depends on which previous frame or subsequent frame is referred to (for example, the nearest previous frame is 8×8 blocks, and the next nearest previous frame is 32 ×8 blocks), this method can also become quite complicated. In addition, the interpolation method can be specified regionally within the frame.

When encoding, the job of the encoder is to determine the optimal or near-optimal usage of bits to choose between the shape and size of the motion vector area, and to select the optimal or near-optimal interpolation method. A decision is needed to specify the number and distance of referenced frames. These assignments are determined by exhaustively testing the size of a large number of candidate motion vector regions, the candidate frame to be referred to, and the interpolation method of each such motion vector region, until the optimal or near optimal encoding is found. The optimum (relative to the selected standard) can be determined by finding the smallest SNR after the coding block or finding the minimum number of bits for a fixed quantization parameter (QP) after the coding block, or by applying other suitable measurements.

The direct mode extension is the traditional direct mode used in B-frame macroblocks in MPEG-4, which is efficient in motion vector coding, and provides the benefits of 8×8 block mode by using simple common increments. The direct mode weights each corresponding motion vector of the subsequent P frame at the corresponding macroblock position based on the time distance, and the P frame refers to the previous P frame. For example, if M=3 (ie two intervening B frames), using simple linear interpolation, the first B frame uses -2/3 times the motion vector of the subsequent P frames to determine the pixel offset relative to this P frame , And 1/3 times the motion vector of the subsequent P frame to determine the pixel offset relative to the previous P frame. Similarly, the second B frame uses -1/3 times the same P frame motion vector to determine the pixel offset relative to this P frame, and 2/3 times the subsequent P frame motion vector to determine relative to the previous P frame. The pixel offset of the frame. In direct mode, small increments are added to each corresponding motion vector . As another aspect of this invention, this concept can be extended to point to one or more B frame references that are n far away from P frames, these P frames in turn refer to one or more previous or subsequent P frames or I frames, by considering the frame distance Determine the frame scale fraction.

Figure 12 is a diagram showing a sequence of P frames and B frames, where the interpolation weight of the B frame is determined as a function of the distance to 2 away from the subsequent P frame, which refers to 1 away from the subsequent P frame. In this illustrative example, M=3, which means that there are two consecutive B frames 1200 and 1202 between the bracketed P frames 1204 and 1206. In this example, each coexisting macroblock in the next nearest subsequent P frame 1208 can point to the intervening (ie nearest) P frame 1204, and the first two B frames 1200, 1202 can refer to the next nearest The subsequent P frame 1208 instead of the nearest subsequent P frame 1204, as in traditional MPEG. Therefore, for the first B frame 1200, the frame ratio score 5/3 times the motion vector mv from the next nearest subsequent P frame 1208 will be used as the pixel offset relative to the P frame 1208, and the second B frame Frame 1202 will use 4/3 times the offset of that same motion vector.

If the nearest subsequent P frame referenced by the B frame points to the next nearest preceding P frame, then the simple frame distance is used again to obtain the appropriate frame scale score applied to the motion vector. Figure 13 is a diagram showing a sequence of P frames and B frames, where the interpolation weight of the B frame is determined as a function of the distance to 1 away from the subsequent P frame, which refers to 2 away from the previous P frame. In this illustrative example, M=3, B frames 1300 and 1302 refer to the nearest subsequent P frame 1304, which in turn refers to 2 far away P frames 1306. Thus, for the first B frame 1300, the pixel offset score is the frame ratio score 2/6 times the motion vector mv from the most recent subsequent P frame 1304, and the second B frame 1302 will have the same 1/6 times that The pixel offset of the motion vector is because the motion vector of the nearest subsequent P frame 1304 points 2 away from the previous P frame 1306, which is 6 frames away.

Generally, in the case of a B frame referring to a single P frame in the direct mode, the frame distance method sets the numerator of the frame ratio score equal to the frame distance from the B frame to its reference or "target" P frame, and sets the denominator equal to from The distance from the target P frame to another P frame referenced by the target P frame. For the measurement from the B frame to the subsequent P frame, the sign of the frame ratio score is a negative sign, and for the measurement from the B frame to the previous P frame, the sign of the frame ratio score is a positive sign. This simple method of applying the frame distance or frame ratio score to the P frame motion vector can achieve effective direct mode coding.

In addition, another aspect of this invention is to allow direct mode to be applied to multiple interpolated motion vector references of a P frame. For example, if one P frame is interpolated from the most recent and the next most recent previous P frame, then the direct mode reference according to this aspect of the invention allows interpolated blending to be used for each multi-reference direct mode B-frame macroblock. Generally, two or more motion vectors of a P frame can be applied with an appropriate frame ratio score. Then, the two or more frame distance corrected motion vectors can be used with the corresponding interpolation weights of each B frame referring to or pointing to that P frame (as described below) to generate interpolated B frame macroblock motion compensation.

Fig. 14 is a diagram showing a sequence of P frames and B frames, where a subsequent P frame has multiple motion vectors referring to the previous P frame. In this example, the B frame 1400 refers to the subsequent P frame P3. This P3 frame has two motion vectors mv1 and mv2, which refer to the corresponding previous P frames P2 and P1. In this example, any one of the two weighting terms or a combination of such weighting terms can be used to interpolate each macroblock of the B frame 1400 in the direct mode.

Each macroblock of the B frame 1400 can be constructed into a mixed form according to the following items: Corresponding pixels of frame P2 replaced by the frame ratio score of mv1 1/3 (these pixels may then be multiplied by a certain ratio weight i) plus Corresponding pixels of frame P3 replaced by the frame ratio score of mv1-2/3 (these pixels may then be multiplied by a certain ratio weight j); and replaced by the frame ratio score of mv2 2/3 (4/6) The corresponding pixels of frame P1 (these pixels may then be multiplied by a certain ratio weight k) plus the corresponding pixels of frame P3 replaced by the frame ratio score of mv2 -1/3 (-2/6) (these pixels may then be multiplied by With a certain proportional weight l).

Like all direct modes, the motion vector increment can be used with each of mv1 and mv2.

According to this aspect of the present invention, as in the case of multiple previous P frames, using the same interpolation method and motion vector frame ratio score application, the direct mode prediction macroblock in the B frame can also refer to multiple subsequent P frames. FIG. 15 is a diagram showing a sequence of P frames and B frames, where the nearest subsequent P frame has a motion vector referring to a previous P frame, and the next nearest subsequent P frame has multiple motion vectors referring to multiple previous P frames. In this example, the B frame 1500 references two subsequent P frames P2 and P3. Frame P3 has two motion vectors mv1 and mv2, which refer to the corresponding previous P frames P2 and P1. Frame P2 has a motion vector mv3, which refers to the previous P frame P1. In this example, 3 weighting terms are used to interpolate each macroblock of the B frame 1500 in the direct mode. If this is the case, the motion vector frame ratio score can be greater than 1 or less than -1.

This form of direct mode B-frame macroblock interpolation weighting can take advantage of the full versatility of the interpolation described here. In particular, in order to obtain the best performance of various image units (for example, quality versus number of bits), each weight or combination of weights can be tested. The interpolation sub-array for this modified direct mode can be assigned to the decoder with a small number of bits per image unit.

Each macroblock of the B frame 1500 can be constructed into a mixed form according to the following items: The corresponding pixels of frame P3 replaced by the frame ratio score of mv1 -5/3 (these pixels may then be multiplied by a certain ratio weight i) plus Corresponding pixels of frame P2 replaced by the frame ratio score of mv1-2/3 (these pixels may then be multiplied by a certain ratio weight j); Corresponding to frame P3 replaced by the frame ratio score of mv2-5/6 Pixels (these pixels may then be multiplied by a certain ratio weight k) plus the corresponding pixels of frame P1 replaced by the frame ratio score of mv2 of 1/6 (these pixels may then be multiplied by a certain ratio weight l); and The corresponding pixels of frame P2 replaced by the frame ratio score of mv3-2/3 (these pixels may then be multiplied by a certain ratio weight m) plus the corresponding pixels of frame P1 replaced by the frame ratio score of mv3 1/3 ( These pixels may then be multiplied by a certain proportional weight n).

Like all direct modes, the motion vector increment can be used with each of mv1, mv2, and mv3.

Note that when the next nearest subsequent P frame refers to the nearest P frame surrounding a candidate B frame, a particularly beneficial direct encoding mode often appears.

The direct mode coding of B frames in MPEG-4 always uses the motion vector of subsequent P frames as a reference. According to another aspect of the present invention, it is also possible for B frames to refer to the motion vectors of the coexisting macroblocks of the previous P frames, and these motion vectors will sometimes prove a beneficial option for direct mode coding reference. If this is the case, when the next nearest preceding P frame is referenced by the motion vector of the nearest preceding P frame, the motion vector frame ratio score will be greater than 1. Figure 16 shows a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame. In this example, the B frame 1600 refers to the previous P frame P2 that is 1 away. The motion vector mv of the frame P2 refers to the next previous P frame P1 (relative to the B frame 1600, 2 is far away). The figure shows the appropriate frame ratio score.

If the nearest previous P frame is interpolated based on multiple vectors and frames, then methods similar to those described in conjunction with FIG. 14 are applied to obtain the motion vector frame ratio score and the interpolation weight. Fig. 17 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has two motion vectors that refer to the previous P frame. In this example, the B frame 1700 refers to the previous P frame P3. One motion vector mv1 of the previous frame P3 refers to the next previous P frame P2, and the second motion vector mv2 refers to the previous P frame P1 far away from 2. The figure shows the appropriate frame ratio score.

Each macroblock of the B frame 1700 can be constructed into a mixed form according to the following items: Corresponding pixels of frame P3 replaced by the frame ratio score of mv1 1/3 (these pixels may then be multiplied by a certain ratio weight i) plus Corresponding pixels of frame P2 with a frame ratio score of 4/3 by mv1 (these pixels may then be multiplied by a certain ratio weight j); and

The corresponding pixels of frame P3 replaced by the frame ratio score of mv2 1/6 (these pixels may then be multiplied by a certain ratio weight k) plus the corresponding pixels of frame P1 replaced by the frame ratio score of mv2 7/6 ( These pixels may then be multiplied by a certain proportional weight l).

When the motion vector of the previous P frame (relative to the B frame) points to the next nearest previous P frame, it is not necessary to use only the next nearest previous frame as an interpolation reference, as shown in FIG. 16. The most recent previous P frame may prove a better choice for motion compensation. If this is the case, the motion vector of the nearest previous P frame is shortened to the frame distance fraction from the B frame to that P frame. Fig. 18 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame. In this example, for M=3, the first B frame 1800 will use 1/3 and -2/3 frame distance scores multiplied by the motion vector mv of the nearest previous P frame P2. The second B frame 1802 will use 2/3 and -1/3 frame distance scores (not shown). The decoder is notified of this choice to distinguish this example from the example shown in FIG. 16.

As with all other coding modes, the use of direct mode preferably includes testing candidate modes against other available interpolation and single vector coding modes and reference frames. For direct mode testing, the nearest subsequent P frame (optionally, and the next nearest subsequent P frame or even further subsequent P frame, and/or one or more previous P frames) can be tested as candidate frames, And a small number of bits (typically one or two) can be used to specify the direct mode P reference frame distance used by the decoder.

The extended interpolation value is specified in the MPEG-1, 2, 4 and H.261 and H.263 standards. The B frame uses the equal weighting of the pixel values of the forward reference frame and the backward reference frame, as replaced by a motion vector of. Another aspect of the present invention includes various useful applications of unequal weights that significantly increase the efficiency of B-frame coding, as well as the extension of such unequal weights to more than two references, which include two or more references. A forward or backward reference in time. This aspect of the invention also includes a method for more than one frame that is referenced and interpolated for a P frame. In addition, when two or more references point forward in time, or when two or more references point backward in time, then it is sometimes useful to use negative weights and weights over 1.0.

For example, FIG. 19 is a frame sequence showing three P frames P1, P2, and P3, where P3 uses two motion vectors using interpolation references, and P1 and P2 each use one. For example, if a continuous change is occurring in the frame range between P1 and P3, then P2-P1 (that is, the pixel value of frame P2 (replaced by the motion vector of P2) minus the pixel value of frame P1 (replaced by the motion vector of P1) Replaced)) will be equal to P3-P2. Similarly, P3-P1 will be twice the size of P2-P1 and P3-P2. In this case, the pixel value of frame P3 can be predicted differently from P2 and P1 by the formula: P3=P1+2×(P2-P1)=(2×P2)-P1. In this case, the interpolation weight of P3 is relative to P2 It is 2.0 and -1.0 for P1.

As another example, FIG. 20 shows a frame sequence of four P frames P1, P2, P3, and P4, where P4 uses an interpolation reference with three motion vectors, and P1, P2, and P3 each have one. Therefore, since P4 is predicted from P3, P2, and P1, three motion vectors and interpolation weights are applied. In this case, if continuous changes are occurring in this frame range, then P2-P1 will be equal to P3-P2 and P4-P3, and P4-P1 will be equal to 3×(P2-P1) and 3×(P3- P2).

Therefore, in this example, the P4 prediction based on P2 and P1 is: P4=P1+3×(P2-P1)=(3×P2)-(2×P1) (weights of 3.0 and -2.0) based on P3 and The P4 prediction of P1 is: P4=P1+3/2×(P3-P1)=(3/2×P3)-(1/2×P1) (weight 1.5 and -0.5) P4 prediction based on P3 and P2 It is: P4=P2+2×(P3-P2)=(2×P3)-P2 (weights of 2.0 and -1.0). However, it is possible that the change closest to P4 (including P3 and P2) is greater than the change that includes P1 The prediction is the more reliable prediction value of P4. Therefore, by weighting each of the above two items including P1 by 1/4 and weighting the items only including P3 and P2 by 1/2, it will result in: 1/2(2 P3-P2)+1/4(3/ 2 P3-1/2 P1)+1/4(3 P2-2 P1)=1 3/8 P3+1/4 P2-5/8 P1 (The weights are 1.375, 0.25, and -0.625) Accordingly, it is sometimes useful to use both a weight greater than 1.0 and a weight less than 0. At other times, if there is a noise-like change from one frame to the next, a positive weighted average with a moderate coefficient between 0.0 and 1.0 can produce the best prediction of the P4 macroblock (or other pixel area). For example, the equal weighting of 1/3 of each of P1, P2, and P3 in Fig. 20 may form the best prediction of P4 in some cases.

Note that the best matching motion vector is used to determine the areas of P1, P2, P3, etc. In this example, P1, P2, P3 are used in the calculation. In some cases, this match may best be an AC match, which allows changing DC terms to be predicted by AC coefficients. Alternatively, if DC matching (such as the sum of absolute differences) is used, then the change in AC coefficients can often be predicted. In other cases, various forms of motion vector matching will form the best prediction with a mixture of various weights. Usually, the best prediction for a particular situation is determined empirically using the methods described here.

These techniques are also applicable to B-frames with two or more motion vectors pointing backward or forward in time. When pointing forward in time, the above-described coefficient mode for P frames is reversed to accurately predict the current P frame backwards. Using this aspect of the invention, it is possible to have two or more motion vectors in both forward and backward directions, and therefore predict in both directions at the same time. The appropriate weighted mixture of these various predictions can be optimized by selecting the mixed weight, which best predicts the macroblock (or other pixel area) of the current B frame.

Figure 21 is a diagram showing a sequence of P frames and B frames, where different P frames have one or more motion vectors that refer to different previous P frames. The figure also shows the respective forward and backward references assigned to the specific B frame. Different weights ae for reference. In this example, the B frame 2100 refers to three previous P frames and two subsequent P frames.

In the example shown in Figure 21, frame P5 must be decoded for this example to work. Sometimes it is useful to sort the frames in the bitstream in the order required for decoding ("transmission order"), and the transmission order does not have to be the order of display ("display order"). For example, in a frame sequence showing cyclic motion (such as the rotation of an object), a particular P frame may be more similar to a distant P frame than the nearest subsequent P frame. Fig. 22 is a diagram showing a sequence of P frames and B frames, where the bit stream transmission order of P frames is different from the display order. In this example, frame P3 is more similar to frame P5 than frame P4. So it is useful to send and decode P5 before P4, but display P4 before P5. Preferably, when such P frames can be discarded (for example, at the end time of n frames in the bitstream sequence or after frame X in the display sequence), each P frame should notify the decoder.

If the weight is selected from a small selection set, then a small number of bits can inform the decoder which weight to use. As with all other weights described here, this can be notified to the decoder once per picture unit, or at any other point in the decoding process where the weight change is useful.

It is also possible to download a new set of weights. In this way, a small set of weights can be effective at a given time. This allows a small number of bits to inform the decoder which valid set of weights will be used at any given point in the decoding process. In order to determine the appropriate weight set, a large number of weights can be tested during the encoding process. If a small subset is found to provide high efficiency, the decoder can be notified of which subset to use. Therefore, the decoder can be notified of the specific elements of the subset with only a few bits. For example, 10 bits can select one of 1024 subset elements. In addition, when a specific small subset is changed to maintain efficiency, the decoder can be notified of the new subset. Therefore, the encoder can compare the number of bits required to update the weight set, and dynamically optimize the number of bits required to select among the weight set elements. In addition, a small number of short codes can be used to represent common useful weights, such as 1/2, 1/3, 1/4, and so on. In this way, a small number of bits can be used to represent the weight set, such as K-forward-vector prediction in P-frames (where K=1, 2, 3...), or K- in B-frames. Forward-vector and L-backward-vector prediction (where K and L are selected from 0, 1, 2, 3...), or K-forward-vector and L-backward-vector prediction in P frames ( Among them, K and L are selected from 0, 1, 2, 3...), as a function of the current M value (that is, the relative position of the B frame relative to the adjacent P (or I) frame).

Fig. 23 is a diagram showing a sequence of P frames and B frames with assigned weights. The B frame 2300 has a weight ae, the value of which is assigned from the table of the B frame weight set 2302. The P frame 2304 has weights m and n, the values of which are assigned from the table of the P frame weight set 2306. Some weights can be static (that is, downloaded to the decoder unchanged) and notified by the encoder. Other weights can be downloaded dynamically and then notified.

This same technique can be used to dynamically update the weight set to select DC interpolation compared to AC interpolation. In addition, the coded value can be expressed, which is compared to the linear interpolation of the transformed value (in the alternative linear or non-linear representation) to select general (linear) interpolation (usually the interpolation of the pixel value represented in the non-linear representation). Similarly, this coded value can indicate which such interpolation is applied to the AC value or the DC value or whether to separate the AC and DC parts of the prediction.

The effective subset can also be used to minimize the number of bits necessary to choose between the currently used sets of weight coefficients. For example, if 1024 downloaded weight sets are saved in the decoder, it may take approximately 16 to be valid during a particular part of the frame. Therefore, by selecting (selected from 1024) which subset of the 16 weight sets are valid, only 4 bits are needed to select which of the 16 weight sets is valid. It is possible to use short codes for these most common subsets to represent these subsets, thus allowing a small number of bits to be selected in the commonly used subsets.

Softening and sharpening are the same as simply separating the DC component from the AC signal by subtracting the average value. In the motion vector compensation prediction process, other filtering operations are also possible. For example, various high-pass, band-pass, and low-pass filters can be applied to pixel regions (e.g., macroblocks) to extract various frequency bands. Then, when performing motion compensation, these frequency bands can be modified. For example, it may often be useful to filter out the highest frequencies on a noisy moving image to soften (make it less clear or slightly blurred) the image. The softer image pixels, combined with a steeper tilt matrix for quantization (the steeper tilt matrix ignores higher frequency noise in the current block), usually form a more efficient encoding method. It is already possible to express the change in the quantization slope matrix of each image unit. It is also possible to download custom oblique matrices of luminance and chrominance. Note that regardless of whether the oblique matrix is changed, the efficiency of motion compensation can be improved. However, it is usually most effective to change both the oblique matrix and the filter parameters applied in the motion compensation process.

A common practice is to use reduced chroma coding resolution and chroma-specific oblique matrices. However, in this example, the chroma coding resolution is static (for example, 4:2:0 encodes half resolution horizontally and vertically or 4:2:2 just encodes half resolution horizontally). According to this aspect of the present invention, coding efficiency can be improved by applying a dynamic filter process to chrominance and luminance (independently or cooperatively) in the motion compensation process, one for each image unit.

US Patent Application No. 09/545233 (cited above) entitled "Enhanced Temporal and Resolution Layering in Advanced Television" describes the use of an improved negative half cycle (truncated sine function) displacement filter. These filters have the advantage that they maintain sharpness when performing the fractional-pixel portion of the motion vector displacement. At the integer pixel displacement point and the fragment pixel displacement point, the use of filters that reduce or increase the definition can better move certain macroblocks (or other useful image areas). For example, for "zoom" (where certain objects in the frame move out of focus over time and other parts of the frame come into focus), transition is one of the changes in sharpness and flexibility. Therefore, a motion compensation filter that can increase the sharpness in some areas of the image and reduce the sharpness in other areas can improve coding efficiency. In particular, if a region of the image is moving out of focus, it may be beneficial to reduce the sharpness, which will soften the image (and thus may produce a better match) and reduce grain and/or noise (and thus may increase coding efficiency). If an area of the image is coming into focus, it may be beneficial to maintain the best sharpness or even use a larger negative half-cycle sine filter value to improve sharpness.

In the encoding process, chroma filtering can also benefit from improving and reducing sharpness. For example, many of the coding efficiency advantages of 4:2:0 encoding (encoding half-resolution chroma horizontally and vertically) can maintain full resolution in the U and/or V channels by using a softer motion compensation filter for chroma Rate to achieve. Only when the color details in the U and V channels are high, it is necessary to choose the clearest displacement filter; a softer filter will be more beneficial where there is high color noise or particles.

In addition to changes in focus, the direction and amount of motion blur changes from one frame to the next are also common. When the frame rate of a moving image film is 24fps, even a simple dialogue scene has a significant change in motion blur from one frame to the next. For example, the upper lip may be blurred in one frame and sharpened in the next frame, which is entirely due to the lip movement during the opening of the shutter in the camera. For such motion blur, it would be beneficial not only to have sharpening and softening (blurring) filters in the motion compensation process, but also to have the direction of sharpening and softening. For example, if the direction of movement can be determined, then softening or sharpening in that direction can be used to move or stop the corresponding image close-up. The motion vector used for motion compensation itself can provide some useful information about the amount of motion and the change of the amount of motion (ie, motion blur) for a specific frame (or area within a frame) relative to any surrounding frame (or corresponding area). In particular, the motion vector is the best motion match between the P frames, while the motion blur comes from the motion in the frames during the opening shutter time.

Figure 24 is a graph of object position versus time within a frame. The camera's shutter is only open during part of the frame time. When the shutter is open, any movement of the object causes blur. The amount of motion blur is represented by the amount of position change during the opening of the shutter. Therefore, the slope of the position curve 2400 when the shutter is open is a measure of motion blur.

The amount of motion blur and the direction of motion can also be determined from a combination of sharpness measurement, surrounding motion vectors (where the image region matches), feature smear detection, and artificially assisted designation of frame regions. The filter can be selected based on the determined amount of motion blur and the direction of motion. For example, comparing the determined motion blur amount and motion direction, the mapping of various filters can be determined empirically.

When combined with other aspects of the present invention, this intelligently applied filter can significantly improve the compression coding efficiency. You can select a small number of such filters and use a small number of bits to inform the decoder. Again, this can be done once per image unit or at other useful points in the decoding process. As with the weight set, a set of dynamically loaded filters can be used, as well as an effective subset mechanism to minimize the number of bits required to select between the most beneficial filter parameter sets.

The present invention can be implemented in hardware or software, or a combination of the two (for example, a programmable logic array). Unless otherwise specified, the algorithms included as part of the present invention are not essentially related to any particular computer or other device. In particular, various general-purpose devices can be used with programs written according to the description herein, or it may be more convenient to construct more special devices (such as integrated circuits) to perform specific functions. Therefore, the present invention can be implemented in one or more computer programs executed on one or more programmable computer systems. Each programmable computer system includes at least one processor, at least one data storage system (including permanent and volatile). Memory and/or storage unit), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described here and generate output information. The output information is applied to one or more output devices in a well-known manner.

Each such program can be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, object-oriented programming language) to communicate with the computer system. In any case, the language can be a compiled language or an interpreted language.

Each such computer program is preferably stored in or downloaded to a storage medium or device (such as a solid-state memory or medium, or a magnetic medium or an optical medium), and these storage media or devices are readable by a general-purpose or special-purpose programmable computer, When the storage medium or device is read by a computer system to execute the program described herein, such a computer program is used to configure and operate the computer. It is also conceivable to implement the system of the present invention as a computer-readable storage medium, which is configured to have a computer program, and the storage medium so configured enables the computer system to operate in a specific and predetermined manner to perform the functions described herein.

A number of embodiments of the present invention have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present invention. For example, some of the steps described above may be order-independent, and therefore may be performed in a different order from the above. Therefore, other embodiments are within the scope of the appended claims.