Video image compressing method

Video image compression method

[0001] Cross reference to related applications

[0002] This application is a partial continuation of the US application serial number 09/904203 filed on July 11, 2001, and claims the priority of the US CI P. application serial number 10/187395 filed on June 28, 2002.

[0003] Technical Field

[0004] The present invention relates to video compression, and more particularly to improved video compression frame interpolation in MPEG-type encoding and decoding systems.

[0005] Background Art

[0006] MPEG video compression

[0007] MPEG-2 and MPEG-4 are international video compression standards. Their respective video grammars define an efficient method to represent image sequences with more compact coded data. The language of the coded bits is the so-called "grammar". For example, several markers can represent the entire sample block (for example, 64 samples for MPEG-2). Both MPEG standards describe the decoding (reconstruction) process, which transforms the encoded bits from a compact representation into an approximation of the original format of the image sequence. For example, a flag in the coded bitstream can indicate whether to use a predictive algorithm to process these bits before using a discrete cosine transform (DCT) algorithm to decode subsequent bits. The algorithm including the decoding process is specified by the semantics defined by these MPEG standards. This grammar can be used to develop common video features, such as spatial redundancy, temporal redundancy, uniform motion, spatial masking, and so on. The MPEG decoder must be able to parse and decode the input data stream, but as long as the data stream conforms to the corresponding MPEG syntax, a variety of possible data structures and compression techniques can be used (although technically this deviates from the standard, because of the semantics Inconsistent). It is also possible to carry the required semantics within a certain substitution grammar.

[0008] These MPEG standards use various compression methods, including intra-frame methods and inter-frame methods. In most video scenes, the background remains relatively stable and the action occurs in the foreground. The background may move, but usually many scenes are redundant. These MPEG standards start compression by creating reference frames called "intra" frames or "I frames." The I frame is compressed without referring to other frames, so the I frame contains a complete frame of video information. The I frame provides an entry point for the data bit stream for random access, but it can only be moderately compressed. Typically, data representing I frames is placed every 12 to 15 frames in the bitstream (although in some cases it is useful to use wider intervals between I frames). According to this, since only a small part of the frames that fall between the reference I frames is different from the bracketing I frames, only the image difference is captured, compressed, and stored. Two types of frames are used for this distinction-predicted frames (P frames) and bidirectionally predicted (or interpolated) frames (B frames).

[0009] P-frames are generally coded according to past frames (I-frames or previous P-frames), and are usually used as references for subsequent P-frames. P-frames are subject to considerable compression. B-frames provide the highest amount of compression, but in order to be encoded, both past reference frames and future reference frames are required. P-frames and I-frames are "referenceable frames" because they can be referenced by P-frames or B-frames.

[0010] A macroblock is an image pixel area. For MPEG-2, the macroblock is a 16×16 pixel group of 4 8×8DCT blocks, one motion vector is added for P frame, and one or two motion vectors are added for B frame. The macroblocks within the P frame can be individually coded using intra-frame or inter-frame (predicted) coding. It is possible to use intra-frame coding, forward predictive coding, backward predictive coding, or simultaneous use of forward and backward (ie, two-way interpolation) predictive coding to separately code the macroblocks in the B frame. The structure used in MPEG-4 video coding is slightly different, but similar.

[0011] After encoding, the MPEG data bit stream includes a sequence of I frames, P frames, and B frames. The sequence can be composed of I frame, P

CN 1663258 Β

Frames and B-frames are composed of almost any mode (there are several minor semantic restrictions regarding their positions). However, it is common in industry practice to have a fixed frame mode (for example, IΒΒΡΒΒΡΒΒΡΒΒΡΒΒ)_ο

[0012] Motion vector prediction

[0013] In MPEG-2 and MPEG-4 (and similar standards such as H.263), the use of B-type (bi-predictive) frames has proven to be beneficial to compression efficiency. The motion vector of each macroblock of this frame can be predicted by any one of the following three methods:

[0014] Mode 1: Forward prediction based on the previous I frame or P frame (ie, non-bidirectionally predicted frames).

[0015] Mode 2: Backward prediction based on subsequent I frames or P frames.

[0016] Mode 3: Bidirectional prediction based on subsequent and previous I frames or P frames.

[0017] Mode 1 is the same as the forward prediction method used for P frames. Mode 2 is the same concept except that it works backwards according to subsequent frames. Mode 3 is an interpolation mode, which combines information from previous and subsequent frames.

[0018] In addition to these three modes, MPEG-4 also supports another interpolation motion vector prediction mode for B frames: direct mode prediction, which uses motion vectors from subsequent P frames plus an increment (6) Value (if the motion vectors from co-located P macroblocks are divided into 8×8 patterns to produce 4 motion vectors for 16×16 macroblocks then this increment is applied to all 4 independent motion vectors in the B frame). The motion vector of the subsequent P frame points to the previous P frame or I frame. A ratio is used to weight the motion vectors of subsequent P frames. The ratio is the relative time position of the current B frame relative to the subsequent P frame and the previous P (or D frame).

[0019] FIG. 1 is a timeline of frames and MPEG-4 direct mode motion vectors according to the prior art. The concept of MPEG-4 direct mode (mode 4) means that the motion of the macroblock in each intervening B frame may be close to the motion used to encode the same position in the subsequent P frame. The increment is used to make small corrections to the proportional motion vector derived from the corresponding motion vector (MV) 103 of the subsequent P frame. Figure 1 shows the proportional weighting given to the motion vectors 101 and 102 of each intermediate B frame 104a, 104b as a function of the "time distance" between the previous P frame or I frame 105 and the next P frame 106. The motion vectors 101 and 102 assigned to the corresponding intermediate B frames 104a and 104b are equal to the assigned weight values (1/3 and 2/3, respectively) multiplied by the motion vector 103 of the next P frame, plus the increment value.

[0020] Regarding MPEG-2, all prediction modes of B-frames are tested in encoding and compared to find the best prediction for each macroblock. If there is no good prediction, then the macroblocks are individually coded into "1" ("intra") macroblocks. The coding mode is selected as the best mode among forward (mode 1), backward (mode 2) and bidirectional (mode 3) coding, or is selected as internal coding. Regarding MPEG-4, internal coding is not allowed. On the contrary, the direct mode becomes the fourth option. Similarly, the optimal coding mode is selected based on a certain best matching criterion. In software encoders based on MPEG-2 and MPEG-4, DC matching (absolute difference or "SAD") is used to determine the best match.

[0021] The number of consecutive B frames in the coded data bit stream is determined by the parameter value "M" in MPEG. M minus 1 is the number of B frames between each P frame and the next P frame (or I frame). Therefore, if M = 3, there are 2 B frames between each P frame (or I frame), as shown in Figure 1. In terms of the limitation of the M value (and therefore the number of consecutive B frames), the main limitation is that the amount of motion change between P frames (or I frames) becomes larger. More B frames means longer time between P frames (or I frames). Therefore, the efficiency of the motion vector and the limitation of the coding range constitute the limit of the number of intermediate B frames.

[0022] It is also important to note that P frames carry "variation energy" and move forward with the moving image stream, because each decoded P frame is used as the starting point for predicting the next subsequent P frame. However, B frames are discarded after use. Therefore, any bits used to create a B-frame are only used for that frame, and unlike P-frames, they do not provide corrections to help decode subsequent frames.

CN 1663258 Β

positive.

Summary of the invention

[0023] The present invention relates to a method, system and computer program for improving the image quality of one or more predicted frames in a video image compression system, wherein each frame includes a plurality of pixels.

[0024] In one aspect, the present invention includes determining the value of each pixel of the bidirectional prediction frame as a weighted ratio of the corresponding pixel value in a non-bidirectional prediction frame surrounded by a bidirectional prediction frame sequence. In one embodiment, the weighting ratio is a function of the distance between the bracketed non-bidirectionally predicted frames. In another embodiment, the weighting ratio is a mixed function of the distance between the bracketed non-bidirectionally predicted frames and the equal average value of the bracketed non-bidirectionally predicted frames.

[0025] In another aspect of the present invention, the interpolated representation of the pixel value is expressed in a linear space, or in another optimized non-linear space different from the original non-linear representation.

[0026] Other aspects of the present invention include systems, computer programs, and methods, which include:

[0027] A video image compression system having a sequence of referenceable frames including image regions, wherein

[0028] At least one image area of at least one prediction frame is coded with reference to two or more referenceable frames.

[0029] A video image compression system having a sequence of referenceable frames including image regions, wherein

[0030] At least one image area of at least one predicted frame is coded with reference to one or more referenceable frames in display order, wherein at least one such referenceable frame is not the referenceable closest to the front of the at least one predicted frame in the display order frame.

[0031] A video image compression system having a sequence of referenceable frames including macroblocks, where at least one macroblock within at least one prediction frame is coded by interpolation from two or more referenceable frames.

[0032] A video image compression system having reference frames including image regions and a sequence of bidirectional predictive frames, wherein at least one image region of at least one bidirectional predictive frame is coded to include at least two or more motion vectors, each of which The motion vector refers to a corresponding image area in at least one referenceable frame.

[0033] A video image compression system having a sequence of referenceable frames including image regions, wherein at least one image region of at least one prediction frame is coded to include at least two motion vectors, and each such motion vector is referenced in the referenceable frame Corresponding image regions in, wherein each such image region of such at least one predicted frame is encoded by interpolation from two or more referenceable frames.

[0034] A video image compression system having a sequence of reference frames and bidirectional predictive frames including image regions, wherein at least one image region of at least one bidirectional predictive frame is encoded as a selected image from two or more referenceable frames The unequal weighting of regions.

[0035] A video image compression system having a sequence of reference frames and bidirectional predictive frames including image regions, wherein at least one image region of at least one bidirectional predictive frame is encoded by interpolation from two or more referenceable frames, Wherein at least one of the two or more referenceable frames is separated by at least one intervening referenceable frame and the bidirectional prediction frame in the display order, and where such at least one image area is encoded as such at least two or more The unequal weighting of the selected image regions of the two referenceable frames.

[0036] A video image compression system having a sequence of reference frames and bidirectional predictive frames including image regions, wherein at least one image region of at least one bidirectional predictive frame is encoded by interpolation from two or more referenceable frames, At least one of the two or more referenceable frames is separated from the bidirectional predictive frame by at least one intervening subsequent referenceable frame in the display order.

CN 1663258 Β

[0037] A video image compression system with a sequence of reference frames and bi-directional predictive frames including image regions, wherein at least one image region of at least one bi-directional predictive frame is encoded as a selected image from two or more referenceable frames The unequal weighting of regions.

[0038] A video image compression system with a sequence of predictive frames and bidirectional predictive frames, each frame includes pixel values arranged in macroblocks, wherein at least one macroblock in the bidirectional predictive frame is based on two or more The direct mode prediction of the motion vector of the predicted frame is determined.

[0039] A video image compression system with reference frames and a sequence of bidirectionally predicted frames, each frame includes pixel values arranged in macroblocks, where at least one macroblock in the bidirectionally predicted frame is based on The direct mode prediction of the motion vector of one or more prediction frames is determined, wherein at least one of such one or more prediction frames is before the bidirectional prediction frame in the display order.

[0040] A video image compression system with reference frames and a sequence of bidirectionally predicted frames, each frame includes pixel values arranged in macroblocks, wherein at least one macroblock in the bidirectionally predicted frame is based on data from one or more The direct mode prediction of the motion vector of the prediction frame is determined, wherein at least one of such one or more prediction frames is after the bidirectional prediction frame in the display order and is separated from the bidirectional prediction frame by at least one intervening reference frame.

[0041] A video image compression system with a sequence of frames, each frame includes a plurality of image regions having a DC value, and each pixel included in each such image region has an AC pixel value, wherein at least one image of at least one frame At least one of the DC value and the AC pixel value of the area is determined as a weighted interpolation of the corresponding respective DC value and AC pixel value from at least one other frame.

[0042] A video image compression system with a sequence of referenceable frames, each frame includes a plurality of image regions having a DC value, and each pixel included in each such image region has an AC pixel value, wherein at least one of the predicted frames At least one of the DC value and the AC pixel value of the at least one image area is obtained by interpolation from the corresponding respective DC values and AC pixel values of two or more referenceable frames.

[0043] Improve the image quality of a sequence of two or more bidirectionally predicted intermediate frames in a video image compression system, each frame includes a plurality of image regions with a DC value, and each such image region includes each pixel Having an AC pixel value, including at least one of the following: determining the AC pixel value of each image area of the bidirectional prediction intermediate frame is a first weighted ratio of the corresponding AC pixel value in the referenceable frame surrounding the bidirectional prediction intermediate frame sequence; and It is determined that the DC value of each image area of the bidirectional prediction intermediate frame is a second weighted ratio of the corresponding DC value in the referenceable frame surrounding the bidirectional prediction intermediate frame sequence. A video image compression system with a sequence of frames, the frame comprising a plurality of pixels having an initial representation, wherein the pixels of at least one frame are interpolated from the corresponding pixels of at least two other frames, wherein the pixels of the at least two other frames The corresponding pixels are interpolated when transformed to a different representation, and cause the interpolated pixels to be transformed back to the original representation.

[0044] In a video image compression system with reference frames and a sequence of bidirectional predictive frames, dynamically determining the coding mode of such a frame with a variable number of bidirectional predictive frames includes: selecting an initial sequence, which starts from The reference frame has at least one subsequent bidirectional prediction frame and ends with a reference frame; the reference frame is added to the end of the initial sequence to create a test sequence; the test sequence is evaluated against the selected evaluation criteria; For each satisfying step of evaluating the test sequence, insert a bidirectional frame before the added reference frame and repeat the evaluation step; and if the evaluation of the test sequence is not satisfied, then the previous test sequence is accepted as the current encoding mode.

[0045] A video image compression system with a sequence of referenceable frames, the referenceable frames are separated by at least one bidirectional predictive frame, wherein the number of such bidirectional predictive frames varies in this sequence, and at least one of these At least one image region of the bidirectional predictive frame is determined using unequal weights corresponding to pixel values of at least two referenceable frames.

CN 1663258 Β

[0046] A video image compression system with a sequence of frames that are encoded by an encoder and decoded by a decoder, wherein at least one image area of at least one frame is based on a weighted interpolation of two or more other frames, such Weighted interpolation is based on at least one set of weights available to the encoder and decoder, wherein a designation of one of the selected at least one set of weights is transmitted from the encoder to the decoder to select one or more currently valid weights .

[0047] A video image compression system with a sequence of frames that are encoded by an encoder and decoded by a decoder, wherein at least one image area of at least one frame is based on a weighted interpolation of two or more other frames, such Weighted interpolation is based on at least one set of weights, of which at least one set of weights is downloaded to the decoder, and then a designation of one of the selected at least one set of weights is transmitted from the encoder to the decoder to select one or more Currently valid weights.

[0048] A video image compression system with a reference frame sequence, the reference frame is encoded by the encoder and decoded by the decoder, wherein the predicted frame in the reference frame sequence is transmitted from the encoder to the decoder, and the transmission order is different It is based on the display order of this predicted frame after decoding.

[0049] A video image compression system with a sequence of referenceable frames, the referenceable frame includes pixels arranged in an image area, wherein at least one image area of at least one predicted frame is coded with reference to two or more referenceable frames , Where each such image area is determined using unequal weights corresponding to the pixel values of such two or more referenceable frames.

[0050] A video image compression system with predicted frames, bi-directional predicted frames and intra-frame sequences, each frame includes an image area, wherein at least one filter selected from the sharpening and softening filter group is in the predicted frame or bi-directional The motion vector of the predicted frame is used in at least one image area of the predicted frame or bidirectionally predicted frame in the process of compensating the prediction.

[0051] The drawings and the following description clarify the details of one or more embodiments of the invention. The other features, objectives and advantages of the present invention are apparent from the drawings, and claims.

Description of the drawings

[0052] FIG. 1 is a timeline of frames and MPEG-4 direct mode motion vectors according to the prior art.

[0053] FIG. 2 is a timeline and proportional pixel weights of a frame according to this aspect of the invention.

[0054] FIG. 3 is a timeline of a frame and mixed, proportional and equal pixel weights according to this aspect of the invention.

[0055] FIG. 4 is a flowchart showing an illustrative embodiment of the present invention as a method that can be implemented by a computer.

[0056] FIG. 5 shows a diagram of multiple previous references in which the current P frame refers to the previous two P frames and one previous I frame.

[0057] FIG. 6A is a diagram of a typical prior art MPEG-2 encoding mode, showing a constant number of B frames between I frame and/or P frame to be surrounded.

[0058] FIG. 6B is a theoretically possible prior art MPEG-4 video coding mode diagram, showing the number of B frames that are surrounded by I frames and/or P frames, and the changes between I frames distance.

[0059] FIG. 1 is a diagram of an encoding mode.

[0060] FIG. 8 is a flowchart showing an embodiment of an interpolation method, and DC interpolation is different from AC interpolation.

[0061] FIG. 9 is a flowchart showing an embodiment of an interpolation method using a replacement representation of luminance pixels.

[0062] FIG. 10 is a flowchart showing an embodiment of an interpolation method using replacement representation of chrominance pixels.

[0063] FIG. 11 is a diagram showing the unique motion vector area size of each of two P frames.

[0064] FIG. 12 is a diagram showing a sequence of P-frames and B-frames, with interpolation weights of B-frames, which are determined to be 2

CN 1663258 Β

A function of the distance from (2-away) subsequent P-frames with reference to a 1-away ("away) subsequent P-frame.

[0065] FIG. 13 is a diagram showing a sequence of P-frames and B-frames, with interpolation weights of B-frames, which are determined as a function of the distance from a 1 to a subsequent P frame, which refers to a 1 far away from the previous P frame .

[0066] FIG. 14 is a diagram showing a sequence of P frames and B frames, where subsequent P frames have multiple motion vectors referring to previous P frames.

15 is a diagram showing a sequence of P frames and B frames, where the nearest subsequent P frame has a motion vector referring to the previous P frame, and the next nearest subsequent ρ frame has multiple references to the previous ρ frame Motion vector.

[0068] FIG. 16 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector that refers to a previous P frame.

[0069] FIG. 17 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has two motion vectors referring to the previous P frame.

[0070] FIG. 18 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame.

[0071] FIG. 19 is a frame sequence showing three P frames P1, P2, P3, where P3 uses an interpolation reference with two motion vectors, one for each of P1 and P2.

[0072] FIG. 20 shows a frame sequence of four P frames P1, P2, P3, and P4, where P4 uses an interpolation reference with three motion vectors, and P1, P2, and P3 each have a motion vector.

[0073] FIG. 21 is a diagram showing a sequence of P frames and B frames, where different P frames have one or more motion vectors that refer to different previous P frames, and show respective forward and backward assigned to a specific B frame Different weights to the reference.

[0074] FIG. 22 is a diagram showing a sequence of P frames and B frames, where the order of the bit stream of the P frames is different from the display order.

[0075] FIG. 23 is a diagram showing a sequence of P frames and B frames with assigned weights.

[0076] FIG. 24 is a time chart of the position of an object in a frame.

[0077] The same reference symbols in different drawings denote the same elements.

Detailed ways

[0078] Overview

[0079] One aspect of the present invention is based on the recognition that a common practice is to use an M value of 3, which provides two B frames between each P (or D frame. But M=2 and M=4 Or higher are useful. It is particularly important to note that the value of M (the number of B frames plus 1) also has a natural relationship with the frame rate. At 24 frames per second (fps), the movies playback rate, A time distance of 1/24 second between frames can cause substantial image changes from frame to frame. But at frame rates of 60fps, 72fps or higher, the time distance between adjacent frames decreases accordingly. The result is that with the frame rate The increase of, the larger the number of B frames (that is, the larger the M value) becomes useful and beneficial in compression efficiency.

[0080] Another aspect of the present invention is based on the recognition that both MPEG-2 and MPEG-4 video compression use too simple interpolation methods. For example, for mode 3, the bidirectional prediction of each macroblock of the frame is the equal average of the subsequent and previous frame macroblocks, which is replaced by two corresponding motion vectors. This equal average is suitable for M = 2 (ie a single intermediate B frame), because the time from the B frame to the previous P (or I) frame and the subsequent P (or D frame) is equal. However, for larger M values, use equal Weighting can only make the best prediction for centrally symmetrical B frames (ie, intermediate frames if M = 4, 6, 8, etc.). Similarly, in MPEG-4 direct mode 4, even if the motion vector is proportionally weighted, every The predicted pixel values of the middle B frames are equal proportions of the corresponding pixels of the previous P (or D frame and subsequent P frames.

CN 1663258 Β

[0081] Therefore, for M>2, it is an improvement to apply an appropriate proportional weight to the predicted pixel value of each B frame. The proportional weight of each pixel of the current B frame corresponds to the relative position of the current B frame with respect to the previous and subsequent P (or I) frames. Therefore, if M = 3, the first B frame uses 2/3 of the corresponding pixel value (adjusted motion vector) of the previous frame and 1/3 of the corresponding pixel value (adjusted motion vector) of the subsequent frame.

[0082] FIG. 2 is a timeline and proportional pixel weighted values of a frame according to this aspect of the invention. The pixel value in each macroblock of each intermediate B frame 20la, 20lb is weighted as a function of the "distance" between the previous P or I frame A and the next P or I frame B, and the farther away from the P or I frame Closer, the greater the weighting value. That is, each pixel value of the bidirectionally predicted B frame is a weighted combination of the corresponding pixel values of the non-bidirectionally predicted frames A and B that are surrounded. In this example, for M=3, the weight of the first B frame 201a is equal to 2/3A+1/3B, and the weight of the second B frame 201b is equal to 1/3A+2/3B. Also shown is the equal average weight assigned under the conventional MPEG system; the MPEG-2 and MPEG-4 weights of each B frame 201a, 201b are equal to (A+B)/2.

[0083] Application of Extended Dynamic Range and Contrast Range

[0084] If M is greater than 2, the proportional weighting of the pixel values in the intermediate B frame will improve bidirectional (mode 3) and direct (effectiveness of MPEG-4 mode 4) coding in many cases. Examples include common film and video editing effects, such as fade and cross fade. These types of video effects are problematic coding examples of MPEG-2 and MPEG-4. This is due to the use of a simple DC matching algorithm, and the commonly used M = 3 (ie two intermediate B frames), resulting in equal B frames proportion. The coding in these cases is improved by using proportional B frame interpolation according to the present invention.

[0085] Proportional B frame interpolation also has a direct application for improving the coding efficiency of the extended dynamic and contrast range. A common event in image coding is a change in brightness, which occurs when the object gradually moves into (or out of) the shadow (soft shadow boundary). If a logarithmic encoding representation is used for brightness (for example, the brightness specifically represented by logarithmic brightness Υ), then the change in lighting brightness will be a change in DC offset. If the lighting brightness is halved, then the pixel value will be reduced by the same amount. Therefore, in order to encode this change, an AC match should be found, and the encoded DC difference should be applied to this area. This DC difference encoded into the P frame should also be applied proportionally in each intervening B frame (see co-pending U.S. Patent Application No. 09/905039, entitled Method and System for Improving Compressed Image Chromainformation, It is assigned to the assignee of the present invention, so it is incorporated herein by reference for additional information regarding logarithmic coding representation).

[0086] In addition to changes in brightness, proportional B-frame interpolation is also beneficial to changes in contrast. For example, as the airplane moves toward the audience, moves out of clouds or mist, its contrast gradually increases. This contrast increase will be expressed as the magnitude of the increase in the AC coefficients of the DCT in the corresponding P-frame coded macroblock. Likewise, the contrast change in the intervening B-frames will be most closely approximated by proportional interpolation, thus improving the coding efficiency.

[0087] As the frame rate becomes larger and as the value of M increases, the use of proportional B frame interpolation to improve dynamic range and contrast coding efficiency becomes increasingly important.

[0088] Applying high M values to temporal stratification

[0089] Utilizing the embodiments of the present invention allows the value of M to be increased, thereby increasing the number of B frames between P frames and/or I frames to be surrounded, while maintaining or increasing coding efficiency. This usage is beneficial for many applications, including temporal stratification. For example, in U.S. Patent No. 5,988,863 entitled Temporal and Resolution Layering for Advanced Television (which is assigned to the assignee of the present invention and is hereby incorporated by reference), it is noted that the B frame is the time for layering (frame ) Appropriate mechanism for rate. The flexibility of this rate is related to the number of consecutive B frames available. For example, a single B frame (M = 2) can support a 36fps decoded time layer in a 72fps stream or a 60fps stream.

CN 1663258 Β

30fps decoded time layer. Three B-frames (M = 4) can support the 36fps and 18fps decoded time layer in the 72fps stream, and the 30fps and 15fps decoded time layer in the 60fps stream. Using M = 10 in a 120fps stream can support 12fps, 24fps, and 60fps decoding time layers. Μ=4 can also be used for 144fps streams to provide a temporal layer decoded at 72fps and 36fps.

[0090] As an improvement to each Nth frame, multiple frames of 120fps or 72fps can be decoded and mixed proportionally, as described in the co-pending US Patent Application No. 09/545233, which is titled^uEnhancements to Temporal and Resolution Layering (which is assigned to the assignee of the present invention and incorporated herein by reference) to improve the motion blur characteristics of the 24fps result.

[0091] Even higher frame rates can be synthesized using the method described in the co-pending U.S. Patent Application No. 09/435277, which is entitled System and Method for Motion Compensaion and Frame Rate Conversion (which is assigned to the present The assignee of the invention is hereby incorporated by reference). For example, through motion compensation frame rate conversion, 72fps camera film (original) can be used to create an effective frame rate of 288 frames per second. Using M = 12, you can get frame rates of 48fps and 24fps and other useful rates, such as 144fps, 96fps and 32fps (of course, the film is 72fps). The frame rate conversion using this method does not need to be an integral multiple. For example, an effective rate of 120fps can be created from a source of 72fps and then used as a source of 60fps and 24fps (using M = 10).

[0092] Therefore, temporal layering is beneficial to optimize the performance of B frame interpolation. The proportional B-frame interpolation described above makes the function of a larger number of consecutive B-frames more efficient, so these benefits can be realized.

[0093] Mixed B frame interpolation ratio

[0094] One reason for using equal-average weighting as the motion compensation mode predictor for the pixel values of the B frame in the conventional system is that the P (or I) frame before or after the specific B frame may be noisy, so it is not ideal Match. Equal mixing in the interpolated motion compensation block will optimize the noise reduction. There is a difference residual that is coded using a quantized DCT function. Of course, the better the match from the motion compensation ratio, the fewer differential residual bits required, and the higher the final image quality.

[0095] In the case of objects moving in and out of shadows or haze, the true ratio when M>2 provides a better prediction. However, when the light and contrast have not changed, equal weighting can prove to be a better predictor, because the error of moving the macroblock forward along the motion vector will be averaged with the error from the backward shifted block, so each The error is halved. Even so, it is more likely that the B frame macroblocks of the closer P (or D frame) are more related to that frame than the farther P (or I) frame.

[0096] Therefore, in some cases, such as changes in the contrast or brightness of the area, it is ideal to use the true ratio (for brightness and color) of the B-frame macroblock pixel weighting, as described above. In other cases, it may be better to use equal proportions, as in MPEG-2 and MPEG-4.

[0097] Another aspect of the present invention utilizes a mixture of these two ratio techniques (equal average and frame-distance ratio) for B frame pixel interpolation. For example, for the case of M=3, 3/4 of the ratio of 1/3 and 2/3 can be mixed with the equal average of 1/4, resulting in the two ratios being 3/8 and 5/8. This technique can be summarized using the "mixing factor" F.

[0098] Weight = F · (frame distance proportional weight) + (1F) · equal average weight

[0099] The useful range of the mixing factor F is from 1 to 0, with 1 being a fully proportional interpolation, and 0 being a perfect equal average (the opposite assignment may also be used).

[0100] FIG. 3 is a timeline of a frame and mixed, proportional and equal pixel weights according to this aspect of the invention. The pixel value of each macroblock of each intermediate B frame 301a and 301b is weighted as a function of the "time distance" between the previous P or I frame A and the next P or I frame B, and the values of A and B, etc. Average function. In this example,

CN 1663258 Β

For M = 3 and blending factor F = 3/4, the blending weight of the first B frame 301a is equal to 5/8A+3/8B (that is, 3/4 of the proportional weighting 2/3A+1/3B plus equal average Weighted (A+B) /2 of 1/4). Similarly, the weight of the second B frame 301b is equal to 3/8A+5/8Bo

[0101] The value of the blending factor F can be set for all encodings as a whole, or set for each group of pictures (GOP), a range of B frames, each B frame, or each region within a B frame (including, Such as refinement to each macro block, or for MPEG-4 direct mode using P vector in 8X8 mode, or even a single 8X8 motion block).

[0102] In order to save bits and reflect the fact that the mixing ratio is usually not important enough to be transmitted with each macroblock, the optimal use of mixing should be related to the type of image to be compressed. For example, for an image that is fading out or fading, or an image whose overall lighting or contrast is gradually changing, the blending factor F is close to or equal to 1 (ie, proportional interpolation) is generally optimal. For continuous images without such changes in lighting or contrast, a lower blending factor value such as 2/3.1/2 or 1/3 may form the optimal choice, thereby retaining some of the benefits of proportional interpolation and some of the equal average interpolation. These benefits. All blending factor values in the range of 0 to 1 are generally useful, because for any given B frame, a specific value in this range proves to be optimized.

[0103] For images with a wide dynamic range and a wide contrast range, the blending factor can be determined depending on the characteristic area of the local area. However, it is generally recommended that a wide range of lighting and contrast ratios supports full proportional interpolation instead of equal average interpolation.

[0104] The optimized mixing factor is usually determined by experience, although experience for a specific type of scene can be used to create a table of mixing factors by scene type. For example, the determination of image change characteristics can be used to select a blending ratio for a frame or region. Alternatively, a number of candidate blending factors (for the entire frame or region) can be used to encode B-frames, and each blending factor can be evaluated to optimize image quality (for example, determined by the highest signal-to-noise ratio (SNR)) and the lowest number of bits . The evaluation of these candidates can then be used to select the best value for the mixing ratio. A combination of image change characteristics and encoding quality/efficiency can also be used.

[0105] A B frame near the middle of the B frame sequence or a B frame from a low value M will not be too much affected by the proportional interpolation, because the calculated ratio is already close to the equal average. However, for a higher value of M, the extreme value of the B frame position can be significantly affected by the selection of the mixing factor. Note that the mixing factors can be different for these extreme positions. These extreme positions use the average more than the more central positions (which gain little or no benefit from deviation from the average) because they already have adjacent P (or D frames). High ratio. For example, if M = 5, the blending factor F that may be used in the first and fourth B frames is mixed with more equal averages, but the second and third B frames may use a strict 2/5 And the equal average ratio of 3/5. If the ratio-to-average blend factor changes as a function of the position of the B frame in the sequence, then the change value of the blend factor can be transmitted in the compressed bitstream or as The auxiliary information is passed to the decoder.

[0106] If a static general mixing factor is required (due to the lack of a way to transmit the value), the value 2/3 is usually close to optimal and can be selected as a static value for B frame interpolation in the encoder and decoder. For example, using a blending factor of F = 2/3, for consecutive frames with M = 3, the ratio will be 7/18 (7/18 = 2/3*1/3+1/3*1/2) and 11/ 18 (11/18 =2/3*2/3+1/3*1/2).

[0107] Linear interpolation

[0108] Video frame pixel values are generally stored in a specific representation, which maps original image information to digital values. This mapping may result in linear or non-linear representations. For example, the brightness value used in compression is non-linear. The use of various forms of non-linear representation includes logarithms, exponents (various screens), and black-corrected exponents (usually

CN 1663258 Β

Used for video signals).

[0109] On a narrow dynamic range or for interpolation in neighboring regions, a non-linear representation is acceptable because these neighboring interpolations represent piecewise linear interpolation. Therefore, small changes in brightness can be reasonably approximated by linear interpolation. However, for wide variations in brightness, such as those occurring in images with a wide dynamic range and wide contrast range, it will be inaccurate to process non-linear signals as linear ones. Even for images in the normal contrast range, linear fading and cross-fade can be reduced by linear interpolation. Some fade and cross fades use nonlinear fade and fade rates, adding further complexity.

[0110] An additional improvement to the use of proportional blending, or even simple proportional or equal average interpolation, is to perform this interpolation on the pixel values represented in linear space, or in other optimized non-linear brightness representations that are different from the original non-linear brightness representation. This interpolation is performed in linear space.

[0111] This can be done, for example, by first converting the two non-linear luminance signals (from the previous and subsequent P (or D frames) into linear representations, or different non-linear representations. Then apply proportional mixing, and then apply the inverse transform After that, the result of mixing is produced in the original non-linear brightness representation of the image. However, the scale function has been performed on a better representation of the brightness signal.

[0112] When colors are fading or becoming more saturated, as occurs in contrast changes associated with changes in mist and cloudy skies, in addition to brightness, this linear or non-linear transformation is advantageously applied to colors The (chromaticity) value is also useful.

[0113] Exemplary Embodiment

[0114] FIG. 4 is a flowchart showing an exemplary embodiment of the present invention as a method that can be implemented by a computer:

[0115] Step 400: In the video image compression system, for calculating the direct and interpolation mode of the B frame, use (1) the frame distance ratio, or (2) a mixture of equal weight and frame distance ratio to determine the interpolation value to be applied For each pixel of the input sequence of two or more bidirectionally predicted intermediate frames, the interpolation value is derived from at least two non-bidirectionally predicted frames that enclose the sequence input from the source (for example, a video image stream).

[0116] Step 401: Optimize interpolation with respect to image units (for example, group of pictures (GOP), frame sequence, scene, frame, region within a frame, macro block, DCT block, or similar useful grouping or pixel selection). The interpolation value can be statically set during the entire encoding period, or dynamically set for each image unit.

[0117] Step 402: Further optimize the interpolation value with respect to the scene type or coding simplicity. For example, the interpolation value can be set: statically (such as 2/3 ratio and 1/3 equal average); proportionally, it is a frame close to the equal average, but the adjacent P (or D frame) should be mixed with equal average; dynamic Ground, based on overall scene characteristics, such as fade and cross-fade; dynamically (and locally), based on local image area characteristics, such as local contrast and local dynamic range; or dynamically (and locally), based on coding performance (For example, the highest coded SNR) and the least coded bits generated.

[0118] Step 403: Transmit an appropriate proportional amount to the decoder, if it is not statically determined.

[0119] Step 404: Optionally, convert the brightness information to a linear or alternative non-linear representation for each frame, and transmit this replacement to the decoder, if it is not statically determined.

[0120] Step 405: Use the determined interpolation value to determine the proportional pixel value.

[0121] Step 406: If necessary (due to step 404), convert to the original representation.

[0122] Extended P frame reference

[0123] As mentioned above, in the prior art MPEG-2 and 4 compression methods, the P frame refers to the previous P or I frame,

CN 1663258 Β

And the B frame refers to the nearest previous or subsequent P and/or I frame. The same technology is used in the DCT compression standards of H. 261 and H. 263 motion compensation, which includes low bit rate compression technology.

[0124] In the H.263++ and H.26L standards under development, the B frame reference is extended to point to P or I frames that do not directly surround the current frame. That is, the macro block in the B frame may point to a P or I frame before the previous P frame, or point to a P or I frame after the subsequent P frame. Since each macro block has one or more bits, skipping the previous or subsequent P frames can be simply represented. Conceptually, using the previous P frame for reference in B requires only storage space. For low bit rate encoding of H. 263++ or H. 26L, this is a small amount of extra storage. For subsequent P frame references, the P frame encoding sequence must be modified relative to the B frame encoding, so that the future P frame (or possibly I frame) must be decoded before the intervening B frame. Therefore, the coding sequence is also a problem for subsequent P frame references.

[0125] The main differences between the P frame and B frame types are: (1) B frames can be bidirectionally referenced (up to two motion vectors per macro block); (2) B frames are discarded after use (this is also It means that they can be skipped during the decoding process to provide temporal layering); and (3) P frames are used as "springboards", one to the next, because each P frame must be decoded to be used as each subsequent P frame. Frame reference.

[0126] As another aspect of the present invention, a p frame (and a B frame) is decoded according to one or more previous P or I frames (excluding the case where each P frame only refers to the nearest previous P frame or I frame). relatively). Therefore, for example, two or more motion vectors of each macroblock can be used in the current P frame, and all motion vectors point to the back in time (that is, point to one or more previously decoded frames). Such P frames still maintain the "springboard" characteristics. FIG. 5 is an example showing multiple previous references. The current P frame 500 refers to two previous P frames 502, 504 and a previous I frame 506.

[0127] In addition, the concept of macroblock interpolation can be applied in such a P frame reference, as described above. Therefore, in addition to expressing a single reference to more than one previous P or I frame, a motion vector can also be used for each such frame reference to mix the proportions of multiple previous P or I frames. For example, the technique described above using the B-frame interpolation mode with two frame references can be applied to allow any macroblock in the P-frame to use two motion vectors to refer to the previous two P-frames or one previous p-frame and one The front ι frame. This technique interpolates between two motion vectors, but it is not bidirectional in time (as in the case of B-frame interpolation), because both motion vectors point to the back in time. The storage overhead is reduced to a point at which it is quite practical to store the previous multiple P or I frames in the memory for this concurrent reference.

[0128] When applying such P frame interpolation, it is definitely necessary to select various useful ratios of the previous two or more P frames (and optionally, one previous I frame) and notify the decoder. In particular, the equal mixing of frames is one of the useful mixing ratios. For example, with the first two ρ frames as a reference, an equal 1/2 of each ρ frame can be mixed. For the first three P frames, an equal mixing of 1/3 can be used.

[0129] Another useful mix of two P frames is 2/3 of the nearest preceding frame and 1/3 of the farthest preceding frame. For the first 3 P frames, another useful mix is 1/2 of the nearest front frame, 1/3 of the middle front frame, and 1/6 of the farthest front frame.

[0130] In any case, a simple set of useful mixes of multiple previous P-frames (and optionally one I-frame) can be utilized, and they can be simply notified from the encoder to the decoder. Whenever it is useful to optimize the coding efficiency of a picture unit, the specific mixing ratio used can be selected. A small number of bits can be used to select many mixing ratios, and these bits can be transmitted to the decoder as long as they are suitable for the desired image unit.

[0131] As another aspect of the present invention, it is also useful to transition to select a single P frame with reference from the nearest previous P (or I) frame to a more "far" previous P (or I) frame. In this way, the P frame will use a single motion vector for each macro block (or alternatively, each 8X8 block in MPEG-4 mode encoding), but will use one or more bits to represent the

CN 1663258 Β

Reference refers to a single specific previous frame. The P frame macroblock in this mode is not interpolated, but will refer to the selected previous frame, which is selected from the possible two, three or more previous P (or D frames) for reference. For example, The 2-bit encoding can designate one of up to 4 previous frames as a single selected frame. This 2-bit encoding can be changed in any convenient image unit.

[0132] Number of adaptive B frames

[0133] A fixed mode of I, P, and B frame types is typically used in MPEG encoding. The number of B frames between P frames is typically constant. For example, it is typical to use two B frames between P (or I) frames in MPEG encoding. FIG. 6A is a typical prior art MPEG-2 encoding mode diagram, showing a fixed number of B frames (ie, two) between the I frame 600 and/or P frame 602 that are surrounded.

[0134] The MPEG-4 video coding standard conceptually allows a variable number of B frames and varying distances between I frames and/or P frames to be surrounded. Fig. 6B is a theoretically possible prior art MPEG-4 video coding mode diagram, showing a variable number of B frames between the enclosed I frame 600 and/or P frame 602, and the change between I frame 600 distance.

[0135] This flexible coding structure can theoretically be used to improve coding efficiency by matching the most effective B-frame and P-frame coding types to moving image frames. Although this flexibility has been specifically allowed, it is rarely studied, and the mechanism for actually determining the positions of B and P frames in this flexible structure is also unknown.

[0136] Another aspect of the present invention applies the concepts described here to this flexible coding structure and simple fixed coding mode in common use. Therefore, the above-mentioned method can be used to interpolate the B frame, and the P frame can refer to more than one previous P frame or I frame and be interpolated according to the original.

[0137] In particular, the macroblocks in the B-frames can utilize the ratio mixing suitable for flexible coding structures as effective as the fixed structure. When the B frame refers to a P frame or I frame that is farther than the closest P frame or I frame for bracketing, proportional mixing can also be used.

[0138] Similarly, in this flexible coding structure, a P frame can refer to more than one previous P frame or I frame, which is as effective as the fixed mode structure. In addition, when these P frames refer to more than one previous P frame (optionally, an I frame is added), the mixing ratio can be applied to the macroblocks in such a p frame.

[0139] (A) Determine position in flexible coding mode

[0140] The following method allows the encoder to optimize the frame encoding mode and utilize the efficiency of the mixing ratio. For the selected range of frames, many candidate encoding modes can be tried to determine the optimal or near optimal (relative to the specified standard) mode. Figure 7 is a diagram of an encoding mode that can be checked. The initial sequence 700 ending with a P or I frame is arbitrarily selected and used as the basis for additional P and/or B frames to be added, and then the additional P and/or B frames (as described below) are evaluated. In one embodiment, a P frame is added to the initial sequence 700 to create the first test sequence 702 for evaluation. If the evaluation is satisfied, then an intervening B frame is inserted to create a second test sequence 704. For each satisfied evaluation, an additional B frame is inserted to create a test sequence 706 to 712 that gradually becomes longer until the evaluation criterion becomes unsatisfied. At that point, the previous coding sequence is accepted. Then repeat this process, using the last P frame of the previously accepted coding sequence as the starting point for adding a new P frame, and then insert a new B frame.

[0141] The optimal or near optimal encoding mode can be selected based on various evaluation criteria, which usually involves the trade-off of various encoding features, such as the image quality of the encoding and the number of encoding bits required. Commonly used evaluation criteria include the minimum number of bits used (in a fixed quantization parameter test), or the best signal-to-noise ratio (in a fixed bit rate test), or a combination of the two.

CN 1663258 Β

[0142] Minimizing a sum of absolute differences (SAD) is also common, and SAD forms a measure of DC matching. As in the co-pending titled Motion Estimation for Video Compression System^>,As described in US Patent Application No. 09/904192 (which is assigned to the assignee of the present invention and therefore incorporated by reference), the AC matching criterion is also a useful measure of the quality of a particular candidate match (this patent application also describes other useful Optimization). Therefore, the AC and DC matching criteria accumulated on the best matching of all macroblocks can be checked to determine the overall matching quality of each candidate coding mode. When this AC/DC matching technique is used with the estimation of the number of coded bits for each frame mode type, it can improve or replace the signal-to-noise ratio (SNR) and tests that use the fewest bits. Typically, the quantization parameter value (QP) used for encoding B-frame macroblocks is higher than that used for P-frames, which affects the number and quality of bits used in various candidate encoding modes (usually measured as signal-to-noise ratio) .

[0143] (B) Mixing ratio optimization in flexible coding mode

[0144] Optionally, for each candidate mode determined according to the foregoing method, the suitability of the mixing ratio with respect to one or more standards (for example, the optimal or close to the optimal mixing ratio) can be tested. This can be done, for example, by testing for best quality (lowest SNR) and/or efficiency (least bits used). The use of one or more previous references for each macroblock in the P frame can also be determined in the same way. The reference mode and mixing ratio of each candidate are tested to determine one or more sets of suitable references.

[0145] Once the coding mode is selected for the next step (step 700 in FIG. 7), the subsequent steps (steps 702-712) can be tested for various candidate coding modes. In this method, more efficient coding of moving image sequences can be determined. Therefore, the efficiency can be optimized/improved as described in subsection (A) above; hybrid optimization can be applied in the coding step of each test.

[0146] DC to AC interpolation

[0147] In many cases of image encoding, such as when using a logarithmic representation of an image frame, the aforementioned interpolation of frame pixel values will optimally encode changes in brightness. However, in alternative video "Y-curves", linear, and other representations, applying different interpolation blending factors to the DC value of the pixel instead of the AC value often proves to be useful. FIG. 8 is a flowchart showing an embodiment of an interpolation method with DC interpolation that is significantly different from AC interpolation. For selected image regions (usually DCT blocks or macroblocks) 802, 802' from the first and second input frames, the average pixel value 804, 804 of each such region is subtracted, so that in the selected region In the DC value (ie the average value of the entire selected area) 806, 806^zSeparate from the AC value (that is, the reserved signed pixel value) 808, 808. Then the respective DC value is 806,806^zCan be multiplied by the interpolation weight 810.810^,, Which is different from multiplying by AC (signed) pixel value 808,808^zThe used 814, 814, o Then, the latest interpolated DC value 812 and the latest interpolated AC value 816 can be merged into 818, resulting in a new prediction 820 for the selected area.

[0148] Like other interpolation values in the present invention, the appropriate weight can be notified to the decoder of each image unit.

It is possible to choose a small number of bits among many coded values, as well as to choose the independent interpolation of the AC to DC aspects of the pixel value.

[0149] Linear & Non-Linear Interpolation

[0150] Interpolation is a linear weighted average. Since the interpolation operation is linear, and because the pixel value in each image frame is usually expressed in a non-linear form (such as video gamma representation or logarithmic representation), further optimization of the interpolation process becomes possible. For example, the interpolation of pixels in a specific frame sequence, and the interpolation of DC values separated from AC values, sometimes expressed by linear pixels will be optimal or close to optimal. However, for other frame sequences, if the pixels are expressed as logarithmic values or other pixel representations, then this interpolation will be optimal or close to optimal. In addition, the optimal representation or near optimal representation used to interpolate U and V (chrominance) signal components may be different from the optimal table of Y (luminance) signal components.

CN 1663258 Β

Show or near optimal representation. Therefore, a useful aspect of the present invention is to convert the pixel representation into an alternative representation as part of the interpolation process.

[0151] FIG. 9 is a flowchart showing an embodiment of a method of interpolation of luminance pixels using alternate representations. Starting from a region or block of luminance (Υ) pixels in the initial representation (such as video gamma or logarithmic) (step 900), the pixel data is transformed into an alternative representation that is different from the initial representation (such as linear, logarithmic) , Video gamma) (step 902). Then, as described above, the transformed pixel area or block is interpolated (step 906), and transformed back to the original representation (step 906). As a result, the interpolated pixel brightness value is obtained (step 908).

[0152] FIG. 10 is a flowchart showing an embodiment of a method of chrominance pixel interpolation using replacement representation. Starting from a region or block of chrominance (U, V) pixels in the initial representation (such as video gamma or logarithm) (step 1000), the pixel data is transformed into an alternative representation that is different from the initial representation (such as linear, Logarithmic, video gamma) (step 1002). Then, as described above, the transformed pixel area or block is interpolated (step 1006), and transformed back to the initial representation (step 1006). As a result, the interpolated pixel chromaticity value is obtained (step 1008).

[0153] The conversion between the various representations can be carried out in accordance with the explanation of US Patent Application No. 09/905039 entitled Method and System for lmproving Compressed Image Chroma Information, which is assigned to the assignee of the present invention and is therefore incorporated by reference . Note that a simple lookup table is usually used for the representation transformation and its inverse transformation of the replacement.

[0154] As a variation of this aspect of the invention, the replacement (linear or non-linear) representation space used for AC interpolation may be different from the replacement representation space used for DC interpolation.

[0155] Like the interpolation weight, which alternative interpolation representation is selected for each luminance (Y) and chrominance (U and V) pixel representation, a small number of bits per selected image unit can be used to inform the decoder.

[0156] Number of motion vectors per macro block

[0157] In MPEG-2, each 16×16 macroblock in a P frame allows one motion vector. In the B frame, MPEG-2 allows at most two motion vectors per 16X16 macroblock, which corresponds to a bidirectional interpolation mode. In MPEG-4 video coding, each 16X16 macroblock in the P frame allows up to 4 motion vectors, corresponding to one motion vector for each 8X8DCT block. In MPEG-4B frame, when using interpolation mode, each 16X16 macroblock allows up to two motion vectors. If the subsequent corresponding P frame macroblock is set to 8×8 mode with 4 motion vectors, a single motion vector increment in MPEG-4 direct mode can result in 4 independent "implicit" motion vectors. This is after converting the time distance (the B frame is closer in time to the previous P frame or I frame reference of the P frame), by adding a motion vector increment carried in the 16X16B frame macroblock to the subsequent P frame Each of the corresponding 4 independent motion vectors in the macro block is implemented.

[0158] One aspect of the present invention includes the option of increasing the number of motion vectors per image region (eg, macroblock). For example, it sometimes proves that each B-frame macroblock has more than two motion vectors. These can be applied by referring to additional P-frames or I-frames and having 3 or more interpolation terms in the weighted sum. Additional motion vectors can also be applied to allow independent vectors of 8X8DCT blocks of B-frame macroblocks. Also, 4 independent increments can be used to extend the concept of direct mode by applying independent increments to each of the 4 8×8 area motion vectors of subsequent P frames.

[0159] In addition, the B frame interpolation technique can be used to modify the P frame to refer to more than one previous frame in the interpolation mode, using the two interpolation term techniques of the B frame described above. This technique can be easily extended to more than two previous P-frames or I-frames, and the resulting interpolation has 3 or more terms in the weighted sum.

[0160] Like other aspects of the present invention (such as pixel representation and DC-to-AC interpolation method), a small number of bits per image unit can be used to pass a specific weighted sum to the decoder.

CN 1663258 Β

[0161] When applying this aspect of the present invention, the correspondence between the 8×8 pixel DCT block and the motion vector domain does not need to be as strict as in MPEG-2 and MPEG-4. For example, it may be useful to use replacement area sizes other than 16X16, 16X8 (used only in interlaced scanning in MPEG-4)> 8X8 for motion vectors. This replacement may include any number of useful area sizes, such as 4X8, 8X12, 8X16, 6X12, 2X8, 4X8, 24X8, 32X32, 24X24, 24X16, 8X24, 32X8, 32X4, etc. Using a small amount of this useful size, a few bits can inform the decoder of the correspondence between the motion vector area size and the DCT block size. In a system using traditional 8X8DCT blocks, a set of simple correspondences to the motion vector domain is useful to simplify the processing in motion compensation. In a system where the DCT block size is not 8X8, then greater flexibility can be achieved in the specified motion vector domain, as shown in the title^uEnhanced Temporal and Resolution Layering in Advanced Television co-pending US Patent Application No. 09/545233, which is assigned to the assignee of the present invention, and therefore is incorporated by reference. Note that the motion vector area boundary does not need to correspond to the DCT area boundary. In fact, it is often useful to define the motion vector area in such a way that the boundary of the motion vector area falls within the DCT block (and not on its boundary).

[0162] The concept of extending the flexibility of the motion vector domain is also applicable to the interpolation aspect of the present invention. As long as the correspondence between each pixel and one or more motion vectors pointing to one or more reference frames is specified, all the versatility of the present invention can be used to apply the above interpolation method to the full flexibility of useful motion vectors. Even when using P frames, for each previous frame reference, the size of the area corresponding to each motion vector can be different, and when using B frames, for each previous and future frame reference, corresponding to each motion vector The area size can be different. If the area size of the motion vector is different when applying the improved interpolation method of the present invention, the interpolation reflects the overlapping common area. The overlapping common area of the motion vector reference can be used as the area on which the DC term is determined when the DC and AC pixel values are respectively interpolated.

[0163] FIG. 11 shows the respective unique motion vector area sizes 1100 and 1102 of the two P frames 1104 and 1106. When calculating the interpolation according to this invention, the joint 1108 of the size of the motion vector area is determined. Union 1108 defines all areas that are considered to have allocated motion vectors.

[0164] Therefore, for example, when the 4×4DCT region of the B frame 1112 is interpolated backward to the previous P frame 1104, the 4×4 region 1110 in the union 1108 will use the motion vector corresponding to the 8×16 region 1114 in the previous P frame. If forward prediction, the 1110 area in the union 1108 will use the motion vector corresponding to the 4×16 area 1115 in the next P frame. Similarly, the backward interpolation of the region 116 in the union 1108 will use the motion vector corresponding to the 8×16 region 1114, and the same region in the forward prediction will use the motion vector corresponding to the 12×16 region 1117.

[0165] In an embodiment of the present invention, two steps are used to realize the interpolation of a universal (ie, non-uniform size) motion vector. The first step is to determine the common area of the motion vectors, as described in relation to FIG. 11. This establishes the correspondence between pixels and motion vectors (that is, the number of motion vectors for each specified pixel area size) for each previous frame or subsequent frame reference. The second step is to use appropriate interpolation methods and effective interpolation factors for each area of the pixel. The task of the encoder is to ensure that the optimal or near optimal motion vector area and interpolation method are specified, and all pixels have their vectors and interpolation methods fully specified. For the fixed mode of the motion vector (for example, a motion vector is specified for each 32X8 block for the entire frame), there is a single specified interpolation method (for example, the fixed ratio mixing of each distance from the reference frame to the reference frame specified for the entire frame), It's very simple. If the size of the motion vector area is changed regionally, and the difference in the area size depends on which previous frame or subsequent frame is referred to (for example, the nearest previous frame is 8X8 blocks, and the next nearest previous frame is 32X8 blocks) , This method can also become quite complicated. In addition, the interpolation method can be specified regionally within the frame.

[0166] During encoding, the job of the encoder is to determine the optimal or near optimal use of bits to shape the motion vector area.

CN 1663258 Β

Choose between shape and size, and choose the optimal or close to optimal interpolation method. A decision is needed to specify the number and distance of referenced frames. These specifications are determined by exhaustively testing the size of a large number of candidate motion vector regions, the candidate frame to be referred to and the interpolation method of each such motion vector region, until the optimal or close to optimal coding is found. The optimum (relative to the selected standard) can be determined by finding the smallest SNR after the coding block or finding the minimum number of bits for a fixed quantization parameter (QP) after the coding block, or by applying other suitable measurements.

[0167] Direct Mode Extension

[0168] The traditional direct mode used in the B-frame macroblocks in MPEG-4 is efficient in motion vector coding and provides the benefits of the 8×8 block mode by using simple common increments. The direct mode weights each corresponding motion vector of the subsequent P frame at the corresponding macroblock position based on the time distance, and the P frame refers to the previous P frame. For example, if M = 3 (ie two intervening B frames), using simple linear interpolation, the first B frame uses -2/3 times the subsequent P frame motion vector to determine the pixel offset relative to this P frame , And 1/3 times the motion vector of the subsequent P frame to determine the pixel offset relative to the previous P frame. Similarly, the second B frame uses -1/3 times the same P frame motion vector to determine the pixel offset relative to this P frame, and 2/3 times the subsequent P frame motion vector to determine the motion vector relative to the previous P frame. The pixel offset of the frame. In direct mode, small increments are added to each corresponding motion vector. As another aspect of this invention, this concept can be extended to refer to one or more B-frames that are far away from the P-frames. These P-frames refer to one or more preceding or subsequent P-frames or I-frames, by considering the frame distance. Determine the frame scale fraction

[0169] FIG. 12 is a diagram showing a sequence of P frames and B frames, where the interpolation weight of the B frame is determined as a function of the distance to 2 away from the subsequent P frame, the P frame with reference 1 away from the subsequent P frame. In this illustrative example, M=3, which means that there are two consecutive B frames 1200 and 1202 between the P frames 1204 and 1206 to be surrounded. In this example, each coexisting macroblock in the next nearest subsequent P frame 1208 may point to the intervening (ie nearest) P frame 1204, and the first two B frames 1200, 1202 may refer to the next nearest The subsequent P frame 1208 instead of the nearest subsequent P frame 1204, as in traditional MPEG. Therefore, for the first B frame 1200, the frame ratio score 5/3 multiplied by the motion vector mv from the next nearest subsequent P frame 1208 will be used as the pixel offset relative to the P frame 1208, and the first B frame Frame 1202 will use 4/3 multiplied by the offset of that same motion vector.

[0170] If the nearest subsequent P frame referenced by the B frame points to the next nearest preceding P frame, then the simple frame distance is used again to obtain a suitable frame scale score applied to the motion vector. Figure 13 is a diagram showing a sequence of P frames and B frames, where the interpolation weight of the B frame is determined as a function of the distance to 1 away from the subsequent P frame, which refers to 2 away from the previous P frame. In this illustrative example, M=3, B frames 1300,1302 refer to the nearest subsequent P frame 1304, and this P frame 1304 refers to 2 far away P frame 1306. Thus, for the first B frame 1300, the pixel offset score is the frame ratio score 2/6 times the motion vector mv from the most recent subsequent P frame 1304, and the second B frame 1302 will have the same 1/6 times that The pixel offset of the motion vector is because the motion vector of the nearest subsequent P frame 1304 points 2 away from the previous P frame 1306, which is 6 frames away.

[0171] Generally, when a B frame refers to a single P frame in the direct mode, the frame distance method sets the numerator of the frame ratio score equal to the frame distance from the B frame to its reference or "target" P frame, and sets The denominator is equal to the distance from the target P frame to another P frame referenced by the target P frame. For the measurement from the B frame to the subsequent P frame, the sign of the frame ratio score is a negative sign, and for the measurement from the B frame to the previous P frame, the sign of the frame ratio score is a positive sign. This simple method of applying the frame distance or frame ratio score to the P frame motion vector can achieve effective direct mode coding.

[0172] In addition, another aspect of this invention is to allow direct mode to be applied to multiple interpolated motion vector references of one P frame. For example, if a P frame is interpolated from the nearest and the next nearest previous P frame, then according to the present invention

CN 1663258 Β

Direct mode reference in this aspect allows interpolation blending to be used for each multi-reference direct mode B frame macroblock. Generally, two or more motion vectors of a P frame can be applied with an appropriate frame scale score. Then, the two or more frame distance corrected motion vectors can be used with the corresponding interpolation weights of each B frame referring to or pointing to that P frame (as described below) to generate interpolated B frame macroblock motion compensation.

[0173] FIG. 14 is a diagram showing a sequence of P frames and B frames, in which a subsequent P frame has multiple motion vectors referring to the previous P frame. In this example, the B frame 1400 refers to the subsequent P frame P3. This P3 frame has two motion vectors mvl and mv2, which refer to the corresponding previous P frames P2 and Pl. In this example, any one of the two weighting terms or a combination of such weighting terms can be used to interpolate each macroblock of the B frame 1400 in the direct mode.

[0174] Each macro block of the B frame 1400 can be constructed in a mixed form according to the following items:

[0175] Corresponding pixels of frame P2 replaced by frame ratio score 1/3 of mv1 (these pixels may then be multiplied by a certain ratio weight i) plus frame P3 replaced by frame ratio score -2/3 of mv1 Corresponding pixels of (these pixels may then be multiplied by a certain proportional weight j); and

[0176] The corresponding pixels of the frame P1 replaced by the frame ratio score of mv2 2/3 (4/6) (these pixels may then be multiplied by a certain ratio weight k) plus the frame ratio score of mv2 -1/ 3 (-2/6) replaces the corresponding pixels of frame P3 (these pixels may then be multiplied by a certain scale weight of 1).

[0177] Like all direct modes, the motion vector increment can be used with each of mv1 and mv2.

[0178] According to this aspect of the present invention, as in the case of multiple previous P frames, using the same interpolation method and motion vector frame ratio score application, the direct mode prediction macroblock in the B frame can also refer to multiple subsequent P frames. frame. FIG. 15 is a diagram showing a sequence of P frames and B frames, where the nearest subsequent P frame has a motion vector referring to a previous P frame, and the next nearest subsequent P frame has multiple motion vectors referring to multiple previous P frames. In this example, the B frame 1500 refers to two subsequent P frames P2 and P3. The frame P3 has two motion vectors mv1 and mv2, which refer to the corresponding previous P frames P2 and P1. Frame P2 has a motion vector mv3, which refers to the previous P frame P1. In this example, 3 weighting terms are used to interpolate each macroblock of the B frame 1500 in the direct mode. If this is the case, the motion vector frame ratio score can be greater than 1 or less than T.

[0179] This form of direct mode B-frame macroblock interpolation weighting can take advantage of all the versatility of the interpolation described here. In particular, in order to obtain the best performance of various image units (for example, quality versus number of bits), each weight or combination of weights can be tested. The interpolation sub-array for this modified direct mode can be assigned to the decoder with a small number of bits per image unit.

[0180] Each macro block of the B frame 1500 can be constructed in a mixed form according to the following items:

[0181] The corresponding pixels of frame P3 replaced by the frame ratio score of mv1 -5/3 (these pixels may then be multiplied by a certain ratio weight i) plus the frame replaced by the frame ratio score of mv1-2/3 The corresponding pixels of P2 (these pixels may then be multiplied by a certain ratio weight j);

[0182] The corresponding pixels of the frame P3 replaced by the frame ratio score of mv2 -5/6 (these pixels may then be multiplied by a certain ratio weight k) plus the frame P1 replaced by the frame ratio score of mv2 of 1/6 Corresponding pixels of (these pixels may then be multiplied by a certain scale weight of 1); and

[0183] Corresponding pixels of frame P2 replaced by the frame ratio score of mv3-2/3 (these pixels may then be multiplied by a certain ratio weight m) plus frame P1 replaced by the frame ratio score of mv3 1/3 Corresponding pixels (these pixels may then be multiplied by a certain proportional weight η).

[0184] Like all direct modes, the motion vector increment can be used with each of mv1, mv2, and mv3.

CN 1663258 Β

use.

[0185] Note that when the next nearest subsequent P frame refers to the nearest P frame surrounding a candidate B frame, a particularly beneficial direct coding mode often appears.

[0186] Direct mode coding of B frames in MPEG-4 always uses the motion vector of subsequent P frames as a reference. According to another aspect of the present invention, it is also possible for the B frame to refer to the motion vectors of the coexisting macroblocks of the previous P frame, and these motion vectors will sometimes prove a beneficial choice for direct mode coding reference. If this is the case, when the next nearest preceding P frame is referenced by the motion vector of the nearest preceding P frame, the motion vector frame ratio score will be greater than 1. Figure 16 shows a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame. In this example, the B frame 1600 refers to the previous P frame P2 far away from T1. The motion vector mv of the frame P2 refers to the next previous P frame P1 (relative to the B frame 1600, 2 is far away). The figure shows the appropriate frame ratio score.

[0187] If the nearest previous P frame is interpolated according to multiple vectors and frames, then methods similar to those described in conjunction with FIG. 14 are applied to obtain the motion vector frame ratio score and interpolation weight. Fig. 17 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has two motion vectors that refer to the previous P frame. In this example, the B frame 1700 refers to the previous P frame P3. One motion vector mvl of the previous frame P3 refers to the next previous P frame P2, and the second motion vector mv2 refers to the previous P frame P1 far away from 2. The figure shows the appropriate frame ratio score.

[0188] Each macro block of the B frame 1700 can be constructed in a mixed form according to the following items:

[0189] The corresponding pixels of the frame P3 replaced by the frame ratio score of mv1 1/3 (these pixels may then be multiplied by a certain ratio weight i) plus the frame P2 replaced by the frame ratio score 4/3 of mv1 Corresponding pixels (these pixels may then be multiplied by a certain proportional weight j); and

[0190] The corresponding pixels of the frame P3 replaced by the frame ratio score of mv2 of 1/6 (these pixels may then be multiplied by a certain ratio weight k) plus the frame P1 replaced by the frame ratio score of mv2 of 7/6 Corresponding pixels (these pixels may then be multiplied by a certain scale weight of 1).

[0191] When the motion vector of the current P frame (relative to the B frame) points to the next nearest previous P frame, it is not necessary to use only the next nearest previous frame as an interpolation reference, as shown in FIG. 16. The most recent previous P frame may prove a better choice for motion compensation. In this case, the motion vector of the nearest previous P frame is shortened to the frame distance fraction from the B frame to that P frame. Fig. 18 is a diagram showing a sequence of P frames and B frames, where the nearest previous P frame has a motion vector referring to the previous P frame. In this example, for M = 3, the first B frame 1800 will use 1/3 and -2/3 frame distance scores multiplied by the motion vector mv of the nearest previous P frame P2. The second B frame 1802 will use 2/3 and -1/3 frame distance scores (not shown). The decoder is notified of this choice to distinguish this example from the example shown in FIG. 16.

[0192] Like all other coding modes, the use of direct mode preferably includes testing candidate modes against other available interpolation and single vector coding modes and reference frames. For direct mode testing, the nearest subsequent P frame (optionally, and the next nearest subsequent P frame or even further subsequent P frame, and/or one or more previous P frames) can be tested as candidate frames, And a small number of bits (typically one or two) can be used to specify the direct mode P reference frame distance used by the decoder.

[0193] Extended interpolation

[0194] It is specified in the MPEG-1, 2, 4 and H. 261 and H. 263 standards that the B frame uses the equal weighting of the pixel values of the forward reference frame and the backward reference frame, as replaced by a motion vector . Another aspect of the present invention includes various useful applications of unequal weights that significantly increase the efficiency of B-frame coding, and the extension of such unequal weights to more than two references, which include two or more references. A reference to forward or backward in time. This aspect of the invention also

CN 1663258 Β

Includes methods for more than one frame to be referenced and interpolated for P-frames. In addition, when two or more references point forward in time, or when two or more references point backward in time, it is sometimes useful to use negative weights and weights that exceed ΐ.ο.

[0195] For example, FIG. 19 is a frame sequence showing three P frames P1, P2, and P3, where P3 uses two motion vectors for interpolation reference, and P1 and P2 each use one. For example, if continuous changes are occurring in the frame range between P1 and P3, then P2-P1 (that is, the pixel value of frame P2 (replaced by the motion vector of P2) minus the pixel value of frame P1 (replaced by the motion vector of P1) Replaced)) will be equal to P3-P2. Similarly, P3-P1 will be twice the size of P2-P1 and P3-P2. In this case, the pixel value of frame P3 can be predicted differently from P2 and P1 through the formula:

[0196] P3 = Ρ1+2X (Ρ2-Ρ1) = (2XP2)-Ρ1

[0197] In this case, the interpolation weight of P3 is 2.0 for P2 and -1.0 for P1.

[0198] As another example, FIG. 20 shows a frame sequence of four P frames P1, P2, P3, and P4, where P4 uses an interpolation reference with three motion vectors and one each for P1, P2, and P3. Therefore, since P4 is predicted from P3, P2, and P1, three motion vectors and interpolation weights are applied. In this case, if continuous changes are occurring in this frame range, then P2-P1 will be equal to P3-P2 and P4-P3, and P4-P1 will be equal to 3X (P2-P1) and 3X (P3-P2) .

[0199] Therefore, in this example, the P4 prediction based on P2 and P1 is:

[0200] P4 = P1+3X (P2-P1) = (3XP2)-(2XP1) (weights are 3.0 and -2.0)

[0201] The P4 prediction based on P3 and P1 is:

[0202] P4 = P1+3/2X (P3-P1) = (3/2XP3)-(1/2XP1) (weights are 1.5 and -0.5)

[0203] The P4 prediction based on P3 and P2 is:

[0204] P4 = P2+2 X (P3-P2) = (2X P3) -P2 (weights are 2.0 and T. 0)

[0205] However, it is possible that the change closest to P4 (including P3 and P2) is a more reliable predicted value of P4 than the prediction that includes P1. Therefore, by giving 1/4 weight to each of the two items including P1 above, and 1/2 weight to the items including only P3 and P2, it will lead to:

[0206] 1/2(2P3-P2) +1/4(3/2P3-1/2P1)+1/4(3P2-2P1)=

[0207] 13/8P3+1/4P2-5/8P1 (weights are 1.375, 0.25 and -0.625)

[0208] Accordingly, it is sometimes useful to use both a weight greater than 1.0 and a weight less than 0. At other times, if there is a noise-like change from one frame to the next, a positive weighted average with a moderate coefficient between 0.0 and 1.0 can produce P4 macroblocks (or other pixel regions). Best forecast. For example, the equal weighting of 1/3 of each of P1, P2, and P3 in Fig. 20 may form the best prediction of P4 in some cases.

[0209] Note that the best matching motion vector is used to determine the areas of P1, P2, P3, etc. In this example, P1, P2, P3 are used in the calculation. In some cases, this matching may be best AC matching, which allows changing DC terms to be predicted by AC coefficients. Alternatively, if DC matching (such as the sum of absolute differences) is used, then the change in AC coefficients can often be predicted. In other cases, various forms of motion vector matching will form the best prediction with a mixture of various weights. Usually, the best prediction for a particular situation is empirically determined using the method described here.

[0210] These techniques are also applicable to B-frames with two or more motion vectors pointing backward or forward in time. When pointing forward in time, the above-described coefficient mode for P frames is reversed to accurately predict the current P frame backwards. Using this aspect of the invention, it is possible to have two or more motion vectors in both forward and backward directions, and therefore predict in both directions at the same time. The appropriate weighted mixture of these various predictions can be optimized by selecting the mixed weight, which best predicts the macroblock (or other pixel area) of the current B frame.

CN 1663258 Β

[0211] FIG. 21 is a diagram showing a sequence of P frames and B frames, where different P frames have one or more motion vectors that refer to different previous P frames, and the figure also shows the respective forwards assigned to the specific B frame reference. Different weights ae and backward reference. In this example, the B frame 2100 refers to three previous P frames and two subsequent P frames.

[0212] In the example shown in FIG. 21, frame P5 must be decoded for this example to work. Sometimes it is useful to order the frames in the bitstream in the order required for decoding ("send order"), and the order of transmission does not have to be the order of display ("display order"). For example, in a frame sequence showing cyclic motion (such as the rotation of an object), a particular P frame may be more similar to a distant ρ frame than the nearest subsequent ρ frame. Fig. 22 is a diagram showing a sequence of P frames and B frames, where the bit stream transmission order of P frames is different from the display order. In this example, frame P3 is more similar to frame P5 than frame P4. So it is useful to send and decode P5 before P4, but display P4 before P5. Preferably, when such P-frames can be discarded (for example, the end time of n frames in the bitstream sequence or after frame X in the display sequence), each P-frame should notify the decoder.

[0213] If the weight is selected from a small selection set, then a small number of bits can inform the decoder which weight to use. As with all other weights described here, this can be notified to the decoder once per picture unit, or at any other point where the weight change during the decoding process is useful.

[0214] It is also possible to download a new set of weights. In this way, a small set of weights can be effective at a given time. This allows a small number of bits to inform the decoder which valid set of weights will be used at any given point in the decoding process. In order to determine the appropriate set of weights, a large number of weights can be tested during the encoding process. If a small subset is found to provide high efficiency, the decoder can be notified of which subset to use. Therefore, the decoder can be notified of the specific elements of the subset with only a few bits. For example, 10 bits can select one of 1024 subset elements. In addition, when a specific small subset is changed to maintain efficiency, the decoder can be notified of the new subset. Therefore, the encoder can compare the number of bits required to update the weight set, and dynamically optimize the number of bits required to select the weight set elements. In addition, a small number of short codes can be used to represent common useful weights, such as 1/2, 1/3, 1/4, and so on. In this way, a small number of bits can be used to represent the weight set, for example, for K-forward-vector prediction in P frames (where K = 1,2,3...), or for B frames Κ-forward-vector and L-backward-vector prediction (where Κ and L are selected from 0,1,2,3-stroke), or K-forward-vector and L-backward-vector in P frame Prediction (where K and L are selected from 0, 1, 2, 3), as a function of the current M value (ie, the relative position of the B frame relative to the neighboring P (or D frame).

[0215] FIG. 23 is a diagram showing a sequence of P frames and B frames with assigned weights. The B frame 2300 has a weight ae, the value of which is assigned from the table of the B frame weight set 2302. The P frame 2304 has weights m and n, the values of which are allocated from the table of the P frame weight set 2306. Some weights can be static (that is, downloaded to the decoder unchanged) and notified by the encoder. Other weights can be downloaded dynamically and then notified.

[0216] This same technique can be used to dynamically update the weight set to select DC interpolation compared to AC interpolation. In addition, the coded value can be represented, and the linear interpolation of the transformed value (in the alternative linear or non-linear representation) selects a general (linear) interpolation (usually the interpolation of the pixel value represented by the non-linear representation). Similarly, this coded value can indicate which such interpolation is applied to the AC value or the DC value or whether to separate the AC and DC parts of the prediction.

[0217] Effective subsets can also be used to minimize the number of bits necessary to choose between currently used weight coefficient sets. For example, if 1024 downloaded weight sets are saved in the decoder, it may take approximately 16 to be valid during a particular part of the frame. Therefore, by selecting (selected from 1024) which subsets of the 16 weight sets are valid, only 4 bits are needed to select which of the 16 weight sets is valid. It is possible to use short codes for these most common subsets to represent these subsets, thus allowing a small number of bits to be selected in the commonly used subsets.

CN 1663258 Β

[0218] Softening and Sharpening

[0219] Like simply separating the DC component from the AC signal by subtracting the average value, other filtering operations are also possible in the motion vector compensation prediction process. For example, various high-pass, band-pass, and low-pass filters can be applied to pixel regions (eg, macroblocks) to extract various frequency bands. Then, when performing motion compensation, these frequency bands can be modified. For example, it may often be useful to filter out the highest frequencies on a noisy moving image to soften (make it less clear or slightly blurred) the image. The softer image pixels, combined with the steeper slope matrix (t set t matrix) used for quantization (the steeper slope matrix ignores the higher frequency noise in the current block), usually form a more efficient encoding method. It is already possible to express the change in the quantization slope matrix of each image unit. It is also possible to download custom oblique matrices of luminance and chrominance. Note that regardless of whether the oblique matrix is changed, the efficiency of motion compensation can be improved. However, it is usually most effective to change both the oblique matrix and the filter parameters applied in the motion compensation process.

[0220] A common practice is to use a reduced chroma coding resolution and a chroma-specific oblique matrix. However, in this example, the chroma coding resolution is static (eg 4:2:0 encodes half resolution horizontally and vertically or 4:2:2 just encodes half resolution horizontally). According to this aspect of the present invention, the coding efficiency can be improved by applying a dynamic filter process to chrominance and luminance (independently or cooperatively) in the motion compensation process, one for each image unit.

[0221] US Patent Application No. 09/545233 entitled Enhanced Temporal and Resolution Layering in Advanced Television (cited above) describes the use of an improved negative half cycle (truncated sine function) displacement filter. These filters have the advantage that they maintain sharpness when performing the fractional-pixel portion of the motion vector displacement. At the integer pixel displacement point and the fragment pixel displacement point, the use of filters that reduce or increase the definition can better move certain macroblocks (or other useful image areas). For example, for "zoom" (where certain objects in the frame move out of focus over time and other parts of the frame come into focus), transition is one of the changes in sharpness and flexibility. Therefore, a motion compensation filter that can improve the sharpness in some areas of the image and reduce the sharpness in other areas can improve the coding efficiency. In particular, if an area of the image is moving out of focus, it may be beneficial to reduce the sharpness, which will soften the image (and thus may produce a better match) and reduce grain and/or noise (and thus may increase coding efficiency). If an area of the image is coming into focus, it may be beneficial to maintain the best sharpness or even use a larger negative half-cycle sine filter value to improve sharpness.

[0222] In the encoding process, chroma filtering can also benefit from improving and reducing sharpness. For example, many of the coding efficiency advantages of 4:2:0 encoding (encoding half-resolution chroma horizontally and vertically) can maintain full resolution in the U and/or V channels by using softer motion compensation filters for chroma Rate to achieve. Only when the color details in the U and V channels are high, it is necessary to choose the clearest displacement filter; a softer filter will be more beneficial where there is high color noise or particles.

[0223] In addition to the change in focus, the direction from one frame to the next frame and the amount of motion blur change are also common. When the frame rate of a moving image film is 24fps, even a simple dialogue scene has a significant change in motion blur from one frame to the next. For example, the upper lip may be blurred in one frame and sharpened in the next frame, which is entirely due to lip movement during the opening of the shutter in the camera. For such motion blur, it will be beneficial not only to have sharpening and softening (blurring) filters in the motion compensation process, but also to have the direction of sharpening and softening. For example, if the direction of movement can be determined, then softening or sharpening in that direction can be used to move or stop the corresponding image close-up. The motion vector used for motion compensation itself can provide some useful information about the amount of motion and the change of the amount of motion (ie, motion blur) for a specific frame (or area within a frame) relative to any surrounding frame (or corresponding area). In particular, the motion vector is the best motion match between P frames, and the motion blur comes from the time of opening the shutter in the frame.

CN 1663258 Β

During the exercise.

[0224] FIG. 24 is a graph of object position versus time in a frame. The camera's shutter is only open during part of the frame time. When the shutter is open, any movement of the object causes blur. The amount of motion blur is represented by the amount of position change during the opening of the shutter. Thus, when the shutter open position of the slope of the curve 2400 is shipped motion blur measure.

[0225] The amount of motion blur and the direction of motion can also be determined from a combination of sharpness measurement, surrounding motion vectors (where the image area matches), feature smear detection, and artificially assisted designation of the frame area. The filter can be selected based on the determined amount of motion blur and the direction of motion. For example, comparing the determined motion blur amount and motion direction, the mapping of various filters can be determined empirically.

[0226] When combined with other aspects of the present invention, this intelligently applied filter can significantly improve the compression coding efficiency. You can select a small number of such filters and use a small number of bits to inform the decoder. Again, this can be done once per image unit or at other useful points in the decoding process. As with the weight set, a set of dynamically loaded filters can be used, as well as an effective subset mechanism to minimize the number of bits required to select between the most beneficial filter parameter sets.

[0227] Realization

[0228] The present invention can be implemented in hardware or software, or a combination of the two (for example, a programmable logic array). Unless otherwise specified, the algorithms included as part of the present invention are not essentially related to any particular computer or other device. In particular, various general-purpose devices can be used with programs written as described herein, or it may be more convenient to construct more special devices (such as integrated circuits) to perform specific functions. Therefore, the present invention can be implemented in one or more computer programs executed on one or more programmable computer systems. Each programmable computer system includes at least one processor, at least one data storage system (including permanent and volatile). Memory and/or storage unit), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described here and generate output information. The output information is applied to one or more output devices in a well-known manner.

[0229] Each such program can be implemented in any desired computer language (including machine, assembly, or high-level procedural, logical, object-oriented programming language) to communicate with the computer system. In any case, the language can be a compiled language or an interpreted language.

[0230] Each such computer program is preferably stored in or downloaded to a storage medium or device (such as a solid-state memory or medium, or a magnetic medium or an optical medium), which is a general-purpose or special-purpose programmable computer When the storage medium or device is read by a computer system to execute the program described herein, such a computer program is used to configure and operate the computer. It is also conceivable to implement the system of the present invention as a computer-readable storage medium, which is configured to have a computer program, and the storage medium so configured enables the computer system to operate in a specific and predetermined manner to perform the functions described herein.

[0231] A number of embodiments of the present invention have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present invention. For example, some of the steps described above may be order-independent, and therefore may be performed in a different order from the described order. Therefore, other embodiments are within the scope of the appended claims.

CN 1663258 Β