Measurement-based and scalable deblock filtering of image data

[0001] Embodiments according to the invention generally relate to data processing, and more specifically to video decoding and deblocking.

[0002] The video contains a series of pictures (or frames), each consisting of a two-dimensional array of pixels. These pixels are divided into blocks (for example, an 8x8 array of pixels). The video is encoded (compressed) to reduce the total bit rate, allowing the video to be transmitted more efficiently to another device, where it is decoded (decompressed) and reconstructed. , Can be displayed.

[0003] Current video and image compression techniques include lossy processing such as perceptual quantization of block transformation coefficients. Lossy encoding of independent blocks in a video frame can result in abrupt transitions known as blocking artifacts at the block edges of the decoded and reconstructed image, giving the rendered image a relatively chunky appearance. May be exhibited.

[0004] To eliminate or reduce the appearance of blocking artifacts, the decoding device performs a deblocking operation. In deblocking, a deblocking filter (eg, a lowpass digital filter) is applied across the block boundaries to smooth the transition between adjacent blocks and improve the perceived quality of the displayed video.

[0005] However, there is a problem in choosing a deblock filter of appropriate intensity. If the selected filter is too weak, the appearance of blocking artifacts cannot be effectively reduced. On the other hand, if the selected filter is too strong, too much detail may be removed.

[0006] There is also a problem in choosing the number of pixels to which the deblock filter is applied. In one extreme case, blocking artifacts may not be effectively reduced or eliminated if the filter is applied only to pixels that are immediately adjacent to the block boundary. In the other extreme case, if the filter is applied across all the pixels in a block, some pixel values will be changed unnecessarily and the details within that block will be lost. Sometimes.

<p> [0007] Therefore, it is important to be able to select a deblock filter that can strike an acceptable balance between the need to reduce or eliminate blocking artifacts and the need to maintain a satisfactory level of detail. It is also important to be able to determine the number of pixels to which the selected deblock filter should be applied to avoid overfiltering the image data and incidental loss of image detail.</p>

<p> [0008] According to an embodiment of the invention, pixel values within a first block of pixels are compared for deblocking, and pixel values in the first block are adjacent to the first block. Compared to the pixel value in the second block of pixels. Based on the results of the comparison, a digital deblocking filter and a "target area" are selected, which are the number of pixels in the first block and the second block to which the selected filter should be applied. It identifies the number of pixels in a block.</p><p> More specifically, in one embodiment, the first value is calculated for the first block of pixels, the second value is calculated for the second block of pixels, said second. Block is adjacent to the first block. In one embodiment, the first value is based on an "artifact count" for the first block and an "activity" within the first block. The artifact count gives a measure of the number of blocking artifacts associated with the first block. The activity provides a measure of the amount of variation in the first block, and more specifically, the activity is the average of the selected pixel values or the activity of the selected pixel value from an intermediate value such as a weighted average. Gives a measure of the amount of deviation. In a similar fashion, the second value is based on the artifact count for the second block and the activity within the second block. In such an embodiment, the first value and the second value are each proportional to the artifact count and inversely proportional to the activity in the first block and the second block, respectively. There is.</p><p> [0010] In one embodiment, the artifact count calculates the difference in pixel values over the boundary between the first block and the second block, and sets the difference as two thresholds. Determined by comparison. The use of the two thresholds allows the visual sensitivity of the artifact to be evaluated more accurately, facilitating the selection of a deblocking filter of appropriate intensity. In such an embodiment, the user (eg, the viewer) selects at least one of the thresholds based on individual preferences.</p><p> [0011] In one embodiment, one of the first value and the second value described above is used as an index for selecting a digital deblock filter from the banks of the filter, and the filter of the bank. The intensity gradually increases as the value of the index increases. In such an embodiment, the smallest of the first value and the second value is used to select the deblock filter. The use of filter banks with different intensity filters can eliminate the drawbacks associated with hard switching between fixed filtering and filtering decisions. Furthermore, the number of filters in the filter bank can be specified according to the type of graphics processing unit (GPU) used to carry out the present invention. For example, when used for lower end or baseline GPUs, a smaller number of filters with fewer taps can be selected. In general, the overall complexity of filtering can be reduced or extended to support different implementations with different levels of computational power.</p><p> [0012] The selected deblock filter then extends over the boundary between the first block and the second block to a particular number of pixels on one side of the boundary and of the boundary. New pixel values can be generated for the first block and the second block that are applied to a particular number of pixels on the other side and smooth the transition across the block boundaries. More specifically, the selected deblock filter is applied to a first number of pixel values in the first block and a second number of pixel values in the second block. In one embodiment, the first number of the pixel values corresponds to the first value described above, and the second number of the pixel values corresponds to the second value described above. Therefore, the first value and the second value are used for both selecting the deblock filter and defining the target area.</p><p> [0013] In short, according to embodiments of the present invention, deblock filtering is adaptive (eg, for the quality of pixel values as measured by artifact count per block and activity) and is scalable. (For example, with respect to the capabilities of the graphics card) and can be controlled by the user. The embodiments according to the invention are not only well adapted for a wide range of processing power, but also well for applications such as a wide range of video quality and enhancement of low bit rate video (eg, internet video). It is a thing.</p><p> [0014] These objectives and other objectives and effects of the various embodiments of the present invention will be appreciated by those skilled in the art after reading the following detailed description of the embodiments exemplified in the various drawings. You can recognize it.</p><p> [0015] The present invention is shown in each of the accompanying drawings as an example, but not limited thereto, in which similar reference numerals refer to similar elements.</p>

<figref num="1">It is a block diagram of an example of the system which can carry out the decoder by embodiment of this invention.</figref><figref num="2">FIG. 6 is a block diagram showing an example element of a decoding and post-processing pipeline according to one embodiment of the present invention.</figref><figref num="3">It is a flowchart of the computer embodiment for deblocking the image data by one Embodiment of this invention.</figref><figref num="4">An example of an array of blocks in an image frame according to an embodiment of the present invention is shown.</figref><figref num="5">Illustrates the thresholds used to count blocking artifacts according to one embodiment of the invention.</figref><figref num="6">(A) shows an example of a row of pixels in a block according to an embodiment of the present invention. (B) shows an example of the distribution of pixel values according to the embodiment of the present invention.</figref><figref num="7">An example of a target area according to an embodiment of the present invention is shown.</figref><figref num="8">It is a flowchart of an example of the computer embodiment for processing the image data by one Embodiment of this invention.</figref>

[0025] Embodiments according to the present invention, of which some embodiments are exemplified in the accompanying drawings, will be described in detail below. Although the present invention will be described in connection with these embodiments, it will be understood that these descriptions are not intended to limit the invention to these embodiments. On the contrary, the present invention is intended to include alternatives, variants and equivalents contained within the spirit and scope of the invention as defined by the claims. Furthermore, the following detailed description of embodiments of the present invention describes many specific details in order to fully understand the invention. However, those skilled in the art will recognize that the present invention can be practiced without these particular details. On the one hand, well-known methods, procedures, components and circuits are not detailed so as not to unnecessarily obscure aspects of embodiments of the present invention.

[0026] Some parts of the detailed description below are made in the form of procedures, steps, logic blocks, processes and other symbolic representations of operations relating to data bits in computer memory. These explanations and expressions are the means used by those skilled in the art to most effectively convey the substance of their achievements to those skilled in the art. Procedures, computer execution steps, logic blocks, processes, etc. are here and generally considered to be a self-consistent series of steps or instructions that lead to the desired result. These steps require physical manipulation of physical quantities. Usually, but not always, these quantities of electrical or magnetic signals that can be stored, transferred, combined, compared, or otherwise manipulated in a computer system. It is in shape. It has been found to be convenient to describe these signals as bits, values, elements, symbols, letters, terms, numbers, etc., sometimes and mainly because they are commonly used.

[0027] However, keep in mind that all of these and similar terms are associated with appropriate physical quantities and are merely expedient labels that apply to these quantities. As is clear from the above, unless otherwise specified, "encoding", "decoding", "deblocking", "receiving", "sending", "using", "applying", etc. Descriptions using terms such as "calculate", "increase", "compare", "select", "add", "weight", "compute or calculate", "access", etc. Manipulates data represented as physical (electronic) quantities in computer system registers and memory, as well as physical quantities in computer system memory or registers or other similar information storage, transmission or display equipment. It should be understood that it refers to the operation and processing of a computer system or similar electronic computing device that converts it into other data represented by.

[0028] FIGS. 3 and 8 are flowcharts of an example of a computer implementation method for processing data according to an embodiment of the present invention. Although these flowcharts show specific steps, such steps are typical examples. That is, embodiments of the present invention are well adapted to perform various other steps or modifications of the steps shown in these flowcharts.

[0029] The flowcharts of FIGS. 3 and 8 are performed as computer-executable instructions present on a computer-usable medium of some form, such as a program module, executed by one or more computers or other devices. Can be done. In general, a program module includes routines, programs, objects, components, data structures, etc. that perform a particular task or perform a particular abstract data type. The functionality of the program module can be combined or distributed as desired in various embodiments.

[0030] As an example, but not limited to, computer-usable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules or other data. .. Computer storage media are, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory or other memory techniques, compact disk ROM (CD). -ROM), digital versatile disc (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other used to store desired information. Includes medium.

Communication media can carry computer-readable instructions, data structures, program modules or other data in modulated data signals such as carrier waves or other transport mechanisms, including any information distribution medium. The term "modulated data signal" means a signal in which one or more of the characteristics of the signal are set or modified in such a way as to encode the information in the signal. .. By way of example, communication media include, but are not limited to, wired media such as wired networks or direct wired connections, and wired media such as acoustic, radio frequency (RF), infrared and other wireless media. Any combination of those mentioned above is also within the range of computer readable media.

[0032] In the following description, unless otherwise stated, "decoder" refers to an element that performs both decoding and deblocking (eg, stand-alone or peripheral or integrated system). "Decoding" refers to basic decoding behavior (eg, variable length decoding, dequantization, inverse transformation and motion compensation) unless otherwise stated, while "deblocking" refers to decoding. It refers to applying a deblock filter to the coding result. "Block" refers to an array of pixels (eg, an NxN array). The term "pixel" refers to a position in a two-dimensional screen space, while the term "pixel value" refers to data associated with a pixel (eg, luma value, chroma value, etc.).

[0033] FIG. 1 is a block diagram of an example of a system 10 in which the decoder according to the present invention can be implemented. In the embodiment of FIG. 1, the system includes a host central processing unit (CPU) 11 coupled to a graphics processing unit (GPU) 12 via bus 15. This GPU is also referred to as a media processing unit (MPU).

[0034] Both the CPU and the GPU are coupled to the memory 14. In the embodiment of FIG. 1, the memory is a shared memory, which stores instructions and data for both the CPU and the GPU. Alternatively, you can have separate dedicated memory for each of the CPU and GPU. This memory can also include a video buffer for storing the pixel data that drives the combined display (screen) 13.

[0035] The system 10 also includes a user interface 16 including an on-screen cursor controller in one implementation. This user interface can include a keyboard, mouse and / or touch screen device (touch pad).

[0036] Generally speaking, the system 10 includes basic components of a computer system platform that implements the functions according to embodiments of the present invention. The system 10 is, for example, any of a number of different types of computer systems (eg, servers, laptops, desktops and notebooks), as well as home entertainment systems such as set-top boxes or digital televisions (eg, servers, laptops, desktops and notebooks). For example, a DVD player), or a portable or handheld electronic device (eg, a mobile phone, a personal digital assistant, or a handheld game device).

FIG. 2 is a block diagram showing elements of the decoder 20 according to one embodiment of the present invention. Generally, the decoder receives the encoded bitstream 21 and decodes the data in that bitstream to generate displayable image data 24.

[0038] In one embodiment, the raw image data (eg, video data) is also known as MPEG-4 Part 10 or MPEG-4 Advanced Video Coding (AVC), H.264. It is compressed (encoded) using a compression scheme such as. Thus, but not limited to, frame types (eg, intracode I-frames, predictive P-frames or bi-predictive B-frames), macroblock predictive modes (eg, interblock vs. intrablock), transforms (eg, discrete). A set of encoding parameters such as cosine transform) coefficients, texture coefficients and motion vector information is generated. These encoding parameters are then converted to variable length code (VLC) using, for example, Huffman coding. The encoded bitstream is a serialized bitstream containing its VLC.

[0039] The decoder essentially reverses the encoding process so as to reconstruct the image data. In the embodiment of FIG. 2, the decoder includes a decoding pipeline 26 that includes a decoding stage 22 and a decoding stage 23.

The decoding stage performs basic decoding (decompression) operations such as variable length decoding, inverse quantization, inverse transformation and motion compensation. In general, the decoding stage parses the encodered bitstream to extract the VLC and then transforms the VLC to reproduce the encoding parameters described above. These encoding parameters are then used to reconstruct the original video data (more specifically, some variant of the original video data is constructed).

[0041] In the deblocking stage, when the reconstructed image is rendered (displayed), a digital deblocking filter is used to smooth the boundaries between adjacent blocks. Applies to. In the embodiment of FIG. 2, the deblocking stage is independent of the type of decoding performed in the decoding stage and can therefore be categorized as an out-of-loop deblocker.

[0042] FIG. 3 is a flowchart 30 that gives an overview of a computer-implemented deblocking method according to an embodiment of the present invention. In step 31, with reference to FIG. 4, access to the decoded data (pixel values) for the first block B0 and the second block B1. In one embodiment, for the deblocking method described herein, only the Luma plane is used to reduce the number and complexity of calculations. Subjective tests show that when it comes to the perception of blocking artifacts, chroma values are not as important as Luma values, so only Luma values can be used to establish the strength of deblocking filtering. Has been done. However, the present invention is not limited to using only Luma values.

[0043] In step 32 of FIG. 3, an "artifact count" is determined for each of blocks B0 and B1. Generally speaking, the artifact count is a measure of the number of blocking artifacts associated with each block. Further information is given in connection with the description in Figure 4.

[0044] In step 33 of FIG. 3, an "activity" is determined for each of blocks B0 and B1. Generally speaking, activity is a measure of the deviation of pixel values within a block. Further information is given in connection with the description of (A) and (B) of FIG.

[0045] In step 34 of FIG. 3, the artifact count and activity for both blocks are used together to select the deblock filter from the filter bank. More specifically, in one embodiment, the first value N0 for the first block is calculated based on the artifact count and artifacts for the first block, and similarly for the second block. The value N1 of 2 is calculated. These values N0 and N1 are used to set the index value N, which index value is then used to select the deblock filter from the filter bank. Further information is given in connection with the description in Figure 7.

[0046] In step 35 of FIG. 3, the artifact count and activity are also used as the basis for selecting the number of pixel values in blocks B0 and B1 to which the selected deblock filter should be applied. More specifically, in one embodiment, the values N0 and N1 specify a "area of interest" that extends across the boundaries of blocks B0 and B1 and determines the number of pixels to which the selected deblock filter should be applied. Used to do. Further information is given in connection with the description in Figure 7.

[0047] In step 36 of FIG. 3, the selected deblock filter is applied to the area of interest across the boundary between blocks B0 and B1. That is, the selected deblock filter was identified in block B0 and in block B1 so as to generate a corresponding number of new (modified) pixel values in each of those blocks. Applies to a number of pixel values. Further information is given in connection with the description in Figure 7.

[0048] In one embodiment, the method described above is in two paths, namely the horizontal path where the vertical boundary is processed and the vertical path where the horizontal boundary is processed, all the horizontal boundaries of the block in the image frame and Applies to vertical boundaries. These two passes are done sequentially, in which case the result of the first pass (changed pixel value) is stored and used as input for the second pass.

Artistic count With reference to FIG. 4, the image frame 40 includes an array of blocks containing blocks B0 and B1. The frame 40 may be one frame in such a sequence of frames, as in some video.

[0050] In FIG. 4, only eight (8) blocks are shown, but in reality, the image frame contains such a large number of blocks. Blocks B0 and B1 are adjacent to each other, i.e. blocks B0 and B1 share an edge or boundary 41. In addition to block B1, there are many other blocks adjacent to block B0, and similarly, in addition to block B0, there are many other blocks adjacent to block B1.

[0051] In the example embodiment of FIG. 4, each block contains an array of 8x8 pixels. However, the present invention is not limited to this, and other block sizes can be used. Note that for the purposes of this discussion, block boundaries are defined as being between pixels. Not all of the pixels are illustrated in FIG. 4, but only the pixels adjacent to the boundaries of blocks B0 and B1 are shown.

[0052] As described above, according to one embodiment of the invention, the artifact count is determined for each block in the image frame. Generally speaking, the artifact count for a block is the selected pixel value (eg, Luma value) that spans the boundary between that block and its neighbors (four blocks that share an edge with that block of interest). It is determined by thresholding the difference between. Measurements for each block are made over all four boundaries of that block. The artifact count per block is stored and then used with the activity scale to select the deblock filter and the number of pixels to which the filter is applied.

More specifically, in one embodiment, for each pair of adjacent pixels, a pixel value is subtracted, where each pair is one pixel on one side of the block boundary and the block boundary. It consists of pixels that are immediately adjacent to the other side. Using blocks B0 and B1 as an example, a subtraction is made between the pixel value for pixel p0 and the pixel value for pixel q0. Similarly, a subtraction is made between the pixel value for pixel p1 and the pixel value for pixel q1, and all such subtractions are made around each of the four edges of block B0. That is, the artifact count for block B0 is the boundary between block B0 and block B1, the boundary between block B0 and block B2, the boundary between block B0 and block B3, and between block B0 and block B4. Based on the difference in pixel values across the boundaries of.

[0054] Such processing is repeated for each pixel along each of the four edges of block B1 and finally for each block in that frame. The absolute value of the difference between two pixel values is used so that the difference is always a positive number.

[0055] Therefore, for any block BN, diff (BN, i) = | [Pixel value for p (i)]-[Pixel value for q (i)] | (1) Here, pixel p (i) is located at the edge of block BN, and q (i) is a pixel immediately adjacent to the block boundary from pixel p (i). Pixel values for corner pixels are used twice, once in the horizontal pass and once in the vertical pass. Therefore, in the embodiment of FIG. 4, i = 1,2, ....., 32.

[0056] In one embodiment, referring to FIG. 5, each of the differences diffs (BN, i) is then compared to both the first threshold thr1 and the second threshold thr2. Here, the second threshold is greater than the first threshold. In such an embodiment, the second threshold is a constant multiple of the first threshold, i.e. thr2 = n * thr1, where n&gt; 1. Therefore, if the first threshold is increased, so is the second threshold. Because of this relationship between the first and second thresholds, the range between the first and second thresholds is the first threshold. If is increased, it is increased.

[0057] Each of these blocks has a counter associated with it, for example, a first counter associated with block B0 and a second counter associated with block B1. In the embodiment of FIG. 5, for each value of "i", if the value of diff (BN, i) is greater than or equal to the first threshold thr1, the counter associated with the block BN is It is incremented (for example, it is incremented by a value of 1). If the value of diff (BN, i) is less than the first threshold thr1, the counter associated with block BN is incremented by the ratio of diff (BN, i) to thr1. If the value of diff (BN, i) is greater than or equal to the second threshold thr2, the counter associated with the block BN is incremented again (eg, incremented by another value of 1). If the value of diff (BN, i) is less than the second threshold thr2 but greater than the first threshold thr1, the counter associated with the block BN will be diff (BN, i). It is increased by the ratio with thr2. Note that in this embodiment, the counter value is not incremented by more than 2 even if the second threshold thr2 is exceeded.

Therefore, for example, for block B0, Artifact count (B0, i) = min (1, [diff (B0, i) / thr1]) + min [1, diff (B0, i) / thr2]) (2) Artifactcount<u style="single"></u>B0 = Σ (Artifact<u style="single"></u>count (B0, i)) (3) Here, Artifact<u style="single"></u>count (B0, i) is the amount by which the counter value for block B0 is incremented for each value of diff (B0, i), Artifact.<u style="single"></u>count<u style="single"></u>B0 is the final value of the counter for block B0. In this way, the artifact count for block B0 is accumulated, and similarly, the artifact count is accumulated for each block in block B1 and the frame.

[0059] Two separate thresholds are used to better evaluate the visibility to blocking artifacts. If, for example, the difference in pixel values across a block boundary is fairly large, and the area surrounding the block boundary has relatively low frequency content (low activity), then the artifact is It will be more visible and interesting. In such situations, the block edge at that boundary is better treated with a strong lowpass filter with increased length (more taps), as it reduces the artifacts to an acceptable degree. Will. To do this, each difference large enough to exceed both thresholds doubles the increment applied to that counter value. If the activity is low, it can be replaced by choosing a strong filter with increased lowpass filtering capability.

[0060] In one embodiment, the threshold is controlled by the user. That is, the user can specify, for example, the first threshold value thr1. This is because the second threshold, thr2, is a multiple of the first threshold, and the second threshold is also, in a sense, a user-controlled value. The strength of the filter is increased by decreasing the threshold thr1 and vice versa. Thus, the user can establish the strength of deblocking filtering, for example, based on his or her personal preference or the display capability of the user's device. Out-of-loop deblocking is independent of decoding, so the same instance of decoded content can be filtered on the one hand for larger screen sizes or higher resolution displays, and on the other hand. It can be filtered for smaller screen sizes or lower resolution displays.

[0061] In one implementation, the user is provided with a graphic user interface (GUI), such as a slider, that the user can position within any range of values (eg, 0 to 100). Therefore, the user does not necessarily have to select a specific deblocking filter, but instead can select a setting that corresponds to the image quality that the user is satisfied with, and that setting gives the desired image quality. It is mapped to a deblocking filter that causes it. Therefore, the GUI is independent of the number of filters in the filter bank and facilitates the scalability of the invention for different types of GPUs.

Activity calculation [0062] FIG. 6 (A) illustrates the rows of pixels p0, p1, ...., p7 in two adjacent blocks B0 and B1 and block B0, while the figure. (B) of 6 exemplifies an example of the relative value of these pixels. These examples show 8 pixels per row, but the invention is not limited to this.

[0063] As mentioned above, according to one embodiment of the invention, a measure of activity is determined for each of these blocks. More specifically, in one embodiment, the measure of activity is determined for each row of each block and each column of each block.

[0064] In general, activity is determined as a function of the deviation of those selected pixel values from the intermediate values derived from the selected pixel values. In particular, in one embodiment, the intermediate value is a weighted average of the selected pixel values. In one such embodiment, the difference between each of the selected pixel values and their weighted average is also weighted. Thus, in one embodiment, the activity is determined as a weighting function of the deviation of those selected pixel values from the weighted average of the selected pixel values.

[0065] As mentioned above, deblock filtering is achieved using two paths: a horizontal path and a vertical path. In the horizontal path, each row of pixels is processed, i.e. the selected pixel values described above correspond to the rows of pixels. In the vertical path, each column of pixels is processed, i.e. the selected pixel values described above correspond to the columns of pixels.

[0066] In one embodiment, the weighted average (WM) of pixels p0, p1, ...., p7 is determined as follows. WM = (w0 * p0 + w1 * p1 + ......... + w7 * p7) &gt;&gt; 5 (4) Where w0, w1, ...... w7 are the weights selected so that w0&gt; w1&gt; ......&gt; w7 and w0 + w1 + ...... + w7 = 32. And, for the sake of simplicity, p0, p1, ....., and p7 represent pixel values. In general, the weighting factor is greater as the pixels are closer to the target boundary. Therefore, in the example of (A) of FIG. 6, since pixel p0 is closer to the boundary between block B0 and block B1, that pixel is weighted more than the other pixels, and so on.

[0067] In one embodiment, the activities associated with pixels p0, p1, ...., p7 are determined as follows: Activity = a * (p0-WM) + b * (p1-WM) + c * (p2-WM) + d * (p3-WM) + e * (p4-WM) + f * (p5-WM) + g * (p6-WM) + h * (p7-WM) (5) Here, a, b, ......, h are fixed weights such that a&gt; b&gt; ......&gt; h. As mentioned above, these weighting factors are greater as the pixels are closer to the target boundary. In this embodiment, those weights are used to suggest a relevance of deviations from the weighted average, which decreases as the Manhattan distance of the pixels from the block boundary increases.

Selection of filters and target areas To summarize this point, according to the embodiments described above, the artifact count is determined for each block and the activity is determined for each row and each column of each block. This information can be used to select a deblock filter from a filter bank stored in memory and to identify the number of pixels (target area) to which the selected filter should be applied.

[0069] In one embodiment, the artifact count per block and activity per row or column is the first value N0 and the second block (eg, block B0) for each row and column of the first block (eg, block B0). For example, it is used to determine the second value N1 for each row and each column of block B1). For the sake of simplicity, the single line of block B0 and the corresponding line of block B1 will be described below.

[0070] FIG. 7 illustrates the row R0 of block B0 and the corresponding row R1 of block B1, that is, the rows R0 and R1 exist along the same horizontal line.

[0071] The first value N0 for line R0 is determined as follows: N0 = exp (-[Activity<u style="single"></u>R0] / [Artifact<u style="single"></u>count<u style="single"></u>B0)) (6) Here, Activity<u style="single"></u>R0 represents the activity associated with row R0, Artifact<u style="single"></u>count<u style="single"></u>B0 represents the artifact count for block B0. Similarly, the second value N1 is determined as follows. N1 = exp (-[Activity<u style="single"></u>R1] / [Artifact<u style="single"></u>count<u style="single"></u>B1)) (7)

Note that the values N0 and N1 are proportional to the artifact count. On the other hand, the values N0 and N1 are inversely proportional to the activity across the block boundaries.

[0073] According to embodiments of the present invention, one of the values N0 and N1 is used to select a deblock filter from a bank of preselected filters. In one embodiment, the smallest of the values N0 and N1 is used to select the deblock filter from the filter bank. By using the smallest of N0 and N1, the continuity of image detail can be guaranteed. That is, by using the smallest of N0 and N1, the selected filter will be weaker than the filter that would have been selected if the maximum value had been used. Therefore, in this embodiment, image detail is more important than removal of blocking artifacts. However, if the removal of artifacts is considered more important, the largest of N0 and N1 can be used instead.

[0074] In one embodiment, the filter bank comprises seven (7) filters of different intensities. By using filter banks with different intensity filters, the drawbacks associated with hard switching between fixed filtering and filtering decisions can be eliminated.

[0075] In one implementation, the filter bank contains a filter (H) such as, where the indexes N = 0,1, ......, 6. Table 1-Typical filter banks H [N] = {0 0 0 0 128 0 0 0 0} {0 0 0 26 76 26 0 0 0} {0 0 -5 42 54 42 -5 0 0} {0 1 7 33 46 33 7 1 0} {-2 2 15 30 38 30 15 2 -2} {8 8 16 16 32 16 16 8 8} {8 16 16 16 16 16 16 16 8}

[0076] In the example above, the filters in the filter bank are arranged in such an order that the strength of the filter increases as the value of index N increases (as for higher artifact counts). The filter coefficients in Table 1 are examples. Different filters with different specifications can also be designed and used.

[0077] As mentioned above, the values N0 and N1 are proportional to the artifact count. When using the two thresholds thr1 and thr2 as described above, the artifact count is incremented by a factor of 2 when the difference between the pixel values across the block boundaries is relatively large. Therefore, if only the artifact count is considered (ie, the activity is ignored), the higher the artifact count, the higher the values of N0 and N1, which will result in stronger lowpass filtering and blocking artifacts. It will be eliminated or reduced more effectively. On the other hand, the values N0 and N1 are inversely proportional to the activity across the block boundaries. Therefore, if only activity is considered (ie, ignoring the artifact count), the higher the activity, the lower the values of N0 and N1, which makes lowpass filtering weaker and adjacent to the block boundary. It is possible to prevent loss of detail in the area to be used.

[0078] In one embodiment, the selected value of N0 and N1 (eg, the smallest) is rounded to the nearest integer and used as the index N to select one of the filters. To. That is, N = min (N0, N1). For example, if N0 and N1 are found to be 3 and 4, respectively, then N is 3 and the following filter is applied to the area of interest. H [3] = {0 1 7 33 46 33 7 1 0}

[0079] The values of N0 and N1 are also used to determine the area of interest (the number of pixels to which the selected filter is applied). In general, the selected filter is applied to N0 pixels in block B0 and N1 pixels in block B1. Therefore, in the above example, as shown in FIG. 7, the filter H [3] is applied to 3 pixels in block B0 and 4 pixels in block B1. Note that the area of interest can contain a different number of pixels on each side of the block boundary, thus providing a considerable degree of flexibility for the area of interest.

[0080] Embodiments according to the invention are not limited to including seven filters in a filter bank. The number of filters in the filter bank can be reduced to reduce the complexity of the calculation. You can also use a filter that uses fewer taps than a typical filter as shown in the table above. For example, for use on lower end or baseline GPUs, you can probably choose a smaller number of filters with fewer taps. In this way, the overall complexity of filtering can be made scalable by supporting different GPU implementations according to the processing power of the GPU.

[0081] A filter bank can also contain more than eight filters. The number of filters in the filter bank can be increased in practices that do not use the same number N to select both the filter and the area of interest. For example, as mentioned above, the values of N0 and N1 can be rounded to the closest integers for the purpose of defining the area of interest, but using their actual values, eight or more entries. It is also possible to determine the filter bank containing.

The calculated values of N0 and N1 can be greater than the number of pixels in a row or column, and the value of N and the number of filters in a filter bank can be greater than the number of pixels in a row or column. Please note. In such an implementation, the values of N0 and N1 can be restricted (eg, reduced or expanded) to a value corresponding to the number of pixels in the row or column to determine the size of the area of interest. You can use non-scalable values to select filters from the filter bank, and if necessary, use the same or different scaling factors used to size the area of interest. Therefore, those values can be made scalable.

[0083] FIG. 8 is a flowchart 80 of an example of a computer implementation method for processing data according to an embodiment of the present invention. In step 81, the first value (N0) for the first block of pixels (block B0) and the second value (N1) for the second block of pixels (block B1) are calculated. The first value N0 is based on a measure of the difference between the selected pixel value for block B0 and the selected pixel value for blocks adjacent to block B0 containing block B1. In one embodiment, these differences are obtained by subtracting pixel values for adjacent pixel pairs, each of which is a pair associated with a pixel in block B0 and an adjacent pixel in an adjacent block (as in equation (1) above). ). In one such embodiment, the counter value associated with block B0 is incremented each time each of these differences exceeds a first threshold (eg, threshold thr1), and these differences are also Each of the above is incremented again each time it crosses a second threshold (eg, threshold thr2). The counter value is also increased by an equivalent amount if these differences are less than the first threshold or fall between the first and second thresholds. Can be (as in equation (2) above). Generally speaking, the first value N0 is based on the artifact count for block B0.

The first value N0 is also based on a measure of deviation in a subset of pixel values for block B0 (eg, row R0). In one embodiment, a weighted average is calculated for this subset of pixel values (row R0), and then the difference between each pixel value in this subset and its weighted average is determined. In one such embodiment, the difference between these pixel values and the weighted average is also weighted (as in equation (5) above). Generally speaking, the first value N0 is also based on the activity in block B0.

[0085] The second value N1 is also based on the artifact count for block B1 and the activity within block B1.

[0086] In step 82, one of a first value N0 and a second value N1 (N0 or N1) selects a digital deblock filter (H) from a filter bank (eg, Table 1). Used as the index (N) of.

[0087] In step 83, the first value N0 and the second value N1 are the number of pixel values to which the selected deblock filter should be applied (eg, in row R0 of block B0 and in the corresponding row of block B1). Used to identify the number of pixels). That is, the first value N0 and the second value N1 are used to determine the target area (as in the example of FIG. 7 above).

[0088] In step 84 of FIG. 8, the selected deblock filter is applied to the pixel values in block B0 and the pixel values in block B1 to generate new pixel values. That is, the selected deblock filter is applied to the area of interest to eliminate or reduce blocking artifacts.

[0089] In short, according to embodiments of the present invention, a deblocking filter is selected that strikes an acceptable balance between processing blocking artifacts and maintaining a satisfactory level of detail. In addition, the number of pixels to which the selected deblock filter should be applied can be appropriately determined to avoid overfiltering of image detail. Moreover, the deblock filtering according to the invention is adaptive (eg, for the quality of pixel values as measured by artifact count and activity per block) and scalable (eg, the capabilities of graphics cards). On the other hand, it is user-controllable and well adapted for applications such as a wide range of video quality and low bitrate video (eg, internet video) enhancements.

[0090] The above description of a particular embodiment of the invention is presented as an exemplary description. These explanations do not mean that this is all, and that the present invention is not intended to be limited to the form itself disclosed herein, and many modifications and modifications are possible in view of the above. Is what. These embodiments provide the best description of the principles of the invention and its practical application, thereby allowing others skilled in the art to adapt the invention and various embodiments to the particular application as intended. It has been selected and described in order to make various changes to it for best use. The scope of the present invention shall be defined by the scope of claims and their equivalents.

10 ... system, 11 ... host central processing unit (CPU), 12 ... graphic processing unit (GPU), 13 ... display (screen), 14 ... memory, 15 ... bus, 16 ... user interface, 20 ... decoder, 21 ... encoded bitstream, 22 ... decoding stage, 23 ... deblocking stage, 24 ... displayable image data, 26 Decoding pipeline, 30 Flowchart, 40 Image frame, 41 Edge or boundary, 80 Flowchart, B0 ~ B4, BN Block, p0 ~ p7 Pixel, q0 ~ q7 Pixel, R0, R1 Row, N0 First value, N1 Second value, N Index, WM Weighted average, w0 ~ w7 Weight