Method and apparatus for image enhancement for low bit rate video compression

CLAIM OF PRIORITY UNDER 35 U.S.C. §119

0001The present Application for Patent claims priority to Provisional Application No. 60/569,369 entitled “Method and Apparatus for Image Enhancement For Low Bit Rate Video Compression” filed May 6, 2004, and assigned to the assignee hereof and hereby expressly incorporated by reference herein.

BACKGROUND

00021. Field

0003The invention relates to data compression in general and to block-based compression in particular.

00042. Background

0005Transform coding is a common image compression technique that divides an image into sub-images or blocks for processing. Block-based compression introduces artifacts between block boundaries because blocks are independently coded. Therefore, the transform does not take into consideration the correlation between block boundaries. Accordingly, the technique typically results in low bit rate images that are heavily compressed, but contain severe compression artifacts such as blocking, ringing and motion smear.

0006Scalable video coding is acquiring widespread acceptance into low bit rate applications, particularly in heterogeneous networks with varying bandwidths (e.g. Internet and wireless streaming). Scalable video coding enables coded video to be transmitted as multiple layers—typically, a base layer contains the most valuable information and occupies the least bandwidth (lowest bit rate for the video) and enhancement layers offer refinements over the base layer. Most scalable video compression technologies exploit the fact that the human visual system is more forgiving of noise (due to compression) in high frequency regions of the image than the flatter, low frequency regions. Hence, the base layer predominantly contains low frequency information and high frequency information is carried in enhancement layers. When network bandwidth falls short, there is a higher probability of receiving just the base layer of the coded video (no enhancement layers). In such situations, the reconstructed video is blurred and deblocking filters may even accentuate this effect.

0007As a result, several post-processing algorithms for deblocking have been proposed to reduce and/or eliminate compression artifacts. However, many involve complex computations and may result in an overall blurring effect on the output image. Other de-blocking filters do not conserve edge information effectively and are typically complex to implement in hardware.

SUMMARY

0008Methods and apparatus for enhancing image quality through blur reduction by emphasizing the high frequency edges in a decoded base layer, are described. A high pass filter can mark, i.e. highlight, the edges of neighboring 4×4 pixel blocks and then overlay the highlights onto the image. In the case of hardware implementations like Mobile Station Modem (MSM) device platforms used in wireless technologies, the disclosed methods and apparatus can use deblocking filter conditionals and circuitry already existent in video cores to introduce a high frequency enhancement feature with little or no additional cost.

BRIEF DESCRIPTION OF THE DRAWINGS

0009<figref idref="DRAWINGS">FIG. 1</figref> is an illustration of an example of a communication system for delivery of streaming video;

0010<figref idref="DRAWINGS">FIG. 2</figref> is an illustration of pixels adjacent to vertical and horizontal 4×4 block boundaries;

0011<figref idref="DRAWINGS">FIG. 3</figref> is an illustration of one example of an image compressor;

0012<figref idref="DRAWINGS">FIG. 4</figref> is an illustration of one example of an image decompressor that is symmetric to the image compressor; and

0013<figref idref="DRAWINGS">FIG. 5</figref> is a flowchart of one example embodiment to process images.

DETAILED DESCRIPTION

0014In the following description, specific details are given to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, electrical components may be shown in block diagrams in order not to obscure the embodiments in unnecessary detail. In other instances, such components, other structures and techniques may be shown in detail to further explain the embodiments.

0015It is also noted that the embodiments may be described as a process, which is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently and the process can be repeated. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

0016Methods and apparatus to enhance the quality of images compressed using block based compression by emphasizing the high frequency edges in a decoded image, are described. A high frequency emphasis filter can amplify, i.e. enhance, the edges of each image block received with the result that the edges of the image blocks are enhanced. As a result of allowing the low frequencies to pass, areas of constant intensity in the input image are basically unaffected by the high frequency emphasis filter. Areas of a strong intensity gradient, containing the high frequencies, are accentuated or amplified in the filter output. A high pass filter could also be utilized with the output of the high pass filter being added, or overlaid onto the image, thereby enhancing the edges of the image. Since the output of the high pass filter is nearly zero for low frequency areas, areas of constant intensity are not affected noticeably by the overlaid output. The edge enhancement feature is useful in a temporally scalable video delivery system when the base layer is received, but some of the enhancement layer data is not received, or received with errors.

0017<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a general communications system for encoding and decoding streaming pictures. The system <b>100</b> includes encoder device <b>105</b> and decoder device <b>110</b>. Encoder device <b>105</b> further includes transform component <b>115</b>, quantization component <b>120</b>, deblocking <b>125</b>, temporal prediction component <b>130</b>, processor component <b>135</b>, and communications component <b>140</b>. Processor <b>135</b> provides the computing platform to perform the processes of the other components. Transform component <b>115</b> transforms video data from the spatial domain to another domain, such as the frequency domain in the case of DCT (discrete cosine transform). The data that is being transformed can be intra-coded data where the actual video data is transformed, or it can be inter-coded data where a motion vector and residual error are transformed. Other digital transforms include the Hadamard transform, DWT (discrete wavelet transform), and integer transforms such as used in H.264.

0018Quantization component <b>120</b> allocates the number of bits to represent each of the transformed coefficients. The quantization of the transformed coefficients can be changed for every macroblock, where a macroblock is a block of 16×16 pixels (a macroblock is made up of a 16×16 Luma block, and two 8×8 Chroma blocks). The quantization parameter, QP, determines the level of quantization that is performed. More digital compression is realized by increasing QP providing for coarser digital representation of the coefficients, but artifacts such as blurring and blocking can also be introduced. Deblocking component <b>125</b> performs low pass filtering and/or edge enhancement. Temporal prediction component <b>130</b> locates a best matching macroblock in reference frame (or frames) and calculates a motion vector that identifies the best matching macroblock.

0019Transform component <b>115</b> and quantization component <b>120</b> also perform inverse transformation and de-quantization, respectively. These inverse operations are performed in the encoder to allow for reconstruction of the compressed video data such that temporal prediction component <b>130</b> can perform its predictions with the same reconstructed video data as a decoder device that is receiving the encoded data. For the same reason, deblocking component <b>125</b> can perform low pass filtering and/or edge enhancement in encoder device <b>105</b>. The deblocking operation is performed after the dequantization and inverse transforming in the reconstruction process.

0020Communications component <b>140</b> contains logic used to receive data to be encoded from external source <b>145</b>. External source <b>145</b> could be, for example, external memory, the Internet, a live video and/or audio feed, and receiving the data can include wired and/or wireless communications. Communications component <b>140</b> also contains logic to transmit (Tx) encoded data over Network <b>150</b>. Network <b>150</b> can be part of a wired system such as telephone, cable, and fiber optic, or a wireless system. In the case of wireless, communication systems, network <b>140</b> can comprise, for example, part of a code division multiple access (CDMA or CDMA2000) communication system or alternately, the system can be a frequency division multiple access (FDMA) system, a time division multiple access (TDMA) system such as GSM/GPRS (General Packet Radio Service)/EDGE (enhanced data GSM environment) or TETRA (Terrestrial Trunked Radio) mobile telephone technology for the service industry, a wideband code division multiple access (WCDMA), a high data rate (1xEV-DO or 1xEV-DO Gold Multicast) system, or in general any wireless communication system employing a combination of techniques.

0021Decoder device <b>110</b> contains similar components as encoder device <b>105</b> including inverse transform component <b>155</b>, de-quantization component <b>160</b>, deblocking component <b>165</b>, temporal prediction decoding component <b>170</b>, communications component <b>175</b> and processor component <b>180</b>. Decoder device <b>110</b> receives encoded data that has been transmitted over network <b>150</b> or from external storage <b>185</b>. Communications component <b>175</b> contains logic used for receiving (Rx) encoded data in conjunction with network <b>150</b>, as well as logic for receiving encoded data from external storage <b>185</b>. External storage <b>185</b> could be, for example, external RAM or ROM, or a remote server. Intra-coded data is processed by dequantization component <b>160</b>, followed by inverse transform component <b>155</b>, followed by deblocking component <b>165</b> resulting in a decoded picture that can be displayed on display component <b>190</b>. Deblocking component <b>165</b> performs low pass filtering and/or edge enhancement.

0022Inter-coded data can only be decoded after the reference frame(s), from which it was predicted, is decoded. The residual error coefficients are processed by dequantization component <b>160</b>, followed by inverse transform component <b>155</b> followed by deblocking component <b>165</b> resulting in a decoded residual error. The residual error is then combined with a best matching macroblock from the reference frame that was located by temporal prediction decoding component <b>170</b>. Deblocking component <b>165</b> then processes the inter-coded frame, resulting in a decoded inter-coded frame. The decoded frames can be displayed with display component <b>190</b>, stored in external storage <b>185</b> or stored in internal memory of processor component <b>180</b>. Display component <b>190</b> can be an integrated part of the decoding device that contains such parts as video display hardware and logic, including a display screen, or it can be an external peripheral device. Communications component <b>175</b> also contains logic used to communicate the decoded frames to external storage component <b>185</b> or display component <b>190</b>. The low pass filtering and edge enhancement performed by deblocking components <b>125</b> and <b>165</b> are discussed in more detail below.

0023A video encoding standard known as H.264 was jointly developed by the Moving Picture Experts Group (MPEG) and the International Telecommunication Union (ITU) and has been added to the MPEG-4 specification as MPEG-4 Part 10 or Advanced Video Coding (AVC).

0024A de-blocking filter specified in H.264 requires decision trees that determine the activity along block boundaries. As originally designed in H.264, block edges with image activity beyond set thresholds are not filtered or weakly filtered, while those along low activity blocks are strongly filtered. The filters applied can be such, as, for example, 3-tap or 5-tap low pass Finite Impulse Response (FIR) filters.

0025In one example, an edge enhancement method is achieved through high-frequency emphasis using the deblocking filter. Deblocking filter designs for block based video compression predominantly follow a common principle, the measuring of intensity changes along block edges, followed by a determination of strength of the filter to be applied and then by the actual low pass filtering operation across the block edges. The goal of deblocking filters is to reduce blocking artifacts through smoothing (low pass filtering across) of block edges. In order to avoid blurring of edge features in the image, edge activity measurement is performed and low pass filtering is normally applied in non-edge regions (the lower the edge activity measurement in the region, the stronger the filter). Through modifications to the deblocking filter, such as, for example, by changing the coefficients of the filters, the deblocking filter can function in a complimentary fashion and enhance the detected edges using high frequency emphasis filters (in an ideal sense, a high frequency emphasis filter is simply 1+high pass filter).

0026Edge enhancement using a deblocking filter also has applications in temporally scalable coded bitstreams that include a base layer and an enhancement layer. In a condition where the decoder receives the base layer but does not receive all of the enhancement layer(s), the reconstructed image sequence will be blurred and enhancing the edge or activity (using high pass filtering or high frequency emphasis) in the received image could improve the perceptive image quality. Implementing edge enhancement in the de-blocker can be more efficient since the filter decisions are already part of the existing design. Moreover, implementing such a pixel domain post-processing function in a digital signal processor (DSP) platform would consume many valuable MIPS that could be utilized for scalability support in the bitstream, as well as many other quality enhancement functions.

0027An in-loop deblocking filter (included after the encoder's decoder enabling prediction on de-blocked reconstructed frames) can operate on all 4×4 block boundaries and impact every pixel of the reconstructed frame. Although complex (in terms of computations), this deblocking feature is required for compliance to H.264 standards and the widespread adoption and proliferation of the standard necessitates its implementation on software, firmware and hardware platforms depending on the target application. As a result, access to every pixel of the reconstructed video is provided. This enables the ability to apply post-processing functions to reconstructed video such as image enhancement, edge enhancement, error concealment (in video transmission applications over error-prone networks), frame rate up conversion, and more. While, the baseline H.264 standard does not offer scalability modes, extensions to H.264 enable Scalable Video Coding (SVC) where high frequency emphasis filtering for edge enhancement can be applied.

0028<figref idref="DRAWINGS">FIG. 2</figref> is an illustration of pixels adjacent to vertical and horizontal 4×4 block boundaries. Vertical boundary <b>200</b> represents any boundary between two side-by-side 4×4 blocks. Pixels <b>202</b>, <b>204</b>, <b>206</b> and <b>208</b>, labeled p<b>0</b>, p<b>1</b>, p<b>2</b> and p<b>3</b> respectively, lie to the left of vertical boundary <b>200</b> while pixels <b>212</b>, <b>214</b>, <b>216</b> and <b>218</b>, labeled q<b>0</b>, q<b>1</b>, q<b>2</b> and q<b>3</b> respectively, lie to the right of vertical boundary <b>200</b>. Horizontal boundary <b>220</b> represents any boundary between two 4×4 blocks, one directly above the other. Pixels <b>222</b>, <b>224</b>, <b>226</b> and <b>228</b>, labeled p<b>0</b>, p<b>1</b>, p<b>2</b> and p<b>3</b> respectively, lie above horizontal boundary <b>200</b> while pixels <b>232</b>, <b>234</b>, <b>236</b> and <b>238</b>, labeled q<b>0</b>, q<b>1</b>, q<b>2</b> and q<b>3</b> respectively, lie below horizontal boundary <b>200</b>. In the example below, the filtering operations affect up to three pixels on either side of, above or below the boundary. Depending on the quantizer used for transformed coefficients, the coding modes of the blocks (intra or inter coded), and the gradient of image samples across the boundary, several outcomes are possible, ranging from no pixels filtered to filtering pixels p<b>0</b>, p<b>1</b>, p<b>2</b>, q<b>0</b>, q<b>1</b> and q<b>2</b>.

0029In one example, a determination is made that there is sufficient edge activity to perform high frequency emphasis filter edge enhancement, i.e., edge activity above a threshold. If the edge activity is below the threshold, then low pass filtering can be performed. Alternatively, high frequency emphasis filtering can be performed on all edges regardless of activity and then a decision to low pass filter can be made. Alternatively, a second threshold could be used where high frequency emphasis filtering is performed if edge activity is above the second threshold and low pass filtering is performed if edge activity is below the first threshold.

0030H.264 deblocking comprises strong filtering methods used in areas of relatively little activity, and hence very little likelihood of edges, and weak filtering which is used in areas with higher activity an therefore more likelihood of an edge. The following discussion involves only the H.264 weak deblocking filter methods. To determine a maximum threshold, below which low pass filtering is performed by the original H.264 weak deblocking filters, the following variables are first computed to determine the strength of filtering and under which activity conditions each 4×4 block edge under consideration is filtered:

0031Threshold t_Cis determined as follows.

0032for luma: <br /><i>t</i>_C<i>=t</i>_C0+((<i>a</i>_p&lt;β)?1:0)+((<i>a</i>_q&lt;β)?1:0)  [1]

0033for chroma: <br /><i>t</i>_C<i>=t</i>_C0+1  [2]<br />where <i>a</i>_p<i>=Abs </i>(<i>p</i>₂<i>−p</i>₀) and <i>a</i>_q<i>=Abs </i>(<i>q</i>₂<i>−q</i>₀)  [3]

0034In H.264, the variable t_C0is the activity threshold based on bS (boundary strength) and indexA, (indexA is an offset from quantization parameter QP, where QP varies from 0 to 51). The activity measurements a_pand a_q, given in equation [3] above, increment t_COby 1 each when they are less than β (see equation [1] above). The larger β, the more likely that an edge is present and the less likely that low pass filtering will be performed. Boundary strength is determined for each 4×4 block edge and ranges from zero to 4. Details of boundary strength determination are known to those of ordinary skill in the art and are not necessary to understand the disclosed method. Once the boundary strengths are derived, indexA values are determined based on the average QP's of the neighboring blocks. Since QP changes on a macroblock basis and is different for luma and chroma, 3 values of luma and chroma average QP (and hence 6 indexA's) are computed: 1) along the leftmost macroblock edge, 2) along the topmost macro block edge and 3) for all 4×4 block edges internal to the macroblock. Once the indexA values are calculated, the values for β are determined in a look-up table and the values for β range from zero to 18. The threshold t_cis then computed in equations [1] and [2] above and the low pass filtering parameter Delta is computed:

0035Delta: <br />Δ=Clip3(−<i>t</i>_C<i>,t</i>_C,((((<i>q</i>₀<i>−p</i>₀)<<2)+(<i>p</i>₁<i>−q</i>₁)+4)>>3))  [4]

0036Low pass Filtering of p_i,q_i, i=0,1,2 for both luma and chroma occurs as follows:

0037<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mtable><mtr><mtd><mrow><msubsup><mi>p</mi><mn>0</mn><mi>′</mi></msubsup><mo>=</mo><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>1</mn></mrow><mo>⁡</mo><mrow><mo>(</mo><mrow><msub><mi>p</mi><mn>0</mn></msub><mo>+</mo><mi>Δ</mi></mrow><mo>)</mo></mrow></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>5</mn><mo>]</mo></mrow></mtd></mtr><mtr><mtd><mrow><msubsup><mi>q</mi><mn>0</mn><mi>′</mi></msubsup><mo>=</mo><mrow><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>1</mn></mrow><mo>⁡</mo><mrow><mo>(</mo><mrow><msub><mi>q</mi><mn>0</mn></msub><mo>-</mo><mi>Δ</mi></mrow><mo>)</mo></mrow></mrow><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>where</mi></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>6</mn><mo>]</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>1</mn></mrow><mo>⁡</mo><mrow><mo>(</mo><mi>x</mi><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>3</mn></mrow><mo>⁡</mo><mrow><mo>(</mo><mrow><mn>0</mn><mo>,</mo><mn>255</mn><mo>,</mo><mi>x</mi></mrow><mo>)</mo></mrow></mrow><mo>⁢</mo><mstyle><mspace width="0.8em" height="0.8ex" /></mstyle><mo>⁢</mo><mi>where</mi></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>7</mn><mo>]</mo></mrow></mtd></mtr><mtr><mtd><mrow><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>3</mn></mrow><mo>⁡</mo><mrow><mo>(</mo><mrow><mi>x</mi><mo>,</mo><mi>y</mi><mo>,</mo><mi>z</mi></mrow><mo>)</mo></mrow></mrow><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mrow><mi>x</mi><mo>;</mo><mrow><mi>z</mi><mo><</mo><mi>x</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mi>y</mi><mo>;</mo><mrow><mi>z</mi><mo>></mo><mi>y</mi></mrow></mrow></mtd></mtr><mtr><mtd><mrow><mi>z</mi><mo>;</mo><mi>otherwise</mi></mrow></mtd></mtr></mtable></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>8</mn><mo>]</mo></mrow></mtd></mtr></mtable></math></maths>

0038The low pass filtering of equations [5] and [6] above is only the first low pass filtering in the H.264 deblocking filter. In addition to this low pass filtering, a second type of low pass filtering is also performed. In this example, it is in this second type of low pass filtering where the disclosed method is implemented. The filtering for Luma in equations [9], [10], [11] and [12] below depends on edge activity measurements a_pand a_qand filters only pixels p<b>1</b> and q<b>1</b> (see pixels <b>204</b>, <b>214</b>, <b>224</b> and <b>234</b> of <figref idref="DRAWINGS">FIG. 2</figref>) and includes low pass filtering when the edge activity measurements are below a threshold β and high frequency emphasis filtering when the edge activity measurements are above the threshold:

0039If (a_p<β), low pass filter as follows: <br /><i>p′</i>₁<i>=p</i>₁+Clip3(−<i>t</i>_C0<i>, t</i>_C0, (<i>p</i>₂+((<i>p</i>₀<i>+q</i>₀+1)>>1)−(<i>p</i>₁<<1))>>1)  [9]

0040else, high pass filter as follows: <br /><i>p′</i>₁<i>=p</i>₁+Clip3(−<i>t</i>_C0<i>, t</i>_C0, ((<i>p</i>₂>>1)−((<i>p</i>₀<i>+q</i>₀+1)>>1)+(<i>p</i>₁>>1))>>1)  [10]

0041If (a_q<β), low pass filter as follows: <br /><i>q′</i>₁<i>=q</i>₁+Clip3(−<i>t</i>_C0<i>, t</i>_C0, (<i>q</i>₂+((<i>p</i>₀<i>+q</i>₀+1)>>1)−(<i>q</i>₁<<1))>>1)  [11]

0042else, high pass filter as follows: <br /><i>q′</i>₁<i>=q</i>₁+Clip3(−<i>t</i>_C0<i>, t</i>_C0, ((<i>q</i>₂>>1)−((<i>p</i>₀<i>+q</i>₀+1)>>1)+(<i>q</i>₁>>1))>>1)  [12]

0043The filtering of the p<b>1</b> and q<b>1</b> pixels was found to offer satisfactory edge enhancement without producing artifacts such as aliasing. It was found that filtering pixels p<b>0</b> and q<b>0</b> resulted in aliasing and/or blocking artifacts.

0044To summarize, when the activity along the block edge is weak (below β), the edge is low pass filtered as follows:

0045<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>p</mi><mn>1</mn></msub><mo>=</mo><mrow><msub><mi>p</mi><mn>1</mn></msub><mo>+</mo><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>3</mn></mrow><mo>(</mo><mrow><mrow><mo>-</mo><msub><mi>tc</mi><mn>0</mn></msub></mrow><mo>,</mo><msub><mi>tc</mi><mn>0</mn></msub><mo>,</mo><mrow><mo>[</mo><mrow><mfrac><msub><mi>p</mi><mn>2</mn></msub><mn>2</mn></mfrac><mo>+</mo><mfrac><mrow><msub><mi>p</mi><mn>0</mn></msub><mo>+</mo><msub><mi>q</mi><mn>0</mn></msub></mrow><mn>4</mn></mfrac><mo>-</mo><msub><mi>p</mi><mn>1</mn></msub></mrow><mo>]</mo></mrow></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>13</mn><mo>]</mo></mrow></mtd></mtr></mtable></math></maths>

0046When the activity is above β, the deblocking filtering is changed to high frequency emphasis as follows:

0047<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mtable><mtr><mtd><mrow><msub><mi>p</mi><mn>1</mn></msub><mo>=</mo><mrow><msub><mi>p</mi><mn>1</mn></msub><mo>+</mo><mrow><mrow><mi>Clip</mi><mo>⁢</mo><mn>3</mn></mrow><mo>(</mo><mrow><mrow><mo>-</mo><msub><mi>tc</mi><mn>0</mn></msub></mrow><mo>,</mo><msub><mi>tc</mi><mn>0</mn></msub><mo>,</mo><mrow><mo>[</mo><mrow><mfrac><msub><mi>p</mi><mn>2</mn></msub><mn>4</mn></mfrac><mo>-</mo><mrow><mo>(</mo><mfrac><mrow><msub><mi>p</mi><mn>0</mn></msub><mo>+</mo><msub><mi>q</mi><mn>0</mn></msub></mrow><mn>4</mn></mfrac><mo>)</mo></mrow><mo>+</mo><mfrac><msub><mi>p</mi><mn>1</mn></msub><mn>4</mn></mfrac></mrow><mo>]</mo></mrow></mrow></mrow></mrow></mrow></mtd><mtd><mrow><mo>[</mo><mn>14</mn><mo>]</mo></mrow></mtd></mtr></mtable></math></maths>

0048In an example embodiment, the high pass filtering operation can be enabled using a determined boundary strength (bS) value, such as, for example, 7 (where 7 is an arbitrary value that does not correspond to any of the H.264 standardized values of bS from zero to 4) such that the high pass filtering operation is activated by the decoder when only the base layer and no enhancement layer is received. Since 3-bits are required to represent the value of bS, any value from 5 thru 7 can be used to indicate that a given 4×4 block edge is required to be enhanced or edge enhancement should be turned on. As discussed above, a second threshold (β<b>2</b> for example) could be used where high frequency emphasis filtering is performed if edge activity is above second threshold β<b>2</b> and low pass filtering is performed if edge activity is below the first threshold β.

0049<figref idref="DRAWINGS">FIGS. 3</figref>, <b>4</b> and <b>5</b> show flow charts illustrating examples of deblocking processes including edge enhancement. The processes illustrated in <figref idref="DRAWINGS">FIGS. 3</figref>, <b>4</b> and <b>5</b> can be performed by devices such as encoder device <b>105</b> and decoder device <b>110</b> of <figref idref="DRAWINGS">FIG. 1</figref>. Process <b>300</b> in <figref idref="DRAWINGS">FIG. 3</figref> utilizes a single threshold similar to an example discussed above and contained in equations [9], [10], [11] and [12]. Deblocking process <b>300</b> can be performed in raster order as macroblocks are decoded or postponed until an entire frame is decoded. A processor such as processor components <b>135</b> and <b>180</b> in <figref idref="DRAWINGS">FIG. 1</figref> accesses from memory, step <b>305</b>, 16×16 pixel macroblocks that have been received, dequantized, and inverse transformed. Receiving means such as communications components <b>140</b> and <b>175</b> in <figref idref="DRAWINGS">FIG. 1</figref>, along with neighboring block determining means such as deblocking components <b>125</b> and <b>165</b> in <figref idref="DRAWINGS">FIG. 1</figref> can perform step <b>305</b>. The edge activity along each edge of each pair of neighboring 4×4 blocks is measured, step <b>310</b>. Edge activity can be measured in a way such as, for example, that shown for a_pand a_qin equation [3] above. Measuring means such as deblocking components <b>125</b> and <b>165</b> in <figref idref="DRAWINGS">FIG. 1</figref> can perform step <b>310</b>. If it is determined that edge activity is above a threshold, step <b>315</b>, then edge enhancement is performed, step <b>320</b>. Edge enhancement can be performed with high frequency emphasis filtering such as in equations [10] and [12] above, or by employing a high pass filter and overlaying (adding) the output of the high pass filter onto the enhanced pixels. Enhancing means such as deblocking components <b>125</b> and <b>165</b> in <figref idref="DRAWINGS">FIG. 1</figref> can perform step <b>320</b>. Comparing means such as deblocking components <b>125</b> and <b>165</b> can perform step <b>315</b>. If, in step <b>315</b>, it is determined that the edge activity is not greater than the threshold, then the pixels are low pass filtered, step <b>325</b>. Low pass filtering can be performed with low pass filtering such as equations [5], [6], [9] and [11] above. After filtering, a 4×4 block is stored into memory, step <b>330</b>. Steps <b>305</b> through <b>330</b> are repeated until an entire frame is low pass filtered and/or high frequency emphasis filtered, at which time the frame can be stored to memory or displayed, step <b>330</b>. Steps <b>305</b> through <b>330</b> are repeated until the encoded bitstream is completed.

0050<figref idref="DRAWINGS">FIGS. 4 and 5</figref> show variations of process <b>300</b>. Process <b>400</b> in <figref idref="DRAWINGS">FIG. 4</figref> performs edge enhancement by high frequency emphasis filtering of each boundary between 4×4 neighboring blocks regardless of the edge activity measurement, step <b>415</b>. If the edge activity, as measured in step <b>410</b>, is at or less than the threshold, step <b>420</b>, then the neighboring blocks are low pass filtered. Process <b>500</b> in <figref idref="DRAWINGS">FIG. 5</figref> utilizes two thresholds. If edge activity is less than the first threshold, step <b>515</b>, then the neighboring blocks are low pass filtered, step <b>520</b>. This threshold could be the standard β threshold as discussed above and presented in equations [9], [10], [11 ] and [12]. If the edge activity is at or higher than the first threshold, it is compared to a second threshold, step <b>525</b>. If the edge activity is above the second threshold, then the pixels of the neighboring blocks are edge enhanced with high frequency emphasis filtering, step <b>530</b>. If step <b>525</b> determines that the edge activity is at or less than the second threshold, then no filtering is performed.

0051It is to be appreciated that the described methods and apparatus can be practiced with modifications on existing architecture of the video core in MSMs, these modifications can be extended to generic hardware implementations of this feature.

0052In the description above, a slice can represent a set of consecutive scan lines other than 16 consecutive scan lines. Also, the data block may be an n×m block, where n is not equal to m, if the block can be subdivided. Moreover, a different color space with the same or different number of color components may be used to represent an image pixel. However, a block size of 16×16 pixels and the YCbCr color space will be used below for purposes of explanation.

0053As discussed above, deblocking components <b>125</b> and <b>165</b> in <figref idref="DRAWINGS">FIG. 1</figref> can choose, as the system allows, different filters depending on the characteristics of the neighboring blocks. Thus the deblocking components <b>125</b> and <b>165</b> can comprise one or more types of filters such as, but not limited to, an FIR filter, an averaging filter, a Gaussian filter, and/or a polyphase filter.

0054Furthermore, as discussed above, the transform components <b>115</b> and <b>155</b> of <figref idref="DRAWINGS">FIG. 1</figref> can comprise different types of transforms other than the H.264 integer transform as discussed above. The H.264 integer transform operates on 4×4 pixel blocks. Thus, each 4×4 block needs to be deblocked and or edge enhanced. Another type of transform is the ABSDCT (ABSolute DCT) technique. Compression techniques using ABSDCT will next be described using a block size of 16×16 pixels. Generally, each of the luminance and chrominance components is passed to a block interleaver. In one embodiment, a 16×16 block is presented to the block interleaver, which orders the image samples within the 16×16 blocks to produce blocks and composite sub-blocks of data for DCT analysis. One −16×16 DCT is applied to a first ordering, four −8×8 DCTs are applied to a second ordering, 16−4×4 DCTs are applied to a third ordering, and 64−2×2 DCTs are applied to a fourth ordering. The DCT operation reduces the spatial redundancy inherent in the image source. After the DCT is performed, most of the image signal energy tends to be concentrated in a few DCT coefficients.

0055For the 16×16 block and each sub-block, the transformed coefficients are analyzed to determine the number of bits required to encode the block or sub-block. Then, the block or the combination of sub-blocks that requires the least number of bits to encode is chosen to represent the image segment. For example, two −8×8 sub-blocks, six −4×4 sub-blocks, and eight −2×2 sub-blocks may be chosen to represent the image segment. The chosen block or combination of sub-blocks is then properly arranged in order.

0056For the 16×16 data block, the ABSDCT technique generates data known as PQR information that represents the block size assignment within the 16×16 block. The PQR information is a variable bit width data and describes to what extent a 16×16 block is subdivided. The R-bit of the PQR field represents whether the 16×16 block is subdivided into four −8×8 blocks.

0057The additional four bits are referred to as ‘Q’ information. Each bit of Q denotes a subdivision of an 8×8 block into four −4×4 blocks. For each bit of Q that is set, four more bits of ‘P’ are present to indicate if any of the 4×4 blocks are subdivided into 2×2. Accordingly, the length of PQR data can be 1 to 21 bits long, depending on the block size assignment within the 16×16 block. Thus, each block may be divided into sub-blocks of sizes 8×8, 4×4, and/or 2×2 depending on the assignment criterion.

0058Thus, the PQR information may be generated and used for determing which block sizes to deblock (low pass filtering and/or high frequency emphasis filtering). The PQR information is used to determine edge content in the image. The greater the edge information in a block, the smaller the block size and the longer the PQR code.

0059Examples of the methods and apparatus described above include:

0060A method for deblocking and edge enhancement of block based digital video that comprises: determining that two blocks share a common boundary, measuring edge activity of the common boundary, comparing the measured edge activity to a first threshold, filtering one or more pixels of the two blocks with a low pass filter if the edge activity is below the first threshold, and enhancing one or more pixels of the two neighboring blocks if the edge activity is at or above the first threshold.

0061An apparatus for deblocking and edge enhancement of block based digital video, that comprises: means for determining that two blocks share a common boundary, means for measuring edge activity of the common boundary, means for comparing the measured edge activity to a first threshold, means for filtering one or more pixels of the two blocks with a low pass filter if the edge activity is below the first threshold, and means for enhancing one or more pixels of the two neighboring blocks if the edge activity is at or above the first threshold.

0062An electronic device for deblocking and edge enhancement of block based digital video, the electronic device configured to determine that two blocks share a common boundary, to measure edge activity of the common boundary, to compare the measured edge activity to a first threshold, to filter one or more pixels of the two blocks with a low pass filter if the edge activity is below the first threshold, and to enhance one or more pixels of the two neighboring blocks if the edge activity is at or above the first threshold.

0063A computer-readable medium embodying means for causing a computer to execute a method for deblocking and edge enhancement of block based digital video, comprising: determining that two blocks share a common boundary, measuring edge activity of the common boundary, comparing the measured edge activity to a first threshold, filtering one or more pixels of the two blocks with a low pass filter if the edge activity is below the first threshold, and enhancing one or more pixels of the two neighboring blocks if the edge activity is at or above the first threshold.

0064Those of ordinary skill in the art would understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

0065Those of ordinary skill would further appreciate that the various illustrative logical blocks, modules, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed methods.

0066The various illustrative logical blocks, modules, and circuits described in connection with the examples disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

0067The steps of a method or algorithm described in connection with the examples disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuit (ASIC). The ASIC may reside in a wireless modem. In the alternative, the processor and the storage medium may reside as discrete components in the wireless modem.

0068The previous description of the disclosed examples is provided to enable any person of ordinary skill in the art to make or use the disclosed methods and apparatus. Various modifications to these examples will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other examples without departing from the spirit or scope of the disclosed method and apparatus.

0069Thus, a method and apparatus for deblocking and edge enhancement of a block base video including low pass filtering and high frequency edge enhancement based on measured edge activity levels of block boundaries have been described.