High precision up-sampling in scalable coding of high bit-depth video

High-precision upsampling in the adaptive coding of high-bit-depth video

High precision up-sampling in scalable coding of high bit-depth video

[Cross-reference related applications]

This application claims the priority of U.S. Provisional Patent Application No. 61/745,050 filed on 2012/12/21, which is incorporated herein by reference in its entirety.

The present invention generally relates to images. In particular, the embodiments of the present invention relate to high-precision upsampling in an adaptive video codec for high-bit-depth video.

Audio and video compression is a key element in the development, storage, distribution, and consumption of multimedia content. The choice of compression method involves a trade-off between coding efficiency, coding complexity, and delay. As the ratio of processing power to computational cost increases, it allows the development of more complex compression techniques that can compress more efficiently. As an example, in video compression, the Motion Graphics Expert Group (MPEG) from the International Standards Organization (ISO) has published MPEG-2, MPEG-4 (Part 2), and H.264/AVC (Or MPEG-4, Part 10) coding standard to continue to improve the original MPEG-1 video standard.

Despite the compression efficiency and success of H.264, a new generation of video compression technology called "High Efficiency Video Coding (HEVC)" is currently under development. It was designed in October 2012 by B.Bross, WJ.Han, GJSullivan, J.-R.Ohm, and T.Wiegand's "High efficiency video coding (HEVC) text specification draft 9 (High efficiency video coding (HEVC) )text specification draft 9)" ITU-T/ISO/IEC Joint Collaborative Team Video Coding (JCT-VC) document JCTVC-K1003 (which is incorporated into this article by reference to its entire content) HEVC is expected to be Through the existing H .264 (also known as AVC) standard to provide improved compression capabilities.

Video signals can be characterized by multiple parameters, such as bit depth, color space, color gamut, and resolution. Modern televisions and video recording and playback devices (for example, Blu-ray players) support various resolutions, including standard definition (for example, 720×480i) and high definition (HD) (for example, 1090×1080p). Ultra High Definition (UHD) is a next-generation resolution format with a resolution of at least 3,840×2,160. Ultra-high definition can also be referred to as Ultra HD, UHDTV, or ultra-high field of view. As used herein, UHD means any resolution higher than HD resolution.

Another aspect of the characteristics of a video signal is its dynamic range. Surrounding. The dynamic range (DR) is the range of the intensity (for example, luminosity, brightness) in the image, for example, from the darkest dark color to the brightest bright color. As used herein, the term "dynamic range" (DR) can be related to the ability of the human mental visual system (HVS) to perceive the intensity (e.g., luminosity, brightness) in an image, for example, from the darkest dark to The brightest bright color. In this sense, DR is about the intensity of "scene-referred". DR can also be related to the ability of the display device to fully or roughly present a specific width of intensity range. In this sense, DR is about the intensity of "display-referred". Unless a specific meaning is clearly indicated to have a particularly important meaning at any point in the description herein, it should be inferred that the term can be used in any meaning (eg, interchangeably).

As used herein, the term high dynamic range (HDR) relates to some 14-15 orders of magnitude DR width of the spanned human visual system (HVS). For example, a human being with substantially normal well-adapted (e.g., in terms of one or more of statistics, biometrics, or ophthalmology) has an intensity range spanning about 15 orders of magnitude. Adapted humans can perceive dim light sources as few as a few photons. However, these same humans in the desert, ocean or snow can perceive the bright intensity of the suns approaching pain at noon (or even a quick glance at the sun, but very briefly to prevent damage). Although this span is used for "adapted" humans, for example, those humans whose HVS has a time period for resetting and adjusting.

Conversely, the DR through which humans can simultaneously perceive the extended width of the intensity range may be slightly shorter than that of HDR. As used in this article In use, the terms "enhanced dynamic range" (EDR), "visual dynamic range", or "variable dynamic range" (VDR) can be used individually or interchangeably to refer to DR that can be sensed synchronously by the HVS. As used herein, EDR can be related to DR with a span of 5-6 orders of magnitude. Therefore, although perhaps slightly narrower than HDR involving real situations, EDR still represents a broad DR width. As used herein, the term "synchronized dynamic range" may be related to EDR.

As used herein, the term "bit depth" of an image or video refers to the number of bits used to represent or store the pixel value of the color element of the image or video signal. For example, the term N-bit video (for example, N=8) represents the pixel value of the color element (for example, R, G, or B), because the video signal can be obtained in the range of 0 to 2.^NValue in the range of -1.

As used herein, the term "high bit depth" means any bit depth value greater than 8 bits (for example, N=10 bits). Please note that although HDR images and video signals are usually associated with high bit depth, high bit depth images may not necessarily have high dynamic range. Therefore, as used herein, high bit depth imaging can be associated with both HDR and SDR imagery.

To support backward compatibility with traditional recording and playback devices and new display technologies, multiple layers can be used to transmit UHD and HDR (or SDR) video data from upstream devices to downstream devices. Given the above-mentioned multi-layer stream, traditional decoders can use the base layer to reconstruct the content version of the HD SDR. The advanced decoder can use both the base layer and the enhancement layer to reconstruct the UHD EDR content version to present it on a more compatible display. As posted here Ming people understand that it is hoped to use an adaptive codec to encode high-bit-depth video improved technology.

The methods described in this section are methods that can be implemented, but not necessarily methods that have been previously conceived or implemented. Therefore, unless otherwise stated, it should not be assumed that any method described in this section is suitable as a conventional technology only because it is included in this section. Likewise, problems identified for one or more methods should not be assumed to have been identified in any prior art based on this section, unless otherwise stated.

<p>100Coding System</p><p>102Enhancement layer input</p><p>104Basic layer input</p><p>105BL encoder</p><p>107BL bit stream</p><p>110BL decoder</p><p>112Signal</p><p>115Interlayer processing unit</p><p>120Inter-layer prediction program</p><p>127 Residual value</p><p>130EL encoder</p><p>132EL bit stream</p><p>210Interlayer structure prediction</p><p>220liter sampling program</p><p>305-320Step</p>

The embodiments of the present invention are illustrated in the figures of the accompanying drawings by way of example rather than limitation, and the same reference numerals refer to similar elements, and among them: Figure 1 depicts the adaptable coding according to the embodiment of the present invention Exemplary implementation of the system; Figure 2 depicts an exemplary implementation of an adaptive decoding system according to an embodiment of the present invention; Figure 3 depicts an exemplary procedure for upsampling of image data according to an embodiment of the present invention.

This article describes the high-precision up-sampling in the adaptive coding of video input with high bit depth. Determine the given parameters related to the bit depth of the intermediate result, the internal input bit depth, and the filter accuracy bit Meta depth, scaling and rounding factors to maintain the accuracy of the operation and prevent overflow.

In the following description, for illustrative purposes, many specific details are presented to provide a comprehensive understanding of the embodiments of the present invention. However, it will be clearly understood that the present invention can be implemented without these specific details. In other cases, well-known structures and devices are not described in detail so as not to unnecessarily obscure the present invention.

summary

The exemplary embodiments described herein are about high-precision upsampling in the hierarchical encoding and decoding of video signals with high bit depth. In response to the bit depth requirements of the video encoding or decoding system, the input data, filter coefficients, and scaling and rounding parameters are determined for a separable amplification filter. First, a first rounding parameter is used to filter the input data across a first spatial direction to generate the first upsampled data. The first intermediate data is generated by scaling the first up-sampled data using a first offset parameter. A second rounding parameter is then used to filter the intermediate data across a second spatial direction to generate second upsampled data. The second intermediate data is generated by scaling the second up-sampled data using a second offset parameter. The final up-sampled data can be generated by editing the second intermediate data.

High-precision separable upsampling

Existing display and recording devices (such as HDTV, set-top boxes, or Blu-ray players) usually support up to 1080p HD resolution (e.g. For example, 60 frames per second (1920×1080) signal. For consumer applications, this type of signal is now usually compressed with a bit depth of 8 bits per pixel per color element in a luma-chroma color format, where chroma elements usually have a lower resolution than luma elements (e.g. : YCbCr or YUV 4:2:0 color format). Due to the 8-bit depth and the corresponding low dynamic range, this type of signal is usually referred to as a signal with a standard dynamic range (SDR).

As new television standards are being developed, such as ultra-high-resolution (UHD), it may be desirable to encode signals with increased resolution and/or a higher bit depth in an adaptable format.

Figure 1 depicts an exemplary implementation of an adaptive coding system. In an exemplary embodiment, the base layer (BL) input signal 104 may represent an HD SDR signal and the enhancement layer (EL) input 102 may represent a UHD HDR (or SDR) signal at a high bit depth. The BL encoder 105 is used to compress (or encode) the BL input 104 to generate an encoded BL bitstream 107. The BL encoder 105 can use any of known or future video compression algorithms (such as MPEG-2, MPEG-4, part2, H.264, HEVC, VP8, etc.) to compress or encode the BL input signal 104.

Given a BL input 104, the encoding system 100 not only generates an encoded BL bit stream 107, but also generates a BL signal 112 that represents the BL signal 107 when it will be decoded by the corresponding receiver. In some embodiments, the signal 112 may be generated by a separate BL decoder (110) after the BL encoder 105. In some other embodiments, the signal 112 may be generated from a feedback loop used to perform motion compensation in the BL encoder 105. As shown in Figure 1 As shown, the signal 112 can be processed by the inter-layer processing unit 115 to generate a signal that will be suitable for use by the inter-layer prediction program 120. In some embodiments, the inter-layer processing unit 115 may amplify the signal 112 to conform to the spatial resolution of the EL input 102 (for example, from HD resolution to UHD resolution). After the inter-layer prediction 120, the residual value 127 is calculated, which is then encoded by the EL encoder 130 to generate an encoded EL bitstream 132. The BL bitstream 107 and the EL bitstream 132 are usually multiplexed into a single coded bitstream for transmission to the appropriate receiver.

The term SHVC refers to the adaptive extension of a new generation of video compression technology called High Efficiency Video Coding (HEVC) [1], which enables substantially higher compression capabilities than the existing AVC (H.264) standard [2]. SHVC is currently being jointly developed by the ISO/IEC MPEG and ITU-T WP3/16 teams. One of the key aspects of SHVC is spatial scalability, where inter-layer structure prediction (for example, 120 or 210) provides the most significant gain. An example of the SHVC decoder is shown in Figure 2. As part of the inter-layer prediction, the up-sampling process (220) up-sampling or up-converting the pixel data from the base layer (215) to match the pixel resolution of the data (e.g., 202 or 230) received in the enhancement layer. In one embodiment, the up-sampling process can be performed by applying an up-sampling or interpolation filter. In the adaptable extension of H.264 (SVC) or SHVC SMuC0.1.1 software [3], a separable multi-phase upsampling/interpolation filter is applied. Although the above filters work well with input data with standard bit depth (for example, an image with 8 bits per pixel per color element), they work well for input data with high bit depth (for example, each color element uses per pixel 10 For images with more than bits), it may overflow.

In 2D upsampling or interpolation procedures, it is common practice to apply separable filters to reduce processing complexity. The aforementioned filters first upsample the image in one spatial direction (for example, horizontal or vertical) and then in another direction (for example, vertical or horizontal). Without loss of generality, in the following description, it is assumed that the vertical upsampling is after the horizontal upsampling. Then, the filtering procedure can be described as:

Horizontal upsampling:

tempArray[x,y]=Σ_i,j(eF[xPhase,i]*refSampleArray[xRef+j,y]) (1)

Vertical upsampling

predArray[x,y]=Clip((Σ_i,j(eF[yPhase,i]*tempArray[x,yRef+j])+offset)&gt;&gt;nshift) (2)

Where eF stores the coefficients of the multi-phase upsampling filter, refSampleArray contains the reference sample values from the reconstruction base layer, tempArray stores the intermediate value after the first 1-D filtering, and predArray stores the final value after the second 1D filtering, xRef and yRef corresponds to the relative pixel position used for upsampling, nshift represents a scaling or normalization parameter, offset represents a rounding parameter, and Clip() represents a clipping function. For example, in the exemplary embodiment, given data x and critical values A and B, the function y=Clip(x,A,B) represents<maths><img he="238" wi="629" file="TWI618397B_D0001.tif" img-content="drawing" img-format="tif" orientation="portrait" inline="no" /></maths>

For example, for N-bit image data, the exemplary values of A and B may include A=0 and B=2^N-1。

In equation (2), by shifting the binary representation of b by c bits to the right, the operation a=b&gt;&gt;c means that b is divided by 2.^c(E.g. a=b/2^c). Please note that in equation (1), for the first stage of filtering, no clipping or offset operations are applied. Please also note that in this implementation, the order of horizontal and vertical filtering is not important. Applying vertical filtration first and then horizontal filtration produces the same results as applying horizontal filtration first and then vertical filtration.

In SMuC0.01[3], the filter accuracy of eF (represented as US_FILTER_PREC) is set to 6 bits. When the internal bit depth of refSampleArray is 8 bits, tempArray can be kept within the target implementation bit depth (for example, 14 or 16 bits). However, when the internal bit depth of refSampleArray exceeds 8 bits (for example, 10 bits), the output of equation (1) may overflow.

In one embodiment, the above-mentioned overflow can be prevented by (a) fixing the order of operations in the up-sampling procedure, and (b) by combining intermediate scaling operations. In one embodiment, when horizontal filtering is before vertical filtering, upsampling can be implemented as follows:

Horizontal upsampling:

tempArray[x,y]=(Σ_i,j(eF[xPhase,i]*refSampleArray[xRef+j,y]+ioffset1))&gt;&gt;nShift1 (3)

Vertical upsampling

predArray[x,y]=Clip((Σ_i,j(eF[yPhase,i]*tempArray[x,yRef+j])+ioffset2)&gt;&gt;nShift2), (4)

Without loss of generality, let INTERM_BITDEPTH represent the bit depth (or bit resolution) requirement for the intermediate filter processing; that is, the result cannot be expressed in more bits than INTERM_BITDEPTH (for example, INTERM_BITDEPTH=16). Let INTERNAL_INPUT_BITDEPTH denote the bit depth used to represent the input video signal in the processor. Please note that INTERNAL_INPUT_BITDEPTH can be equal to or greater than the original bit depth of the input signal. For example, in some embodiments, INTERNAL_INPUT_BITDEPTH=10 can be used to internally represent 8-bit input video data. In addition, in another example, INTERNAL_INPUT_BITDEPTH=14 can be used to represent 14-bit input video.

In an embodiment, the scaling parameters in equations (3) and (4) can be calculated as nShift1=(US_FILTER_PREC+INTERNAL_INPUT_BITDEPTH)-INTERM_BITDEPTH, (5)

nShift2=2* US_FILTER_PREC-nShift1. (6)

In an embodiment, the values of nShift1 and nShift2 may not be allowed to be negative numbers. For example, a negative value for nShift1 indicates that the bit resolution allowed for intermediate results is sufficient to prevent overflow; therefore, when it is a negative number, nShift1 can be set to zero.

If rounding is used in both (3) and (4) (highest complexity, highest precision): then iOffset1=1&lt;&lt;(nShift1-1), (7)

iOffset2=1&lt;&lt;(nShift2-1), (8)

Where a=1&lt;&lt;c is expressed as a binary left shift of "1" by c bits, that is, a=2^c. In addition, when rounding is not used in (3) and (4) (lowest complexity, lowest precision): iOffset1=0, (9)

iOffset2=0. (10)

In addition, if rounding is used in (3) but not rounding in (4): then iOffset1=1&lt;&lt;(nShift1-1), (11)

iOffset2=0. (12)

In addition, if rounding is used in (4) but not rounding in (3) (this is the general case): then iOffset1=0, (13)

iOffset2=1&lt;&lt;(nShift2-1); (14)

In an exemplary embodiment, let INTERM_BITDEPTH=14, US_FILTER_PREC=6, and INTERNAL_INPUT_BITDEPTH=8, and then, according to equations (5) and (6), nShift1=0 and nShift2=12. In another example, for US_FILTER_PREC=6, if INTERNAL_INPUT_BITDEPTH=10 and INTERM_BITDEPTH=14, then nShift1=2 and iOffset1=0 or 2, depending on the selected rounding mode. In addition, nShift2=10 and iOffset2=0 or 2⁹, Depends on the selected rounding mode.

Please note that using the implementation shown in equations (3) and (4), performing vertical filtering before horizontal filtering may produce different results from performing horizontal filtering before vertical filtering. Therefore, in the decoder, appropriate filtering can be All decoders (for example, by decoding standards or specifications) are fixed or predetermined, or in some embodiments, the proper order can be signaled from the encoder to the decoder using appropriate flags in the metadata.

Figure 3 depicts an exemplary procedure for upsampling of image data according to an embodiment of the present invention. First (305), the encoder or decoder in the layered coding system determines the appropriate filtering order (for example, horizontal filtering before vertical filtering) and scaling and rounding parameters. In one embodiment, it can be based on the bit depth required by the intermediate storage (for example, INTERM_BITDEPTH), filter coefficients (for example, US_FILTER_PREC), and internal input representation (for example, INTERNAL_INPUT_BITDEPTH), according to equations (5)-(14 ) To determine the scaling and rounding parameters. In step 310, the image data is up-sampled in the first direction (for example, horizontal). The output result of this stage is rounded and scaled before using the intermediate storage of the first offset parameter (for example, nShift1) and the first rounding parameter (for example, iOffset1). Then (315), the intermediate result is up-sampled in the second direction (for example, vertical). Use the second offset parameter (for example, nShift2) and the second rounding parameter (for example, iOffset2) to round and scale the output result of this stage. Finally (320), the output result of the second stage is edited before the final output or storage.

The method described in this article can also be applied to the use of high bit depth Other image applications that can separate and filter the image data, such as reduction, noise filtering, or frequency conversion.

Demonstration of computer system implementation

The embodiments of the present invention can be computer systems, systems and components configured in electronic circuits, integrated circuit (IC) devices such as microcontrollers, field programmable gate arrays (FPGA), or another configurable or programmable logic A device (PLD), a discrete-time or digital signal processor (DSP), an application-specific integrated IC (ASIC), and/or equipment including one or more of the above-mentioned systems, devices, or components are implemented. The computer and/or IC can perform, control, or execute instructions related to high-precision upsampling, such as those described herein. The computer and/or IC can calculate any of the various parameters or values for the high-precision upsampling as described herein. The encoding and decoding embodiments can be implemented in various combinations of hardware, software, firmware, and above.

Certain implementations of the invention include a computer processor that executes software instructions that cause the processor to perform the methods of the invention. For example, one or more processors of a monitor, encoder, set-top box, transcoder, or the like can be implemented by executing software instructions in a program memory accessible by the processor. The high-precision up-sampling method. The present invention can also be presented in the form of a program product. The program product may include any medium that carries a set of computer-readable signals, and the set of computer-readable signals may include instructions that, when executed by the data processor, cause the data processor to perform the method of the present invention. The program product according to the present invention can be in any of various forms. For example, program products can include physical media such as magnetic data storage media, including floppy disks, Hard disk drives, optical data storage media including CD ROM, DVD, electronic data storage media including ROM, flash RAM, or the like. The computer-readable signal on the program product can be selectively compressed or encrypted.

Unless otherwise specified, the components (for example, software modules, processors, components, devices, circuits, etc.) herein refer to the above-mentioned components (including the reference to "tools") and should be construed as including and performing the components Any element whose function is equivalent (for example, functionally equivalent) includes an element that is not structurally equivalent to the disclosed structure for performing the function in the illustrated exemplary embodiment of the present invention.

Equivalents, extensions, substitutes and miscellaneous

This explains an exemplary embodiment of high-precision upsampling in adaptive coding of high-bit-depth video. In the foregoing specification, many specific details of the embodiments of the present invention that may be different from implementation to implementation have been described. Therefore, what the present invention is, and the unique and exclusive instructions for the invention formulated by the applicant are as specified in the scope of the patent application issued from this application (in the specific form issued in the scope of the above-mentioned patent application), including any Follow-up correction. Any clear definition of the terms included in the scope of the above-mentioned patent application shall stipulate the meaning of the above-mentioned terms as used in the scope of the above-mentioned patent application. Therefore, without limitation, elements, characteristics, features, advantages or attributes that are not explicitly stated in the scope of the patent application shall limit the scope of the above-mentioned patent application in any way. Therefore, the present specification and drawings are to be regarded as illustrative rather than restrictive.

refer to

[1] B. Bross, WJ. Han, GJ Sullivan, J.-R. Ohm, and T of ITU-T/ISO/IEC Joint Cooperative Team Video Coding (JCT-VC) document JCTVC-K1003 in October 2012 . "High efficiency video coding (HEVC) text specification draft 9" by Wiegand.

[2] ITU-T and ISO/IEC JTC 1, "Advanced Video Coding for generic audio-visual services", ITU T Rec. H.264 and ISO/IEC 14496-10 (AVC).

[3] SMuC0.1.1 software for SHVC (adaptable extension of HEVC):<u style="single">https://hevc.hhi.fraunhofer.de/svn/svn_SMuCSoftware/tags/0.1.1/.</u>