High dynamic range codecs

Technical Field

0001The invention relates to encoding image data. The invention has particular application for encoding images or for encoding sequences of video data.

Background

0002Dynamic range is a measure of the relative brightness of the brightest and darkest parts of an image. Until recently, most televisions, computer monitors and other display devices have been capable of reproducing dynamic ranges of only a few hundred to one. This is far less than the dynamic range that can be appreciated by the human eye. Display devices having greater dynamic ranges are becoming available. Such high dynamic range display devices can provide images that are much more natural and realistic than the images produced by conventional "low dynamic range" display devices.

0003High dynamic range display devices are beneficial in a wide range of applications. For example, high dynamic range display devices may be used for displaying realistic video images ranging from movies and game visuals, to visual displays in simulators such as flight simulators. High dynamic range display devices also have application in demanding imaging applications such as medical imaging.

0004Many current image data formats specify pixel values using 24 or fewer bits per pixel. These bits specify both the brightness and color for the pixel. 24 bits is too few to specify both a full range of colors and a brightness that can be varied smoothly over the range that a high dynamic range display is capable of reproducing. To obtain full benefit from a high dynamic range display it is necessary to provide image data capable of specifying a wide range of pixel values. Various high dynamic range data formats that provide larger numbers numbers of bits per pixel have been developed or proposed. Such high dynamic range data formats are not typically backward compatible with prior lower dynamic range data formats.

0005For example, HDRV "Perception-motivated HDR Video Encoding" as described in <nplcit id="ncit0001" npl-type="s"><text>R. Mantiuk, G. Krawczyk, K. Myszkowski, and H-P. Seidel. Perception-motivated high dynamic range video encoding. ACM Transactions on Graphics (Proc. of SIGGRAPH 2004), 23(3):730- 38, 2004</text></nplcit> is a lossy HDR video compressionrnethod, which, does not offer backward compatibility. The method encodes HDR pixels using 11 bits for luminance and twice 8 bits for chrominance. The resulting video stream does not contain any information on LDR frarnes.

0006JPEG HDR is described in <nplcit id="ncit0002" npl-type="b"><text>Greg Ward and Maryann Simmons. Subband encoding of high dynamic range imagery. In APGV '04: Proceedings of the 1st Symposium on Applied perception in graphics and visualization, pages 83-90, New York, NY, USA, 2004. ACM Press</text></nplcit>. This method involves sub-sampling a subband layer, which can lead to the loss of high frequencies, To prevent it, the method suggest three approaches: pre-correction of LDR layer, to encode within this layer high frequencies that can be lost due to sub-sampling; post-correction which tries to restore high frequencies that has been lost rather than modifying LDR image and full-sampling, which means that no sub-sampling is performed.

0007There remains a need for practical methods and apparatus for encoding and decoding HDR images, especially HDR video images. There remains a particular need for such methods and apparatus that provide backward compatibility with existing hardware for reproducing lower-dynamic-range images

0008US Patent Application <patcit id="pcit0001" dnum="US20050259729A1"><text>US 2005/0259729 A1</text></patcit> discloses a method of coding a quality scalable video sequence. An N-bit input frame is converted to an M-bit input frame, where M is an integer between 1 and N. To be backwards compatible with existing 8-bit video systems, M may be selected to be 8. The M-bit input frame is encoded to produce a base-layer output bit stream. An M-bit output frame is reconstructed from the base-layer output bit stream and converted to a N-bit output frame. The N-bit output frame is compared to the N-bit input frame to derive an N-bit image residual that is encoded to produce an enhancement layer bit stream.

Summary of the Invention

0009This invention provides methods and apparatus for encoding high dynamic range image data and for decoding the data to provide both lower dynamic range image data and higher dynamic range image data. The methods and apparatus may be applied to encoding video data. In some embodiments of the invention the lower dynamic range data is encoded in a standard format such as a MPEG (Motion Pictures Experts Group) format.

0010One aspect of the invention provides a method for encoding a high dynamic range image. The method comprises obtaining a lower dynamic range image corresponding to the high dynamic range image; identifying groups of pixels in the high dynamic range image for which corresponding pixels in the lower dynamic range image all have the same pixel value; generating a prediction function based at least in part on pixel values of the pixels in the high dynamic range image belonging to each of a plurality of the groups; applying the prediction function to the lower dynamic range image to obtain a predicted image; computing a residual image representing differences between pixel values in the predicted image and corresponding pixel values in the high dynamic range image; and, encoding and storing data representing the lower dynamic range image,the prediction function and the residual image.

0011Other aspects of the invention provide methods for decoding high dynamic range images that have been encoded according to the invention and apparatus for encoding and/or decoding high dynamic range images.

0012Further aspects of the invention and features of specific embodiments of the invention are described below.

Brief Description of the Drawings

0013In drawings which illustrate non-limiting embodiments of the invention, <ul id="ul0001" list-style="none" compact="compact"><li><figref idref="f0001">Figure 1</figref> is a flow chart illustrating an encoding method according to an embodiment of the invention;</li><li><figref idref="f0002">Figure 1A</figref> is a histogram of pixel values from a high dynamic range image for which corresponding pixels in a lower dynamic range version of the image all have the same pixel value;</li><li><figref idref="f0003">Figure 2</figref> is a flow chart illustrating a decoding method according to the invention;</li><li><figref idref="f0004">Figure 3</figref> is a flow chart illustrating an MPEG encoding method according to a specific embodiment;</li><li><figref idref="f0005 f0006 f0007">Figures 4A through 4F</figref> show the relationship between luma values in corresponding HDR and LDR images for various tone mapping algorithms;</li><li><figref idref="f0008">Figure 5</figref> shows a method for filtering residual image data according to an embodiment of the invention; and,</li><li><figref idref="f0009">Figure 6</figref> is a plot illustrating bit rate as a function of an image quality parameter for a prototype encoding system.</li></ul>

Description

0014Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.

0015<figref idref="f0001">Figure 1</figref> shows a method <b>10</b> for encoding a frame of image data according to a basic embodiment of the invention. Method <b>10</b> encodes both high dynamic range (HDR) data <b>12</b> and lower dynamic range (LDR) data <b>14</b> into encoded image data <b>38.</b> As described below, encoded image data <b>38</b> can be decoded to reconstruct both LDR data and HDR data.

0016By way of example only, HDR data <b>12</b> may be represented in a color space such the absolute CIE XYZ (2E standard observer) color space in which the color and brightness of each pixel are specified by three floating point numbers. LDR data <b>14</b> may be represented in a color space such as the sRGB color space in which the color and brightness of each pixel are specified by three bytes. In some embodiments, LDR data <b>14</b> is derived from HDR data <b>12</b> (or a precursor of HDR data <b>12)</b> by a suitable dynamic range reduction process <b>16.</b>

0017Dynamic range reduction may comprise tone mapping and/or gamut mapping, for example. Any suitable tone mapping operator or gamut mapping may be used. For example, a tone mapping operator may be chosen to saturate both luminance and color, change color vales and enhance local contrast. Such changes may result in a lower compression ratio, but both LDR and HDR frames will be preserved in the resulting video stream.

0018In block <b>18,</b> method <b>10</b> establishes a prediction function <b>19.</b> Prediction function <b>19</b> provides as an output a predicted pixel value for a pixel in HDR data <b>12</b> based upon the pixel value for the corresponding pixel in LDR data <b>14.</b> Since the goal is to be able to reproduce HDR data <b>12</b> and LDR data <b>14</b> from encoded image data <b>38</b> it is preferable to base prediction function <b>19</b> on a version of LDR data <b>14</b> that can be reconstructed from encoded image data <b>38.</b>

0019Where LDR data <b>14</b> is encoded and compressed using a lossy algorithm, it is not possible to guarantee that the reconstructed version of LDR data <b>14</b> will be identical to the original LDR data <b>14.</b> For this reason, <figref idref="f0001">Figure 1</figref> shows that block <b>19</b> receives as an input reconstructed LDR data <b>26.</b> Reconstructed LDR data <b>26</b> is obtained by encoding/compressing LDR data <b>14</b> in block <b>20</b> to provide encoded compressed LDR data <b>22</b> and then decoding/ decompressing encoded compressed LDR data 22 in block <b>24.</b> Encoded compressed LDR data <b>22</b> is included in encoded image data <b>38.</b> Line <b>15</b> illustrates a less accurate alternative in which block <b>18</b> uses LDR data <b>14</b> directly to establish prediction function <b>19.</b>

0020Prediction function <b>19</b> preferably exploits statistical relationships between pixel values in reconstructed LDR data <b>26</b> and corresponding pixel values in HDR data <b>12.</b> In general, if one takes all pixels in reconstructed LDR image <b>26</b> for which the pixels all have the same particular pixel value, the corresponding pixels in HDR image data <b>12</b> will not all have the same pixel value. That is, in general, there is a one-to-many relationship between LDR pixel values and HDR pixel values.

0021<figref idref="f0002">Figure 1A</figref> is a histogram in which the horizontal axis ranges over all possible HDR pixel values and the vertical axis indicates how many pixels in the image represented by HDR image data <b>12</b> have that value. There may be a significant number of pixel values for which the image does not have any pixels that have that value. The shaded bars in <figref idref="f0002">Figure 1A</figref> represent values of pixels in HDR image data <b>12</b> for which the corresponding pixels in reconstructed LDR image data <b>26</b> all have the same pixel value <i><b>X_LDR</b>.</i> The HDR pixel values that correspond to LDR pixel value <b><i>X_LDR</i></b> range from <b><i>A</i></b> to <b><i>B</i></b>. All of the HDR pixel values for pixels that correspond to the same pixel value in reconstructed LDR image data <b>26</b> may be called a bin. It is typical, but not mandatory, that different bins do not overlap.

0022A prediction function <b>19</b> for an image may be obtained from HDR image data <b>12</b> and reconstructed LDR image data <b>26</b> by collecting HDR pixel values into bins and statistically analyzing each of the bins. Collecting the HDR pixel values into bins may comprise: <ul id="ul0002" list-style="none" compact="compact"><li>$ taking reconstructed LDR image data <b>26,</b> and for each pixel value represented in reconstructed LDR image data <b>26</b> identifying the set of all pixels that have that pixel value;</li><li>$ for each of the sets of pixels identifying the corresponding pixels in HDR data <b>12</b> and determining the pixel values of those corresponding pixels to yield a set of all of the HDR pixel values that corresponds to each LDR pixel value.</li></ul>

0023Prediction function <b>19</b> may be obtained by any of: <ul id="ul0003" list-style="none" compact="compact"><li>$ finding the arithmetic mean of the HDR pixel values in each bin;</li><li>$ finding the median of the HDR pixel values in each bin;</li><li>$ finding the average of the values <b><i>A</i></b> and <b><i>B</i></b> that delimit the bin;</li><li>$ some combination of these; or</li><li>$ the like.</li></ul> It is believed that for many applications the arithmetic mean provides a good combination of accuracy and computational efficiency.

0024Given a prediction function <b>19</b> it is only necessary to encode differences between the values predicted by prediction function <b>19</b> and the actual values from HDR image data <b>12.</b> Such differences are usually close to zero and therefore can be efficiently compressed in residual frames.

0025Prediction function <b>19</b> needs to be defined only for the possible pixel values in the LDR data <b>14</b> (256 values in the case where pixel values are represented by an 8-bit number). Prediction function <b>19</b> may comprise a lookup table that is indexed by valid values for LDR pixels. Prediction function <b>19</b> may be implemented as a lookup table having one output value corresponding to each index value. For example, where LDR pixels have 8-bit values, the lookup table may comprise 256 different values indexed by integers in the range of 1 to 256. Prediction function <b>19</b> does not need to be continuous since its major role is to make the values of residual frames as small as possible. In the alternative, prediction function <b>19</b> may be partly or entirely represented by a suitably parameterized continuous curve.

0026In block <b>28</b> method <b>10</b> obtains a predicted HDR image by applying prediction function <b>19</b> to reconstructed LDR data <b>26.</b> The pixel value for each pixel of reconstructed LDR data <b>26</b> is applied as an input to prediction function <b>19</b> and the pixel value is replaced with the resulting output from prediction function <b>19</b> to yield a predicted HDR image <b>29.</b>

0027Block <b>30</b> computes a difference between predicted HDR image <b>29</b> and the image of HDR data <b>12</b> to provide a residual image <b>32.</b> Residual image <b>32</b> is encoded/compressed in block <b>34</b> and output as residual image data <b>35</b> to encoded image data <b>38.</b> Block <b>34</b> may comprise filtering and quantizing residual image <b>32</b> to remove information that will not have a discernable effect (or, with more aggressive filtering and/or quantization an excessively deleterious effect) on the fidelity of a HDR image reconstructed from encoded image data <b>38.</b>

0028<figref idref="f0003">Figure 2</figref> shows a method <b>40</b> for decoding encoded image data <b>38.</b> LDR data <b>22</b> may be extracted from encoded image data <b>38</b> and decoded/decompressed in block <b>32</b> to yield LDR data <b>43</b> that is output as an LDR data output <b>44.</b> If LDR data output <b>44</b> is all that is required then no further processing need be done.

0029If an HDR data output <b>56</b> is also required then prediction function <b>37</b> is decoded at block <b>46</b> to yield prediction function <b>47</b> and residual image data <b>35</b> is decoded / decompressed at block <b>50</b> to yield residual image <b>52.</b>

0030In block <b>48,</b> prediction function <b>47</b> is applied to LDR data <b>43</b> to yield a predicted HDR image <b>49.</b> In block <b>54</b> the predicted HDR image <b>49</b> is combined with residual image <b>52</b> to yield HDR data output <b>56.</b> A decoder that operates as shown in <figref idref="f0003">Figure 2</figref> can be backwards- compatible with systems and devices that require LDR data output <b>44</b> while providing high quality HDR data at HDR data output <b>56.</b>

0031Methods <b>10</b> and <b>40</b> may be performed by: <ul id="ul0004" list-style="none" compact="compact"><li>$ programmed data processors, which may comprise one or more microprocessors, digital signal processors, some combination thereof, or the like executing software that causes the data processors to implement the methods;</li><li>$ hardware circuits, for example circuits that include functional blocks that cooperate to implement the method - the circuits may comprise, for example, suitably configured field-programmable gate arrays ("FPGAs") or application-specific integrated circuits ("ASICs"); or,</li><li>$ performing some parts of the methods in programmed data processors and other parts of the methods in suitable hardware circuits.</li></ul>

0032<figref idref="f0004">Figure 3</figref> shows a method <b>70</b> according to a more specific example embodiment. Method <b>70</b> encodes video frames in a way that complies with the standards set by the Motion Pictures Experts Group (MPEG) standards. Method <b>70</b> receives two incoming streams of video data. A stream containing HDR frames <b>74</b> is received at input <b>72.</b> A stream containing LDR frames <b>76</b> is received at input <b>78.</b> LDR frames <b>76</b> may be derived from HDR frames <b>74</b> or some precursor of HDR frames <b>74</b> upstream from input <b>78.</b>

0033An encoder operating as shown in <figref idref="f0004">Figure 3</figref> produces three compressed streams: a LDR stream <b>80,</b> which may be fully compatible with MPEG; a residual stream <b>82,</b> which contains differences between LDR frames <b>76</b> and the corresponding HDR frames <b>74;</b> and an auxiliary stream, <b>84</b> which contains auxiliary data for reconstructing HDR frames <b>74.</b> The best performance can be achieved when residual stream <b>82</b> and auxiliary stream <b>84</b> do not duplicate the information encoded in LDR stream <b>80.</b>

0034LDR frames <b>76</b> are encoded in block <b>88</b> using a suitable encoder. For example, block <b>88</b> may use an MPEG video encoder compatible with the ISO/IEC 14496-2 standard. Other video encoders may be used in the alternative. The resulting video stream may be encapsulated in a suitable media container format, such as Audio Video Interleave (AVI) or QuickTime™, so that it can be recognized and played back by existing software.

0035In block <b>90</b> the MPEG encoded LDR frames are decoded. To minimize computation, the decoding in block <b>90</b> may be performed by the MPEG encoder used in block <b>88.</b> MPEG encoders typically decode frames internally for use in estimating motion vectors. Block <b>90</b> may comprise accessing the decoded frames generated by the MPEG encoder. In the alternative, block <b>90</b> may be performed independently from block <b>88.</b>

0036The output from block <b>90</b> will generally different from the input to block <b>88</b> because MPEG is a lossy compression method. LDR frames that are MPEG encoded and then decoded are not exactly the same as the original LDR frames but contain compression artifacts.

0037In blocks <b>92A</b> and <b>92B</b> the color spaces of one or both of LDR frames <b>76</b> and HDR frames <b>74</b> are transformed, if necessary, to provide LDR frames and HDR frames that are represented in mutually compatible color spaces. Which transformations, if any, are performed in blocks <b>92A</b> and <b>92B</b> depends upon the color spaces of incoming LDR frames <b>76</b> and HDR frames <b>74.</b> In some cases blocks <b>92A</b> and <b>92B</b> are not required. In other cases, only one of blocks <b>92A</b> and <b>92B</b> is required.

0038HDR and LDR color spaces are compatible when color channels of both the LDR and HDR color spaces represent approximately the same information. It is also desirable that the HDR and LDR color spaces be perceptually uniform. Perceptual uniformity facilitates the estimation of color differences according to perceivable, rather than arithmetic, differences. It is also desirable that the HDR color space preserve a broad color gamut, ideally the full visible color gamut, even though the full visible color gamut cannot be displayed on the existing displays.

0039The inventors consider that a good color space for use in representing HDR image data is a combination of the CIE 1976 Uniform Chromacity Scales (<i>u₀</i>, <i>v₀</i>) with the gamma correction of the sRGB color space. Other color spaces could also be used. In one example, incoming LDR frames <b>76</b> are represented in the sRGB color space while incoming HDR frames <b>74</b> are represented in the CIE XYZ (2E standard observer) color space. In this case, block <b>92A</b> comprises converting LDR pixels from the sRGB color space to the l_ldru_ldrv_ldr space. This can be done by computing the CIE XYZ color coordinates and then computing luma and u' and v' color coordinates from the XYZ values. The XYZ values may be determined using the sRGB conversion formulas given in IEC 61966-2-1:1999. <i>Multimedia systems and equipment - Colour measurement and management - Part 2-1: Colour management - Default RGB colour space - sRGB.</i> International Electrotechnical Commission, 1999. For example for the <i>R</i>_8-bit is the 8-bit color coordinate: <maths id="math0001" num="(1)"><math display="block"><mi mathvariant="italic">Rʹ</mi><mo>=</mo><mfrac><msub><mi>R</mi><mrow><mn>8</mn><mo>-</mo><mi mathvariant="italic">bit</mi></mrow></msub><mn>255</mn></mfrac></math><img file="EP2290983B1_D0001.tif" /></maths><maths id="math0002" num="(2)"><math display="block"><mi>R</mi><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mi mathvariant="italic">Rʹ</mi><mo>/</mo><mn>12.92</mn><mo>;</mo><mi mathvariant="italic">if Rʹ</mi><mo>≤</mo><mn>0.04045</mn></mtd></mtr><mtr><mtd><msup><mfenced separators=""><mfenced separators=""><mi mathvariant="italic">Rʹ</mi><mo>+</mo><mn>0.055</mn></mfenced><mo>/</mo><mn>1.055</mn></mfenced><mn>2.4</mn></msup><mo>;</mo><mi mathvariant="italic">otherwise</mi></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0002.tif" /></maths>

0040The <i>G</i>_8-bit and <i>B</i>_8-bit color coordinates may be converted to floating point values similarly and then <i>X, Y,</i> and <i>Z</i> may be determined from: <maths id="math0003" num="(3)"><math display="block"><mfenced open="[" close="]"><mtable><mtr><mtd><mi>X</mi></mtd></mtr><mtr><mtd><mi>Y</mi></mtd></mtr><mtr><mtd><mi>Z</mi></mtd></mtr></mtable></mfenced><mo>=</mo><mfenced open="[" close="]"><mtable><mtr><mtd><mn>0.4124</mn></mtd><mtd><mn>0.3576</mn></mtd><mtd><mn>0.1895</mn></mtd></mtr><mtr><mtd><mn>0.2126</mn></mtd><mtd><mn>0.7152</mn></mtd><mtd><mn>0.0722</mn></mtd></mtr><mtr><mtd><mn>0.0193</mn></mtd><mtd><mn>0.1192</mn></mtd><mtd><mn>0.9505</mn></mtd></mtr></mtable></mfenced><mo>⋅</mo><mfenced open="[" close="]"><mtable><mtr><mtd><mi>R</mi></mtd></mtr><mtr><mtd><mi>G</mi></mtd></mtr><mtr><mtd><mi>B</mi></mtd></mtr></mtable></mfenced></math><img file="EP2290983B1_D0003.tif" /></maths>

0041The example matrix in Equation (3) assumes the white point D65. Luma can be computed for each LDR pixel by using suitable corrected color values. For example, luma may be given by: <maths id="math0004" num="(4)"><math display="block"><msub><mi>l</mi><mi mathvariant="italic">ldr</mi></msub><mo>=</mo><mn>0.2126</mn><mo>×</mo><msub><mi>R</mi><mrow><mn>8</mn><mo>-</mo><mi mathvariant="italic">bit</mi></mrow></msub><mo>+</mo><mn>0.7152</mn><mo>×</mo><msub><mi>G</mi><mrow><mn>8</mn><mo>-</mo><mi mathvariant="italic">bit</mi></mrow></msub><mo>+</mo><mn>0.0722</mn><mo>×</mo><msub><mi>G</mi><mrow><mn>8</mn><mo>-</mo><mi mathvariant="italic">bit</mi></mrow></msub></math><img file="EP2290983B1_D0004.tif" /></maths> where: <i>l_ldr</i> is the luma value for an LDR pixel. Luma is the weighted sum of the non-linear R' G' B' components after gamma correction has been applied.

0042The chromaticities <i>u'</i> and <i>v'</i> may be obtained from: <maths id="math0005" num="(5)"><math display="block"><mi mathvariant="italic">uʹ</mi><mo>=</mo><mfrac><mrow><mn>4</mn><mo>⁢</mo><mi>X</mi></mrow><mrow><mi>X</mi><mo>+</mo><mn>15</mn><mo>⁢</mo><mi>Y</mi><mo>+</mo><mn>3</mn><mo>⁢</mo><mi>Z</mi></mrow></mfrac></math><img file="EP2290983B1_D0005.tif" /></maths> and <maths id="math0006" num="(6)"><math display="block"><mi mathvariant="italic">vʹ</mi><mo>=</mo><mfrac><mrow><mn>9</mn><mo>⁢</mo><mi>Y</mi></mrow><mrow><mi>X</mi><mo>+</mo><mn>15</mn><mo>⁢</mo><mi>Y</mi><mo>+</mo><mn>3</mn><mo>⁢</mo><mi>Z</mi></mrow></mfrac></math><img file="EP2290983B1_D0006.tif" /></maths> then 8-bit numbers <i>u_ldr</i> and <i>v_ldr</i> can be obtained by multiplying each of <i>u'</i> and <i>v'</i> by a suitable scaling factor such as: <maths id="math0007" num="(7)"><math display="block"><msub><mi>u</mi><mi mathvariant="italic">ldr</mi></msub><mo>=</mo><mi mathvariant="italic">uʹ</mi><mo>×</mo><mn>410</mn></math><img file="EP2290983B1_D0007.tif" /></maths> and <maths id="math0008" num="(8)"><math display="block"><msub><mi>v</mi><mi mathvariant="italic">ldr</mi></msub><mo>=</mo><mi mathvariant="italic">vʹ</mi><mo>×</mo><mn>410</mn></math><img file="EP2290983B1_D0008.tif" /></maths> In the transformed color space, each pixel of the LDR data is represented by the pixel values <i>l_ldr</i>, <i>v_ldr</i>, <i>u_ldr</i>.

0043Block <b>92B</b> may transform color values of DR frames <b>74</b> in substantially the same manner as described above for the LDR pixel values. Ordinary gamma correction typically cannot be used for the range of luminance values that may be specified in an HDR frame. Therefore, some embodiments use a perceptually uniform luminance representation that has been derived from the contrast detection measurements for human observers. This space has properties similar to a space in which LDR pixel values are gamma corrected but can encode the full visible range of luminance (using 11-12 bits for example).

0044In an example embodiment, HDR luminance, <i>y</i>, is transformed into 12-bit HDR luma, <i>l_hdr,</i> by the formula: <maths id="math0009" num=""><math display="block"><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mfenced><mi>y</mi></mfenced><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mi>a</mi><mo>⋅</mo><mi>y</mi></mtd><mtd><mi mathvariant="italic">if y</mi><mo>&lt;</mo><msub><mi>y</mi><mi>l</mi></msub></mtd></mtr><mtr><mtd><mi>b</mi><mo>⋅</mo><msup><mi>y</mi><mi>c</mi></msup><mo>+</mo><mi>d</mi></mtd><mtd><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mi mathvariant="italic">y</mi><mi>l</mi></msub><mo>≤</mo><mi>y</mi><mo>&lt;</mo><msub><mi>y</mi><mi>h</mi></msub></mtd></mtr><mtr><mtd><mi>e</mi><mo>⋅</mo><mi>log</mi><mfenced><mi>y</mi></mfenced><mo>+</mo><mi>f</mi></mtd><mtd><mi mathvariant="italic">if y</mi><mo>≥</mo><msub><mi>y</mi><mi>h</mi></msub></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0009.tif" /></maths> where constants are listed in Table I below. The inverse transformation is given by: <maths id="math0010" num="(10)"><math display="block"><mi>y</mi><mfenced><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub></mfenced><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mi mathvariant="italic">aʹ</mi><mo>⋅</mo><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub></mtd><mtd><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mo>&lt;</mo><msub><mi>l</mi><mi>l</mi></msub></mtd></mtr><mtr><mtd><mi mathvariant="italic">bʹ</mi><mo>⁢</mo><msup><mfenced separators=""><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mo>+</mo><mi mathvariant="italic">dʹ</mi></mfenced><mi mathvariant="italic">cʹ</mi></msup></mtd><mtd><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mi>l</mi><mi>l</mi></msub><mo>≤</mo><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mo>&lt;</mo><msub><mi>l</mi><mi>h</mi></msub></mtd></mtr><mtr><mtd><mi mathvariant="italic">eʹ</mi><mo>⋅</mo><mi>exp</mi><mfenced separators=""><mi mathvariant="italic">fʹ</mi><mo>⋅</mo><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub></mfenced></mtd><mtd><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mo>≥</mo><msub><mi>l</mi><mi>h</mi></msub></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0010.tif" /></maths> where the various constants used in Equations (9) and (10) are listed in Table I below. <tables id="tabl0001" num="0001"><table frame="all"><tgroup cols="6"><colspec colnum="1" colname="col1" colwidth="19mm" /><colspec colnum="2" colname="col2" colwidth="22mm" /><colspec colnum="3" colname="col3" colwidth="17mm" /><colspec colnum="4" colname="col4" colwidth="17mm" /><colspec colnum="5" colname="col5" colwidth="15mm" /><colspec colnum="6" colname="col6" colwidth="17mm" /><thead><row><entry namest="col1" nameend="col6" align="center" valign="middle">TABLE I - Example Constants for Equations (9) and (10)</entry></row><row><entry align="center" valign="middle"><i>a</i></entry><entry align="center" valign="middle"><i>b</i></entry><entry align="center" valign="middle"><i>c</i></entry><entry align="center" valign="middle"><i>d</i></entry><entry align="center" valign="middle"><i>e</i></entry><entry align="center" valign="middle"><i>f</i></entry></row></thead><tbody><row><entry align="center" valign="middle">17.554</entry><entry align="center" valign="middle">826.81</entry><entry align="center" valign="middle">0.10013</entry><entry align="center" valign="middle">-884.17</entry><entry align="center" valign="middle">209.16</entry><entry align="center" valign="middle">-731.28</entry></row><row><entry align="center" valign="middle"><i>y_l</i></entry><entry align="center" valign="middle"><i>y_h</i></entry><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /></row><row><entry align="center" valign="middle">5.6046</entry><entry align="center" valign="middle">10469</entry><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /></row><row><entry align="center" valign="middle"><i>a'</i></entry><entry align="center" valign="middle"><i>b'</i></entry><entry align="center" valign="middle"><i>c'</i></entry><entry align="center" valign="middle"><i>d'</i></entry><entry align="center" valign="middle"><i>e'</i></entry><entry align="center" valign="middle"><i>f'</i></entry></row><row><entry align="center" valign="middle">0.056968</entry><entry align="center" valign="middle">7.3014e-30</entry><entry align="center" valign="middle">9.9872</entry><entry align="center" valign="middle">884.17</entry><entry align="center" valign="middle">32.994</entry><entry align="center" valign="middle">0.00478</entry></row><row><entry align="center" valign="middle"><i>l_l</i></entry><entry align="center" valign="middle"><i>l_h</i></entry><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /></row><row><entry align="center" valign="middle">98.381</entry><entry align="center" valign="middle">1204.7</entry><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /><entry align="center" valign="middle" /></row></tbody></tgroup></table></tables>

0045Block <b>94</b> generates a prediction function for the HDR image data. The prediction function attempts to predict a pixel value for a pixel in the HDR image data based upon a corresponding pixel value for the corresponding pixel in the LDR image data. The prediction function is ideally chosen to minimize the number of pixels in the HDR image data that have values that differ significantly from the values predicted by the prediction function. The prediction function is preferably non-linear in the logarithmic domain.

0046In cases where pixel values representing chromaticity in the HDR data are almost the same as the corresponding pixel values in the LDR image data, it is not beneficial to compute a prediction function for the chromaticity pixel values (e.g. <i>u'</i> and <i>v'</i>). In such cases, it is only necessary to provide a prediction function for the brightness values (e.g. luma, luminance or the like).

0047Since LDR frames <b>76</b> and HDR frames <b>74</b> contain similar information, these frames are strongly correlated. Where LDR frames <b>76</b> are obtained by applying a tone mapping algorithm to HDR frames <b>74,</b> the particular nature of the correlation depends upon what tone mapping algorithm was used.

0048<figref idref="f0005 f0006 f0007">Figures 4A through 4F</figref> show how the luma values of a LDR frame relate to the luma values of a corresponding HDR frame. Each of these Figures applies a different tone mapping function to derive an LDR image from an example HDR image. These tone mapping functions provide generally linear relationships between <i>l_ldr</i> and <i>l_hdr</i> at lower values. There is more variation between the tone mapping functions for higher luminance values. In each of <figref idref="f0005 f0006">Figures 4A to 4D</figref>, LDR luma values are plotted on the horizontal axis and HDR luma values are plotted on the vertical axis. The points marked X indicate the pixel values of corresponding pixels in the LDR and HDR images.

0049<figref idref="f0005 f0006 f0007">Figures 4A to 4F</figref> correspond respectively to the tone mapping functions disclosed in: <ul id="ul0005" list-style="none" compact="compact"><li>$<nplcit id="ncit0003" npl-type="s"><text> S. Pattanaik, J.E. Tumblin, H Yee, and D.P. Greenberg. Time dependent visual adaptation for realistic image display. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, pages 47-54, July 2000</text></nplcit>.</li><li>$ <nplcit id="ncit0004" npl-type="s"><text>Erik Reinhard, Michael Stark, Peter Shirley, and Jim Ferwerda. Photographic tone reproduction for digital images. ACM Trans. on Graph., 21(3):267-276, 2002</text></nplcit>.</li><li>$ <nplcit id="ncit0005" npl-type="s"><text>Frédo Durand and Julie Dorsey. Fast bilateral filtering for the display of high-dynamic-range images. ACM Trans. on Graph., 21(3):257-266, 2002</text></nplcit>.</li><li>$ <nplcit id="ncit0006" npl-type="s"><text>Raanan Fattal, Dani Lischinski, and Michael Werman. Gradient domain high dynamic range compression. ACM Trans. on Graph., 21(3):249-256, 2002</text></nplcit>.</li><li>$ <nplcit id="ncit0007" npl-type="s"><text>Frédéric Drago, Karol Myszkowski, Thomas Annen, and Norishige Chiba. Adaptive logarithmic mapping for displaying high contrast scenes. Computer Graphics Forum, proceedings of Eurographics 2003, 22(3):419-426, 2003</text></nplcit>.</li><li>$ <nplcit id="ncit0008" npl-type="b"><text>Rafal Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. A perceptual framework for contrast processing of high dynamic range images. In APGV'05: Proceedings of the 2nd Symposium on Applied Perception in Graphics and Visualization, pages 87-94, New York, NY, USA, 2005. ACM Press</text></nplcit>.</li></ul>

0050The prediction function may be generated as described above. Where the prediction function is defined as the arithmetic mean of the values of all HDR pixels falling into a corresponding bin, then the prediction can be written as: <maths id="math0011" num="(11)"><math display="block"><mi mathvariant="italic">RF</mi><mfenced><mi>l</mi></mfenced><mo>=</mo><mfrac><mn>1</mn><mrow><mi mathvariant="italic">card</mi><mfenced><msub><mi mathvariant="normal">Ω</mi><mi>l</mi></msub></mfenced></mrow></mfrac><mstyle displaystyle="false"><mstyle displaystyle="true"><munder><mo>∑</mo><mrow><mi>i</mi><mo>∈</mo><msub><mi mathvariant="normal">Ω</mi><mi>l</mi></msub></mrow></munder></mstyle><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mfenced><mi>i</mi></mfenced></mstyle></math><img file="EP2290983B1_D0011.tif" /></maths> where Σ_i={i=1...N* <i>l_ldr</i> (<i>i</i>)=<i>l</i>}, <i>l</i>=0...255; <i>N</i> is the number of pixels in a frame and <i>l_ldr</i>(<i>i</i>) and <i>l_hdr</i>(<i>I</i>) are the luma values for the <i>i</i>^th pixel in the LDR and HDR frames respectively. The prediction function is preferably updated for each frame.

0051In <figref idref="f0005 f0006 f0007">Figures 4A to 4F</figref>, the prediction functions are shown as solid lines. The prediction functions will depend on the image content as well as on the tone-mapping function used. <figref idref="f0005 f0006 f0007">Figures 4A through 4F</figref> show prediction functions for typical HDR images. <figref idref="f0005 f0006 f0007">Figures 4A to 4F</figref> show that typical prediction functions tend to be slowly changing with an increasing slope over significant portions of their range. Therefore, in some embodiments, instead of encoding the values of the prediction function for every bin, the differences between prediction function values for two consecutive bins are encoded. To further reduce the number of bits, these differences can be compressed, for example, using an adaptive Huffman algorithm as indicated in block <b>95.</b> The size of auxiliary data stream <b>84</b> is 1% or less of the total stream size in some embodiments. Therefore the storage overhead of a prediction function can be almost insignificant. Prediction functions or parts of prediction functions may also be represented in other ways, for example, as parameterized polynomial curves, spline curves, or other parameterized functions.

0052In block <b>96</b> residual frames are computed. The pixel values in the residual frame each represent the difference between the pixel value for the corresponding pixel of the HDR frame and the pixel value for that pixel predicted by applying the prediction function to the pixel value of the corresponding pixel of the LDR frame. Block <b>96</b> may be performed separately for each pixel value (<i>l, u</i> and <i>v</i> in this example). For luminance values, each pixel <i>r_l</i>(<i>i</i>) in the residual frame may be calculated as: <maths id="math0012" num="(12)"><math display="block"><msub><mi>r</mi><mi>l</mi></msub><mfenced><mi>i</mi></mfenced><mo>=</mo><msub><mi>l</mi><mi mathvariant="italic">hdr</mi></msub><mfenced><mi>i</mi></mfenced><mo>-</mo><mi mathvariant="italic">RF</mi><mfenced separators=""><msub><mi>l</mi><mi mathvariant="italic">ldr</mi></msub><mfenced><mi>i</mi></mfenced></mfenced></math><img file="EP2290983B1_D0012.tif" /></maths> for chromatic values, the prediction function may be an identity function, in which case: <maths id="math0013" num="(13)"><math display="block"><msub><mi>r</mi><mi>u</mi></msub><mfenced><mi>i</mi></mfenced><mo>=</mo><msub><mi>u</mi><mi mathvariant="italic">hdr</mi></msub><mfenced><mi>i</mi></mfenced><mo>-</mo><msub><mi>u</mi><mi mathvariant="italic">ldr</mi></msub><mfenced><mi>i</mi></mfenced></math><img file="EP2290983B1_D0013.tif" /></maths> and <maths id="math0014" num="(14)"><math display="block"><msub><mi>r</mi><mi>v</mi></msub><mfenced><mi>i</mi></mfenced><mo>=</mo><msub><mi>v</mi><mi mathvariant="italic">hdr</mi></msub><mfenced><mi>i</mi></mfenced><mo>-</mo><msub><mi>v</mi><mi mathvariant="italic">ldr</mi></msub><mfenced><mi>i</mi></mfenced></math><img file="EP2290983B1_D0014.tif" /></maths>

0053A properly chosen prediction function can reduce the amount of data that encodes HDR frames significantly. Despite this saving, residual frames can still contain a significant amount of noise that does not visibly improve the quality of reconstructed HDR images. The compression ratio can be improved without causing a noticeable reduction in image quality by filtering the residual frames to reduce or eliminate this noise. Block <b>98</b> filters the residual frames. The signal in residual frames is often relatively close to the visibility threshold. Therefore, filtering can result in significant reduction of data without significant degradation in the quality of HDR images reconstructed from the data.

0054An output of block <b>98</b> is a residual frame in which high frequencies have been attenuated in those regions where they are not visible. <figref idref="f0008">Figure 5</figref> shows a method <b>110</b> that may be applied for filtering residual frames. Method <b>110</b> may be performed in the context of an encoding method according to the invention but also has application in other contexts where it is desired to reduce the amount of data representing an image without introducing visible artifacts into the image.

0055The description that follows describes processing that is done on a luma channel. The same processing may be also applied to chroma channels. The chroma channels may be subsampled, for example to half of their original resolution to reduce processing. This reduction approximately accounts for differences in luminance and chrominance CSF.

0056Method <b>110</b> receives a residue frame <b>112</b> and a HDR frame <b>114</b> that masks the residue frame. In blocks <b>116</b> and <b>118</b> a Discrete Wavelet Transform (DWT) is applied to split each of masking frame <b>114</b> and residue frame <b>112</b> into several frequency- and orientation-selective channels. Other suitable transforms, such as the cortex transform described in <nplcit id="ncit0009" npl-type="s"><text>A.B. Watson. The cortex transform: Rapid computation of simulated neural images. Computer Vision Graphics and Image Processing, 39:311-327, 1987</text></nplcit>, may be applied in place of the DWT. The cortex transform can be very computationally-intensive and so is practical only if sufficient computational resources are available.

0057A prototype embodiment uses the CDF 9/7 discrete wavelet basis (which is used also for the lossy compression of images according to the JPEG-2000 standard). This wavelet basis gives a good trade-off between smoothness and computational efficiency. In the prototype, only the three finest scales of the wavelet decomposition are used since filtering of lower spatial frequencies at coarser scales could lead to noticeable artifacts.

0058In block <b>120</b> a function such as a contrast sensitivity function (CSF) is applied to account for the lower sensitivity of the human visual system for high spatial frequencies. Applying the CSF involves weighting each band of wavelet coefficients by a constant value. Example weighting factors for a viewing distance of 1700 pixels are given in Table 2. <tables id="tabl0002" num="0002"><table frame="all"><tgroup cols="4"><colspec colnum="1" colname="col1" colwidth="14mm" /><colspec colnum="2" colname="col2" colwidth="19mm" /><colspec colnum="3" colname="col3" colwidth="19mm" /><colspec colnum="4" colname="col4" colwidth="19mm" /><thead><row><entry namest="col1" nameend="col4" align="left" valign="top">TABLE 2 - CSF coefficients</entry></row><row><entry valign="top">Scale</entry><entry valign="top">LH</entry><entry valign="top">HL</entry><entry valign="top">HH</entry></row></thead><tbody><row><entry>1</entry><entry>0.275783</entry><entry>0.275783</entry><entry>0.090078</entry></row><row><entry>2</entry><entry>0.837755</entry><entry>0.837755</entry><entry>0.701837</entry></row><row><entry>3</entry><entry>0.999994</entry><entry>0.999994</entry><entry>0.999988</entry></row></tbody></tgroup></table></tables>

0059Human visual channels have limited phase sensitivity. This provides a further opportunity to discard information without noticeable degradation of reconstructed images. A masking signal does not only affect regions where the values of wavelet coefficients are the highest, but it may also affect neighboring regions. Phase uncertainty also reduces the effect of masking at edges, as opposed to textures which show higher amounts of masking.

0060Phase uncertainty may be modelled with the L_0.2 norm, which is also used in JPEG-2000 image compression. The L_0.2 norm is given by: <maths id="math0015" num="(15)"><math display="block"><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><mi mathvariant="italic">card</mi><mfenced><mi mathvariant="normal">Θ</mi></mfenced></mrow></mfrac><mo>⁢</mo><msup><mfenced><mstyle displaystyle="false"><mstyle displaystyle="true"><munder><mo>∑</mo><mi mathvariant="normal">θ</mi></munder></mstyle><msup><mfenced open="|" close="|"><msub><mi>L</mi><mi mathvariant="italic">CSF</mi></msub></mfenced><mn>0.2</mn></msup></mstyle></mfenced><mfrac><mn>1</mn><mn>0.2</mn></mfrac></msup></math><img file="EP2290983B1_D0015.tif" /></maths> and mathematical equivalents thereof where 1 denotes the neighbourhood of a coefficient (a 13H13 box is used as the neighbourhood in the prototype implementation), <i>L_CSF</i> is a wavelet coefficient that has been weighted by applying a CSF factor and <i>Lcs_F</i> is the CSF-weighted wavelet coefficient after taking phase uncertainty into account.

0061Block <b>124</b> predicts how threshold contrast changes in the presence of the masking signal from original HDR frame <b>114.</b> To model contrast masking, one can employ a threshold elevation function. The threshold elevation function may, for example, have the form: <maths id="math0016" num="(16)"><math display="block"><msub><mi>T</mi><mi>e</mi></msub><mfenced><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub></mfenced><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd columnalign="left"><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub><mo>≤</mo><mi>a</mi></mtd></mtr><mtr><mtd><msup><mrow><mfenced><mrow><mi>c</mi><mo>×</mo><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub></mrow></mfenced></mrow><mi>b</mi></msup></mtd><mtd columnalign="left"><mi mathvariant="italic">otherwise</mi></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0016.tif" /></maths> In the prototype embodiment, the constants in Equation (16) are given by <i>a</i>=0.093071 and <i>b</i>=1.0299 and <i>c</i>=11.535.

0062Each CSF-weighted coefficient for the residual frame, <i>R_CSF</i>, is compared to the value of the corresponding threshold elevation <i>T_e</i> calculated from original HDR frame <b>114.</b> If <i>R_CSF</i> is smaller than the visibility threshold <i>T_e</i> from Equation (16), the coefficient may be set to zero without introducing changes in the eventual reconstructed image that are noticeable. This may be expressed by: <maths id="math0017" num="(17)"><math display="block"><msub><mi>R</mi><mi mathvariant="italic">filt</mi></msub><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mn>0</mn></mtd><mtd><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mi>T</mi><mi>e</mi></msub><mfenced><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub></mfenced><mo>≤</mo><msub><mi>R</mi><mi mathvariant="italic">CSF</mi></msub></mtd></mtr><mtr><mtd><mi>R</mi></mtd><mtd><mi mathvariant="italic">otherwise</mi></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0017.tif" /></maths>

0063Finally, the filtered wavelet coefficients, <i>R_filt</i> are transformed back to the image domain. The prefiltering method presented above can substantially reduce the size of a residual stream. The filtering is a reasonable trade-off between computational efficiency and accuracy of the visual model. Filtering as described herein typically increases encoding time by no more than about 80%. Filtering during encoding does not increase decoding times.

0064Returning to <figref idref="f0004">Figure 3</figref>, block <b>100</b> quantizes the filtered residual frames. Although the magnitudes of the differences encoded in residual frames are usually small, they can take values in the range of !4095 to 4095 (for 12-bit HDR luma encoding). Obviously, such values cannot be encoded using 8-bit MPEG encoder. Although the MPEG standard provides an extension for encoding luma values on 12 bits, such an extension is rarely implemented, especially in hardware.

0065Quantization block <b>100</b> permits the magnitude of residual values to be reduced, preferably sufficiently that those values can be encoded using a standard 8-bit MPEG encoder. Various quantization schemes may be used. For example, some embodiments apply a non-linear quantization, where large absolute values of residual are heavily quantized, while small values are preserved with maximum accuracy. Since very few pixels contain a residual having a large magnitude, most pixels are not affected by the strong quantization.

0066Strong quantization can result in some images having poor visual quality. This is because even a few pixels that have large quantization errors can stand out in a way that detracts from perceived image quality.

0067A simple clamping of residual values (for example, to an 8-bit range) can produce visually better results at the cost of losing details in very bright or dark regions. Furthermore, in typical images, with suitably chosen prediction functions, only a very few pixels have residual values that exceed an 8-bit range.

0068In some embodiments, to reduce clamping at the cost of a stronger quantization, residual values are divided by a constant quantization factor. The factor can be chosen based upon a trade-off between errors due to clamping and errors due to quantization. Such quantization factors can be set separately for each bin, based on the maximum magnitude of the residual for all pixels that belong to that bin. Therefore, the residual values after quantization can be computed as: <maths id="math0018" num="(18)"><math display="block"><msub><mover><mi>r</mi><mo>^</mo></mover><mi>l</mi></msub><mfenced><mi>i</mi></mfenced><mo>=</mo><msup><mfenced open="[" close="]"><mmultiscripts><msub><mo>/</mo><mrow><mi>q</mi><mfenced><mi>l</mi></mfenced></mrow></msub><mprescripts /><none /><mrow><msub><mi>r</mi><mi>l</mi></msub><mfenced><mi>i</mi></mfenced></mrow></mmultiscripts></mfenced><mrow><mo>-</mo><mn>127</mn><mo>+</mo><mn>127</mn></mrow></msup><mo>,</mo><mi mathvariant="italic">where l</mi><mo>=</mo><mi>k</mi><mo>⇔</mo><mi>i</mi><mo>⊂</mo><msub><mi mathvariant="normal">Ω</mi><mi>k</mi></msub></math><img file="EP2290983B1_D0018.tif" /></maths> where: <ul id="ul0006" list-style="none" compact="compact"><li>$ the operator [.]^-127÷127 rounds the value inside the square brackets to the closest integer and then clamps the value if it is larger than 127 or smaller than -127;</li><li>$ <i>q</i>(<i>l</i>) is a quantization factor that is selected separately for each bin Σ<i>_k</i>. The quantization factor may be given by: <maths id="math0019" num="(19)"><math display="block"><mi>q</mi><mfenced><mi>l</mi></mfenced><mo>=</mo><mi>max</mi><mfenced separators=""><msub><mi>q</mi><mi>min</mi></msub><mo>⁢</mo><mfrac><mrow><msub><mi>max</mi><mrow><mi>i</mi><mo>∈</mo><msub><mi mathvariant="normal">Ω</mi><mi>l</mi></msub></mrow></msub><mfenced><mfenced open="|" close="|" separators=""><msub><mi>r</mi><mi>l</mi></msub><mfenced><mi>i</mi></mfenced></mfenced></mfenced></mrow><mn>127</mn></mfrac></mfenced></math><img file="EP2290983B1_D0019.tif" /></maths></li></ul> where <i>q_min</i> is a minimum quantization factor which may be, for example, 1 or 2.

0069The quantization factors <i>q</i>(<i>l</i>) may be stored together with the prediction function in auxiliary data stream <b>84.</b> This data may be first compressed as in block <b>95.</b> In most cases, most of quantization factors <i>q</i>(<i>l</i>) will have the value <i>q_min</i>. Therefore, run-length encoding followed by Huffman encoding is an effective way to compress the data representing the quantization factors.

0070In block <b>102</b> the residual values are encoded. When the residual values are 8-bit values they can be encoded using ordinary MPEG compression (e.g. MPEG-4 compression). In a prototype embodiment, the quantized residual values, <i>r̂</i>₁, and chroma residual values <i>r_u</i> and <i>r_v</i> are MPEG encoded after rounding them to the nearest integer value. Note that the operations applied to obtain residual values are approximately linear in cases where the prediction function is nearly linear and the effect of the adaptive quantization of Equation (18) is minimal. In such cases, the visual information of a residual frame is in the same frequency bands as the original HDR frame, and the DCT quantization of the residual has a similar effect as for the original HDR pixel values. Therefore, a standard DCT quantization matrix can be used for encoding the residual frames.

0071Since the MPEG encoding in blocks <b>88</b> and <b>102</b> are independent, it is possible to separately set MPEG quality parameters for each of blocks <b>88</b> and 102. In most applications, setting two sets of MPEG quality parameters is neither intuitive nor convenient. In preferred embodiments, a single quality control sets quality parameters for both of blocks <b>88</b> and <b>102.</b> It has been found that setting the quality parameters in blocks <b>88</b> and <b>102</b> to be equal to one another provides generally satisfactory results.

0072Some quality settings for blocks <b>88</b> and <b>102</b> may give better compression results than others. To achieve the best quality HDR images, block <b>102</b> should comprise encoding using the best quality. The quality settings in block <b>88</b> primarily affect the quality of LDR images reconstructed from stream <b>80</b> but may have some impact on the quality of HDR images also.

0073Some embodiments of the invention exploit the fact that both LDR and HDR frames contain the same scenes. Therefore the optical flow should be the same for both of them. In such embodiments, the same motion vectors computed for LDR frames are also used for residual frames. Data structure <b>38</b> may include only one set of motion vectors. In alternative embodiments of the invention, motion vectors are computed separately for LDR and residual frames and both sets of motion vectors are stored in encoded image <b>data 38.</b>

0074Software for performing methods according to the invention may be implemented in various ways. In a prototype embodiment, the software is implemented as a dynamic library to simplify integration with external software. A separate set of command line tools permits for encoding and decoding of video streams from and to HDR image files.

0075Since HDR video playback involves decoding two MPEG streams, <b>80</b> and <b>82,</b> achieving an acceptable frame rate is more challenging than in the case of ordinary LDR video playback. Playback frame rate can be boosted by performing some parts of the decoding process using graphics hardware. For example, both color space conversion and up-sampling of color channels can be computationally expensive when executed on a CPU and yet can be performed extremely efficiently in a graphics processor (GPU) as fragment programs. Additionally, some color conversion functions can be significantly accelerated with the use of fixed point arithmetic and lookup tables.

0076<figref idref="f0009">Figure 6</figref> illustrates the performance of the prototype embodiment as a function of the quality setting. The lower points correspond to LDR stream <b>80</b> whereas the upper points correspond to the sum of LDR stream <b>80</b> and residual stream <b>82.</b> It can be seen that for lower values of the qscale quality parameter (i.e. for higher quality images) the percentage of the overall data stream made up by residual stream <b>82</b> is smaller than it is for higher values of the quality parameter (corresponding to lower-quality LDR images).

0077Codecs as described herein may be used to encode and decode both individual images and video sequences. Such codecs may be used to encode and decode movies to be stored on media such as DVDs, or other storage media that may become common for storing movies in future.

0078Some aspects of the invention provide media players that include an output for HDR images to which a HDR display device is connected or can be connected. The media players include hardware, software, or a combination of hardware and software that implement decoding methods as shown in <figref idref="f0003">Figure 2</figref> for example.

0079Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a data processing system may implement the encoding methods of <figref idref="f0001">Figures 1</figref> or <figref idref="f0004">3</figref> or the decoding method of <figref idref="f0003">Figure 2</figref> by executing software instructions stored in a memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any medium which carries a set of computer-readable signals comprising instruction which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

0080Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention. <ol id="ol0001" ol-style=""><li>1. A method for encoding a high dynamic range image, the method comprising: <ul id="ul0007" list-style="none" compact="compact"><li>obtaining a lower dynamic range image corresponding to the high dynamic range image;</li><li>identifying groups of pixels in the high dynamic range image for which corresponding pixels in the lower dynamic range image all have the same pixel value;</li><li>generating a prediction function based at least in part on pixel values of the pixels in the high dynamic range image belonging to each of a plurality of the groups;</li><li>applying the prediction function to the lower dynamic range image to obtain a predicted image;</li><li>computing a residual image representing differences between pixel values in the predicted image and corresponding pixel values in the high dynamic range image; and,</li><li>encoding and storing data representing the lower dynamic range image, the prediction function and the residual image.</li></ul></li><li>2. A method wherein obtaining the lower dynamic range image comprises encoding the lower dynamic range image and decoding the lower dynamic range image.</li><li>3. A method comprising transforming the high dynamic range image, the lower dynamic range image or both the high dynamic range image and the lower dynamic range image between color spaces before establishing the prediction function.</li><li>4. A method wherein, immediately prior to generating the prediction function the lower dynamic range image and higher dynamic range image are both expressed in color spaces that include a luma or luminance pixel value and two or more chromaticity pixel values.</li><li>5. A method wherein the lower dynamic range image is represented in a color space comprising one intensity pixel value and two or more chroma pixel values.</li><li>6. A method wherein the prediction function is nonlinear in the logarithmic domain.</li><li>7. A method wherein generating the prediction function comprises computing an arithmetic mean of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups.</li><li>8. A method wherein generating the prediction function comprises computing a median of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups.</li><li>9. A method generating the prediction function comprises computing an average of the highest and lowest pixel values of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups.</li><li>10. A method wherein generating the prediction function comprises one or more of: <ul id="ul0008" list-style="none" compact="compact"><li>computing an arithmetic mean of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups;</li><li>computing a median of the pixel vales, of the pixels in the high dynamic range image belonging to each of the plurality of the groups;</li><li>computing an average of the highest and lowest pixel values of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups;</li><li>computing a centroid of a subset of the pixel values of the pixels in the high dynamic range image belonging to each of the plurality of the groups; and,</li><li>combinations thereof.</li></ul></li><li>11. A method wherein the lower dynamic range image and high dynamic range image each comprise a frame in a video sequence.</li><li>12. A method comprising generating a new prediction function for each frame in the video sequence.</li><li>13. A method comprising monitoring a difference between successive frames in the video sequence and generating a new prediction function each time the difference indicates that the current frame is significantly different from a previous frame.</li><li>14. A method wherein the pixel values are pixel intensity values.</li><li>15. A method wherein the pixel intensity values comprise luminance values, luma values or radiance values.</li><li>16. A method comprising generating a chroma prediction function for each of one or more chroma values and, for each of the one or more chroma values: <ul id="ul0009" list-style="none" compact="compact"><li>applying the corresponding chroma prediction function to corresponding chroma values for pixels in the lower dynamic range image to obtain a predicted image;</li><li>computing a residual chroma image representing differences between the chroma values for pixels in the predicted and high dynamic range images; and,</li><li>encoding and storing data representing the chroma prediction functions and the chroma residual images.</li></ul></li><li>17. A method comprising downsampling the residual image prior to storing the data representing the residual image.</li><li>18. A method comprising filtering the residual image to remove noise prior to storing the data representing the residual image.</li><li>19. A method wherein filtering the residual image comprises: <ul id="ul0010" list-style="none" compact="compact"><li>applying a discrete wavelet transform to the residual image and to the high dynamic range image to yield a transformed residual image and a transformed high dynamic range image;</li><li>establishing threshold values for coefficients in the transformed residual image based upon values of coefficients in the transformed high dynamic range image; and,</li><li>setting values for the coefficients in the transformed residual image to zero if the coefficients have values that do not exceed the corresponding thresholds.</li></ul></li><li>20. A method wherein establishing the threshold values comprises applying a threshold elevation function to the coefficients in the transformed high dynamic range image.</li><li>21. A method wherein the threshold elevation function comprises raising the coefficients to a predetermined constant power.</li><li>22. A method wherein the threshold elevation function comprises multiplying the coefficients by a predetermined constant number.</li><li>23. A method according to any one of claims 20 to 22 wherein the threshold elevation function is given by: <maths id="math0020" num=""><math display="block"><msub><mi>T</mi><mi>e</mi></msub><mfenced><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub></mfenced><mo>=</mo><mrow><mo>{</mo><mtable><mtr><mtd><mn>1</mn></mtd><mtd columnalign="left"><mi mathvariant="italic">if</mi><mspace width="1em" /><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub><mo>≤</mo><mi>a</mi></mtd></mtr><mtr><mtd><msup><mrow><mfenced><mrow><mi>c</mi><mo>×</mo><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub></mrow></mfenced></mrow><mi>b</mi></msup></mtd><mtd columnalign="left"><mi mathvariant="italic">otherwise</mi></mtd></mtr></mtable></mrow></math><img file="EP2290983B1_D0020.tif" /></maths> or a mathematical equivalent thereof.</li><li>24. A method comprising applying predetermined contrast sensitivity function weighting factors to the coefficients of the transformed high dynamic range image prior to applying the threshold elevation function to the coefficients.</li><li>25. A method comprising applying a phase uncertainty function to the coefficients of the transformed high dynamic range image prior to applying the threshold elevation function to the coefficients.</li><li>26. A method wherein the phase uncertainty function is given by: <maths id="math0021" num=""><math display="block"><msub><mover><mi>L</mi><mo></mo></mover><mi mathvariant="italic">CSF</mi></msub><mo>=</mo><mfrac><mn>1</mn><mrow><mi mathvariant="italic">card</mi><mfenced><mi mathvariant="normal">Θ</mi></mfenced></mrow></mfrac><mo>⁢</mo><msup><mfenced><mstyle displaystyle="false"><mstyle displaystyle="true"><munder><mo>∑</mo><mi mathvariant="normal">θ</mi></munder></mstyle><msup><mfenced open="|" close="|"><msub><mi>L</mi><mi mathvariant="italic">CSF</mi></msub></mfenced><mn>0.2</mn></msup></mstyle></mfenced><mfrac><mn>1</mn><mn>0.2</mn></mfrac></msup></math><img file="EP2290983B1_D0021.tif" /></maths> or a mathematical equivalent thereof, where 1 denotes the neighbourhood of a coefficient <i>L_CSF</i> is a wavelet coefficient and <i>L_CSF</i> is the wavelet coefficient after applying the phase uncertainty function.</li><li>27. Apparatus for encoding a high dynamic range image, the apparatus comprising a data processor executing instructions that cause the data processor to: <ul id="ul0011" list-style="none" compact="compact"><li>obtain a lower dynamic range image corresponding to the high dynamic range image;</li><li>identify groups of pixels in the high dynamic range image for which corresponding pixels in the lower dynamic range image all have the same pixel value;</li><li>generate a prediction function based at least in part on pixel values of the pixels in the high dynamic range image belonging to each of a plurality of the groups;</li><li>apply the prediction function to the lower dynamic range image to obtain a predicted image;</li><li>compute a residual image representing differences between pixel values in the predicted image and corresponding pixel values in the high dynamic range image; and, encode and store data representing the lower dynamic range image, the prediction function and the residual image.</li></ul></li><li>28. Apparatus for encoding a high dynamic range image, the apparatus comprising: <ul id="ul0012" list-style="none" compact="compact"><li>a means for obtaining a lower dynamic range image corresponding to the high dynamic range image;</li><li>a means for identifying groups of pixels in the high dynamic range image for which corresponding pixels in the lower dynamic range image all have the same pixel value;</li><li>a means for generating a prediction function based at least in part on pixel values of the pixels in the high dynamic range image belonging to each of a plurality of the groups;</li><li>a means for applying the prediction function to the lower dynamic range image to obtain a predicted image;</li><li>a means for computing a residual image representing differences between pixel values in the predicted image and corresponding pixel values in the high dynamic range image; and,</li><li>a means for encoding and storing data representing the lower dynamic range image, the prediction function and the residual image.</li></ul></li><li>29. Apparatus for decoding a high dynamic range image, the apparatus comprising a data processor executing instructions that cause the data processor to: <ul id="ul0013" list-style="none" compact="compact"><li>retrieve data representing a lower dynamic range image corresponding to the high dynamic range image, a prediction function and a residual image;</li><li>apply the prediction function to the lower dynamic range image to obtain a predicted high dynamic range image; and,</li><li>combine the residual image with the predicted high dynamic range image to obtain the high dynamic range image.</li></ul></li><li>30. Apparatus for decoding a high dynamic range image, the apparatus comprising: <ul id="ul0014" list-style="none" compact="compact"><li>means for retrieving data representing a lower dynamic range image corresponding to the high dynamic range image, a prediction function and a residual image;</li><li>means for applying the prediction function to the lower dynamic range image to obtain a predicted high dynamic range image; and,</li><li>means for combining the residual image with the predicted high dynamic range image to obtain the high dynamic range image.</li></ul></li></ol>