Encoding, decoding and representing high dynamic range images

Technical Field [0001] The application claims the benefits of US Application No. 60 / 564.608 filed April 23, 2004.

Technical field [0002] The present invention relates to digital images with a wide dynamic range. The invention particularly relates to a method and apparatus for encoding and decoding images with a wide dynamic range and to data structures containing digital images with a wide dynamic range.

Background of the invention [0003] Human vision has the ability to evaluate contrast ratios up to 1: 10,000. This means that a person may notice a scene where some parts are 10,000 times brighter than other parts of the scene and notice details in both the lightest and darkest parts of the scene. In addition, human vision can adapt its sensitivity to lighter or darker scenes in the range of 6 orders of magnitude.

[0004] Most conventional digital image formats (so-called 24-bit formats) use up to 24 bits to record color and luminance information for each pixel of the image. For example, each of the red, green or blue (RGB) values ​​for a pixel can be saved in one byte (8 bits). Such formats can represent changes in brightness in the range of up to about two orders of magnitude (each byte can store one of the possible 256 values). There are a number of standard formats for representing digital images (which include both photographic and video images). They include JPEG (Joint Photographic Experts Group), MPEG (Motion Picture Experts Group), AVI (Audio Video Interleave, interlaced audio and video), TIFF Tagged Image File Format, BMP (file format with graphic features) Bit Map, Bitmap), PNG (Portable Network Graphics) <a href="http://pl.wikipedia.org/wiki/Grafika_rastrowa">raster </a><a href="http://pl.wikipedia.org/wiki/Format_pliku">file format</a> graphic), GIF (Graphical Interchange Format) and others. Such formats can be called "output reference standards" because they do not attempt to preserve image information, which can be reproduced by electronic displays of the most available types. Until recently, displays such as computer monitors, televisions, digital movie projectors and the like were unable to accurately reproduce images having a contrast ratio greater than 1: 1000 or similar.

[0005] Display technologies developed by the beneficiary and others can reproduce images having a high dynamic range (HDR). Such displays can reproduce images that represent real scenes more accurately than conventional displays. There is a need for formats for storing HDR images for reproduction on these displays and other HDR displays that will be available in the future.

[0006] A number of formats have been proposed for storing HDR images as digital data. All these formats have various disadvantages. A number of these formats provide hindering large image files that can only be viewed using specialized software. Some camera manufacturers propose their own RAW formats. These formats tend to be camera-specific and excessive in terms of data storage requirements.

[0007] There is a need for a convenient frame for storing, exchanging and reproducing images with a wide dynamic range. Particularly needed are frameworks that are backwards compatible with existing image viewer technologies. There is a particular need for backward compatibility in cases where it may be necessary to reproduce the image in inherited devices such as DVD players that have hardware image decoders.

Durand et al .: "Fast Bilateral Filtering for the Display of High-Dynamic-Range Images" (ACM TRANSACTIONS ON GRAPHICS, ACM, vol. 21, no. 3, July 2002, pages 257-266) presents a technique for displaying images about wide dynamic range that reduces contrast while maintaining details. This technique is based on a two-scale breakdown of the image into a basic layer encoding large-scale changes and a detail layer. Only the base layer has a reduced contrast thereby maintaining the details. The base layer is obtained using a non-linear edge-retaining filter, where the weight of each pixel is calculated from the Gaussian function in the spatial domain multiplied by the function of influence in the intensity domain, which reduces the weight of pixels with large differences in intensity.

Erdem et al, Compression of 10-bit video using the tools of MPEG-2 (SIGNAL PROCESSING. IMAGE COMMUNICATION, vol. 7, no. 1, March 1995, pages 27-56) solves the problem of video compression containing 10-bits per pixel using MPEG-2 standard tools, which is mainly directed to 8-bit video per pixel. The authors show that it is possible to develop a compression scheme with scaled amplitude for 10-bit video using the syntax and tools of MPEG-2.

US Patent 6,301,393 B1, dated October 9, 2001, presents a method for representing a digital image having color values ​​with an expanded range of colors in a color storage space with a limited range of colors including the steps of: adjusting the color value of a digital image with an extended range of colors to match a limited range of colors to create a digital image with a limited range of colors; presenting a digital image with a limited range of colors in the color storage space; determining a cropped digital image with a limited range of colors in which areas heavily quantized in a digital image with a limited range of colors have been cropped; determining a residual image that shows the difference between a digital image with an expanded color gamut and a cropped digital image with a limited gamut color; and assigning the residual image to the digital image with a limited range of colors in the color storage space such that the assigned residual image and the digital image with a limited range of colors in the color storage space are adapted for use to create a reconstructed digital image with an expanded range of colors.

The creator of the above patent, Spaulding et al., Also described in Spaulding, Kevin E. et al .: "Extending the color gamut and dynamic range of an sRGB image using a residual image", Color Research &amp; Application, vol. 28, no . 4, pp. 251 - 266,<a href="http://dx.doi.org/10.1002/col.10160">http://dx.doi.org/10.1002/col.10160</a> August 2003, a solution developed to maintain compatibility with existing file formats and software applications, while maintaining information about the extended dynamic range and range of colors associated with the original scenes. With this departure, the input raw image from a digital camera or film scan is first transformed into scene-related ERIMM RGB coding, then a rendered sRGB image is created in the usual way and saved to a conventional image file (e.g. a standard JPEG file). A residual image showing the difference between the original image with extended dynamic range and the final rendered image created and stored in the image file using custom metadata tags. This creates a mechanism for archiving information about the extended dynamic range / color gamut, which is normally removed during the rendering process, without compromising interoperability. The corresponding enabled application can decode the residual image metadata and use it to reconstruct the ERIMM RGB image, while applications that do not read the metadata will ignore them and will only have access to the sRGB image. The residual image is created in such a way that it will have negligible pixel values ​​for those parts of the image that are in the sRGB range and therefore can be subjected to high compression. Tests on a sample of 950 real client images showed that scene information with extended dynamic range can be stored in an average size file about 8% larger compared to the sRGB image itself.

Brief Description of the Invention [0008] One aspect of the present invention provides a method of encoding image data with a wide dynamic range. The method includes obtaining a tonal map data corresponding to a given image with a wide dynamic range. The tonal map data has a smaller dynamic range than the wide dynamic range image data. The method calculates coefficient data comparing value coefficients for wide dynamic range image data with corresponding tonal map data values; generates information on a wide dynamic range based on coefficient data; generates tonal map information based on tonal map data; and stores wide dynamic range information and tonal map information in a data structure.

[0009] Data structure can be read by inherited image viewers. Inherited image viewers can read tonal map information and ignore information about a wide dynamic range. In some embodiments, the data structure includes a JFIF file and tonal map information includes a JPEG image. In some embodiments, the data structure includes an MPEG file and the tonal map information includes an MPEG video frame.

[0010] Another aspect of the present invention provides a data structure for representing a wide dynamic range image having an initial dynamic range. The data structure contains a part which is a tonal map and a part which is information with a wide dynamic range. The tonal map portion contains tonal map information representing the image and has a dynamic range smaller than the initial dynamic range. Part of the information about the wide dynamic range contains information describing the ratio of the luminance value in the part of the tonal map to the luminance value of the image with a wide dynamic range.

[0011] Another aspect of the present invention provides a device for encoding images with a wide dynamic range.

[0012] Further aspects of the invention and features of specific embodiments of the invention are described below.

Brief description of the figures [0013] In the figures, which show non-limiting embodiments of the invention,

Figure 1 is a block diagram of a data flow illustrating a method of creating an HDR image file in accordance with the general embodiment of the invention;

Figure 2 is a block diagram containing general information about the methods of encoding and decoding an HDR image according to the invention;

Figure 3 is a block diagram of a data flow illustrating a method of creating an HDR image file in accordance with one specific embodiment of the invention;

Figure 4 is a block diagram illustrating methods according to some embodiments of the invention that provides corrections to artifacts resulting from compression and / or reduction of sampling values; and

Figure 5 is a block diagram illustrating a method according to one embodiment of the invention that provides corrections to artifacts resulting from compression and / or reduction of sampling values ​​during HDR image reconstruction.

Description [0014] In the following, specific details will be provided to provide a more accurate understanding of the invention. However, the invention can be implemented without these details. In other words, well-known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Specifications and drawings, respectively, should be treated in an illustrative and non-limiting sense.

[0015] One aspect of the present invention offers a data structure for representing HDR images (HDR data structures). In preferred embodiments, HDR data structures allow viewing images in a standard dynamic range mode using standard image viewing software and allow viewing versions of the same image with a wide dynamic range in wide dynamic range mode using an HDR browser and a suitable HDR monitor.

[0016] Figure 1 shows a system 10 according to the invention for creating HDR data structures 16 and for viewing images represented by HDR data structures

16. Figure 2 shows the method 30 implemented by the system 10 to create HDR data structures and alternative methods 31A and 31B for displaying images based on data in HDR data structures 16.

[0017] System 10 includes an encoder 14 for creating an HDR image data structure 16 based on the original HDR image data 12. The data structure 16 may be decoded by a standard decoder 18 to obtain an image with a standard dynamic range 19. In some embodiments, the standard decoder 18 includes an "inherited" hardware decoder or software decoder such as the corresponding image viewer software. The data structure 16 can be decoded by the HDR decoder 20 to obtain the reconstructed HDR image 21.

[0018] Method 30 starts at block 32 by obtaining HDR image data 12. HDR image data 12 includes information that directly or indirectly determines the luminance of the pixels in the image. HDR 12 image data can be in any useful format and can be obtained by using the appropriate HDR camera (if possible in combination with multiple exposure) or rendered directly on the computer. The HDR 12 image data source is irrelevant to the practicality of the invention.

[0019] The method 30 also obtains (block 34) the tonal map data 15 corresponding to the HDR image 12. The tonal map data 15 represents the similarity of the HDR 12 image but has a smaller dynamic range than the HDR image data 12. The tonal map data 15 can be generated from HDR 12 image data as indicated on line 13, or otherwise obtained from data having a common source with HDR 12 image data. If tonal map data 15 is not obtained from HDR image data 12, then the order in which tonal map data 15 and HDR image data 12 are obtained (i.e., order of blocks 32 and 34) is not valid.

[0020] The encoder 14 generates a data structure 16. The data structure 16 includes a part of the tonal map 16A, which is based on the data of the tonal map 15, and a part of the HDR information 16B, which contains information that can be combined by the HDR decoder 20 with data from the part. 16A tonal map for the reconstruction of HDR 12 image data or its close approximation. Method 30 generates (block 36) a portion containing HDR information by comparing tonal map data 15 (or, equivalent, tonal map data reconstructed from tonal map portion 16A of data structure 16) and HDR image data 12. In block 38, method 30 stores a portion of the map tonal 16A and part of HDR 16B information in the data structure 16.

[0021] In preferred embodiments of the invention, the data structure 16 has a format that can be read by a standard decoder to create a lower dynamic range (LDR) image. The standard decoder 18 may implement a decoding method 31A. The standard decoder 18 generates a standard LDR 19 image recovering part of the 16A tonal map and displaying the image represented by the part of the 16A tonal map (block 39). The standard decoder may ignore the part containing HDR 16B information.

[0022] The data structure 16 can also be read by the HDR decoder 20. The HDR decoder 20 implements the decoding method 31B and generates the HDR image 21 based on information from both part of the tonal map 16A and part of the HDR information 16B. Method 31B retrieves data from part of the tonal map 16A and part of the HDR information 16B of the data structure 16 in block 40. In block 42, a reconstructed HDR image is created by modifying the tonal map extracted from the 16A tonal map portion according to the HDR information derived from the HDR 16B portion of the information. The reconstructed HDR image is displayed in block 44.

[0023] The portion of the tonal map 16A may be in any useful format. For example, part of the 16A tonal map can be in JPEG, MPEG, AVI, TIFF, BMP, GIF or some other useful formats. Part of the 16A tonal map contains information that directly or indirectly determines the luminance of the pixels in a dynamic range image, which is smaller than the range of the original HDR 12 image. While the HDR image data 12 determines the color image, the portion of the tonal map 16A preferably includes information determining the color for pixels in the image.

[0024] In some embodiments of the invention, the data structure 16 includes a file formatted as JFIF (JPEG File Interchange Format, an extension of the JPEG standard). In such embodiments, part of the tonal map 16A may be included in the image part of the JFIF file, while part of the HDR information 16B may be stored in one or more parts of the JFIF file application extension and / or in one or more parts of the JFIF file commentary. In this embodiment, any standard JPEG viewer can open the data structure 16 and display the image contained in part of the tonal map 16A in a dynamic range smaller than the original HDR 12 data range or the reconstructed HDR 21 image.

[0025] Standard JPEG viewers ignore application extensions in JFIF files that they do not support. Therefore, the presence of some HDR 16B information does not significantly affect the display of the image from the data structure 16 using a standard JPEG viewer. When HDR 16B information is in the JFIF file comment field, HDR 16B information is preferably encoded as ASCII text because some applications may try to read the JFIF file comment fields. Such applications can expect that the comment fields contain only text and may behave incorrectly when attempting to open a comment field containing unexpected data. Version 1.2 is one of the JFIF versions. The JFIF 1.2 version is described in detail in Annex B to ISO DIS 10918-1, which is incorporated herein by reference.

[0026] In some embodiments of the invention, the data structure 16 includes a file formatted as MPEG. In such embodiments, part of the tonal map 16A may be included in the image part of the MPEG file while part of the HDR information 16B may be stored in one or more application extension of the MPEG file and / or in one or more parts commenting on the MPEG file. In this embodiment, any standard MPEG viewer can open the data structure 16 and display the image contained in the tonal map part 16A in a dynamic range smaller than the range of the original HDR 12 data or the reconstructed HDR 21 image. One part of HDR 16B information can be associated with each of the frames MPEG video file or, in the case of MPEG versions using key frames, some HDR 16B information can only be associated with key frames. Conventional MPEG key frame interpolation techniques can be used to create key frames of inner frames (i.e., frames that are positioned between key frames).

[0027] Standard MPEG browsers ignore channels in MPEG files that they do not support. Therefore, the presence of some HDR 16B information does not significantly affect the display of the image from data structure 16 using any standard MPEG viewer. When HDR 16B information is in the MPEG file comment field, HDR 16B information is preferably encoded as ASCII text because some applications may try to read MPEG file comment fields. Such applications may expect that only the text is placed in the comment field and may behave incorrectly when attempting to open a comment field containing data of the unexpected type.

[0028] The portion of the tonal map 16A may be formed from the tonal map data 15 in any useful manner. For example, part of the tonal map 16A may be generated by a suitable tonal mapping operator. The tonal mapping operator preferably has properties such that:

• the original HDR input data (ie the original HDR 12 image data) are smoothly mapped to the output domain with a standard dynamic resolution (usually 24-bit);

• no operator output elements for tonal mapping take the value 0 or 255;

• a shade is maintained for each pixel; and • if the tonal mapping operator changes the saturation value, it only performs mild changes that can be described by reversible functions.

The inventors have found that the two-sided filter described in Durand and Dorsey, Quick Two-Sided Filtering for Wide Dynamic Range Display, ACM Transactions on Graphics, 21, 3, 249-256 (2002) offers the appropriate tonal mapping operator. Part of the 16A tonal map can be encoded using a suitable encoder such as a JPEG encoder or MPEG encoder.

[0029] Part of the tonal map 16A may display pixel color values ​​in any useful manner. For example, pixel color values ​​can be represented as RGB values ​​(red, green and blue), CMYK values ​​(cyan, magenta, yellow and black, cyan, magenta, yellow and black), YCbCr values ​​(luminance and chrominance, ang luminance and chrominance) or similar. The data in part of the 16A tonal map can be compressed using any suitable compression scheme. For example, data in part of the 16A tonal map can be compressed in a manner consistent with the JPEG or MPEG standards.

[0030] In some embodiments, the HDR information portion 16B includes ratios between the values ​​specified by the tonal map portion 16A for individual pixels, and the values ​​specified by the original HDR image 12 for the same pixels. In such embodiments, HDR information 16B can be generated by dividing the values ​​specified by the original HDR image 12 by the corresponding values ​​specified by part of the tonal map 16A. Data resulting from this operation can be stored as part of HDR 16B information. The accuracy with which the data values ​​from the HDR 16B information part are represented can be selected to ensure acceptable quality of the reconstructed HDR images. In some embodiments, each of the data values ​​in the HDR 16B information portion before compression is represented by one byte (8 bits).

[0031] In some embodiments of the invention, part of the HDR information 16B determines the relationship between the pixel luminance of the reconstructed HDR image 21 and the luminance specified for the corresponding pixels by the tonal map information 16A. In such embodiments, part of the HDR 16B information does not need to include color information.

[0032] Part of the HDR information 16B may include luminance ratios specified by the original HDR image 12 for areas or pixels in the image to the luminance determined by part of the tonal map 16A for the respective areas or pixels. In such embodiments, the color information is transmitted through part of the 16A tonal map. In such embodiments, the HDR 16B portion may have the same structure as the gray image. For example, when the HDR 16 data structure includes a JFIF file, the HDR 16B portion may be encoded as a grayscale JPEG image. When the HDR 16 data structure includes an MPEG file, the HDR 16B portion can be encoded as an MPEG image in the gray room.

[0033] Figure 3 shows the HDR encoder 50 according to an embodiment of the invention, in which the HDR information used to form part of the HDR information 16B of the data structure 16 includes ratios of pixel values ​​in the HDR image 12 to the corresponding values ​​specified by part of the tonal map 16A. Encoder 50 receives HDR 12 image data. Encoder 50 obtain tonal map data 15 either by extracting tonal map data 15 from HDR image data 12, as shown in dashed line 13, and tonal mapping device 17, or by receiving tonal map data 15 from another source, as shown in dashed line 13A. The tonal mapping device 17 preferably does not cut the color or luminance values ​​and maintains the color and luminance factors for each pixel of the tonal map data 15.

[0034] In the illustrated embodiment of the invention, the encoder 50 comprises a standard encoder 52. The standard encoder encodes tonal map data 15 to produce coded tonal map data 15A. The encoded 15A tonal map data can be read using a standard browser. For example, the standard encoder 52 may include an encoder that encodes tonal map data 15 as tonal map data encoded as JPEG or MPEG that can be read by a JPEG or MPEG viewer. The coded tonal map data is written to part of the tonal map data 16A of the HDR 16 data structure.

[0035] In some embodiments of the invention, the encoder 50 receives the encoded tonal map data 15A from an external source. In such embodiments, the encoder 50 need not include the standard encoder 52.

[0036] The encoded tonal map data 15A is decoded by a decoder 54 to obtain a reconstructed tonal map data 55. The HDR image data 12 is divided by the reconstructed tonal map data 55 by a divider 56 to obtain a factor 57 data. The factor 57 data is optionally compressed by data compressor 58 for obtaining HDR 16B information. The data compressor 58 may preferably include a JPEG or MPEG encoder. In some embodiments, the same JPEG or MPEG encoder is used to encode both the 16A tonal map portion and the HDR 16B portion of the HDR 16 data structure.

[0037] In some embodiments of the invention, the coefficient 57 data includes some functions of the ratio of HDR image data values ​​12 to the corresponding values ​​determined by the tonal map data (or part of the tonal map 16A). For example, coefficient 57 data may contain information specifying the logarithm of such a coefficient.

[0038] In some alternative embodiments, the tonal map data 15 is provided directly to the divider 56 as indicated by line 53. In such embodiments, a decoder 54 is not required. When part of the tonal map 16A is encoded using a lossy algorithm such as JPEG or MPEG encoding, it is preferable to base some of the HDR 16B information on the reconstructed tonal map data 55 instead of the tonal map data 15. Basing part of HDR information 16B on the reconstructed tonal map data 55 enables a more accurate reconstruction of HDR image data 12 from HDR data structure 16 in cases where part of tonal map 16A is encoded in a lossy coding process. Part of tonal map information 16A, instead of tonal map data 15, will be used to reconstruct HDR image 21 (figure 1).

[0039] The compressor 58 may take any of a number of forms. In some embodiments, the compressor 58 performs one or more of the following operations:

• reducing the sample value of the 57 factor data;

• 57 data compression.

Any suitable compression can be used. In a currently preferred embodiment of the invention, the compressor 58 both reduces the sampling value of the coefficient 57 data and encodes the coefficient data with the reduced sampling value. When the 57 factor data has a reduced sampling value, some HDR 16B information has an image size smaller than the image size of the 57 factor data or tonal map data (i.e. part of the HDR 16B information specifies values ​​for the number of pixels that is less than the number of pixels for which the values ​​determine the coefficient 57 data or the tonal map data 15). In these cases, some HDR 16B information has a lower spatial resolution than the tonal map data 15.

[0040] In those embodiments of the invention in which the coefficient 57 data is subject to downsampling or other lossy compression mechanisms, HDR information 16B may not have the details necessary to accurately reconstruct the HDR 12 image data. Distortions resulting from the lossy compression of coefficient 57 data may be at least partially compensated by applying correction to part of the 16A tonal map and / or part of the HDR 16B information.

[0041] Figure 4 is a block diagram illustrating the operation of methods 60 that relate to data correction in a portion of tonal map 16A or a portion of HDR information 16B to reduce artifacts resulting from coding a lossy portion of tonal map 16A and / or a portion of HDR information 16B. Method 60 obtains HDR 90 image data and tonal map data 91 in blocks 62 and 64. HDR 90 image data and tonal map 91 data can be obtained in any suitable manner in this manner as described above. In some embodiments, the tonal map data 91 is obtained from HDR image data 90 as indicated by arrow 65.

[0042] At block 66, tonal map data 91 is encoded to obtain coded tonal map data 92. In some embodiments, coding block 66 includes JPEG or MPEG encoding. Then, at block 68, the encoded tonal map data 92 is decoded to obtain reconstructed tonal map data 94. Reconstruction block 68 may include passing coded tonal map data 92 to a suitable decoder such as a JPEG or MPEG decoder in the case where block 66 includes JPEG or MPEG encoding.

[0043] Block 70 generates coefficient 96 data by using a function that receives as input values ​​data from the HDR image 90 (prime values) and corresponding values ​​from the reconstructed tonal map data 94 (second values). The function includes dividing the first values ​​by the second values ​​or vice versa. In a simple embodiment of the invention, the coefficient 96 data include the RI value for each pixel in the image described by:

RI (x, y)

L (HDR (x, y))

L (TM (x, y)) (1) where: (x, y) are the coordinates identifying the pixel; L is a function that returns the luminance of a pixel from the pixel data; HDR (x, y) is the pixel data in HDR 90 image data with (x, y) coordinates; and TM (x, y) are pixel data in the reconstructed tonal map data 94 (or tonal map data 91) for a pixel having coordinates (x, y). In some embodiments, the coefficient data stores the log RI, the square root of RI, or another RI function.

[0044] Blocks 72 and 74 encode coefficient 96 data. In this embodiment, the coding includes reducing the sample size of the coefficient 96 data in block 72 to obtain coefficient data with reduced sampling 98, and then compressing the coefficient data with reduced sampling 98 to obtain the encoded data factor 100. The amount of downsampling performed by a small portion of the HDR 16B image in block 72 can be selected based on competing goals and perform the HDR image reconstructed from the HDR 16 data structure reproducing the HDR 90 image data with the highest fidelity. In some embodiments of the invention, the coefficient 96 data have sufficiently reduced sampling that such coefficient data with reduced sampling 98 has fewer pixels than the coefficient 96 data by a multiplier in the range of 4 to 15.

[0045] For example, sampling reduction can be performed using a Gaussian filter according to weighting formula e^{- (XA2 / RA2)}, where x is the distance from the center of the output pixel in the input image, and R is the sampling reduction radius. The sampling reduction radius can be defined as the area in which the weights of the input pixel components add up to a significant part of the total value of the output pixel.

[0046] Any suitable form of data compression may be performed at block 74. In some embodiments, block 74 performs JPEG encoding. In other embodiments, block 74 performs MPEG encoding.

[0047] In block 76, reconstructed factor 102 data is created by decoding the encoded factor 100 data. The reconstructed factor 102 data will usually not be identical to the factor 96 data due to data loss in blocks 74 and 76.

[0048] In block 78, reconstructed HDR image data 104 is created by applying a factor of 102 to the reconstructed data, the inverse of the function used in block 70 to the coefficient data, and then, for each pixel, multiplying by the luminance result for the pixel in the reconstructed tonal map data 94 . For example, when coefficient 96 data stores RI values ​​as defined in equation (1), then reconstructed HDR image data 104 can be obtained by multiplying the luminance for each pixel in reconstructed tonal map data 94 by the corresponding RI value with reconstructed coefficient data 102. For example, when the coefficient data stores the natural log ln (RI) values, the reconstructed HDR image data 104 can be obtained by raising e, the base of the natural logarithm, to the power of the values ​​in the reconstructed coefficient data 102, and then multiplying the result by the luminance for each pixel in the reconstructed data tonal map 94.

[0049] The reconstructed HDR image data 104 will differ from the original HDR 90 image data because the reconstructed factor 102 data is not the same as the original factor 96 data and usually, less important because of rounding errors in the factor 96 data. Block 80 optional compares the reconstructed HDR 104 image data with the original HDR 90 image data to determine if any correction is required and how the correction should be performed. Correction may be performed by correcting data of the portion of the tonal map 16A and / or correcting the data of the portion of HDR 16B information. Some methods simply do one or the other of these corrections.

[0050] Block 82 obtains corrected tonal map data 106. Corrected tonal map data 106 can be obtained by dividing the original HDR image data 90 by the reconstructed factor 102 data. The corrected tonal map data 106 can now be encoded if necessary as block 83 indicates and saved , as part of the tonal map data 16A of the HDR data structure 16 in block 84. This pre-correction can be made at any time after the reconstructed factor 102 data is available. For many purposes, this pre-correction does not significantly degrade the image, which can be seen by viewing part of the 16A tonal map using a conventional image viewer. This correction tends to get the image represented by part of the 16A tonal map slightly sharper than in the absence of correction. The reconstructed HDR image data 104 may be stored as part of the HDR information 16B of the HDR data structure 16 in block 86.

[0051] In some cases, it is undesirable to change the tonal map data recorded in the tonal map portion 16A. For example, the encoded tonal map data 92 may be carefully optimized to provide the best image quality when viewed on a particular browser, such as, for example, an MPEG decoder in a DVD player. In such cases, the encoded tonal map data 92 may be stored in the tonal map portion 16A of the data structure 16, and the coefficient data 96 may be stored in the HDR information portion 16B of the HDR data structure 16. Corrections to the appearance of the HDR image formed from the data structure 16 may be performed by correcting part of the HDR 16B information during HDR image reconstruction. For example, the data in part of the HDR 16B information may be corrected by a browser capable of processing HDR images.

[0052] Figure 5 is a block diagram illustrating the operation of a method 110 that applies final correction to data in a portion of HDR information 16B to reduce artifacts resulting from lossy coding of a portion of HDR information 16B. Method 110 can be implemented in a processor capable of processing HDR images. Part of the tonal map data 16A is decoded at block 112 using a standard decoder to obtain a standard image 19. The decoded tonal map information is used to correct some of the HDR 16B information in block 114. The corrected HDR information is decoded in block 116 using the HDR decoder to obtain the reconstructed HDR 21 image.

[0053] In simple cases, when the spatial frequency content of the full resolution image represented by part of the 16A tonal map data is substantially the same as for the coefficient 96 data then corrected coefficient data can be obtained by performing the calculation:

<img file="PL1743301T3_D0001.tif" />

where: RICORRECTED is the corrected RI value on which the corrected HDR information is based; RI is the ratio for the pixel of the coefficient 96 data; L (TM) is the luminance of a pixel from the tonal map data 91; and L (TMR) is the luminance for the corresponding pixel of the tonal map data that has been subjected to reduced sampling in the same manner as it is performed in block 72 to obtain the reduced sampling rate data 98. The tonal map data can be subjected to reduced sampling and then again increased sampling in the same way as the RI image, so that TM and TMR have the same resolution.

[0054] Such simple correction is not always adequate, because the spatial frequencies present in the coefficient 96 data are not, for all images, the same as the spatial frequencies present in the tonal map data 91. Therefore, it is preferable to include a factor that takes into account the variance in the correction function the ratio between the RI values ​​in the coefficient 96 data and the corresponding L (TMR) values. One way to account for this variance is to generate adjusted RICORRECTED values ​​by:

<img file="PL1743301T3_D0002.tif" />

where: σ is a measure of the variance of the ratio between the RI values ​​in the data of the coefficient 96 and the corresponding L values ​​(TMR). In some embodiments, σ ​​is calculated according to:

var (7? Z) ^{σ =} var (£ (n /_s)) ⁽⁴⁾ [0055] The variance function var (x) can be defined as the difference between the maximum and minimum value of x for pixels in the environment, divided by the average value of x in the environment or divided by the value of x for a pixel located centrally in the environment. For example, the variance can be calculated for a block of pixels centered on the discussed pixel. The size of the environment for which σ is calculated is preferably equal to the sampling reduction radius for reducing the sampling of block 72.

[0056] Since the final correction implemented by block 114 can introduce artifacts, conservative behavior in the selection of the correction amount is desirable. For example, when var (L (TMR)) is greater than the error to be corrected, you can set σ to zero. The magnitude of the error can be determined by comparing block 80 with the stored data structure 16. It is also desirable to ensure that 0 <σ <1. Allowing σ to take values ​​such as σ> 1 can lead to undesirably high values ^RICORRECTED^. [0057] In embodiments of the invention that optionally perform pre-correction of blocks 82, 83 and 84, and which also allow block 83 to be performed while displaying the HDR image, it is desirable to insert a flag in the data structure indicating whether or not it has been performed. initial correction. The flag is preferably placed in the comment field or in the application extension field, where it can be ignored by standard displays that do not support HDR images.

[0058] In some cases, HDR displays may render colors from outside the conventional display's color gamut. It is desirable to provide a mechanism that reproduces with high fidelity the colors specified by the original HDR image data. One way to provide an improved color is to scale the color information so that any color that has a basic component out of range that can be effectively handled by the encoder used to encode part of the 16A tonal map (which can be, for example, a JPEG or MPEG encoder) is backscaled to the extent that it can be handled by the encoder. Coefficient data can be adjusted to correctly reproduce the scaled color.

[0059] One way to provide improved color is to provide global desaturation of the image when creating part of the 16A tonal map. The desaturation size can be selected to ensure that all colors in the image are within a range that can be effectively handled by the JPEG encoder or another used to encode parts of the 16A tonal map. This method is more preferred than the method described above because it can handle colors having negative main components. Negative main components are acceptable in some HDR formats and may be necessary to represent colors outside the standard RGB range. The desaturation process can be reversed during decoding by the HDR browser.

[0060] The saturation level of the input color can be defined as:

"Min (R, G, B) _f

5 = 1- - 3-7 — L ·. / (5) where: S is the saturation level; R, G and B are the values ​​for the red, green and blue component of the primary color respectively; and Y is the overall luminance. The saturation level will be higher than one if the image contains any negative values ​​for the primary color components.

[0061] When the saturation level is zero, no additional image processing is needed. If the saturation level is different from zero, the saturation level can be modified according to:

5 '= axS ^ (6) where: α and β are parameters; and S 'is the corrected saturation. The α parameter determines how much saturation to keep in coded colors.

[0062] A change in saturation level can be obtained by acquiring new values ​​for the basic components for each pixel of the image. This is done in some embodiments according to:

<img file="PL1743301T3_D0003.tif" />

and

<img file="PL1743301T3_D0004.tif" />

and

<img file="PL1743301T3_D0005.tif" />

where R ', G' and B 'are scaled values ​​for R, G and B, respectively.

[0063] Please note that this transformation does not change luminance, Y. The primary component that was the smallest before the transformation remains the smallest after the transformation. The original color values ​​can be recovered by reversing equations (7), (8) and (9). For example, if the component of the primary color having the lowest value for a pixel was blue, then the inverse transformation for the blue channel for that pixel will be determined by:

<img file="PL1743301T3_D0006.tif" />

and inverse transformations for the red and green channels will be described by:

<img file="PL1743301T3_D0007.tif" />

and

<img file="PL1743301T3_D0008.tif" />

Examples [0064] A series of HDR images were stored in HDR 16 data structures as described above. Original images were compared with HDR images reconstructed from HDR 16 data structures. A preliminary formula was used to assess Daly's Visual Differences Predictor (VDP) as described in Daly, S., The visual differences predictor: An algorithm for the assessment of image fidelity, In Digital Images and Human Vision, AB Watson editor, MIT Press, Cambridge Massachusetts, 1993, to assess what percentage (i.e. has a probability greater than 75%) of pixels in reconstructed HDR images can be seen by people as different from the corresponding pixels of the original HDR image under typical viewing conditions. It was found that VDP is an excellent preliminary formula for forecasting when differences between images will be noticed.

[0065] The first set of experiments involved using different tonal mapping operators to produce a portion of tonal map 16A, and for each tonal map operator, correcting either part of tonal map 16A or part of HDR information 16B in accordance with one of the correction methods described above. Part of the 16A tonal map and HDR 16B information were encoded using JPEG encoding at two quality levels 90 and 100. A set of experiments gave the results shown in Table I.

<td colspan="4">Table I - Image quality for several tonal mapping operators</td>

<td>Mapping Operator</td><td rowspan="2">JPEG quality</td><td>VDP with application</td><td>VDP with application</td>

<td>tonal</td><td>initial correction</td><td>final correction</td>

<td>Double-sided filter</td><td> 90</td><td> 0.93%</td><td> 5.4%</td>

<td></td><td> 100</td><td> 0.02%</td><td> 1.8%</td>

<td>Reinhard</td><td> 90</td><td> 2.5%</td><td> 4.7%</td>

<td>Global</td><td> 100</td><td> 0.09%</td><td> 2.8%</td>

<td>Histogram Adj.</td><td> 90</td><td> 5.9%</td><td> 21%</td>

<td></td><td> 100</td><td> 0.63%</td><td> 17%</td>

<td>Gradient</td><td> 90</td><td> 7.5%</td><td> 36%</td>

<td></td><td> 100</td><td> 3.0%</td><td> 34%</td>

[0066] The VDP values ​​in Table I are average values ​​for a number of images. It can be seen that the choice of the tonal mapping operator can have a significant impact on the quality of the HDR image, which will be reconstructed from the HDR 16 data structure. Of the tonal mapping operators used in this experiment, on average, a double-sided filter gives the best results.

[0067] Some implementations of the invention include computer processors executing software instructions that cause the processors to perform the method of the invention. For example, one or more processors in a computer system may implement the method of any of figures 1 to 5 by executing software instructions in program memory available to processors. The invention may also be provided in the form of a software product. The software product may contain any medium carrying a set of computer-readable signals containing instructions that, when executed by a computer processor, will cause the data processor to perform the inventive method. The software product of the invention may be in one of a large range of forms. The software product may contain, for example, a physical medium such as a magnetic data storage medium including floppy disks, disk drives, optical storage media such as CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM or the like, or such transmission media like digital or analog communication links. Instructions may optionally be present in the signals read by the computer in a compressed and / or encrypted format.

[0068] If components (e.g., software module, processor, assembly, device, circuit, etc.) are mentioned above, unless otherwise indicated, reference to this element (including reference to "center") should be interpreted as including equivalents of this element, any elements that perform the functions of the element described (i.e. which are functionally equivalent), including elements that are not structural equivalents of the disclosed structure that performs functions in the illustrated embodiments of the invention.

Prepared and verified

Anna Stenzel Patent Attorney