Image formats and related methods and apparatuses

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No. 61/539,438, filed Sep. 26, 2011, hereby incorporated by reference in its entirety.

TECHNICAL FIELD OF THE INVENTION

The invention relates to the media industry and has application in delivery of image content such as video and/or still images to consumers who view the images on displays such as televisions, digital cinema displays, media viewers and the like. The invention has particular application to formats for transmission of image data, displays, and methods for encoding, decoding and/or displaying image data.

BACKGROUND OF THE INVENTION

Video content is typically prepared and distributed by way of a combination of systems and technologies that may be termed a “video delivery pipeline”. <figref idref="DRAWINGS">FIG. 1</figref> is a flowchart of a conventional video delivery pipeline <b>100</b> showing various stages from video capture to video content display. A sequence of video frames <b>101</b> is captured at block <b>102</b>. Video frames <b>101</b> may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g. using computer animation) to provide video data <b>103</b>. Alternately, video frames <b>101</b> may be captured on film by a film camera. The film is converted to a digital format to provide video data <b>103</b>. In a production phase <b>104</b> video data <b>103</b> is edited to provide a video production <b>105</b>.

Video data of production <b>105</b> is provided to a processor at block <b>106</b> for post-production editing. Block <b>106</b> post-production editing may include adjusting or modifying colors or brightness in particular areas of an image to enhance the image quality or achieve a particular appearance for the image in accordance with the video creator's creative intent. This is sometimes called “color timing”. Other editing (e.g. scene selection and sequencing, image cropping, addition of computer-generated visual special effects, etc.) may be performed at block <b>106</b> to yield a final version <b>107</b> of the production for distribution. During block <b>106</b> post-production editing, video images are viewed on a reference display <b>108</b>. In block <b>106</b> the final production <b>107</b> is viewed on reference display <b>108</b> or another reference display for approval. It is not mandatory that the same display be used for color timing and approval.

Following post-production, video data of final production <b>107</b> is delivered at block <b>116</b> to a display subsystem <b>120</b>. As seen in <figref idref="DRAWINGS">FIG. 1A</figref>, block <b>116</b> delivery includes an encoder stage <b>117</b>A which generates encoded video data <b>118</b> embodying the content of video data <b>107</b> to be distributed by way of a video distribution medium <b>115</b> (e.g. satellite, cable, DVD, wireless communication link, internet, local area network, broadcast, etc.). A decoder stage <b>117</b>B is located downstream from encoder stage <b>117</b>A to decode video data <b>118</b> transmitted over medium <b>115</b>.

Display subsystem <b>120</b> may perform video processing <b>120</b>A and displaying <b>120</b>B. Video processor <b>120</b>A may be integrated with displaying <b>120</b>B or may be separate. At block <b>120</b>A, video data <b>118</b> is provided to a video processor for processing and/or decoding. Video data <b>118</b> is output to a display <b>122</b> at block <b>120</b>B to display a sequence of images to a viewer.

Encoded video data <b>118</b> may have a format selected with reference to properties of medium <b>115</b> (for example to fit within bandwidth requirements and/or format requirements of medium <b>115</b>). To improve the quality of displayed images, encoded video data <b>118</b> may be driven through video delivery pipeline <b>100</b> at a relatively high bit rate so as to facilitate an increased bit depth for defining RGB or chroma values for each chrominance (color) channel. For example, video data <b>118</b> may comprise 8, 10 or 12 bits of data for each color channel of a pixel. The video data may be compressed.

Despite using a high bit depth for each chrominance channel, variations in display characteristics (such as luminance range, gamut, etc.) may affect the appearance of an image rendered on a display so that the image rendered does not match the creative intent of the video's creator. In particular, the perceived color or brightness of an image rendered on a particular display subsystem may differ from the color or brightness of the image as viewed on reference display <b>108</b> during post-production block <b>106</b>.

The same video content <b>107</b> may be displayed on any of a wide variety of different types of electronic displays including televisions, computer displays, special purpose displays such as medical imaging displays or virtual reality displays, video game displays, advertising displays, displays on cellular telephones, tablets, media player displays, displays in hand-held devices, displays used on control panels for equipment of different kinds and the like. Displays may employ any of a wide range of technologies. Some non-limiting examples are plasma displays, liquid crystal displays (LCDs), cathode ray tube (CRT) displays, organic light emitting diode (OLED) displays, projection displays that use any of various light sources in combination with various spatial light modulation technologies, and so on.

Different displays may vary significantly with respect to features such as: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0011">the color gamut that can be reproduced by the display;</li><li id="ul0002-0002" num="0012">the maximum brightness achievable;</li><li id="ul0002-0003" num="0013">contrast ratio;</li><li id="ul0002-0004" num="0014">resolution;</li><li id="ul0002-0005" num="0015">acceptable input signal formats;</li><li id="ul0002-0006" num="0016">color depth;</li><li id="ul0002-0007" num="0017">white level;</li><li id="ul0002-0008" num="0018">black level;</li><li id="ul0002-0009" num="0019">white point; and</li><li id="ul0002-0010" num="0020">grey steps. <br /> Consequently, the same image content may appear different when played back on different displays. Image content that matches a creator's creative intent when displayed on some displays may depart from the creator's creative intent in one or more ways when viewed on other displays. The appearance of displayed images is also affected by the environment in which a display is being viewed. For example, the luminance of ambient lighting, the color of ambient lighting and screen reflections can all affect the appearance of displayed images. </li></ul></li></ul>

With the increasing availability of high-performance displays (e.g. displays that have high peak luminance and/or broad color gamut) comes the problem of how to adjust images for optimum viewing on a particular display or type of displays. Addressing this problem in simplistic ways can result in noticeable artifacts in displayed images. For example, consider the case where an image that appears properly on a display having a moderate peak luminance is displayed on a target display having a very high peak luminance. If one expands the luminance range of the image data to take advantage of the high peak luminance of the target display, the result may be poor due to objectionable artifacts that are rendered apparent by the range expansion. Artifacts may include, for example, one or more of banding, quantization artifacts, visible macroblock edges, objectionable film grain and the like. On the other hand, if the image is displayed on the target display without range expansion, no benefit is gained from the high peak luminance that the target display can achieve.

Video formats used in many current imaging systems (e.g. HDTV, UHDTV) are based on defining black and white levels with a power response. This makes it very difficult to ensure consistent midtones, contrast, and color when video content derived from the same video data <b>107</b> is viewed on different displays.

The system pipeline response can be represented by an end-to-end transfer function that compares an image on a reference display and the same image displayed on a target display. Makers of displays typically make assumptions about the response of the reference display (and environment) in which images were approved. Displays may process image data in various ways (e.g. applying a power function response, making color saturation adjustments, adjusting brightness adjustment, adjusting contrast etc.) to arrive at an image that the display maker thinks will be best appreciated by viewers.

For accurate reproduction of video images, video distribution systems of the type illustrated in <figref idref="DRAWINGS">FIGS. 1 and 1A</figref> typically require that the system response of the pipeline be tailored to match characteristics of the reference display on which the content was approved and/or color graded. Display makers can attempt to provide image processing that achieves a desired system response, for example by defining a response curve between a minimum and maximum luminance. The minimum and maximum luminance may be implied from assumed capabilities of a reference display. Displays that perform image processing based on wrong assumptions regarding the characteristics of a reference display on which the image data was approved or color graded may produce images that are not faithful to the image as approved by its creator. For example different interpretations of the same video signal can result in inconsistent midtones, and other characteristics of images displayed on different displays.

Another issue with video distribution systems of the type illustrated in <figref idref="DRAWINGS">FIGS. 1 and 1A</figref> is that changing to a different reference display or altering the configuration of the reference display or its viewing environment can require changes to the system response of the pipeline if one wishes to ensure that viewers have the highest quality viewing experience. One approach to dealing with this issue is to provide metadata along with video data <b>118</b>. The metadata can specify the source gamut. The target displays may then be made to adapt themselves by providing image processing selected based on the metadata to achieve the desired system response. This adds complexity and is prone to failure for some distribution channels (e.g. broadcast).

There is a general desire for systems, apparatus and methods for generating, delivering, processing and displaying video data to preserve the content creator's creative intent. There is a general desire for systems, apparatus and methods for providing information which may be used to guide downstream processing and/or display of video data.

The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.

SUMMARY OF THE INVENTION

This invention has several aspects. These include methods for displaying digital images, methods for preparing digital images for distribution, video delivery pipelines and their respective components.

One aspect of the invention provides image data encoding methods which fix mid-tones of both reference and target displays to a common response and allow highlights and shadows to be extended and/or compressed based on the capabilities of the target display. The result is an imaging system in which image processing at the target display can be performed depending only on characteristics of the target display or characteristics of the target display and its environment (and does not need to be based on characteristics of the image source). In some embodiments image data is assigned to image values represented by image data according to a sigmoidal response curve. The same sigmoidal response curve may be used in the reference display used in creation and/or approval of the source content.

Another aspect of the invention provides image data encoding methods in which the sizes of the steps between levels in which luminance is encoded increase more slowly and/or become smaller as luminance increases above a mid-tone range and/or decreases below the mid-tone range.

Another aspect of the invention provides methods for encoding image data. The methods comprise processing image data to provide image values corresponding to quantized luminance steps wherein a step size between the quantized luminance steps is smaller in a highlight region than in a mid-tone region.

Another aspect of the invention provides methods for displaying images (which may be still or video images) on a target display. The methods comprise processing image data that has been reviewed on a reference display and encoded such that a predetermined image value of the image data corresponds to a predetermined mid-tone value displayed on the reference display. The processing comprising applying a mapping to the image data that maps the predetermined image value to the predetermined mid-tone value when viewed on the target display. In certain embodiments the mapping provides a log-linear mid-tone region having a slope n that matches or approximates a slope of a corresponding mid-tone region of a response function of the reference display. In an example embodiment the mapping is specified by a sigmoidal function. An example sigmoidal mapping function is given by the equation:

<maths id="MATH-US-00001" num="00001"><math overflow="scroll"><mrow><msub><mi>L</mi><mi>OUT</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>c</mi><mn>1</mn></msub><mo>+</mo><mrow><msub><mi>c</mi><mn>2</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow><mrow><mn>1</mn><mo>+</mo><mrow><msub><mi>c</mi><mn>3</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow></mfrac></mrow></math></maths><img file="US8988552B2_D0001.tif" /><br /> which is discussed below.

Some embodiments involve adjusting the mapping in response to ambient lighting. The adjusting may, for example, comprise one or more of: increasing the mid-tone value to which the predetermined image value is mapped in response to an increase in the ambient lighting; and decreasing mid-tone contrast of the mapping function.

Some embodiments comprise modeling the effect of the ambient lighting on light adaptation of the human visual system to obtain an estimated adaptation and adjusting the mapping based on the estimated adaptation. Modeling the effect of the ambient lighting on the adaptation of the human visual system may optionally comprise modeling a plurality of adaptation aspects to obtain a corresponding plurality of estimated adaptations. In such cases the method may comprise adjusting different characteristics of the mappings based on different ones of the estimated adaptations. For example, adjusting the mapping may comprise one or more of: controlling a luminance to which the predetermined image value is mapped based at least in part on an estimate of a general brightness adaptation; controlling mid-range contrast based at least in part on an estimate of lateral brightness adaptation; and controlling contrast in highlight and shadow regions based at least in part on an estimate of local brightness adaptation.

In cases where the target display has a greater peak luminance than the reference display the method may comprise applying a mapping configured to expand the luminance corresponding to image values corresponding to a highlight region above the mid-tone region. In cases where the target display has a lower black level than the reference display the method may comprise applying a mapping configured to expand the luminance corresponding to image values corresponding to a shadow region below the mid-tone region.

Another aspect of the invention provides cameras and other image capture devices that map detected light to output image values according to a response function. In some embodiments the response function is a sigmoidal function. An example image capture device according to this aspect comprises a light-sensing array operable to detect light and a mapping unit connected to represent intensity of light sensed by elements of the light-sensing array as digital image values according to a response function. An exposure compensation mechanism is connected to control the mapping unit such that a fixed point in the mid-range of the response function is made to correspond to a midpoint of the output image values. In some embodiments the response function is given by the equation:

<maths id="MATH-US-00002" num="00002"><math overflow="scroll"><mrow><msub><mi>L</mi><mi>OUT</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>c</mi><mn>1</mn></msub><mo>+</mo><mrow><msub><mi>c</mi><mn>2</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow><mrow><mn>1</mn><mo>+</mo><mrow><msub><mi>c</mi><mn>3</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow></mfrac></mrow></math></maths><img file="US8988552B2_D0002.tif" /><br /> and the exposure compensation mechanism controls the value of one or both of n and c₂based at least in part on a measured exposure.

Other example aspects of the invention provide displays and image processing apparatus.

Further aspects of the invention and features of specific embodiments of the invention are described below.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate non-limiting example embodiments of the invention.

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a video distribution pipeline.

<figref idref="DRAWINGS">FIG. 1A</figref> is a block diagram illustrating distribution of image content over a distribution medium.

<figref idref="DRAWINGS">FIG. 2</figref> is a graph which illustrates a possible gamma-based response curve for a reference display.

<figref idref="DRAWINGS">FIG. 2A</figref> is a graph which illustrates possible response curves for two different example target displays.

<figref idref="DRAWINGS">FIG. 2B</figref> is a graph which illustrates system response curves for the case where video data approved on a display having a response as shown in <figref idref="DRAWINGS">FIG. 2</figref> is displayed on two different target displays having the response functions of <figref idref="DRAWINGS">FIG. 2A</figref>.

<figref idref="DRAWINGS">FIG. 3</figref> shows a response curve for an example reference display.

<figref idref="DRAWINGS">FIG. 3A</figref> shows response curves corresponding to three example target displays.

<figref idref="DRAWINGS">FIG. 3B</figref> shows curves representing a system response for different combinations of two different target displays and two different reference displays having different peak luminance.

<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram for an example video delivery pipeline.

<figref idref="DRAWINGS">FIG. 5</figref> shows example response curves that may be provided by a mapping unit for dim and bright ambient conditions.

<figref idref="DRAWINGS">FIG. 5A</figref> shows system response curves corresponding to the response curves of <figref idref="DRAWINGS">FIG. 5</figref>.

<figref idref="DRAWINGS">FIG. 6</figref> shows possible response functions that may be provided in capture devices for different ambient lighting conditions.

<figref idref="DRAWINGS">FIG. 7</figref> shows example system response curves corresponding to the curves of <figref idref="DRAWINGS">FIG. 6</figref>.

DESCRIPTION OF THE INVENTION

Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.

Luminance and other values in image data (“image values”) are typically gamma encoded. Simple gamma encoding involves applying a power law to image data, for example according to: <br /><i>T</i>_OUT<i>=AT</i>_IN^γ<br /> where T_INis the input value for luminance or other image value, T_OUTis the output value for the luminance or other image value, A is a constant (typically 1) and γ is the gamma value. Images are typically encoded with values for γ that are less than one (for example, about 0.45). When gamma-encoded images are displayed, the display process applies a gamma expansion (gamma value larger than 1—for example 2.2 or 2.4) which results in a desired image being displayed.

To date, standard dynamic range (SDR) reference displays have power function response characteristics. The inventors have determined that this response does not preserve end-to-end system luminance levels when content is shown on displays having luminance ranges different from that of the reference display. This distorts image appearance. <figref idref="DRAWINGS">FIG. 2</figref> shows a generalized example response curve <b>201</b> for an example reference display. <figref idref="DRAWINGS">FIG. 2A</figref> shows example response curves <b>202</b>A and <b>202</b>B for two different target displays. All three displays have the same gamma (2.4 in this example) but the target displays have higher peak luminance than the reference display and the target display corresponding to curve <b>202</b>B has a higher black level than the reference display.

<figref idref="DRAWINGS">FIG. 2B</figref> shows the resulting system response curves <b>203</b>A and <b>203</b>B for images respectively viewed on the two example target displays having response curves <b>202</b>A and <b>202</b>B. For a perfect match of appearance (the image would appear the same on the target display as on the reference display) luminance levels would fall on dashed line <b>204</b>. However, since both target displays have a higher peak luminance than the reference display, the output levels far exceed the intent. If the visual systems of viewers viewing the target displays are adapted to the same ambient lighting conditions as for the reference display, the result would be an image which appeared much brighter than intended. Also, system responses <b>203</b>A and <b>203</b>B are altered near black. Response curve <b>203</b>A is substantially steeper in the dark region <b>205</b> than the ideal response curve <b>204</b>. This has the effect of stretching levels apart, or increasing contrast, further distorting the creative intent.

Curves <b>206</b>A and <b>206</b>B in <figref idref="DRAWINGS">FIG. 2B</figref> are response curves for the case where the reference display has a higher peak luminance (e.g. 600 nits as opposed to 100 nits) but with the same gamma=2.4 response. These curves show how the system response curves change drastically throughout the entire range depending on the characteristics of the reference display, indicating that appearance is dependent on the characteristics of the reference monitor. <figref idref="DRAWINGS">FIGS. 2 through 2B</figref> demonstrate that mismatch between the capabilities of the reference and target displays in a gamma-based system result in appearance not being preserved through the video pipeline. This can degrade the viewing experience.

The inventors have determined that at least some of the problems with the prior art approach described above arise because values in encoded video data do not correspond to specific luminance values but instead to values determined with reference to the ranges of the displays on which they are displayed. This is a special problem with mid-range values (e.g. values in the range of about 0.1 nits to 100 nits in a dim viewing environment). Pixels have mid-range values in the most salient portions of most images. This issue may be addressed by providing image data formats and systems in which mid-range values are consistently mapped into the same range.

One way to achieve this is illustrated in <figref idref="DRAWINGS">FIGS. 3 and 3A</figref>. These figures illustrate response curves for a number of displays that are constructed such that mid-range values in image data are consistently mapped into the same luminance range. In these example curves, values outside of the mid-range are compressed or expanded smoothly and gently into the range of the display. <figref idref="DRAWINGS">FIG. 3</figref> shows a response curve <b>301</b> corresponding to an example reference display. <figref idref="DRAWINGS">FIG. 3A</figref> shows response curves <b>302</b>A, <b>302</b>B, and <b>302</b>C corresponding to three example target displays. In each case, for image values in a mid-range region <b>303</b> the mappings of image values to luminance is essentially the same.

A feature of response curve <b>301</b> is that for higher image values the luminances increase relatively slowly with increasing values (the luminances increase in relatively small steps). This results in redundant precision at such higher values (since the human visual system cannot distinguish between luminances that differ by less than a threshold amount). This, in turn, facilitates expansion of the luminance corresponding to higher image values without creating perceptible quantization artifacts such as banding and the like that can result when expansion causes adjacent image values to correspond to noticeably different luminance levels.

In some embodiments, response curve <b>301</b> is such that the difference in luminance specified by adjacent levels is smaller than a just noticeable difference (JND) at least in the portion of response curve <b>301</b> toward higher luminance values (for example, at least in the right hand ¼ of curve <b>301</b>). There are various ways to quantify the JND. A simple way to quantify JND is Weber's ratio which can be given by:

<maths id="MATH-US-00003" num="00003"><math overflow="scroll"><mrow><mi>R</mi><mo>=</mo><mfrac><mrow><mi>Δ</mi><mo>⁢</mo><mstyle><mspace width="0.3em" height="0.3ex" /></mstyle><mo>⁢</mo><mi>L</mi></mrow><mi>L</mi></mfrac></mrow></math></maths><img file="US8988552B2_D0003.tif" /><br /> where ΔL is the luminance step size and L is the background luminance level. One estimate of the JND is that the JND corresponds to a Weber's ratio of 1%. In some embodiments, the response function <b>301</b> for a reference display is such that the difference in luminance specified by adjacent image values is less than ½, ⅕ or 1/10 of a JND. This permits expansion of the luminance range by a factor of up to 2, 5 or 10 while preserving the desirable quality that the difference in luminance specified by adjacent quantized image values does not exceed a JND. For example, adjacent image values may correspond to adjacent luminances for which the Weber's ratio is less than 0.005, 0.002 or 0.001. In other embodiments another estimate of JND is applied. for example, as described in G. Ward, <i>A Contrast</i>-<i>Based Scalefactor for Luminance Display</i>, Graphics Gems IV, Ed. by P. S. Heckbert, pp. 415-421, 1994 or J. A. Ferwerda, et al. <i>A Model of Visual Adaptation for Realistic Image Synthesis</i>, Computer Graphics, pp. 249-258, 1996. In such cases, the response curve <b>301</b> for a reference display may be such that the difference in luminance specified by adjacent image values is less than ½, ⅕ or 1/10 of a JND to provide for the possibility of a desired range of expansion without causing artifacts which result from adjacent quantized luminance values differing by significantly more than a JND when displayed on a target display.

In the embodiment illustrated in <figref idref="DRAWINGS">FIGS. 3 and 3A</figref> the response functions of the reference and target displays are all sigmoidal functions. In each case, the sigmoidal function comprises parameters that allow a fixed point to be set. The fixed point is a point corresponding to a specific mid-range image value and a specific corresponding luminance value. The sigmoidal function also has one or more parameters that allow control over the slope of the response function in a mid-range region that includes the fixed point. These parameters can be set so that the response functions for the reference and target displays all pass through the fixed point and all have the same slope in a mid-range region.

The sigmoidal response functions may differ in the ends of the range of image values. This flexibility allows the ends of the response functions to be arranged to extend to the display's capabilities. In this illustrated example embodiment, on all displays the input image values are in a range of zero to one. In each case, the input image value 0.5 maps to 10 nits with the same mid-range slope (contrast). The ends of each response function smoothly roll off according to the display's capabilities.

The response functions of target displays may be specified as a standard. For example, a sigmoidal response may be specified in such a manner that the shape of the response curve is specified by parameters that specify a minimum, a maximum, a mid-range value and a mid-range slope (or contrast). The standard may specify the mid-range value and a mid-range slope. The reference display and target displays may all use response curves characterized by the same mid-range value and a mid-range slope. Different target displays may use different maximum and minimum values. The response curves in the target displays may smoothly roll off to the maximum and minimum values.

A specification does not necessarily fix a mid-point in an absolute sense. As an alternative, a specification could specify a rule for determining the mid-point based on capabilities of the target display. For example the mid-point may be specified based on a geometric mean luminance of the target display. In an example, the mid point may be set for a target display by determining an intermediate value Lat′ according to: <br /><i>L</i>_at′<i>=A</i>√{square root over ((<i>L</i>_min<i>·L</i>_max))}<br /> where Lmin and Lmax are the extremes of the range of the target display and A is a constant and then determining the mid-point for the target display according to: <br /><i>L</i>_at=√{square root over (<i>L</i>_as<i>·L</i>_at′)}<br /> where Las is a predetermined midpoint (e.g. 10 nits) and Lat is the mid-point to be used for the target display.

Similarly, mid-range contrast need not be specified as a fixed value but could, in some embodiments, be based in part on capabilities of a target display. For example, mid-range contrast could be based on a value n′ determined according to:

<maths id="MATH-US-00004" num="00004"><math overflow="scroll"><mrow><msup><mi>n</mi><mi>′</mi></msup><mo>=</mo><mrow><mfrac><mn>1</mn><mn>2</mn></mfrac><mo>⁢</mo><mrow><msub><mi>log</mi><mn>10</mn></msub><mo>⁡</mo><mrow><mo>(</mo><mfrac><msub><mi>L</mi><mi>max</mi></msub><msub><mi>L</mi><mi>at</mi></msub></mfrac><mo>)</mo></mrow></mrow></mrow></mrow></math></maths><img file="US8988552B2_D0004.tif" /><br /> the mid-range slope, n, for the response curve of the target display may then be given by: <br /><i>n</i>=√{square root over (<i>n</i>_S<i>·n</i>′)}<br /> where ns is a specified standard slope.

Roll off of the response functions for a reference or target display at high image values may be determined based on the peak luminance of the display. A display with 120 nit peak luminance will have a much more shallow rolloff than a display with 1200 peak luminance. The exact slope at any point in the rolloff region for a reference display is not too important from the point of view of a colorist as the colorist will compensate for it during mapping.

<figref idref="DRAWINGS">FIG. 3B</figref> shows curves <b>310</b>A and <b>310</b>B representing the system response for two different target displays for the case where the reference display has a maximum luminance of 600 nits and curves <b>311</b>A and <b>311</b>B for the case where the reference display has a maximum luminance of 5000 nits. It can be seen that the mid-range is not affected by these changes and the system response in the mid-range is consistent across the target displays. The top and bottom ends of the system response functions stretch to exploit the capabilities of the target displays.

In an example embodiment, the response curves for the displays are given by:

<maths id="MATH-US-00005" num="00005"><math overflow="scroll"><mrow><msub><mi>L</mi><mi>OUT</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>c</mi><mn>1</mn></msub><mo>+</mo><mrow><msub><mi>c</mi><mn>2</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow><mrow><mn>1</mn><mo>+</mo><mrow><msub><mi>c</mi><mn>3</mn></msub><mo>⁢</mo><msubsup><mi>L</mi><mi>IN</mi><mi>n</mi></msubsup></mrow></mrow></mfrac></mrow></math></maths><img file="US8988552B2_D0005.tif" /><br /> where L_INis the input value for luminance or other image value. For example, L_INmay be a luminance in a range [min_S, max_S] where min_Sand max_Sare respectively standard minimum and maximum values. L_OUTis the output value for the luminance or other image value, c₁, c₂and c₃and n are parameters. Here, the value of n sets the mid-range contrast and the location of the mid-range fixed point is determined primarily by the value of c₂. The parameters c₁and c₃may be used to tune the response function to exploit the full dynamic range of a display.

The image values present in image data may be further encoded. In an example embodiment, the image values represent the logarithms of luminance levels. In an example of such a case, L_INmay be given in the range of [0.001, 1000] by: <br /><i>L</i>_IN=10^(6V-3)<br /> where V is a normalized image value in the range of zero to one.

As noted above, response functions may be determined with reference to a standard range [min_S, max_S]. It is possible to select the standard range to provide some degree of backward compatibility with video signals that are gamma encoded. For example, video data represented using a sigmoidal response function as defined above can be made to display nearly correctly on existing displays which assume standard gamma-encoded video input by appropriate selection of the standard mid-point, standard mid-range slope and standard range. For example, selecting [min_S, max_S] to be [0.005, 120] with suitable choices for ns and Las can yield a video signal that can be viewed on typical televisions with acceptable results.

Response functions having properties like that of the example response function of Equation (2) may be applied in various ways to provide media delivery pipelines. One example video delivery pipeline <b>400</b> is illustrated in <figref idref="DRAWINGS">FIG. 4</figref>. Pipeline <b>400</b> processes and distributes video content based on source video <b>402</b>. A color grading station <b>404</b> includes a reference display <b>406</b>. A mapping unit <b>410</b> maps color graded video <b>405</b> to be viewed on reference display <b>406</b>. Mapping unit <b>410</b> may, for example, implement a mapping function of the type illustrated by Equation (2). Mapping unit <b>410</b> may be a stand-alone device or may be integrated with reference display <b>406</b>, color grading station <b>404</b>, some other device, or a combination of these.

A color grader can use controls of color grading station <b>404</b> to adjust image values from source video <b>402</b> to yield color graded video <b>405</b>. The color grader can view color graded video <b>405</b> on reference display <b>406</b> and continue to make adjustments to color graded video <b>405</b> using controls provided by color grading station <b>404</b> until the color graded video <b>405</b>—as mapped by mapping unit <b>410</b>—has a desired appearance when viewed on reference display <b>406</b>.

Color graded video <b>405</b> may then be distributed over a distribution channel <b>412</b> for viewing on one or more target displays <b>420</b>. The color graded video <b>405</b> may be suitably encoded and decoded in distribution channel <b>412</b>. In some embodiments the color graded video <b>405</b> is quantized, encoded and represented in a data format apropos to the particular distribution channel <b>412</b> being used. In some embodiments color graded video <b>405</b> is formatted according to a VDR (visual dynamic range) format in distribution channel <b>412</b>. VDR format is a video format described in commonly assigned PCT Application No. PCT/US2010/022700 entitled “EXTENDED DYNAMIC RANGE AND EXTENDED DIMENSIONALITY IMAGE SIGNAL CONVERSION AND/OR DELIVERY VIA LEGACY VIDEO INTERFACES” which is hereby incorporated herein by reference for all purposes.

Each target display <b>420</b> is associated with a mapping unit <b>422</b> that maps color graded video <b>405</b> for display on the target display <b>420</b>. Mapping units <b>422</b> may be integrated with target displays <b>420</b> or a suitable mapping unit <b>422</b> may be otherwise provided upstream from each target displays <b>420</b>.

Mapping units <b>422</b> do not require information such as a reference gamut or environmental conditions regarding reference display <b>406</b> because the mappings performed by mapping units <b>410</b> and <b>422</b> have a common fixed mid-point and mid-tone contrast so that the mid-point and mid-tone contrast of images viewed on target displays <b>420</b> are automatically the same as those of the images when viewed on reference display <b>406</b>.

A mapping having parameters that allow control over the luminance that a specific mid-point will be mapped to and control over mid-range contrast (as, for example are provided by the parameters c2 and n of Equation (2)) can be applied to advantage in adjusting displays to account for ambient viewing conditions.

The human visual system reacts differently to images depending on its adaptation to light. For example, a viewer will perceive the same images on a television differently depending on whether the television is being viewed in an otherwise dark room or in a brightly lit room. Lateral adaptation describes the adaptation of a viewer's visual system to the brightness of the environment (including a display and its surroundings). Chromatic adaptation describes the adaptation of a viewer's visual system to the chromaticity of the viewer's surroundings (for example the color temperature of ambient lighting). Another environmental effect that can alter a viewer's perception of images on a display is viewing flare. Viewing flare describes the effect of screen reflections.

In general, lateral adaptation to brighter levels has the effect of raising what is perceived as “mid grey”, and also the perceived contrast. Lateral adaptation may be compensated for by raising the midpoint and lowering the mid-tone contrast. In some embodiments, mapping units <b>422</b> receive input from an ambient light sensor <b>425</b> and are configured to increase the luminance to which the mid-point image value is mapped (for example, by adjusting the value of c₂in a mapping function as provided by Equation (2)) and/or to decrease mid-tone contrast (for example, by adjusting the value of n in Equation (2)) in response to the input from ambient light sensor <b>425</b> indicating greater ambient light intensity.

Viewing flare has the effect of raising black level due to reflections of ambient light from a display screen. The result is typically a reduced dynamic range of the display and a crushing of dark detail. Viewing flare typically raises the level of the deepest black but does the level of the peak white is mostly unaffected. Viewing flare can also cause desaturation of dark colors. Viewing flare can be compensated for by increasing the contrast in dark regions affected by the reflections. In some embodiments, mapping units <b>422</b> alter the mapping of dark levels in response to input from ambient light sensor <b>425</b> to increase contrast in dark regions in response to increases in ambient light. In some embodiments light sensor <b>425</b> is oriented to selectively detect light directed toward a screen of display <b>420</b> or a separate light sensor <b>425</b>A is provided that selectively detects light directed toward the screen of display <b>420</b>. In such embodiments, the boost in dark range contrast may be based on a measure of the ambient light directed toward the screen of display <b>420</b>. The function used to boost dark contrast advantageously has an effect that tapers of with increasing image values such that mid-tones and highlights are not significantly affected. In an embodiment which applies the mapping function of Equation (2), increase in dark-range contrast may be achieved by adjusting the parameter c₁. Such a function may be implemented in various alternative ways including by pre-processing image values to increase dark level contrast prior to the normal mapping performed by mapping unit <b>422</b> or post-processing after the normal mapping performed by mapping unit <b>422</b> or by implementing an alternative mapping in mapping unit <b>422</b> that has one or more parameters permitting control of dark-range contrast.

<figref idref="DRAWINGS">FIG. 5</figref> shows example response curves <b>501</b>A and <b>501</b>B that may be provided by a mapping unit <b>422</b> for dim and bright ambient conditions respectively. <figref idref="DRAWINGS">FIG. 5A</figref> shows corresponding system response curves <b>502</b>A and <b>502</b>B. In <figref idref="DRAWINGS">FIG. 5A</figref>, line <b>503</b>A indicates an appearance match between the reference and target displays for the case of dim ambient conditions and line <b>503</b>B indicates an appearance match between the reference and target displays for the case of bright ambient conditions.

Some embodiments estimate reflections of ambient light from a screen of display <b>420</b>. Such reflections may be estimated from measurements of the ambient light by sensor(s) <b>425</b> and/or <b>425</b>A and known optical characteristics of the display screen. In some embodiments a signal representing measured ambient light is multiplied by a factor which is determined empirically or based on knowledge of the optical characteristics of the display screen to obtain an estimate of reflected light that is added to the luminance created by the display of images on the display <b>420</b>.

Adaptation of the human visual system to light may be estimated by inputting information about the history of light exposure to a mathematical model of the behavior of the human visual system. In some embodiments, mapping units <b>422</b> implement algorithms that apply such mathematical models using as inputs values received from one or more ambient light sensors <b>425</b> and/or information regarding the brightness of content displayed on display <b>420</b>. The modeled adaptation will, in general be a function of past ambient conditions. The modeled adaptation may take into account light emitted by a display instead of or as well as other ambient light at a viewing location. In some embodiments, ambient light may be estimated based at least in part on the video data output by a mapping unit <b>422</b>. In some embodiments, mapping units <b>422</b> implement methods and apparatus as described in U.S. application No. 61/433,454 filed on 17 Jan. 2011 and entitled “METHODS AND APPARATUS FOR ESTIMATING LIGHT ADAPTATION LEVELS OF PERSONS VIEWING DISPLAYS” which is hereby incorporated herein by reference for all purposes.

Light adaptation has a number of different aspects. In some embodiments, control over different aspects of the mapping performed by mapping units <b>422</b> is based on estimates of different adaptation aspects. For example, control over the luminance to which a mid-point image value is mapped may be based on an estimate of a general brightness adaptation. The general brightness adaptation may be based on the average brightness to which the viewer has been exposed over a time period characteristic of the general adaptation of the human visual system. Control over mid-range contrast may be based on an estimate of lateral brightness adaptation. Mapping unit <b>422</b> may apply a plurality of models to estimate a corresponding plurality of different types of adaptation and may control a corresponding plurality of different aspects of the mapping based on outputs of the plurality of models.

A mapping unit <b>422</b> may have any of a wide variety of constructions and may be implemented in software and/or hardware. Mapping units <b>422</b> may implement features as described in the commonly-assigned U.S. application Nos. 61/453,107 filed on 15 Mar. 2011 and entitled “METHODS AND APPARATUS FOR IMAGE DATA TRANSFORMATION” and/or 61/473,691 filed on 8 Apr. 2011 and entitled “IMAGE RANGE EXPANSION CONTROL METHODS AND APPARATUS” which are both hereby incorporated herein by reference for all purposes.

It can be appreciated that some embodiments provide a content delivery system in which image data distributed by the system is tailored for viewing on a “virtual display” having specified capabilities. The capabilities of the virtual display are specified in advance. When the content is to be viewed on a particular physical display (either a reference display or a target display), the content may be mapped as described herein in a manner which ensures that mid-range contrast and a mid-point value are displayed consistently while dark values and bright values are displayed according to the capabilities of the physical display. The distributed data may have redundant precision (i.e. greater precision than is required for displaying the data on a reference display) at least at values corresponding to brightness above a mid-range. The mappings may all have a specified functional form (e.g. a sigmoidal form). In such a system the approved appearance of distributed content may be preserved across a wide range of target displays.

For broadcast and other applications, a mapping as performed by mapping unit <b>410</b> may be provided in a camera or other image capture device. For example, a mapping unit <b>410</b> that implements a mapping according to Equation (2) may be associated with a camera. Some or all of the parameters c1, c2, c3, and n may be adjusted manually or automatically to yield image data that provides a desired image on target display <b>420</b>. The mapping may be adjusted to exploit the limited dynamic range and precision of the light sensor(s) in the capture device under the lighting conditions that the capture device is experiencing. <figref idref="DRAWINGS">FIG. 6</figref> is a plot of possible response functions that may be provided in capture devices for different ambient lighting conditions. Curve <b>601</b>A is an example response function for a capture device having a relatively low dynamic range operating to capture a dimly lit scene. Curve <b>601</b>B is an example response function for the same capture device operating to capture a brightly lit scene. Curve <b>602</b> is an example response function for a capture device having a relatively high dynamic range operating to capture a dimly lit scene.

The gain and aperture on the capture device operate to shift the response curves curve up or down such that a fixed point in the mid-range of the curve is made to correspond to a midpoint of the output image values. The contrast is adjusted depending on the exposure.

It can be seen that curve <b>602</b> provides markedly increased contrast for extreme regions <b>603</b>A and <b>603</b>B of its dynamic range. These regions may be respectively called a dark end region and a bright end region. Such ranges may be outside of the range that can be realistically displayed by most target displays. Providing increased contrast in ranges <b>603</b>A and <b>603</b>B results in relatively more signal values being allocated to ranges that can be reproduced by typical displays and relatively fewer signal values being allocated to ranges <b>603</b>A and <b>603</b>B. This means that luminance values in regions <b>603</b>A and <b>603</b>B will be compressed when displayed on most target displays.

Where a capture device implements a response function as described above, the capture device may be operated to encode linear light levels into image values such that the midpoint and contrast are correct when processed by a mapping unit <b>422</b> and displayed on a display <b>420</b> without further processing. The overall system response from the combination of processing at the image capture device and at the display may: <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0096">cause the middle of the exposure of the captured scene to be displayed in the middle of the display's dynamic range;</li><li id="ul0004-0002" num="0097">allow for slight changes in mid-tone contrast to account for different viewing environments; and</li><li id="ul0004-0003" num="0098">fit the full captured range into the dynamic range of display <b>420</b> without further clipping of darks or brights. <br /><figref idref="DRAWINGS">FIG. 7</figref> shows some example system response curves <b>701</b>A, <b>701</b>B, and <b>702</b> that correspond respectively to curves <b>601</b>A, <b>601</b>B, and <b>602</b> of <figref idref="DRAWINGS">FIG. 6</figref>. In <figref idref="DRAWINGS">FIG. 7</figref>, the “dim” environment corresponded to the center of the response curves at 10 nits. When displaying in the same environment, we see that the appearance is preserved by displaying this same portion at 10 nits. Curve <b>702</b>B shows that in an image captured in a brighter environment with a mid-tone value at 100 nits, the mid-tone value gets mapped to 10 nits on the with a corresponding increase in contrast to account for the dimmer environment. Curves <b>701</b>A and <b>702</b> show that, when operated to capture images in a dim environment, both standard dynamic range (SDR) and high dynamic range (HDR) sensors map the midrange correctly and smoothly roll-off in black and white according to the combined capabilities of the camera or other image capture device and display. </li></ul></li></ul>

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors may implement methods as described herein by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.

As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the scope thereof.