Method and apparatus for image data transformation

Description

TECHNICAL SCOPE

The invention relates to image display and processing. The invention relates in particular to methods and apparatus involving tone and / or scale mapping. The methods and apparatus described herein can be used to provide high-quality images on target displays while maintaining the creative intent. The invention may be practiced with, for example, electronic displays such as, for example, televisions, computer monitors, media players, handheld phones and other portable devices with video function, specialized displays such as, for example, virtual reality displays, advertising displays, and the like. as well as for pre-image processing equipment such as set-top boxes (STBs). set-top boxes), access points and the like.

BACKGROUND OF THE INVENTION

[0002] Patent publications in the general field of the invention include the following:

US20010050757;

US20020075136;

US20020080245;

US20070127093;

US20080094515;

US20080170031;

US20080186707;

US20090201309;

US20090267876;

US20100007599;

US201000118008;

US7158673;

US6989859;

US5276779; and

JP2002092655.

[0003] A creator of a video or other image production (e.g., a director, colorist, or the like) may adjust the tones and colors of pixels in an image such that, on viewing, the image has a desired appearance that conforms to the creator's creative intent. For example, the creator

-2 may wish some scenes had a darker, more overwhelming character than others. The creator may wish to make certain features depicted in the scene stand out or be less visible. Adjusting the tones and colors of pixels in an image may include performing color grading / color timing on the source video data. Color grading may be performed using a hardware / software system that allows the user to alter the video data in various ways to obtain the desired appearance.

[0004] Various display technologies are currently available. For example, there are plasma displays, LCD displays illuminated by various kinds of light sources, such as, for example, various kinds of LEDs, fluorescent lamps or high-intensity incandescent lamps, CRT displays, digital cinema displays, etc. The specific display connects the display hardware with video signal processing components that receive the video signals and make the display hardware display the video content of the video signals.

[0005] Different displays can vary widely with respect to features such as, for example:

• the range of colors that can be reproduced by the display;

• maximum achievable brightness;

• contrast ratio;

• resolution;

• acceptable input signal formats;

• color depth;

• whiteness level;

• black level;

• white point;

• degrees of gray;

• etc.

As a consequence, the same picture content may appear different when displayed on different displays. Image content that is in accord with the creator's creative intent when displayed on certain displays may deviate from the creator's creative intent in one or more ways when displayed on other displays.

[0006] Some current displays may perform better than those that were prior art at a time when content was created in one or more items. For example, new displays may provide images with brighter patches of light, greater contrast, and / or wider color gamuts than older displays.

-3 displays. It may be desirable to utilize these enhanced capabilities without significantly departing from the intent of the content being viewed.

[0007] It may be desirable to play video content created for the use of high performance displays on older displays or displays that are less capable. It would be desirable to provide methods and apparatus for customizing the appearance of video and other images displayed on different displays in order to preserve as much as possible the creative intent of the image data.

[0008] The perception of color and luminance can be influenced by environmental conditions. A video or other images presented in a cinema environment (low ambient light) may be perceived by viewers significantly differently than the same film or other images would be perceived under strong ambient light conditions. In addition, the property (e.g., color temperature) of ambient lighting can influence the viewer's perception of the video content. It would be desirable to display video or other images taking into account the viewing environment in order to preserve as much as possible the creative intent of the movie or other images.

[0009] There is a need to provide image viewers (including still images and / or video) that provide viewing experiences that utilize the capabilities of the displays on which the images are viewed. There remains a need for a device and methods that can be used to adjust image data such that the video or other image content encoded with the image data has the desired appearance during playback.

SUMMARY OF THE INVENTION

[0010] The invention is implemented by an apparatus according to claim 1 and a method according to claim 8 and has a number of objects. They include, without limitation, a device that characterizes the gamut transform functionality; how to transform the gamut, ways to adjust the display of the image content to take into account the ambient lighting conditions; computer-readable program products which, when executed by a data processor, cause the data processor to perform the method of the invention.

[0011] One non-limiting object provides an apparatus including a pixel coordinate mapping unit that is configured to transform the image data according to a transfer function. The transfer function is characterized by a number of hooking points and a free parameter. The carry function has a mid-range slope controlled by a free parameter. The floating parameter has no effect on the transformations at the snap points. anchor point). Such a device may be useful, for example, for transforming graded content for display on a specific target display.

[0012] In some embodiments, the device comprises or receives a signal from an ambient light sensor and the circuit connected to receive the signal

The ambient lighting from the ambient lighting sensor is configured to control one or more free parameters and the coordinates of one of the anchor points based at least in part from the ambient lighting signal.

[0013] In some embodiments, the image data comprises sets of pixel values ​​for the pixels in the image. Pixel value sets contain color values ​​for each of a number of primary colors (for example, values ​​for the primary colors red, green, and blue). The device includes a plurality of pixel coordinate mapping units each connected to transform the corresponding one of the color values. The parameters for the shift function in different coordinate mapping units may be the same or different. With appropriately selected different parameters, the transformations can perform color correction as well as gamut translation.

Another aspect of the invention includes methods of mapping image data for display on a target display. The methods include transforming pixel values ​​of the image data to corresponding transformed pixel values ​​according to a transfer function. The carry-over function has multiple hooks and a floating parameter. The carry function has a mid-range slope controlled by a free parameter. The floating parameter has no effect on transformations at anchor points. In some embodiments, one or more of the floating parameters and mid-range anchor positions are automatically changed to account for ambient lighting and / or to accommodate the viewers' vision systems.

Another object includes methods of mapping image data to be displayed on a target display by combining the global tonal mapping transform with a local large scale tonal mapping operation.

[0016] Another object provides a color manipulation device. For example, the device may include a workstation for modifying still or video images. The device may be used to modify the color values ​​in the image data. The color manipulation device comprises a first memory or input for the source image data and a second memory or output for the modified image data. The pixel coordinate mapping unit is connected to access the first memory, i.e. the input, and configured to transform the source image data according to a transfer function characterized by a plurality of hooks and a floating parameter. The carry function has a mid-range slope controlled by a floating parameter, and the floating parameter has no effect on the transformations at the snap points to obtain the modified image data and to deliver the modified image data to the second memory, i.e., the output. The user input is configured to take a value from the user for the free parameter. The display is connected to display the modified image data. The user can adjust the value of the free parameter to obtain the desired appearance of the image displayed on the display.

[0017] Further objects of the invention and features of specific embodiments of the invention are described below.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

[0018] The attached figures illustrate non-limiting embodiments of the invention.

Fig. 1 shows a schematic representation of a video distribution channel.

Fig. 2 shows an apparatus according to one embodiment of the invention.

Fig. 3 illustrates an exemplary carry function.

Fig. 4 is a block diagram illustrating a method that uses target display information and input image data to determine appropriate values ​​for parameters defining a carry function.

Fig. 5 is a block diagram illustrating an image data processing method in accordance with one embodiment.

Fig. 6 is a block diagram illustrating a method that combines a global tonal mapping operator with a local large scale tonal mapping operator according to one embodiment.

DESCRIPTION OF THE INVENTION

[0019] In the following description, specific details are provided in order to provide a more detailed understanding of the invention. However, the invention can be practiced without these details. In other instances, well-known elements have not been shown or described in detail in order to avoid unnecessarily obscuring the invention. Accordingly, the specifications and figures are to be considered illustrative rather than restrictive.

[0020] Fig. 1 shows schematically a video distribution channel 20. The raw video data 22 is extracted and edited in the editing package 23 to deliver the raw video production 24. The tones and / or colors in the raw video production are adjusted in a color correction station 26 by a colorist (e.g., a human who uses the tools provided by the station to color correct using an appropriate user interface) to obtain color corrected video production. The color correction station 26 includes a professional monitor 30 on which the colorist watches the video production. Using the tools and controls provided by the color correction station 26, the colorist adjusts the tones and / or colors of all or part of the images that make up the video production to obtain an overall appearance that, when displayed on the display 30, conforms to artistic intention of the colorist.

[0021] If all viewers of the color corrected video production 27 watched the video production on a display identical to the display 30 under environmental conditions identical to those experienced by the colorist, then, apart from individual differences in human perception of images, all viewers would see the video production accurately from the intentions of the colorist (i.e. in accordance with the artistic intention

-6 colorist). Given the very wide range of displays that are in use, it is unrealistic to expect all viewers to have the same display, or even that displays where different viewers will watch your video production will have similar characteristics, such as maximum brightness, black level and color gamut.

One object of the invention provides mapping methods and a device that can be used automatically to map tones and / or colors from image data, such as, for example, color corrected video production to be displayed on a specific target display in a manner that accurately maps. a colorist's visual experience.

[0023] In some embodiments, the mapping methods and device provide direct control of one or more of:

average image brightness (adaptation point);

mid-tone local contrast;

color saturation;

the level at which input black is displayed; and the level at which the input white is displayed.

These parameters influence the visual experience.

[0024] Fig. 2 shows an apparatus 40 in accordance with one embodiment of the invention. In this example, device 40 has an input 42 for receiving video data 43 to be displayed on the screen 44 of the target display 41 for viewing by a viewer V. The video data 43 may include color corrected video data, embodying a designer's intention. Device 40 includes a color space translator 46 that translates pixel values ​​for video data 43 into a color space that is native to the target display 41. In the illustrated embodiment, the native color space of the target display 41 is an RGB color space that defines the colors in terms of color intensity. primary target display 41.

[0025] The color space translator 46 may include, for example, a matrix multiplier that multiplies the vector of pixel values ​​in the video data 43 by a 3x3 matrix to obtain the native color space vector of the display 41 (e.g., RGB values). The transfer matrix may be determined taking into account the primary colors and the white point of the target display 41. In some embodiments, the color space translator 46 may be configured to use a color space conversion matrix without scaling for peak luminance. As explained below, this may make parameter selection for subsequent image processing operations more intuitive.

[0026] In the example below, the pixel values ​​of the video data 43 are represented in the ΧΥΖ color space, while the color space translator 46 translates from the color space into positive RGB values. The invention is not limited to data

-7 colors shown in the ΧΥΖ color space. The video data 43 may be presented in any suitable color space.

[0027] Negative RGB values ​​may produce results for translating combinations of pixel values ​​that are out of gamut (e.g., colors that cannot be reproduced using any available combination of primary colors used by the display). Any negative RGB values ​​generated by the color space translator 46 may be trimmed to a low non-negative value. Alternatively, out-of-gamut pixel values ​​may be mapped to pixel values ​​in the gamut before translation (e.g. as mapped in the color space of the video data 43). This can be performed, for example, by a separate mapping unit or by a color space translator component 46.

[0028] After the translator 46 has processed the color spaces, the video data 43 includes the values ​​48R, 48G and 48B which correspond to the primary colors red, green and blue (RGB) of the target display 41 as above.

[0029] The values ​​48R, 48G and 48B are each independently mapped to a new value by the mapping unit 50. Mapping units 50R, 50G and 50B are shown. Each mapping unit maps a corresponding input value obtained from color space translator 46 to a transformed value. In the illustrated embodiment, the transformed values ​​are labeled 48R ', 48G' and 48B 'accordingly.

Each mapping unit 50 maps its input value to an output value according to a carry function 55. Advantageously, the carry function (s) 55 may / may have a plurality of fixed points, which may be referred to as "hooks", and a free parameter which adjusts the slope of the carry function in the midrange area. This slope corresponds to the mid-range contrast. Adjusting the free parameter provides a means for controlling the mid-range contrast. The transfer function can be linear or approach linearity in the mid-range region.

[0031] Fig. 3 shows an example carry function. In Fig. 3, input values ​​are indicated on the horizontal axis and output values ​​are indicated on the vertical axis. Each axis has a logarithmic scale. The carry function 55 has a maximum value 56A for the outputs, a minimum value 56B for the outputs, and a substantially linear middle region 56C. The carry functions 55R, 55G and 55B applied to the red, blue and green channel signals by the mapping units 50A, 50B and 50C may be identical or different. The mapping units 50A, 50B, and 50C may be completely independent or may share hardware and / or software components.

[0032] In one embodiment, the carry function 55 is given by the following equation:

l + C₃Vⁿ

- 8 where Ci, C2 and Ci are constants, V is the input value for the color channel, V 'is the output value for the color channel, and n is the parameter. The carry function according to Equation (1) is an example of a parameterized sigmoid tone curve function.

[0033] Other parameterized carry-over functions may be used alternatively. In some embodiments, the carryover function includes parameters that provide control over one or more of the lower end slopes, the upper end slopes, and the "sharpness" of sliding at the upper and lower ends of the carryover function.

One method of determining values ​​for the parameters in Equation (1) in a specific case is illustrated by the method 70 of Fig. 4. The method 70 uses the target display information and the display information used in color correction or validation of video input (" color correction display ”) to determine the appropriate values ​​for the parameters of Equation (1). Block 71 identifies three luminance anchor points on curve 55. The first anchor point 57A has horizontal and vertical coordinates, respectively, equal to the black levels of the color grading display and the target display. In some embodiments, information about the display to be color graded is inferred from the input signal. For example, the black level for a color grading display can be deduced from the input signal by taking a small percentile (e.g., 0.1 percentile) of the input signal luminance channel. The black level for the target display is the black level for the target display.

[0035] The second anchor point 57B has the white point for the display to be color graded as its horizontal coordinate and the white point for the target display as its vertical coordinate. For example, the white point for a color correction display may be inferred from the input signal as the maximum value of any color channel in the input signal.

[0036] The position of the center hook point 57C affects the overall brightness of the projected image (eg, the image "key"). Proper selection of the centering point 57C facilitates the perception of the input image as bright enough on the target display.

[0037] The horizontal position of the point 57C can be adjusted in various ways; belong to them:

• calculating the geometric mean of the input luminance;

• selecting a fixed value that would be viewed in a color corrected environment as an appropriate mean value. For example, in some embodiments, this value may be set to a level such as 10.

[0038] The vertical value for point 57C may be based on a luminance level corresponding to middle gray for the target display. For example, on a display that can generate luminance values ​​ranging from 1 cd / m² up to 400 cd / m², the average gray is about 20 cd / m² (which is logarithmically halfway between 1 and

-9400 cd / m²). Therefore, a suitable value for point 57C may be gray middle value (about 20 cd / m in this example²). In embodiments where the color space translator 46 is configured to use a color space conversion matrix without peak luminance scaling, a value of 20 will correspond to an average gray of 20 cd / m.².

[0039] In some embodiments, the mid-tone anchor point 57C is selected such that the ratio of the mid-tone anchor point to the white anchor point coordinate is equal, within a desired ratio, for both the input and output of the carry function.

[0040] In some embodiments, a different carry function for each of the RGB coordinates may be used to provide a transform such that the white point of the video data is transformed to match the white point of the target display and / or the target viewing environment. One way to achieve this is to express the white point of the input video data in terms of chromaticity coordinates (such as, for example, CIE chromaticity coordinates x, y) and convert it to scaled values ​​ΧΥΖ given by the following equations:

v

X = - (2) f = i (3)

Z = ^ (4) y

Then these ΧΥΖ values ​​can be converted to the RGB color space for the target display to obtain the white point for the input data which can be denoted as (R, G, B) wp, and_n. In cases where the source and destination white points are the same, both white points should be (111) in normalized RGB coordinates. Then the coordinates for the hook points 57A, 57B, 57C for the red, green, and blue channels can be obtained by multiplying the luminance control values ​​by the white point values ​​as follows:

<td>(R, G, B ^ .- Yminin (R&gt; you</td><td> (5)</td>

<td>. = Y_maxand_n(R, G, B ^ ^k '' max, in / ηαχ, υίκ&gt;&gt; 'wp.in</td><td> (6)</td>

<td>(R, G, B)_midin = Y_mid, in (R&gt; G, B)_pin</td><td> (7)</td>

<td>(R, G, B)_{min out} = Y_min, out (R&gt; G, B ')_{wp mt}</td><td> (8)</td>

<td>(R, G, B ')_midout = Y_mid, out (R, G, B}_wpout</td><td> (9)</td>

<td>{R, G, B)_maxout = Ymax, out (R&gt; G, B)_wpout</td><td> (10)</td>

Where subscript 'in' represents the input image data, subscript 'out' represents the output (ie, data to be displayed); (Y_max ,, in, Ymax, out) denote the uncorrected coordinates of the hook point 57B; (Ymin, ίη, Ymin, out) denotes the uncorrected coordinates of the attachment point 57A; while (Y_mid, in, ymid, out) mean

- 10 uncorrected coordinates of the hook point 57C; while (R, G, B) wp, out represent the RGB coordinates of the target display's white point.

[0041] Equations (5) through (10) provide a set of three hooks for each color channel. For example, the anchor point 57A for the red channel is specified by (R_max, in, Rmax, out); the anchor point 57B for the red channel is defined by (R_mm, in, Rmin, out); while the anchor point 57C for the red channel is defined by (R_mid, in, Rmid, out). In cases where the white points for the input video data and the target display are not the same, the sets of the hooks will be different, resulting in a different transfer function for each color channel.

[0042] The transfer function for each color channel as provided by Equation (1) may be obtained from the coordinates of the respective hook points by performing the calculation:

<img file="PL2687005T3_D0001.tif" />

* 3Ż3 G1 - * 2) + * 2Ζ2 (* 3- * 1) + * 1Ζ1 (* 2 - * 3) (x₃y₃ - x₂y₂) \ (x₃ - x₂)

-yi) (w - ^ 3½) (χι-χ4 (x₂y₂ - xiyi) (¾ - Xi) (11) where xi, X2 and Χ3 are given by:

*2 *3

V. . n min, in

V ... n ηιια, ιη

V max, in, (12) while yl, y2 and y3 are given by:

Ti u λ ^ min, out Vmid, out Vmax, out, (13)

[0043] One feature of the carry functions described above is that n remains a floating parameter. This allows the mid-tone contrast to be adjusted to any desired level. Note that the log slope at the mid-tone hook will slightly differ from the value of n if the mid-tone hook is not centered in the input and output ranges. However, the mid-tone contrast can be set by adjusting the value for n. A good starting point for the mid-tone parameter w is 1. This value for n ensures that the mapped scene has substantially similar local mid-tone contrast in the target display and in the original scene.

[0044] Using the carry function as mentioned above, the display linear luminance values ​​for each of the red, green, and blue color channels can be expressed as follows:

n _ ^clR^{+ c}2R^Rtn ^out 1 + ^ «(14) ^ out

Bout l +^c3G^C.in ^clB ^{+ c}2B^Btn i + C₃B ^_n (15) (16)

- 11 These values ​​can be used to control the target display to display the image. In some embodiments, these values ​​may be corrected for the target display's response to linear input (e.g., normalized) values ​​before being used to control the target display.

[0045] In some embodiments, the normalized control values ​​(R_Wellrm, Gnorm, Bnorm) for the target display are calculated using the following relationships:

η _ Rout ~ Rout, min / 17Ί

K-norm ~ _DD '

Κοηί, πιαχ ^ out.min _ Gout ~ Gout, min ^ norm ~ _yyGout, max «out.min n _ ^ out ~ ^ out, min / 1ΟΊ ^ norm ^—_D _d ^andout, max ^andout, min

The normalized values ​​can be scaled to the range of control signals for the target display (e.g., to the range of 0 to 255 for an 8-bit target display).

[0046] Optionally, the colors in the image may be enhanced by increasing the color saturation. This can be done, for example, using the following dependencies:

_ ^ nonn + ^ dnorm + c ^ iiorm (20) yKnorm / yKnorm / // sg '= R (21) ^ norm ·'

B = r '(22)

The values ​​for a, b, and c in Equation (20) can be defined with respect to the elements of the inverse transformation matrix M, corresponding to the translator 46 of the inverted color space ([X, Y, Z]^T. = M * [R, G, B]), in particular a can be given by: a = M (2, l \ b can be given by: b = M (2,2), while c can be given by: c = M (2,3). In equations (20), (21) and (22), S is the free parameter. Values ​​for S greater than 1 will increase the color saturation. Values ​​for S less than 1 will decrease the color saturation (i.e. will make the colors more saturated).

[0047] When necessary or desired, the normalized control values ​​may be gamma corrected. This can be done, for example, according to the following dependencies:

<td>^ corrected</td><td>= R ^ ^ norm</td><td> (23)</td>

<td>Gorrected</td><td>^at norms</td><td> (24)</td>

<td>^ corrected</td><td> = ^Dnorms</td><td> (25)</td>

- where γ is the display response, γ is around 2.2 in some target displays. When the normalized control values ​​are re-saturated, the gamma correction may be performed on the re-saturated control values ​​(R ', G' and B ').

[0048] In some embodiments, the colors in the image are re-saturated to restore, at least approximately, the saturation lost from tonal compression. In cases where tonal compression is not constant across the entire tonal range of an image, different levels of tonal compression applied to different tones cause different colors to re-saturate to different degrees. In general, the greater the tonal compression, the greater the saturation value. The amount of tonal compression can be quantified by the log slope of the tone curve. As an illustrative example, the sigmoid tone curve function plotted as curve 55 in Fig. 3 has a substantially steeper log slope in the substantially linear mid-tone region 56C than it is around the maximum 56A and minimum 56B. Accordingly, the tonal compression going from the input (horizontal coordinate) to the output (vertical coordinate) is greater around 56A and 56B compared to the substantially linear mid-tone region 56C.

[0049] Applying the global re-saturation method can re-saturate all pixels regardless of the degree of saturation caused by tonal compression. Some embodiments re-saturate transformed pixels of image data as a function of tonal compression of the transformed pixels of the image data. Given that the degree of tonal compression corresponds to the log slope of the tone curve, the degree of tonal compression for the input value Lin can be derived from the carry-over function L_out = i {Lin) in the input value Li_n. The log slope of this carry function can be determined by setting Lin = G and L_out = e^y and solving for dyldx, which represents the log slope. For the tone curve according to Equation (1) above, y can be expressed as:

y = log (q + c₂e^nx) - log (l + c₃e^nx) (26) while the log slope c {Lin) at any point on the tone curve can be calculated as the derivative of y with respect to x in Lin.

_{r (1} \ ^? n (cz-ciC3)

[0050] For the R, G and B color channels, the re-saturated control values ​​(Rre-sat, Gresat, Bre-sat) may be set to the normalized control values ​​as follows:

<td>Rre — sat Rnorm।</td><td>GA 'Lin'</td><td></td><td> (28)</td>

<td>Ge-sat ^ norm।</td><td>(Ld sLi_nj</td><td>} Bc) -c</td><td> (29)</td>

<td>Bre-sat ^— Bnorm</td><td>(Gi</td><td></td><td> (30)</td>

where f (c) is given as:

- 13 / (c) = 1 + fc₁C.^k2 (31) while ki and k2 are constants. In some embodiments, k1 = 1.6774. In some embodiments, k1 = 1.677. In some embodiments, k1 = 1.68. In some embodiments (including, without limitation, some embodiments where ki = 1.6774, ki = 1.677, or ki = 1.68), k2 = 0.9925. In some embodiments (including, without limitation, some embodiments where ki = 1.6774, ki = 1.677, or ki = 1.68), k2 = 0.992. In some embodiments (including, without limitation, some embodiments where ki = 1.6774, ki = 1.677, or ki = 1.68), k2 = 0.99. It should be noted that acceptable results may be obtained using different values ​​for ki and k2. It should also be noted that the re-saturated control values ​​Rre-sat, Gre-sat, and Bre-sat can be calculated from the display linear luminance value for each of the red, green, and blue color channels (R_ABOUTut, G_out and B_out).

[0051] It should be noted that the above-described tonal compression dependent re-saturation method can be practiced without parameters (automatic).

[0052] Fig. 5 is a block diagram illustrating a method 80 according to another embodiment. The method 80 includes a number of optional steps. At block 81, method 80 converts the image data to a color space of the target display. In the illustrated example, the target display has the primary colors red, green, and blue and the color space is the RGB color space. Block 81 may include performing a transform that takes into account the white point and the primary colors of the target display. Block 81 is optional in case the image data is already in the native color space of the target display.

Block 82 determines the chromatic white points for the source and target. The white points can, for example, be represented as chromaticity coordinates in any suitable color space and converted to the native color space of the target display.

The block 83 establishes an initial black level and white level anchors for the carry over function. The starting anchors can be set based on the black and white levels for the source and target displays.

[0055] Block 84 establishes a starting mid-tone anchor point for the carry function. The center-tone anchor point may be established by analyzing the source image data (e.g., by determining the geometric mean luminance of the source image data) and determining target display properties (or properties of the target display and the current viewing environment on the target display).

[0056] Block 85 corrects the snap points based on white points determined at block 82 (using, for example, Equations (5) through (10)).

[0057] Block 86 maps the image data using carry functions determined by corrected hooks determined at block 85.

Block 87 calculates control values ​​for the target display based on the mapped image data in block 86.

[0059] Optional block 88 adjusts color saturation (block 88 may, for example, use Equations (20) to (22) or (28) to (30)).

[0060] Block 89 gamma corrects the control values.

[0061] The control values ​​generated by using the method 80 may be used to control the target display to display the images and / or save them or send them for later display on the target display.

[0062] The apparatus and methods as described herein may be used to optimize the target display for a particular viewing environment. The general type of transference functions described above can be dynamically shifted to account for changes in ambient lighting and the resulting changes in the level of human visual system (HVS) adaptation. The ideal luminance center point for the target display can be a function of the ambient lighting. The vertical component of the mid-tone anchor point may be selected based on the ambient lighting conditions.

[0063] In some embodiments, the determination of the center of the latch point 57C is performed based partly on ambient illumination or based on an estimation of the viewers' eyes adaptation (which may itself be based at least partly on measured ambient illumination or a combination of measured ambient illumination and displayed content. ) as well as the properties of the target display. For example, the vertical coordinate of point 57C may be adjusted based on the ambient lighting near the target display. For example, the vertical coordinate may be lowered to a lower luminance value if the display is in dark ambient lighting conditions (or it is estimated that the viewer's eyes are aligned with darkness), while this value may be raised to a higher value when the target display is in position. in an environment with high ambient light (or it is estimated that viewers' eyes are adjusted to brighter conditions).

[0064] In some embodiments, the degree of saturation adjustment (e.g., according to equations (20), (21), and (22) and equations (28), (29), and (30)) is based in part on ambient lighting or an estimate. adaptation of the viewers' eyes (which may itself be based at least in part on measured ambient lighting or on a combination of measured ambient lighting and displayed content) as well as on target display properties. For example, the parameter S may be adjusted based on the ambient lighting near the target display or a combination of measured ambient lighting and displayed content. For example, the value of the S parameter may be set relatively lower if the display is in dark lighting conditions (or it is estimated that the viewer's eyes are adjusted to darkness), while the value may be set relatively higher if the target display is

- 15 is in an environment with high ambient light (or it is estimated that viewers' eyes are adjusted to brighter conditions). Some embodiments provide a re-saturation control unit that receives a signal from an ambient light sensor and / or signals including displayed image content and / or signals indicative of the overall brightness of the displayed content. The re-saturation control unit may be configured to set new values ​​for a parameter (e.g., S parameter) that affects the degree of re-saturation based on the received signal (s).

[0065] In some embodiments, the spectral properties of ambient lighting are considered. For example, the position of the points 57C in the carry functions for each color channel may be separately set partly based on the degree of ambient illumination in the spectral range corresponding to the color channel.

[0066] Additionally or alternatively, the slope of the transfer function may be controlled based on ambient illumination (or an estimation of the viewer's eye adaptation). In places where the ambient lighting is brighter, reflections off the display surface tend to raise the black level. This effectively reduces the range of the target display. In high ambient lighting conditions (the eyes of viewers are estimated to be adapted to the light), the slope of the transfer curve in the mid-tone region can be lowered to provide an improved visual experience under ambient conditions. For example, for bright (dark) ambient lighting conditions, the perception of contrast is reduced. This can result in the image becoming "flat". Therefore, the slope of the midtone portion of the carry function may be increased from a 1: 1 slope to a greater slope such as a slope of 1: 1.5 or the like (e.g. slope 1.3) to increase the contrast level for the dark-adjusted eyes. This can be done by varying the value of the free parameter n in cases where the transfer functions of the type illustrated by Equation (1) are used. The tilt can be controlled in response to an input from the ambient light sensor.

[0067] In some embodiments, a lighting adaptation circuit is provided that estimates the level of adaptation of the human visual system in response to input signals that may include an ambient light sensor signal, which signal represents a weighted average or other indicator of the displayed content brightness signal. image or the like. The lighting adaptation circuit can be based on a model of the human visual system, for example. Various algorithms for estimating the level of adaptation of the human visual system are known in the art. The lighting adaptation circuit may implement such algorithms in any suitable manner, including software executing on one or more programmable data processors, fixed logic circuits, or combinations thereof. The values ​​for the mid-tone contrast and / or position of the points 57C in the carry functions can be automatically controlled in response to an output from the light adaptation circuit.

[0068] In some embodiments, the carryover functions are set once for the target display. The transfer functions may, for example, be embedded in the target display and made into one or more programmable processors executing the firmware or other software that performs mapping in accordance with the transfer functions as described above; look up tables that implement the carryover functions described above; wired or configurable logic circuits which are arranged to provide an output based on the carry function as described above; or the like.

[0069] In some embodiments, the control values ​​for the red, green, and blue channels of the target display are converted to a bit depth that matches that of the display. For example, the display may use 8-bit control values. If the carry-over functions are used using floating point or other high-precision calculations, the conversion may be, for example, rounding the control values ​​to the nearest corresponding 8-bit value.

[0070] In the above embodiments, minimum and maximum luminance values ​​for the input video data may be generated to map the minimum and maximum brightness values, respectively, to the display pixels. In addition, a selected midpoint from the input video signal may be generated to map a selected midpoint to the display. The mid-tone contrast remains a free parameter. Another feature of the carry-over functions described above is that they provide compression or expansion for both low and high values ​​while maintaining local contrast in the midrange.

[0071] In some embodiments, specific images (e.g., a specific video frame or sequence of video frames) have a relatively low average luminance (low key), while other images (e.g., e.g., frames or groups of frames) may be intentionally captured to do so. to have a relatively high average luminance value (high key). In some embodiments, the intended picture key information is provided in the form of metadata. Metadata can, for example, be created and associated with image data during a color correction operation. For example, the metadata may be embedded in or otherwise associated with a signal carrying color corrected video data. In such embodiments, a picture key as indicated by the metadata may be used to determine the mid tone hook / mid tone anchor points used in carry functions. In case the metadata points to a low-key image, the vertical coordinate of the anchor point may be shifted to a lower value, thus reproducing the key on the target display.

[0072] Different video content may be color corrected for different reference displays. When following the approach described above, it may be desirable to map the content differently to any particular target display depending on the characteristics of the reference display on which the performance was performed.

- 17 has been color correction. The information identifying the reference display or its properties may, for example, be carried in metadata embedded in or otherwise associated with the image data. The target display can store parameters for many different sets of carry-over functions. Different sets of carry-over functions may correspond to and be used for video data that has been color corrected using different reference displays.

Another feature of the exemplary carry functions as provided by Equation (1) is that the same carry function may provide either compression or extension at the high and low ends of the range depending on the parameters selected. For example, in the case where the target display has a larger luminance range than the input data, the target display may be configured with carry functions that extend the range of the image data to match or come closer to that of the target display.

[0074] One advantage of the methods and apparatus according to some embodiments described herein is that the mapping is performed in the RGB color space of the target display. This can save a great deal of computation and / or reduce the complexity of the hardware required to perform the mapping.

[0075] The mapping may be performed in real time.

[0076] The methods of some embodiments provide direct control over each of: 1) average image brightness ("adaptation point"), 2) local center tone contrast (as set by the slope of the tone curve), 3) black input maps for minimum display luminance. and 4) white input maps for the maximum luminance of the display. These variables have been found to be fundamental to providing images that replicate the creative intent as contained in the original image data. In the exemplary embodiments, these variables clearly correspond to separate parameters. Such methods consistently provide a simple and efficient method of performing color mapping that takes original image data (which may, for example, include high dynamic range (HDR) data. high dynamie rank) and / or color corrected image data) and maps the original image data to a limited 3-dimensional gamut of the specified output display.

[0077] The color mapping methods and device as described herein may also or alternatively be used in color correction / content creation. The colorist may be provided with a filter that performs the transformations as described above. The filter can contain controls that allow the colorist to directly set parameters for the move function. A colorist may use these controls, for example, to adjust the black level, etc. In some embodiments, the controls include controls that allow you to directly set one or more of: one or more coordinates for one or more of the white level anchor point, the black level anchor point, and the mid-tone anchor point (e.g., respectively, points 57A, 57B and

- 1857C) and mid-tone contrast (e.g. parameter n). Such controls enable the colorist to adjust the white level, black level, and key without significantly affecting the mid-tone slope and vice versa.

[0078] In some embodiments, the device is set to automatically determine an initial set of parameters that may approximate what the colorist intends. These initial parameters may, for example, be generated based on information characterizing the input video content (e.g. minimum and maximum values ​​for pixel color / luminance coordinates) and information characterizing the target display (e.g. white level, black level and optional metadata (e.g. metadata indicating the key of processed image data)).

[0079] Video production is the creation of different versions for displays having greater and lesser capabilities. For example, standard dynamic range (SDR) correction may be performed to produce video for display on older displays. The tool as described in the document can be used to automatically create SDR video versions. The colorist can control the operation of the tool to obtain optimal results.

[0080] Moreover, if the colorist has set parameters to provide a version to be viewed on a lower capable display, then the parameters to be used in performing mapping for intermediate capable displays may be determined based on the parameter values ​​selected by the colorist for the lower capable display. This may be done, for example, by interpolating the parameter values ​​determined by the colorist for displays with higher and lower capabilities than the intermediate capable display.

[0081] The methods and apparatus described herein are not limited to use in conjunction with professional level color correction. Color correction tools are available to amateurs and even in cases where color correction is performed on an uncalibrated monitor (e.g. home computer display, television, etc.), the methods and device described in the document may be used to translate content created on an uncalibrated monitor to another display (e.g., by evaluating the ability of an uncalibrated display to be color graded). The technology as described in the document is also applicable to signals that are not color corrected.

Combining a global tonal mapping operator with a local large-scale tonal mapping operator

[0082] The color mapping methods and apparatus as described herein may also be combined with other tonal mapping techniques, e.g. local tonal mapping operators (TMOs). Fig. 6 shows an embodiment where a global tonal mapping operator as described herein is linked to a local large-scale tonal mapping operator as described by GJ Ward in US Provisional Patent Application No.

- 1961 / 448,606 pt. "A Local Multiscale Tone-Mapping Operator" (referred to as reference to "Ward"), also filed as International Patent Application No. PCT / US2012 / 027267, filed March 1, 2012, incorporated herein by reference in its entirety. The exemplary embodiment combines the predictability and stability of a global TMO with the ability to maintain lighting and color fidelity while using a local multi-scale operator (MS TMO). mutli-scale operator).

[0083] As shown in Fig. 6, method 60 starts in step 62 by accessing the input image or video data. This data may be stored or transmitted in a variety of color formats such as, for example, YCbCr, RGB, ΧΥΖ and the like. In step 63, the luminance component, e.g., Y, can be extracted from the input data. Depending on the format of the input data (e.g. RGB), this step may require a color conversion (e.g. RGB to ΧΥΖ). Step 64 can apply global tonal mapping operator 55 as described in Equation (1) to the Y color component of the input data. In one embodiment, the global anchor points TMO 55 may be selected such that the luminance range of the input data is mapped to the range [4 * ^ ^_η, 1/2*£^_&amp;χ], where and Z ^ Max denote the minimum and maximum luminance of the target display. Scaling factors 4 and / 2 are typical but adjustable.

[0084] In one embodiment, the output from step 64 may be denoted globally tonally mapped Ftm luminance data. In step 65, a local large scale tonal mapping operator (MS TMO) as described in "Ward" may be applied to the Ftm data. For example, you can calculate the global logarithmic image first

A = log ^, (32) defined as the logarithm of the global tonally mapped luminance data divided by the original luminance pixels. Due to the global image of the log Rl as described in Ward, the output MS TMO (e.g., step 65) may be an image of the locally tonally mapped luminance denoted as Ems. Using Tms it is possible to compute (33) i

with_MS = · (34)

[0085] In step 66, Xms, Yms and Zms data can be converted back to Rms, Gms and Bms (RGBms) data with primary colors and white and black levels determined by the target display. Negative or out of gamut RGBms values ​​may be trimmed to very small positive values ​​or may be remapped to RGB values ​​in the gamut using any of the known gamma mapping algorithms.

[0086] The RGBms data in the gamut from step 66 may in step 67 reapply global tonal mapping operator 55 to all of the color components to provide globally mapped tonally corrected RGBg-ms data. The use of the second global tonal mapping operation ensures that the output MS TMO data is in the range of the target display. Finally, in step 68, before displaying the image data (step 69), the RGBg-ms data may be gamma corrected for the output display, if necessary.

[0087] Some embodiments of the invention include computer processors that execute software instructions that cause the processors to perform the inventive method. For example, one or more processors in the display, a color grading station, set-top box, transcoder, or the like may implement image data conversion methods as described above by executing software instructions in a program memory accessible to the processors. The invention may also be provided as a software product. The software product may include any medium that carries a set of computer readable signals including instructions that, when executed by the data processor, cause the data processor to perform the inventive method. The program products of the invention may be in any of a wide variety of forms. The software product may contain, for example, physical media, such as, for example, magnetic data carriers, including floppy disks, hard drives, optical data media, including CD ROMs, DVDs, electronic data carriers, including ROM, flash memory. RAM and the like. The computer readable signals in the software product may optionally be compressed or encrypted.

[0088] In cases where an item (e.g., a software module, processor, assembly, instrument, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including reference to "means") is interpreted as containing, as equivalents of that component, any component that functions as the component being described (i.e. which is functionally equivalent), including components that are not structurally equivalent to the disclosed structure that performs a function in the illustrated embodiments of the invention.

[0089] Some non-limiting embodiments may (e.g., depending on the circumstances) provide one or more of the following advantages:

Mapping according to the tone mapping curve with the black anchor point may prevent the dark input content from being over-compressed;

• tone mapping curve mapping with black and / or white anchor points may use a larger range of the target display luminance range than tone mapping curve without one or both of such anchor points;

• Color channel specific mapping functions that maximize the luminance range can be used in the RGB color space of the target display

-21 (e.g. after converting from input color space to target RGB color space); and • the white point, brightness and / or average contrast of the output video data for the target display can be adjusted in the carry function (e.g., instead of before or after the mapping according to the carry function).

Some embodiments may not provide any of the above benefits; some embodiments may provide other advantages (e.g., rather than or in addition to the above advantages).