Enhancing dynamic ranges of images

[Refer to related applications] This application claims priority under US Patent Application No. 60/962708, named "ENHANCING DYNAMIC RANGES OF IMAGES", filed July 30, 2007. For the United States -In this application, US Patent Application No. 60/962708 filed on July 30, 2007, claiming the benefit of Article 119 of 35 U.SC (US Patent Law) under the name "ENHANCING DYNAMIC RANGES OF IMAGES", referred to in this document And incorporate. -This application is a continuation of US Patent Application No. 12/182121, named "ENHANCING DYNAMIC RANGES OF IMAGES," filed July 29, 2008. -This application claims the benefit of US Patent Application No. 12/182121 filed on July 29, 2008, Article 120 of 35 U.SC (US Patent Law) under the name "ENHANCING DYNAMIC RANGES OF IMAGES".

[Technical field] The present invention relates to digital images. The present invention specifically relates to devices and methods for increasing the dynamic range of images (including still and / or video images). The present invention is read by, but not limited to, electronic displays, media players (such as DVD players), image processing subsystems used in electronic displays and / or media players, and data processors, and then executed and performed on media. Performed in the provided computer software.

The human eye can perceive a very wide range of light intensity. In order to accurately reproduce the actual scene, it is preferable that the image has a high dynamic range. A high-performance image sensor such as a high-performance CCD array can acquire an image having a high dynamic range. The new generation of display devices undoubtedly offers an improved dynamic range over traditional display technology.

Most of today's movies, videos and still images are recorded in formats that offer a much lower dynamic range than these new generation displays produce. In the future, evolving camera technology and file formats will provide these display devices with high quality content. In the near future, it is desirable to provide a way to increase the dynamic range of media with low dynamic range (eg, image data in low dynamic range (LDR) format). This will allow the observer to benefit from at least some high dynamic range displays while using existing media.

Observers of cinema image systems (projectors) and home theater systems are very discerning. For these and other applications, it is desirable to provide images that are essentially free of visible artifacts.

Depending on the application, it is preferable to increase the dynamic range of the image in real time (for example, to generate an increased image signal).

The present invention has many aspects. In one aspect, we provide a method of receiving image data in low dynamic range (LDR) format as input and generating increased image data with a dynamic range greater than the dynamic range of the input image data as output. In some embodiments, the method is used for video data and is performed in real time (ie, processing the video frame to increase the dynamic range of the video frame is completed on average at least at the frame rate of the video signal. To do).

Further aspects of the invention and features of specific embodiments of the invention will be described below.

The accompanying drawings illustrate, but not limited to, embodiments of the present invention.

<figref num="1">FIG. 1 is a flow chart of a method for increasing the dynamic range of an image according to an embodiment of the present invention.</figref><figref num="1A">FIG. 1A is a flow chart of a method of linearizing input image data according to an exemplary embodiment.</figref><figref num="2">FIG. 2 is a flow chart illustrating an exemplary method of generating a luminance increasing function and using the luminance increasing function in image data.</figref><figref num="2A">FIG. 2A is a flow diagram illustrating an exemplary method of generating a mask that constitutes an edge stopping function.</figref><figref num="2B">FIG. 2B is a flow diagram illustrating the vicinity of a pixel and an exemplary method of determining the degree of gradation at that pixel.</figref><figref num="3">Figure 3A shows an exemplary LDR input image, Figure 3B shows the corresponding smooth component, Figure 3C shows the corresponding brightness enhancement feature modified by the edge stopping component, and Figure 3D shows the high dynamic range (HDR). The output image is shown. It will be found that the media in the patent drawing does not reproduce the dynamic range of the input and output images.</figref><figref num="4A">FIG. 4A is a flow diagram illustrating an exemplary method of generating a smooth component of the luminance increasing function.</figref><figref num="4B">FIG. 4B is a flow diagram illustrating an exemplary method of generating an edge stopping component of the luminance increasing function.</figref><figref num="4C">FIG. 4C illustrates the image pyramid used to generate the luminance increasing function, for example, by the methods of FIGS. 4A and 4B.</figref><figref num="4D">FIG. 4D particularly illustrates the concept of FIG. 4B, a concept related to the generation of edge stopping components.</figref><figref num="5">FIG. 5 illustrates an apparatus according to an embodiment of the present invention.</figref><figref num="6">6 and 6A illustrate a method according to a particular embodiment that increases the dynamic range of the image data and generates a drive value for the modulator of a dual modulation display in order to project the increased image data.</figref>

Specific details are provided for a better understanding of the present invention through the following detailed description. However, the present invention can be practiced without those details. In other examples, well-known elements are not shown or described in detail in order to avoid unnecessarily blurring the invention. Therefore, the specification and drawings should be regarded as descriptive rather than restrictive.

FIG. 1 is a flowchart (flow chart) illustrating a method 20 for increasing the dynamic range of a digital image defined by image data 21 according to an embodiment of the present invention. The image data 21 may be low dynamic range (LDR) image data. Block 22 linearizes the pixel values from the input image data 21. Block 22 is a space in which the brightness changes linearly with the pixel value, and is unnecessary when the pixel value of the input image data 21 is already represented. In some embodiments, the linearization of block 22 is avoided by doing extra processing on the downstream side. The output from block 22 is linearized image data 23. Each pixel of the linearized image data 23 has a value that is at least approximately proportional to the brightness of the pixel.

The specific process performed by the linearization of block 22 depends on the image encoding of the input image data 21. For example, typical image and television data is encoded with a gamma curve with a gamma value in the vicinity of 2.2. This gamma curve is designed to correct for non-linearity in traditional display techniques such as cathode ray tubes (CRTs). The linearization of such data is done by using a function that inverts the gamma curve. Inverting the gamma curve provides a pixel value that is approximately proportional to the brightness of the original scene.

The linearization function of block 22 is performed using a look-up table (LUT). For example, the procedure for linearizing block 22 refers to the pixel value of the input image data 21, searches the LUT for the corresponding linearized pixel value, and linearizes the corresponding linearized pixel value. Includes output to data 23. In another embodiment, the linearization function of block 22 takes the pixel values from the input image data 21 as input and produces the corresponding linearized pixel values for the linearized image data 23 as output. , May be run by software or hardware running on the appropriate data processor.

In some embodiments, the block 22 linearization process comprises selecting one of the most suitable linearization functions for linearizing the input image data 21. Some data formats include information that clearly or implicitly identifies the non-linear relationship between pixel values and brightness. Such information can be obtained, for example, from the header accompanying the input image data 21.

FIG. 1A shows how to perform linearization of block 22 according to a particular embodiment, where the input image data 21 has a non-linear relationship between the pixel values of the input image data 21 and the desired brightness. Contains encoding information that identifies clearly or implicitly. Referring to Figure 1A, the linearization process for block 22 is: reading the encoding information (block 20A); 1 of multiple linearization functions 25 (distinguished as 25A, 25B, 25C ...) that match the encoding information. It may include: applying the selected linearization function 25 to the input image data 21 (block 20C) to generate linearized image data 23;

In some embodiments, the pixel value of the linearized image data 23 comprises the absolute output intensity projected on the display for the corresponding pixel.

Referring to FIG. 1 again, the contrast of the linearized image data 23 is stretched (enhanced) by the block 30 to generate the stretched image data 31. Contrast stretching of block 30 is performed in various ways. In one particular embodiment, the stretching of block 30 is performed by the following equation.<maths num="1"><img he="15" id="000002" wi="75" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Where LDR_ijIs the pixel value from the linearized image data 23 (indexed by the codes i, j), HDR_ijIs the corresponding pixel value from the stretch image data 31 (indexed by the codes i, j), α is a parameter set equal to the black level of the display on which the image is projected, and β is a coefficient. In some embodiments, α is 1 cd / m²Is less than. In one embodiment, α is 0.05-0.6 cd / m.²Is the range of. In certain embodiments, α is 0.3 cd / m²It shows dark black under normal observation conditions. The factor β is chosen to produce the desired contrast ratio, which is not high enough for the resulting image to have an unnatural appearance (eg, artifacts). Stretching the contrast to about 5000: 1 (ie, β up to about 5000) has been found to be possible without the inclusion of unacceptable artifacts in the various images. This threshold is on the safe side. For many images, a larger factor β can be used to achieve remarkable results. However, if this threshold value is exceeded, the appearance of some images may deteriorate.

The coefficient β may be set with reference to the white level on the display on which the image is displayed. For example, β may be selected so that the saturated pixel value of the linearized image data 23 is mapped to the intensity value corresponding to the white point. For example, the white point is 1000 cd / m²May be exceeded. In the prototype embodiment, the white point is about 1200 cd / m²Was chosen to be. The values of α and β are chosen to fit the target display. The values of α and β may be set regardless of the characteristics of the image represented by the linearized image data 23.

In the optional block 40 (FIG. 1), the stretch image data 31 is provided with the filtered stretch image data 41 by using an image filter. An optional block 40 filter reduces noise and quantization artifacts. Contrast stretch (block 30) and non-linear mapping of pixel values (block 22) amplify quantization artifacts and noise. The LDR input image is typically quantized to 256 pixel values and typically covers the dynamic range of the HDR display with a minimum discrimination threshold (JND) step precision using 1000 or more different values. Lossy video compression can further reduce the number of intensity levels available in the local image area. The optional block 40 filtering process can take advantage of unused intensity levels to remove artifacts that would otherwise result from this amplification of quantization artifacts.

In some embodiments, block 40 comprises applying a bidirectional filter to stretch image day 31. Suitable bidirectional filters are described in Tomasi, Manduchi, "Bilateral filtering for gray and color images," ICCV'98 Proceedings, 839. Generally, a bidirectional filter has the following form.<maths num="2"><img he="19" id="000003" wi="127" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Where h (x) is the output of the filter for the pixel at position x, A (x) is the normalization factor, f (x) is the pixel value at position x, and c (ξ-x) is the pixel at position ξ. A weighting function that decreases with the distance of the pixels at position x (c is called the "closeness" function), s (f (ξ) -f (x)) with the difference between f (x) and f (ξ) It is a decreasing weight function (s is called a "similarity" function). The integral of Eq. (2) is calculated over the neighborhood N (x) of position x.

When the bidirectional filter is given by Eq. (2), the normalization factor A (x) is given by Eq.<maths num="3"><img he="22" id="000004" wi="128" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

In some embodiments, the proximity function (c) and the similarity function (s) are Gaussian functions of their respective independent variables. For example, c is given by<maths num="4"><img he="24" id="000005" wi="128" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Where d (ξ-x) is the Euclidean distance between ξ and x, σ_dIs the parameter that defines the variance (ie, how quickly c decreases with increasing distance between ξ and x). The similar function (s) is given by the following equation.<maths num="5"><img he="22" id="000006" wi="127" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Where δ is the appropriate magnitude of the distance in the intensity space between the pixel values at position ξ and position x, σ_rIs the parameter that defines the variance (ie, how quickly s decreases with increasing distance between f (ξ) and f (x)).

In some embodiments, the variance σ of the similar function (s)_rUse a modified function (s) so that is increased with the value of f (x). In such an embodiment, the variance σ is proportional to the stretch derived by the nonlinear intensity mapping of the local pixel values of block 30._rAnd after stretching block 30, make the photometric variance equal to the fixed value of the quantization level, preferably 2.

As mentioned above, σ_rThe effect of having to change with f (x) is similar to doing a bidirectional filter with a fixed variance before stretching block 30. However, since the bidirectional filter of the block 40 after stretching the block 30 can be performed by fixed-point arithmetic, it is convenient to perform the bidirectional filter of the block 40 after the stretch of the block 30. Since bidirectional filtering is computationally expensive, it is preferable to perform bidirectional filtering in a relatively small neighborhood N (x) of each pixel when computer resources are limited. For example, depending on the embodiment, the bidirectional filter of the block 40 may be performed in the vicinity including only the pixels in the space of about 4 pixels of the pixel being processed.

In the LDR display of an image, if the pixels in the brightest area of the image are aligned (for example, if the pixel value of the LDR image is an integer in the range 0-255 (corresponding to 8-bit display), then the pixels in the brightest area of the image will be. Align the pixel value to 255). Since 255 is the maximum possible pixel value, the LDR image lacks information on how bright the original scene was above the minimum threshold that yields a pixel value of 255. In order to increase the dynamic range of the image, it is preferable to exceed the contrast stretch made by the block 30 and increase the extracted pixel value larger than the contrast stretch.

Further, in order to obtain the best HDR image based on the LDR image, it is preferable to increase the brightest pixel values, even if those pixel values are not retrieved. For example, it is preferable to increase the pixel value of pixels having a value at or above the white level of the LDR image. In these areas, information is lost because the scene intensity exceeds the capabilities of the camera, recording means, or image data format.

With reference to FIG. 1 again, block 50 generates a luminance increasing function and applies it to filtered image data 41 (or stretch image data 31 if there is no filter between blocks 30 and 50). The output image 51 is generated as a result of applying the brightness increasing function of the block 50. The brightness increasing function of block 50 increases the brightness of the output image 51, especially in the region where the pixel value for at least one color channel exceeds the threshold of the filtered image data 41. Such regions are referred to herein as "enhancement regions."

The luminance enhancement feature of block 50 attempts to modify the filtered image data 41 to produce an output image 51 that gives the observer an intuitive response similar to that associated with viewing the original scene. This is possible even if it is not possible to accurately replace the information lost from the original scene when generating the input image data 21.

FIG. 2 shows a method of performing the brightness increasing function of block 50 according to a specific embodiment of the present invention. As shown in FIG. 2, the block 50 comprises calculating the luminance increasing function 53 applied to the filtered image data 41 to generate the output image data 51 in the block 50A. As described below, the luminance increasing function 53 should be characterized by preventing the capture of noticeable spatial or temporal artifacts that significantly degrade the output image data 51. The fidelity required for the output image data 51 varies depending on the application. In the following example, the luminance increasing function 53 increases the pixel value of the filtered image data 41 to generate the value used to produce the output image data 51. In other embodiments, the luminance increasing function 53 may be applied to the filtered image data 41 using a method other than increasing.

The brightness increasing function 53 is the filtered image data 41, and has an effect of changing mainly smoothly (smoothly) and spreading beyond the edge of the increasing region. The result is not only for the pixels of the filtered image data 41 of the color channels that exceed the threshold (or other luminance measurements that meet the criteria of being included in the augmentation region), but also the region surrounding such pixels. Increases the brightness for the pixels of. As described below, the brightness enhancement function 53 includes sharp edges in the region of strong image gradation of the filtered image data 41. Depending on the embodiment, the brightness increasing function 53 may be generated by combining the smoothly changing component 53A and the edge stop component 53B. As will be described in more detail below, the edge stop component 53B identifies the position of the sharp gradation of the filtered image data 41.

The smoothly changing component 53A of the brightness enhancement function 53 is based on a map that identifies pixels of filtered data 41 that have values above the threshold (or meet the criteria of being included in other augmented regions). May be decided. Pixels with at least one color channel above the intensity threshold (or pixels with values that meet the criteria by which other luminance measurements are included in the augmented region) have a value (eg 1) and all other pixels are different. It is convenient to make a binary mask 55 with a value (eg 0). In a depiction of image data showing a single brightness value or even, the binary mask 55 uses one value (for example, 1) for pixels whose brightness exceeds the threshold and another value (for example, 0) for other pixels. You may make it by setting it to.

In general, it is preferable to set the threshold value for including the pixel in the augmentation region to be somewhat lower than the aligned value (that is, the maximum value allowed in the input image data 21). Typically, the video format uses 235 white levels to indicate that the perfect white of the reflector corresponds to the 235 pixel values of each color channel. A typical video stream also includes larger "supersaturated" pixel values that correspond to specular highlights and light sources. With Lossy video compression, pixel values may be replaced in several steps. When processing image data 21 input in RGB format on a color channel with pixel values in the range 0-255, using a threshold of 230 separates the growing region from other regions where there is Lossy video compression. It turned out to be useful for. The threshold is preferably equal to or lower than the white point of the image in question. In a typical photo, the 254 threshold was found to be sufficient in the presence of artifacts captured by Lossy compression.

The methods described here are not very sensitive to the specific thresholds chosen to distinguish between bright or saturated pixels and pixels that should be enhanced in HDR images. The threshold value may change to some extent without significantly impairing the appearance of the output image. You do not have to use sharp or fixed thresholds.

If the pixel values are specified in RGB or similar formats where the luminance information is specified separately for multiple color channels, it is not mandatory, but convenient and practical to use the same threshold for each color channel. .. Acceptable results can also be obtained by using one threshold for one color channel (eg, 229) and another threshold for one or more other color channels (eg, 231).

The smoothly changing component 53A of the brightness increasing function 53 is generated from the binary mask 55 by blurring the mask 55 with a large kernel (center) having a Gaussian shape or a nearly Gaussian shape. The result is a grayscale image 57 with a value for each pixel. The value of the grayscale image 57 is maximal in the region corresponding to the central portion of the augmented region of the filtered image data 41 and decreases smoothly away from the central portion of such augmented region. The value of the grayscale image 57 is then mapped to the range from 1 to a, where a is the magnification used to produce the smoothly changing luminance increasing component 53A. The mapping of the values in grayscale image 57 from 1 to a may be linear.

The blur kernel used to generate the grayscale image 57 is advantageously large enough that, in the expected observation situation, the spatial spectrum of the blur filter used to blur the binary mask 55 is human. It mainly contains angular frequencies that are small enough to be unobtrusive to the visual system. For example, the angular frequency is 1 cycle / degree or less, preferably 0.5 cycle / degree or less. The human visual system is less sensitive to changes in brightness that occur at such low spatial frequencies.

The standard deviation of the blur filter with respect to the space between pixels depends on the display dimensions and the expected range of observation distance. For example, on a 37-inch (diagonal) display with a resolution of 1920 x 1080 pixels, an example embodiment uses a blur filter with a standard deviation of 150 pixels. This corresponds to 1.2 degrees at an observation distance of 3 m. The standard deviation of the blur filter corresponds to at least 0.75 degrees, preferably 1 degree, more preferably 1.1 degrees. As a result, the spatial spectrum of the blur filter primarily includes low angular frequencies and does not include high angular frequencies that can be visually disturbing artifacts.

Most computer monitors are designed to be observed from a distance of approximately 0.5 meters. Such a monitor with a width of 30 cm extends to an observation angle of about 30 degrees. The recommended viewing angles for television screens in home theater systems are also typically in the range of 30-36 degrees. If the intended viewing angle is 30 degrees, the standard deviation equal to 0.025 for the horizontal resolution of the display extends to about 0.75 degrees, and the standard deviation equal to 0.05 for the horizontal resolution of the display extends to about 1.5 degrees.

When the method described in this document is used to generate an image to be displayed on a television, the standard deviation of the blur filter is at least about 0.025 of the horizontal resolution of the display, and more preferably at least about 0.033 of the horizontal resolution of the display ( Here, "about" means ± 15%). For example, on a display with a horizontal resolution of 1920 pixels, the standard deviation of the blur filter is preferably at least about 50 pixels, and more conveniently at least about 65 pixels. As mentioned above, good results on this horizontal resolution display are achieved with a standard deviation of 150 pixels.

The value of the luminance amplification factor a is selected according to the performance of the target display. The luminance amplification factor a must not be large enough to produce an output value that is significantly greater than the output the display can tolerate. In an example embodiment, 4 × 1200 = 4800 cd / m²It was found that the value of a = 4, which corresponds to the peak intensity of, gives good results on the Brightside® DR37 display. Due to the large blur radius, only peak intensities are reached in large increasing regions. A higher or lower luminance amplification factor a can also be used. Depending on the image, values of a up to about 32 can be used without capturing significant artifacts. When using this method for a wide range of images without adjustment, a safer value of a, such as a value in the range 2-9 or 10, is preferred. Depending on the embodiment, a may be in the range of 3 to 12.

The smooth component 53A of the brightness increasing function 53 stretches the overall contrast by itself when used in the filtered image data 41, and produces an image that looks clearer than the stretched image data 31 when observed on an HDR display. However, the smooth component 53A does not enhance the local contrast around sharp edges. In order to further improve the appearance in such a state, the brightness increasing function 53 includes an edge stopping component 53B. The edge stopping component 53B of the brightness increasing function 53 limits the influence of the smooth component 53A of the image area separated from the increasing area by the sharp edge.

The edge stopping component 53B may include a binary mask having a pixel value indicating whether or not the smooth component 53A is applied to the pixels. The edge stopping component 53B and the smooth component 53A may be combined by identifying the pixels of the smooth component 53A corresponding to the pixels of the edge stopping function 53B having a value indicating that the smooth component 53A should not be applied. .. The pixel value of the so-identified smooth component 53A can be set to 1 (so as not to affect the corresponding value in filter image 41).

FIG. 2A shows a method of generating a mask constituting the edge stopping function 53B based on the mask 55 and the gradation image 59. The gradation image 59 may be in the form of a binary mask generated from the filtered image data 41 and having a pixel value indicating whether or not the gradation of each pixel of the filtered image data 41 exceeds a threshold value.

Then, using the binary mask 55 as a seed, a flood fill algorithm is applied, and the flood fill is increased only from the area where the flood fill is increased. Generates the edge stopping function 53B by expanding outward until it reaches the boundary of the region that affects the pixel or smooth component 53A of the.

The gradation of the gradation image 59 is calculated using the difference quotient method. For example, the gradation of pixel 200 in FIG. 2B can be obtained by calculating the difference between pixels 201A and 201B that are close to each other in the vertical direction and pixels 202A and 202B that are close to each other in the horizontal direction. In the exemplary embodiment, the gradation is calculated as follows:<maths num="6"><img he="15" id="000007" wi="128" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Here, G is the gradation, A is the pixel value of pixel 201A, B is the pixel value of pixel 201B, C is the pixel value of pixel 202A, and D is the pixel value of pixel 202B. For robustness, it is preferable to use a wide reference line of some pixels (ie 201A and 201B are some distant pixels and 202A and 202B are some distant pixels). In the embodiment shown in FIG. 2B, the reference line is 5 pixels. This has been found to help provide thick edges in the gradation image 59 that prevent the floodfill algorithm from leaking across the edges.

In some embodiments, a morphological "open" operator (usually ""^{o o}It is preferable to further process the edge stopping component 53B (indicated by the symbol) to slightly blur the result and compress the aliasing. An open operator (not shown in Figure 2A) smoothes the contours and eliminates narrow gaps. The open operator works by gradually reducing all edges with one pixel and adding pixels adjacent to any edge in the resulting image. The further processed edge stopping component 53B is then combined with the smooth component 53A as described above to produce the brightness increasing function 53. The resulting brightness increasing function 53 is multiplied over the filtered image data 41 to produce the output image data 51.

Figures 3A, 3B, and 3C show the exemplary LDR input image 60, the corresponding smooth component 53B, and the corresponding brightness increasing function 53 modified by combining the smooth component 53A with the edge stopping component 53B, respectively.

One computationally effective method for producing the smooth component 53A and the edge stopping component 53B involves downsampling and upsampling the image data as shown in FIGS. 4A and 4B, FIG. 4A. And FIG. 4B show a method 70 for generating the smooth component 53A of the luminance increasing function 53 and a method 71 for generating the edge stopping function 53B according to a specific embodiment of the present invention, respectively. Smooth component 53A is produced by method 70 in FIG. 4A. Method 70 begins with mask 55. The mask 55 is the same as the above-mentioned mask 55 (FIG. 2), and is obtained by the same process as the above-mentioned process. At block 72, mask 55 is downsampled N times and downsample mask 73 is sought. Each step of N times block 72 downsampling reduces the number of pixels by an appropriate factor in each dimension. In some embodiments, each step of N times block 72 downsampling is downsampled so that the number of pixels in each dimension is reduced by a factor of 2 (the total number of pixels is reduced by a factor of 4). It is convenient.

In the illustrated embodiment, the smooth component 53A is then determined from the downsample mask 73 via loop 74. Loop 74 consists of N iterations, each iteration applying a blur filter in block 74A (applying a Gaussian blur with a small kernel, eg, applying a Gaussian blur to 3x3 pixels in the vicinity of each pixel. , Includes); and upsampling the subsequent results in block 74B (including the closest neighbor interpolation); This technique is described as the image pyramid method. The use of the image pyramid is described in Burt P., Adelson E., "The Laplacian pyramid as a compact image code" IEEE (1983), Communications Minutes 31, 4, 532-540. The result of method 70 is a smooth component 53A.

In some embodiments, the edge stopping component 53B is generated using method 71 shown in FIG. 4B, starting with a gradation image 75 showing the gradation of the filtered image 41. At block 76, the gradation image 75 is downsampled N times to produce the downsampled gradation image 77. Then the edge stopping function 53B,<u style="single">Downsample gradation image 77</u>Obtained from N times in loop 79, loop 79 using the closest neighbor interpolation<u style="single">Downsample gradation image 77</u>Upsample (block 78A) and apply a morphological "expansion" operation to the result (block 78B). Fixed expansion operation (usually indicated by the number 7 symbol below) to be performed in small (eg 3x3 pixels) blocks (ie with 3x3 square structural elements) and stop at the pixel corresponding to the edge (For example, an edge image of the corresponding resolution is marked to have high gradation).<maths num="7"><img he="12" id="000008" wi="15" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

FIG. 4D further illustrates these concepts, especially those related to the edge stopping component 53B. As shown, method 71 begins with a gradation image 75 showing the gradation of the filtered image 41. The gradation of the gradation image data 75 can be obtained by using the same process (for example, difference quotient) as the process of the gradation image data 59 described above. Method 71 also begins with the downsample mask 73 (FIG. 4A), which is determined by the same method as described above. At block 76, the gradation image 75 is downsampled N times to produce a set of N downsampled gradation images 77, each of which has a corresponding resolution. One of the downsample mask 73 and the downsample gradation image 77 is then subjected to a morphological expansion operation (block 78A). The expansion operation of block 78A (usually indicated by the number 8 symbol below) is performed on a small (eg, 3x3 pixel) block of the downsample mask 73 (ie, using a 3x3 square structural element). The expansion operation of block 78A comprises one of the downsampling gradation images 77 having the same or similar resolution as that of the downsampling mask 73. The expansion operation of block 78A stops at the pixel corresponding to the edge (eg, a pixel marked as having high gradation in a given pixel or a corresponding image with the same resolution of the downsampled gradation image 77). It may be modified as follows.<maths num="8"><img he="12" id="000009" wi="15" file="JP5184635B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>

The result of the expansion operation of block 78A is subjected to loop 79 and used to obtain the edge stopping function 53B. Loop 79 comprises N iterations, and each iteration upsamples the result of the previous loop 79 iteration (or the result of the expansion operation of block 78A in the case of the first loop 79 iteration) (block 78B). ); Includes applying a morphological "expansion" operation to the upsampled results of 78B (block 78C); The block 78B upsampling procedure may include the closest neighbor interpolation. The expansion operation of block 78C is similar to the operation of block 78A described above, but the expansion operation of block 78C is a downsample gradation image 77 with the same or similar resolution as the output of the upsample process of block 78B. In the expansion operation of block 78C and taking one of the inputs, the pixels corresponding to the edges (eg, the fixed pixels or the resolution of the downsampled gradation image 77 are equal) are marked as having high gradation in the corresponding image. Except that it may be modified to stop at the attached pixel). In the illustrated embodiment, there were N upsampling operations and N + 1 expansion operations at the end of loop 79. In another embodiment, the expansion operation of the first block 78A is not required-that is, there are N upsampling operations and N expansion operations. The output of loop 79 is the edge stopping component 53B.

Advantageously, the radius (block size) on which the expansion operation (blocks 78A, 78C) acts is the same as the radius on which the blurring operation (FIG. 4A) of block 74A takes place. This causes the boundaries of the area affected by the blur operation (blocks 74A) and the expansion operations (blocks 78A, 78C) to extend outward at the same rate over subsequent upsample iterations.

FIG. 4C shows a downsampled image and a gradation image that provide an image pyramid applied in the process of performing methods 70, 71 of FIGS. 4A, 4B. In particular, column (1) in Figure 4C shows the downsampling operation of block 72 (Figure 4A); column (2) in Figure 4C shows the blurring of block 74A in loop 74 and the upsampling operation of block 74B (Figure 4A). ); Column (3) of Figure 4C shows a downsample of block 76 of gradation image 75 to obtain a set of downsampled images 77 (Figure 4B); Column (4) of Figure 4C shows a block of loop 79. The expansion of 78A and 78C and the upsampling operation of block 78B are shown (Fig. 4B).

The exemplary methods described herein are implemented in a manner that provides advantageous features, including one or more of the following: -The method is implemented as an algorithm executed by the graphics processor unit ("GPU"). The method is implemented as an algorithm that can be run on a signal processor, application specific integrated circuit (ASIC) or field programmable gate array (FPGA), which is placed in displays, media players, etc. The method is efficient enough to be performed on a dynamic HDTV resolution video stream. -No user input is required. All parameters are preselected based on the hardware characteristics of the display on which the image is projected. -The method is robust in the sense that it avoids the generation of disturbing artifacts. The display quality of the HDR output image is at least as good as the quality of the input image over a very wide range of content. The output video stream is temporally coherent (color and intensity should not change abruptly unless it is in the input image).

FIG. 5 shows an apparatus 80 according to an exemplary embodiment of the present invention. The device 80 includes an input 82 that receives the input image data (FIG. 1). The image data passes through the linearizer 84, the contrast stretcher 86, and the optional filter 88 in this order. The output of the filter 88 is sent to the threshold comparison system 90 and the spatial filter 92 to generate data defining the smooth component 53A. The spatial filter 92 performs, for example, the operation of method 70 (FIG. 4A). The output of the filter 88 also passes through the gradation computer 93, the filter 94, and the threshold comparison system 96 in that order, producing an edge stopping component 53B. The data defining the smooth component 53A and the edge stopping component 53B are provided to the luminance augmentation generation component 98, which produces the luminance augmentation function 53. The outputs of the luminance increasing function 53 and the filter 88 are provided to the multiplier 99, and in the illustrated embodiment, the multiplier 99 multiplies the outputs of the filter 88 and the luminance increasing function 53 (for example, multiplication by pixels). In the illustrated embodiment, the output from the multiplier 99 is provided to the output 95. In other embodiments, the multiplier 99 performs other forms of mapping or other function (ie, other than multiplication in pixels) and takes the output of the filter 88 and the brightness increase function 53 as input to the resulting data 95. Is output. In some embodiments, the output data at output 95 is stored in a data store or continues to follow the data path of the display displaying the output data. The device 80 may process the data received at input 82 in real time.

The elements shown in FIG. 5 may be implemented in any suitable manner. For example, these elements include software running on a suitable data processor, configurable hardware or parts thereof designed to perform necessary functions such as FPGAs.

One HDR display is a type that has two modulators called a dual modulation HDR display. The first modulator produces a light pattern and the second modulator modulates the light pattern generated by the first modulator to produce an image. The first modulator is moved to produce a relatively low resolution depiction of the image. The low resolution depiction is modulated by a second modulator to provide a high resolution image visible to the observer. The first modulator comprises a matrix or array of actively modulated light sources, such as light emitting diodes (LEDs), or, as an alternative, a modulator that modulates the light emitted from a light source that is functionally separated from the modulator. To be equipped with. The first modulator is referred to as a light emitting layer or a light source layer. The amount of light emitted as a function of the position of the light emitting layer can be controlled. The second modulator is a liquid crystal display (LCD), depending on the embodiment. Such a dual-modulated HDR display produces separate drive signals for the first and second modulators.

Several methods of generating drive signals for the first and second modulators on a dual-modulated display were filed May 27, 2005, entitled "RAPID IMAGE RENDERING ON". DUAL-MODULATOR DISPLAY S) is described in the international patent application PCT / CA2005 / 000807. This application is published as WO2006 / 010244 and is incorporated herein by reference.

There is a synergistic effect between the methods described above for improving dynamic range and the methods used to generate modulator drive signals in dual-modulated displays. In particular, some intermediate results (eg, different levels of downsampled / upsampled image data) can be used for both methods. In some embodiments, the methods and devices described herein to improve dynamic range are combined with methods and devices that generate drive signals for dual modulation displays. Advantageously, in such embodiments, data is shared between those methods. This saves hardware and / or computational resources. Some embodiments provide certain savings, and some downsampled image data is used both for the purpose of improving the dynamic range of the image and for generating the appropriate drive signal (eg, the drive signal for one of the modulators). The improved image is projected on the dual modulation display. In some embodiments, the device according to the invention is incorporated into a video processor chip used in a display or a display driver chip used in a display.

6 and 6A show a method 100 of improving and projecting an image according to an exemplary embodiment. Method 100 is performed, for example, in the circuit of a dual modulation display. At block 102, the first LDR image 101A is linearized in intensity space to provide linearized image 101B. If the first LDR image 101A is encoded with a gamma curve, block 102 obtains the linearized image data 101B with a gamma that corrects the luminance value of the LDR image 101A.

Block 104 downsamples the linearized image 101B (eg, to a resolution that matches the resolution of the elements in the light source layer (ie, first modulator) of the dual-modulated display used to project the image). Generate 105. Downsampling of block 104 is performed in one or more stages. The light source layer comprises, for example, a matrix of light sources such as light emitting diodes (LEDs), a controllable array of pixels of a reflective or transmissive modulator that modulates the transfer of light from a normal light source or set of light sources. The resolution of the downsample image 105 is typically greater than the resolution of the downsample gradation image 77 (see FIG. 4B) or greater than the lowest resolution image 107 used for the purpose of improving dynamic range.

The downsample image 105 is saved (eg, stored in an appropriate memory, etc.). At block 106, the downsampled image 105 is further downsampled, yielding the lowest resolution image 107. The lowest resolution image 107 has the resolution required to generate the luminance enhancement function 53 (eg, to perform methods 70, 71 of FIGS. 4A, 4B). The downsampling of block 106 may be done in a series of downsampling steps.

The mask 109 that identifies the augmented area is provided in block 108. Block 108 comprises comparing the pixel value of the lowest resolution image 107 with one or more thresholds, for example, as described above, and generating a binarization mask 109. Mask 109 (FIG. 6) corresponds to the downsample mask 73 (FIGS. 4A, 4B) described above and is generated in block 108 using a process similar to the process described above. In some embodiments, the full resolution binarization mask (similar to mask 55 in FIGS. 2 and 4A) is generated directly from the linearized image data 101B, after which the full resolution binarization mask itself is downsampled to mask 109. obtain.

At block 110, the gradation image 111 is calculated from the linearized image data 101B. The gradation image 111 (FIG. 6) corresponds to the gradation image 75 (FIG. 4B), and is calculated in the block 110 by the same method as the above method. At block 112, the gradation image 111 is downsampled to the same resolution as the lowest resolution image 107 and mask 109. In the illustrated embodiment, the downsampling of block 112 is performed in a series of downsampling steps to produce a set of downsampled gradation images 113 with different resolutions. One set of downsample gradation images 113 (FIG. 6) corresponds to downsample gradation images 77 (FIG. 4B) and is generated in block 112 in the same manner as described above.

At block 114, mask 109 is upsampled the number of times until it reaches the resolution of linearized image 101B. A Gaussian blur (block 74A in FIG. 4A) is applied before the upsampling step for each block 114 as described above in loop 74 (FIG. 4A). The result of the upsample of block 114 is grayscale image 115. The grayscale image 115 corresponds to the grayscale image 57 (FIG. 2) and / or the smooth component 53A of the luminance increasing function (FIGS. 2 and 4A).

At block 116, mask 109 is upsampled to the same resolution as downsampled image 105. The result of the upsample operation of block 116 is saved as the upsample image 117 (for example, stored in an appropriate memory or the like). As described above in Method 71 (FIG. 4B), the expansion operation (blocks 78A, 78C in FIG. 4B) is applied during the upsampling step of each block 116. As described above for the expansion operations of blocks 78A, 78C, the gradation image 113 of the corresponding resolution is used as an edge stop in the upstream step of each block 116 (for example, the range of the expansion operation and / or the corresponding brightness increasing function corresponds to it. Limit the range). For example, the pixels corresponding to the high-gradation pixels of the corresponding gradation image 113 are set to a value that provides a luminance increasing function that affects these pixels to a small extent or does not affect them at all.

At block 118, the upsampled image 117 is further upsampled to the resolution of the linearized image 101B. The result of the upsampling of block 118 is the upsampling image 119. Although not explicitly shown in FIG. 6, the upsampling procedure for block 118 also includes an expansion operation similar to the expansion operation for blocks 78A, 78C (FIG. 4B). Again, in the upsampling step of each block 118, the tonal image 113 of the corresponding resolution is used as an edge stop (eg, limiting the scope of the expansion operation and / or the corresponding scope of the luminance enhancement function). The upsample image 119 corresponds to the edge stop component 53B (Fig. 4B) of the brightness increasing function.

At block 120, the grayscale image 115 is multiplied by the upsampled image 119 (eg, multiplied by pixels) to yield an enhancement image 121. In other embodiments, block 120 may include other mappings that input grayscale image 115 and upsample image 119 and output improved image 121. An antialiasing filter is used on the improvement image 121 at block 122 to produce a saturation extension image 123. In other embodiments, block 122 includes other techniques for removing or subtracting aliasing from the augmented image 121, otherwise antialiasing, to yield the augmented image 123. The extended image 123 corresponds to the above-mentioned luminance increasing function 53 (FIG. 2).

At block 124 (FIG. 6A), the saturated extended image 123 is multiplied by the linearized image data 101B (eg, multiplied by pixels) to yield the HRD image 125. In some embodiments, block 124 includes a mapping (eg, from 1 to a value) before performing the multiplication. In other embodiments, block 124 may include other mappings that input saturated extended image 123 and linearized image data 101B and output HDR image 125.

In the illustrated embodiment of Method 100, the control signal 131 for the light emitting layer (eg, the first modulator of the specific gravity modulated display) is generated in blocks 126-130. At block 126, the brightness of the downsampled image 105 is fixed so that the brightness does not exceed a threshold (eg, the threshold is related to the maximum brightness that the LEDs in the light emitting layer can emit). Block 126 yields a fixed image 127.

At block 128, the luminance collection step is performed on the fixed image 127, resulting in a collection LED image 129. In some embodiments, block 128 comprises applying a blur filter to the fixed image 127. Block 128 is useful when the light emitting elements of the first modulator are arranged in a pattern different from the grid used in image processing. For example, the LEDs or other light sources of the first modulator are placed on a hexagonal grid, but the image processing steps of Method 100 are processed on a square or rectangular grid (for convenience of image processing algorithms and / or hardware). ). In such cases, the rectangular grid element does not correspond to the LED or other light emitting element of the first module. The blur filter operation is performed on block 128, extending the intensity to the corresponding nearby elements corresponding to the LEDs and other light emitting elements of the first modulator.

At block 130, the collected LED image 129 is replaced to produce the first modulator drive value 131. The exchange operation of block 130 increases the intensity of the light supplied by the first modulator up to the region of the second modulator corresponding to the augmented region. The replacement operation of block 130 and the light irradiation field simulation of block 132 try to supplement the superposition effect of multiple LEDs of the first aurator. The block 130 exchange operation receives upsampled image 117 as input. The light intensity at the position of the second modulator is increased by increasing the output of the light source of the first modulator around the light source corresponding to that position. As described below, the pixel values of the second modulator are set based on the light field simulation (block 132). The light field simulation of block 132 takes into account the light produced by the first modulator when the first modulator is driven at the drive value 131. By this method, the light field simulation of block 132 prevents the area surrounding the increased position of the light intensity pattern generated by the first modulator from becoming excessively bright in the image seen by the observer.

In the illustrated embodiment, the light field simulation of block 132 is performed using the first modulator drive value 131 as an input. The first modulator drive value 131 generated in exchange operation 130 takes into account the collected LED image 129 and upsampled image 117. In another embodiment, the light field simulation of block 132 optionally receives the collected LED image 129 and / or upsample image 117. Upsample image 117 provides information about the dynamic range increase applied to the elements of the light source layer. The light field simulation of block 132 yields a luminance map 133. The luminance map 133 is incident on the pixels of the second modulator that results when driving the light source layer (first modulator) with the drive value 131 corresponding to the collected LED image 129 modified in the upsample image 117. Estimate the brightness of the light.

At block 134, the HDR image 125 is divided by the luminance map 133 to yield a drive value 135 for the element of the second modulator. Depending on the embodiment, the division operation of block 134 may include division in pixels. In another embodiment, block 134 may include mappings of other forms that take HDR image data 125 and luminance map 133 as inputs and generate a second modulator drive value 135 from them. Block 134 may also include adjusting the image value with the response function (gamma) of the second modulator.

The display shows a rendering of the HDR image 125 when the first modulator is driven by the drive value 131 and the second modulator is driven by the drive value 135.

In some embodiments, the first modulator drive value 131 is sent downstream to project the drive circuit onto a "spare" scan line in an image format that includes a second modulator drive value 135. The drive circuit extracts the first modulator drive value 131 from the spare scan line and drives the first modulator using the first modulator drive value 131. This is often practical because the first modulator has far fewer elements than the second modulator and the data format can carry one or more scanlines that are not needed by the second modulator. For example, the first modulator consists of fewer LEDs than the number of pixels on a single scan line of the second modulator. In this case, the first modulator drive value 131 for all LEDs is contained in a single scan line in a data format that does not need to convey the pixel drive value of the second modulator.

The display or components within the display are configured to perform method 100 of FIGS. 6, 6A on the incoming image data. In some embodiments, the method comprises determining whether the incoming image data requires increased dynamic range. If dynamic range increase is not required (for example, if the incoming image data defines a high dynamic range image in the appropriate HDR data format), the display switches to a mode in which the dynamic range increase is extinguished. The steps in Method 100 (Figures 6, 6A) are on one or more processors, such as a graphics processor, digital signal processor or microprocessor, and / or hardware subs such as properly configured ASICS, FPGAs, logic circuits, etc. Runs on the system. In some embodiments, the steps of Method 100 (FIGS. 6, 6A) are performed in real time (ie, at least on average at the frame rate of the video signal) on frames of a sequence of video frames.

Some embodiments of the invention include a computer processor that executes software instructions that cause the processor to execute the methods of the invention. For example, in Figure 1, 1A, 2, 2A, 4A, 6 and / or 6A, one or more processors of a display or display controller or media player execute software instructions in program memory accessible by the processor. Do the method. The present invention is also provided in the form of a product of the program. The product of the program comprises a medium having a set of computer-readable data with instructions that cause the data processor to perform the methods of the invention when executed on the data processor. The products of the program according to the invention may be in a wide variety of forms. For example, program products include floppy disks, magnetic data storage media including hard disk devices, optical data storage media including CD / ROM and DVD, electronic data storage media including ROM and flash RAM, and the like. Computer-readable data on the program's product may optionally be compressed or encrypted.

When a component (eg, software module, processor, assembly, device, circuit, etc.) is referenced above, the reference to that component (including the reference to "means") is, unless otherwise specified. A component that performs the function of the component described in the illustrated embodiment of the present invention, including a component that is not necessarily structurally equal to the disclosed component. It is understood that any component to be implemented is included as equivalent to that component.

Many modifications and modifications can be made in the practice of the present invention without departing from the ideas and scope of the present invention, as will be apparent to those skilled in the art in light of the field so far. For example In the exemplary method described above, the brightness increasing function 53 is applied (eg, in block 50) after the contrast has been stretched (eg, in block 30). This order is not mandatory. In an alternative embodiment, the brightness enhancement function can be applied before stretching the contrast. -The method described in this document is facilitated by the operation during drawing, and the pixel value changes linearly with the brightness. This is convenient, but not required. The method described in this document can be carried out in a non-linear space with appropriate modifications. In some applications, it is practical to provide an option that allows the user to fine-tune one or more parameters that affect the increase in dynamic range to obtain an HDR image with the desired appearance. Embodiments of such applications include a user interface that provides access to that parameter. The user can then select the desired values for those parameters and view the image produced from the original image using the methods described herein using those parameters. Any parameter can be made user adjustable. Some of the non-limiting examples of user-adjustable parameters are: the parameters that define the linearization function; the thresholds that identify the augmented area; the parameters that define the dark and white points; the parameters that define the overall applied contrast stretch amount. Parameters to be used; Parameters related to the size of the area affected by the brightness increasing function; Parameters related to the maximum value of the brightness increasing function, etc. -The scope of this application and the appended claims refers to low dynamic range (LDR) image data and high dynamic range (HDR) image data. These references should be understood to be related to each other. That is, LDR data is understood to have a dynamic range lower than that of HDR data, and vice versa. However, unless otherwise stated, there is no limit to the absolute dynamic range of LDR or HDR data.

Therefore, the scope of the present invention is to be construed in accordance with the contents defined in the appended claims.