Image enhancement on JPEG compressed image data

In the past, a typical application for copiers or scan-to-print image processing systems was to reproduce an input image as accurately as possible, i.e., render a copy. Thus, copies have been rendered as accurately as possible, including flaws in the source image. However, as customers have become more sophisticated in their document reproduction requirements, they have recognized that an exact copy is often not desired. Instead, an inexact copy that is perceived as a higher quality image has been deemed more desirable.

Until recently, image quality from the output of a copier or a scan-to-print system was directly related to the quality of the input image. While this has been satisfactory for reproducing images of text or line drawings, it has been found to be sub-optimal for purposes of reproducing photographs and other more complex images. With photographs, in particular, reproduction is complicated given the inexact nature of the discipline, variations in equipment, aging of photographs, and the like. Given that the input image is often poor quality, it has been deemed desirable to render an output image that is perceived to be the "best" possible image (or at least superior to the input image), rather than an exact copy.

U.S. Patents to Eschbach et al. and Fuss et al. disclose an Automated Image Enhancement (AIE) system. This system receives an optionally sub-sampled description of the input image, and alters the tone reproduction curve (TRC) -- a curve that defines the relationship of an input image signal to an output image signal for purposes of enhancement -- and/or derives or alters a sharpness filter for that image on an image-by-image basis, as appropriate, so that the resulting output image is perceived to be superior to the input image. AIE is used, for example, to alter perceived exposure, luminance, sharpness, saturation, color balance, and the like to provide an output image that is preferable to the input image. It is important to note that AIE performs well even if it has to rely upon a sub-sampled image to statistically analyze the overall image. AIE does not require that each and every item of image information be analyzed. Of course, this subsampling speeds image enhancement operations and reduces the size/number of memory buffers required to implement the AIE system.

More particularly, for example, U.S. Patent No. 5,414,538 entitled "Image-Dependent Exposure Enhancement" discloses a method of altering the perceived exposure of an output image produced from an input image that includes: (a) receiving the input image defined in terms of red-green-blue (RGB) signals; (b) converting the RGB signals to corresponding luminance-chrominance signals including at least one signal that represent overall image intensity; (c) comparing the intensity signal to upper and lower intensity threshold signals that define the acceptable levels of brightness and darkness in the image; (d) if one of the thresholds is exceeded, the image signal representative of image intensity is processed according to a select equation, and a TRC associated with the image is adjusted so that exposure characteristics of the resulting output image are perceived to be superior to those of the input image.

U.S. Patent No. 5,450,502 entitled "Image-Dependent Luminance Enhancement" discloses a method of altering the perceived luminance of an output image produced from an input image that includes: (a) receiving the input image data defined in terms of a color space; (b) if required, converting the input image data into a luminance-chrominance color space wherein at least one term bears a relationship to overall intensity of the input image; (c) deriving a global intensity histogram for the overall input image; (d) filtering the histogram signal to flatten high peaks and low valleys without altering its relatively flat portions; and, (e) utilizing the filtered histogram signal to control TRC mapping in a device with which the image is to be rendered. In accordance with another aspect of the disclosed luminance enhancement method, the input image can be divided into plural regions, and a local intensity histogram signal can be derived for each region. If any of the local histogram signals are flatter than the global histogram signal, the local signals are summed and used in place of the global histogram as input to the histogram flattening filter.

The methods described in the Eschbach et al. '538 and '502 patents, and the other Eschbach et al. and Fuss et al. patents noted above, are described in connection with uncompressed image data. However, in many image processing operations, the image data is retrieved from an image storage device or other location or is otherwise supplied in a compressed form to minimize image storage space. Most commonly, the image data is compressed according to the Joint Photographic Expert Group (JPEG) recommendation ISO DIS 10918-1 that has become an international standard for lossy compression of still images.

Thus, heretofore, image enhancement operations according to the AIE systems described above have required that the JPEG (or otherwise) compressed images first be decompressed or "decoded" for image enhancement operations. This is undesirable in that the decompression operation slows the overall enhancement operation, the image processing apparatus must be provided with additional memory to accommodate large amounts of uncompressed image data, and each JPEG or other lossy compression operation, itself, further degrades the image data due to data loss.

In light of the foregoing, it has been deemed desirable to provide a novel and non-obvious method for utilizing JPEG compressed image data in the image enhancement operations described in the above-noted Eschbach et al. and Fuss et al. AIE patents. This will allow JPEG compressed images to be enhanced for subsequent rendering without the image being decompressed, enhanced, and then recompressed.

In accordance with the present invention, a method and apparatus for image enhancement of JPEG compressed image data are provided. In accordance with a first aspect of the present invention, a method of processing JPEG compressed image data comprises: (i) receiving JPEG compressed image data that represent an input digital image, the JPEG compressed image data including a plurality of DC components; (ii) extracting a plurality of the DC components from the JPEG compressed image data; (iii) inputting at least some of the extracted DC components to an automated image enhancement system; (iv) using the DC components input to the automated image enhancement system to derive a final correction tone reproduction curve and/or a sharpness filter for enhancement of the input digital image represented by the JPEG compressed data; (v) decompressing the JPEG compressed image data to obtain decompressed image data that represent the input digital image ; and, (vi) using the final tone reproduction curve and/or the sharpness filter to enhance the decompressed image data.

One advantage of the present invention resides in the provision of a method and apparatus for enhancing JPEG compressed image data, without fully decompressing the data. <ul id="ul0001" list-style="none" compact="compact"><li>FIGURE 1 is a block diagram illustrating image enhancement of JPEG compressed image data in accordance with the present invention;</li><li>FIGURE 2 is a partial illustration of an input digital image to be compressed according to a conventional JPEG data compression operation;</li><li>FIGURE 3 is a diagrammatic illustration of an 8x8 pixel block of JPEG compressed data, and extraction of the DC component therefrom for purposes of image enhancement in accordance with the present invention; and,</li><li>FIGURE 4 illustrates a image enhancement apparatus and method for enhancement of JPEG compressed data in accordance with the present invention.</li></ul>

FIGURE 1 illustrates an example of an image processing system implementing the present invention. Input image data defining an input image is supplied from either a scanner <b>10a</b> or memory <b>10b</b>. In the case where the input image data is supplied from a scanner <b>10a</b>, the scanner can be a black-and-white or color scanner that scans a printed input image and that derives digital image data that defines the scanned printed image. Typically, the scanner <b>10a</b> outputs digital image data defined in terms of red, green, and blue (RGB) color separations, although the data can alternatively be supplied in any other suitable color space. A color space converter <b>11</b> receives the digital image data from the scanner <b>10a</b> and, if required, converts the data into a luminance-chrominance color space, such as <b>YC</b>_<b>b</b><b>C</b>_<b>r</b> color space or the like as is required for JPEG data compression of the input image data. The input image data is output by the color space converter <b>11</b> to a JPEG compression unit <b>12</b> that compresses the data using a conventional JPEG compression process.

Alternatively, the input image data is supplied from a memory <b>10b</b> or other conventional source in JPEG compressed format as is well known to minimize memory requirements. In either case, whether the input image data is supplied from the scanner <b>10a</b>, the memory <b>10b</b>, or another suitable source, the data can define the input image monochromatically or in terms of multiple color separations that, together, define a color image.

As is described in full detail below, JPEG compressed data includes DC components and, in accordance with the present invention, these DC components are extracted from the JPEG compressed data by a DC component extractor <b>14</b> that receives the JPEG compressed data from either the JPEG compression unit <b>12</b> or another source such as the memory <b>10b</b>. The DC components extracted by the DC component extractor <b>14</b> are input to the Automated Image Enhancement (AIE) system <b>16</b> that is fully described in the aforementioned Eschbach et al. and Fuss et al. patents. The unaltered JPEG compressed image data, including the DC components, is also input from the JPEG compression unit <b>12</b> or the memory <b>10b</b> to a TRC and Filter Binder <b>18</b>. It should be noted that optionally one may choose to incorporate a modified DC component extractor as described in U.S. Statutory Invention Registration No. H0001684. In such case, a small subset of the low frequency AC components is also decoded in an efficient way. In this case, some of the performance advantages of the pure DC component extraction with respect to processing time are traded off against an increased set of data for the statistical analysis. This trade-off can be advantageous, e.g., for images with a limited number of sampling points. For simplicity of description, it is intended that the terms "DC" and "DC components" used in the following encompass this optional implementation.

The AIE system <b>16</b>, in turn, derives a correction tone reproduction curve TRC and/or filter based upon only the DC components input from the DC component extractor <b>14</b>. The correction TRC and/or filter are input to the TRC &amp; filter binder <b>18</b> wherein the TRC and filter are bound to or otherwise operatively associated with the JPEG compressed image data. Once the correction TRC and/or filter are bound to or otherwise operatively associated with the relevant JPEG compressed image data, the correction TRC and filter are available to any downstream image processing module, and the TRC and filter can be applied once the JPEG compressed image data are decompressed. For example, when the input image is to be viewed by way of an image output terminal <b>22</b>, such as a printer or visual display, the JPEG compressed data, including the correction TRC and/or filter bound thereto, are input to a decompress and enhance unit <b>20</b> wherein: (i) the JPEG compressed data are decompressed in a conventional manner to provide uncompressed input image data; and, (ii) the uncompressed input image data are enhanced via application of the corrective TRC and/or filter. Thereafter, the enhanced, uncompressed input image data are input to the image output terminal for display/printing.

According to the essential properties of JPEG compression, an input image <b>I</b> (FIGURE 2) is defined in terms of a plurality of pixels <b>P</b>, each defined in terms of a luminance-chrominance color space such as YC_bC_r. This image <b>I</b> to be compressed is divided into a 2-dimensional array of typically square blocks <b>B</b> of pixels <b>P</b> (the individual pixels <b>P</b> in only one of the blocks <b>B</b> are labeled for clarity). Most commonly, the original image <b>I</b> is divided into square blocks with each block comprising 8x8 = 64 pixels <b>P</b> from the original image <b>I</b>.

A discrete cosine transform (DCT) is then performed on the pixel data <b>P</b> in each block <b>B</b>. The forward DCT has the effect of transforming each of the blocks <b>B</b> into the spatial frequency domain and, following the DCT, the elements in a block <b>B</b> still completely describe the original input image data, but larger values tend to cluster at the top left corner of the block <b>B</b>, in a low spatial frequency region. Simultaneously, the elements located more toward the lower right hand portion of the block <b>B</b> will tend toward zero for most photographic images. The top-left entry <b>30</b> in each block <b>B</b>, which represents the average value all pixels <b>P</b> in the block <b>B</b>, is known as the DC component or DC coefficient (identified in FIGURE 3 as "<b>DC</b>") of the block <b>B</b>, and all the other entries <b>32</b> in the block are referred to as the AC coefficients or AC components (identified in FIGURE 3 as "<b>AC</b>") of the block. In the case of color image data, the DC component <b>DC</b> of a block <b>B</b> represents the average luminance-chrominance value of the pixels <b>P</b> defining that block <b>B</b>. Of course, in the case of black-and-white data, the DC component <b>DC</b> of a block <b>B</b> represents only the average luminance of the pixels <b>P</b> defining that block <b>B</b>.

Following the DCT step, individual entries <b>DC,AC</b> in the block <b>B</b> are quantized, or in effect made into smaller numbers, and rounded. Then, the quantized entries are Huffman-encoded to yield a string of bits. There may be other lossless compression steps to encode the quantized DCT coefficients, but the final product is inevitably a string of bits for each block <b>B</b>, wherein each block <b>B</b> is converted into a string of bits of a different length. Of course, numerous variations on the above-described JPEG compression technique are well known and may be implemented without departing from the overall scope and intent of the present invention.

FIGURE 3 illustrates operation of the DC component extractor <b>14</b> as previously described in relation to FIGURE 1. In FIGURE 3 it may be seen that when a block <b>B</b> of JPEG compressed data is input to the DC component extractor <b>14</b>, the DC component extractor <b>14</b> identifies and outputs the DC component <b>DC</b> of the input block <b>B</b>. As noted, this extracted DC component <b>DC</b> represents the average luminance or the average luminance-chrominance value for the pixels <b>P</b> of a block <b>B</b>.

FIGURE 4 is a more detailed illustration of an image processing apparatus that is adapted for enhancement of JPEG compressed image data in accordance with the present invention. JPEG compressed image data is received at <b>50</b> from a memory <b>10b</b>, a JPEG compression unit <b>12</b>, or another source, and the DC component extractor <b>14</b> extracts at least some, and preferably all, of the DC components <b>DC</b> from the JPEG compressed data. Those of ordinary skill in the art will recognize that all of the DC components <b>DC</b> need not necessarily be extracted to provide a good subsampling of the JPEG compressed data -- only some of the DC components <b>DC</b> are required to obtain a usable statistical sampling of the JPEG compressed image data. The JPEG compressed data stream is unaltered by the DC component extractor <b>14</b>, and from the DC component extractor <b>14</b>, it is input to the TRC/Filter binder unit <b>18</b> by way of a data channel <b>100</b>.

The extracted DC components <b>DC</b> are input via data channel <b>102</b> to a DC component control module <b>104</b> that controls the input of the extracted DC components <b>DC</b> into the automated image enhancement (AIE) unit <b>16</b>. In particular, the DC component control unit <b>104</b> makes the extracted DC components <b>DC</b> available on an information channel <b>108</b> that is, in turn, operatively connected to a plurality of AIE modules, e.g., an AIE exposure module <b>112</b>, an AIE color balance module <b>122</b>, and AIE contrast module <b>132</b>, and AIE luminance module <b>182</b>, and an AIE sharpness module <b>192</b>, by way of buses <b>110, 120, 130, 180, 190,</b> respectively. Those of ordinary skill in the art will recognize that other AIE modules can form a part of the AIE unit <b>16</b>, and it is not intended that the invention be limited to the particular AIE modules shown. The AIE modules can also return information to the information channel <b>108</b> for use by other AIE modules. The AIE modules <b>112, 122, 132, 182, 192,</b> in the AIE unit <b>16</b> are described in the aforementioned Eschbach et al. and Fuss et al. U.S. Patents in greater detail.

Each of the exposure, color balance, contrast, and luminance AIE modules <b>112, 122, 132, 182</b> use the data supplied on the information channel <b>108</b> to calculate the input signals for the respective TRC generators <b>114, 124, 134, 184</b> that generate intermediate TRC's required to effect the required image modifications as determined by each module <b>112, 122, 132, 182</b>. Each of the TRC generators <b>114, 124, 134, 184,</b> directs a generated intermediate TRC to a TRC combiner <b>140</b> that combines the respective intermediate TRC's into a single, final TRC 160 defined by a look-up table (LUT). The final TRC is then directed to the bind TRC unit <b>150</b> of the TRC/filter binder <b>18</b> wherein it is bound to the JPEG compressed image data for subsequent use. Similarly, a sharpness module <b>192</b> receives the DC components <b>DC</b> from the information channel <b>108</b> and uses same to generate a sharpness filter <b>194</b> that is directed to the TRC/filter binder <b>18</b>, in particular to a bind filter unit <b>200</b> thereof, wherein the filter is bound to the JPEG compressed data (along with any TRC that has already been bound to the JPEG compressed data by the bind TRC unit <b>150</b>. The combined TRC/Filter and JPEG compressed image data is output at <b>250</b> to downstream image processing modules such as the decompress/enhance unit <b>20</b> and image output terminal <b>22</b> described in connection with FIGURE 1. It should be recognized by those of ordinary skill in the art that the final TRC <b>160</b> and/or sharpness filter <b>194</b> can be bound or otherwise operatively associated with the JPEG compressed image data for any desired length of time, including a fraction of a second or months, years, or longer such as for long-term archival storage. Furthermore, the final TRC 160 and/or filter 194 can be used immediately once derived, assuming the JPEG compressed image data has been suitably decompressed.

Those of ordinary skill in the art will recognize that the DC components <b>DC</b> extracted from the JPEG compressed data provide a subsampling of the uncompressed input image data, without requiring that the image data be further decompressed. Thus, the DC components can be used by the AIE systems described above without significant modifications to these systems. On the other hand, it is necessary to keep in mind that the DC components <b>DC</b>, themselves, represent only an average value of the data in each 64 pixel (or other size) block <b>B</b> of the input image <b>I</b>. Therefore, it may be desirable in certain situations to use more conservative parameters in the AIE system <b>16</b> to account for the fact that any given pixel <b>P</b> in a block <b>B</b> may vary significantly from the average value <b>DC</b>. Also, as noted above, it is to be recognized that not all the extracted DC components <b>DC</b> need to be input to the AIE system <b>16</b>. For example, half of the DC components <b>DC</b> may provide a good subsampling of the JPEG compressed data.