Universal fixed-pixel-size ISP scheme

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of co-pending U.S. patent application Ser. No. 11/255,373, filed Oct. 21, 2005 and entitled “UNIVERSAL FIXED-PIXEL-SIZE ISP SCHEME,” which is incorporated herein by reference in its entirety and for all purposes.

BACKGROUND

1. Field of the Invention

The invention relates to a scheme of using a fixed-size Image Signal Processing unit in imaging applications, and more particularly to a scheme where the line buffer size is a fraction of the image size of the target sensors, other input devices or sources.

2. Description of the Related Art

Traditionally an ISP (Image Signal Processing unit) needs many line buffers in order to perform the functions like Color Interpolation, zooming/Scaling, etc. The size of the line buffers (and their associated control logic) is typically tailored to the maximum image size of the target sensors. The larger the image is, the bigger the line buffers are and the higher the power consumption and manufacturing cost is. In addition in an ISP of the related art, the maximum image size is limited by the line buffers. These disadvantages severely restrict the desire to offer larger image sizes. To cope with the trend of adapting higher pixel sensors in digital camera applications, a Universal Fixed-Pixel-Size ISP scheme is hereby proposed. This new scheme uses a fixed-size line buffer to handle large images, which usually require large size line buffers in a traditional design. By using this universal ISP scheme there is, theoretically, no limit on the ISP in terms of the maximum image size it can process. Since its native line buffer size is much smaller, this universal ISP scheme has the benefits of smaller die-size, manufacturing cost and lower power consumption.

U.S. Pat. No. 6,870,538 (MacInnis et al) discloses a display engine for processing graphics. The graphics may include multiple graphics layers or windows. The processing elements process two or more graphics layers in parallel to generate blended graphics. This parallelism however differs significantly in nature from the two slice buffers of the invention disclosure. The input (both analog and digital video) for receiving data and the processing elements may be integrated on an IC chip.

U.S. Pat. No. 6,480,631 (So et al.) presents an image processing apparatus for performing rotation, enlargement, reduction, clipping or overlapping processing where each scan line is read fully. The applications are different. The purpose of the proposed invention is to enable a short size ISP to handle large images where a long scan line is divided into many shorter ones.

It should be noted that the above-cited example of the related art does not address the use of fixed-size line buffer for large image sizes of target sensors whereby such fixed-size line buffers are small and have a universal use because there is virtually no maximum limit in the image size that could be processed by such a fixed-size line buffer.

SUMMARY

It is an object of at least one embodiment of the present invention to provide a scheme and a method for an Image Signal Processing unit (ISP) with a fixed-size line buffer, regardless of how much larger the size of the image is.

It is another object of the present invention to allow the ISP to perform function such as color interpolation, color correction, gamma correction, shapeness control, brightness control, contrast control, color effects, color-space conversion, resampling (which includes zooming and scaling), image filtering, and image cropping, using the fixed-size line buffer of the invention.

It is yet another object of the present invention to have low power consumption.

It is still another object of the present invention to reduce cost by reducing the size of the real estate required.

It is a further object of the present invention is to accommodate future increases in the image size of the target sensors, other input devices or sources.

These and many other objects have been achieved by dividing the image data into regions which are sequentially loaded into the fixed-size line buffer of the ISP for processing functions such as color interpolation, zooming and scaling. Since these functions of the ISP operate with neighboring pixels of the target pixel, margins of the regions need to be transmitted as well. After processing by the ISP, JPEG encoding starts which includes a DCT, Quantization, and VLC. Instead of JPEG encoding some other encoding means may be used as well. The result is then stored in segments in a buffer storage. VLC also inserts a Restart Marker which is used as a pointer to stitch together again the all the partitioned segments to produce a new and seamless image. The stitching is done sequentially by a CPU, DMA or any other suitable processing unit from the JPEG compressed code by parsing the Restart Marker.

These and many other objects and advantages of the present invention will be readily apparent to one skilled in the art to which the invention pertains from a perusal of the claims, the appended drawings, and the following detailed description of the preferred embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a first preferred embodiment of the present invention.

<figref idref="DRAWINGS">FIG. 2</figref> is a diagram of the image buffer of the present invention.

<figref idref="DRAWINGS">FIG. 3</figref> is a diagram of JPEG code segments in a storage device and of their stitching order.

<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of a second preferred embodiment of the present invention.

Use of the same reference number in different figures indicates similar or like elements.

DETAILED DESCRIPTION

The present invention describes a scheme and a method by which a fixed-size line buffer of an Image Signal Processing unit (ISP) need only be a fraction of the image size of the target sensors or other input devices or sources. This is done by partitioning the image into multiple segments or regions and subsequently loading one scan line of each segment or region into the fixed-size line buffer (the number of fixed-size line buffers may be more than one). In particular, the width of the fixed-size line buffer is smaller than the width of the image buffer.

<figref idref="DRAWINGS">FIG. 1</figref> illustrates in block format the Universal Fixed-Pixel-Size ISP scheme <b>10</b> of a first preferred embodiment of the present invention. Block <b>11</b> is the sensor which feeds the raw Image Buffer <b>12</b>, which is a storage means such as a DRAM or an SDRAM or any other suitable storage device. Image Buffer <b>12</b> is shown by way of example to have four regions: Region <b>0</b> (<b>120</b>), Region <b>1</b> (<b>121</b>), Region <b>2</b> (<b>122</b>), and Region <b>3</b> (<b>123</b>). The image in the Image Buffer <b>12</b> is transferred sequentially, region by region into the Image Signal Processing unit (ISP) <b>13</b>. The Image Signal Processing unit can process exactly one region in a fixed-size line buffer (not shown). A larger image must therefore be subdivided into more regions of a certain width, so that the width of a region can fit into the width of the fixed-size line buffer of the Image Signal Processing unit. In the preferred embodiment of the present invention the fixed-size line buffer is 1-dimensional, thus it can only store one scan line (<b>1</b>D) of a region (<b>2</b>D). The output of ISP <b>13</b> feeds JPEG encoder <b>14</b> (other encoding methods may be used as well), which typically comprises a discrete cosine transform unit (DCT) <b>15</b>, a Quantization unit <b>16</b>, and a Variable Length Coding unit (VLC) <b>17</b>. DCT <b>15</b> may also perform, instead of or in addition to the discrete cosine transform, a wavelet transform or other suitable transform. The output of the JPEG encoder is stored in Main Buffer <b>18</b>, which comprises four compressed JPEG code regions: Region <b>0</b> (<b>180</b>), Region <b>1</b> (<b>181</b>), Region <b>2</b> (<b>182</b>), and Region <b>3</b> (<b>183</b>). The stitching processing in handled by a CPU or DMA engine <b>19</b> (depending on the particular implementation of the invention; any other suitable processing units may be used as well) which parses/searches the RSTm markers in order to identify the end of the slice of each region. The RSTm or restart marker will be discussed under the heading RSTm Insertion below. The output of block <b>19</b> goes to the output device (not shown) or is stored back to the Main Buffer <b>18</b>. Image Buffer <b>12</b> and Main Buffer <b>18</b> may be part of the same memory such as a DRAM, SDRAM or any other such storage device.

To illustrate the concept of the Universal Fixed-Pixel-Size ISP, this invention, by way of example only, chose 640-pixels (VGA i.e. Video Graphics Array) for the line buffer width (the actual width will be slighter more than 640 as will be discussed later on) and a 5M-pixel (2560.times.1920.apprxeq.5 million) sensor. It is obvious to those skilled in the art that other size line buffers and sensors can be used. Prerequisites/assumptions used in this present invention are:

Existence of an Image Buffer <b>12</b>—to store a full image in Bayer or YUV formats and other applicable formats such as RGB. The Bayer format is a color filter, the smallest unit of which comprises a 2.times.2 square, where 2 green squares are diametrically opposite and where a red and blue square are diametrically opposite; a plurality of such 2.times.2 squares is a Bayer filter.

Partitioning Regions

The data from the sensor <b>11</b> is Image data and is transferred from the Image Buffer <b>12</b> (typically SDRAM) to ISP <b>13</b> during image processing. Since the ISP line buffer width (for example, 640-pixels) in this present example is too small to accommodate the 2560 pixels, the image data must be divided into a multi-region structure were the width of each Region <b>0</b>-<b>3</b> (<b>120</b>-<b>123</b>) less than or equal to 640 in the example.

<figref idref="DRAWINGS">FIG. 2</figref> shows a more detailed view of the 5M pixel image data Buffer <b>12</b> of <figref idref="DRAWINGS">FIG. 1</figref>, divided into four VGA-wide Regions <b>0</b>-<b>3</b> (<b>120</b>-<b>123</b>). The image depth is 1920 pixels. Clearly the Image Buffer <b>12</b> must have the dimension of 2560.times.1920. The partition is performed when transferring the image data to ISP <b>13</b>. DMA only moves about 640 pixels instead of a full 2560 pixels for each scan line. After the transfer of one region is done, DMA continues on the next region until the entire image data transfer is completed. Some of the ISP functions operate with neighboring pixels of the target pixel. Which means that additional pixels (represented as a in <figref idref="DRAWINGS">FIG. 2</figref>) located beyond the boundary of the 640-pixel-wide region should be transferred to the ISP as well. The operation starts with the processing of Region <b>0</b> (<b>120</b>) as a (640+.alpha.).times.1920 image. The data transferring from SDRAM to the ISP is still in a traditional scan line pattern. Next Region <b>1</b> (<b>121</b>), followed by Regions <b>2</b> (<b>122</b>) and <b>3</b> (<b>123</b>), are processed as either a (640+2.alpha.).times.1920 or (640+a).times.1920 image.

The output of ISP <b>13</b> will immediately be loaded to 2 slice buffers (640-pixel wide). This might be in an alternating fashion for example. Once one slice buffer (not shown) is full, the JPEG encoding (<b>14</b>) starts on that slice. Since four regions of image are encoded, four segments of SDRAM space Region <b>0</b>-<b>3</b> (<b>180</b>-<b>183</b>) in the Main Buffer <b>18</b> are allocated for storing the JPEG bit steams.

Since the 5M image is vertically divided into four regions, and each region is encoded individually, a naive concatenation of 4 JPEG files would not restore a seamless 5M-pixel JPEG image.

RSTm Insertion

To solve this problem we introduce here the Restart Marker (RSTm), which is one of the JPEG makers inserted in the JPEG bit stream for error recovery/resilience purposes. The RSTm can be programmed to be inserted into JPEG bitstream in every fixed-number minimum coding unit (MCU) interval. Where the definition of the MCU is either a group of (2Y blocks+IU+IV blocks) in 422 image format or (4Y blocks+IU+IV blocks) in 420 format. Each block is 8.times.8 pixels. The height of a slice is one MCU, i.e., 8 pixels in 422 image format or 16 pixels in 420 image format The code assignment of RSTm is X‘FFD<b>0</b>’ to X‘FFD<b>7</b>’, where ‘m’ is a modulo-8 number (which loops from <b>0</b> to <b>7</b>). This invention takes advantage of the Restart Marker to perform stitching of the JPEG compressed files from multiple partitioned image regions.

For each 640-pixel wide region, at the end of every slice, an RSTm (m=N) is inserted with the appropriate number of padding bits (1′b1) to make it end at a byte boundary. At the end of the following slice, another RSTm (m=modulo (N+4, 8)) is inserted again, aligned to the byte boundary by padding bits. I.e., the region numbering goes from <b>0</b> to <b>3</b>, while the Restart Marker numbering goes from <b>0</b> to <b>7</b>. In this example of the present invention, the interval of 2 RSTm codes is 40 MCUs in 420 format. The number 40 (the length of a partitioned slice) is derived from 640 (width of a region) divided by 16 (the pixel width of 2Y blocks) for the 420 format. So that 40 is the number which has to be specified in the JPEG file header, for the present example, and set in the variable length code (VLC) section. In general, the number of minimum coding units in each of the slices is determined by the ratio of the pixel width of the slices to the minimum coding unit width.

The reason for ending RSTm at a byte boundary is to make the stitching of four regions of JPEG code easier later on. It is very difficult for either software or hardware to implement bitstream stitching in the non-byte aligned condition. The ‘m’ value of RSTm has to be carefully arranged so as not to violate JPEG standards once all four regions of JPEG data are stitched together. This is a deterministic pattern of ‘m’ for every slice of each region. If the CPU (<b>19</b>) does the VLC tasks, the ‘m’ value calculation can be done by software. This calculation can also be done by the hardware with some programmability.

<figref idref="DRAWINGS">FIG. 3</figref> illustrates the four segments of JPEG (compressed) data in the Main Buffer <b>18</b>, starting from slice <b>0</b> of each Region <b>0</b>-<b>3</b> (<b>180</b> to <b>183</b>) and the sequence of code stitching. For every slice, the JPEG code terminates with an RSTm at the byte boundary. For each region Slices <b>0</b> to <b>2</b> are identified while slices following Slice <b>2</b> are only labeled as “slices” since any number of slices could follow. Similarly, for each region Restart Markers are identified as RST<b>0</b> to RST<b>7</b>, followed by the first four restart markers RST<b>0</b> to RST<b>3</b> of the next sequence, as explained above.

Stitching

After the encoding of four regions is done with Regions <b>0</b>-<b>3</b> (<b>180</b>-<b>183</b>) of JPEG data in the Main Buffer <b>18</b> (SDRAM), proper stitching of these four regions constitutes a single JPEG data. The sequence of stitching is illustrated in <figref idref="DRAWINGS">FIG. 3</figref> as follows. RST<b>0</b> of slice <b>0</b>, Region <b>0</b> points to the beginning of slice <b>0</b> of Region <b>1</b>. RST<b>1</b> of that slice points to the beginning of slice <b>0</b>, Region <b>2</b>. RST<b>2</b> of that slice points to the beginning of slice <b>0</b>, Region <b>3</b>. RST<b>3</b> of that slice points to the beginning of slice <b>1</b>, Region <b>0</b>. RST<b>4</b> of that slice points to the beginning of slice <b>1</b> of Region <b>1</b>. RST<b>5</b> of that slice points to the beginning of slice <b>1</b>, Region <b>2</b>. RST<b>6</b> of that slice points to the beginning of slice <b>1</b>, Region <b>3</b>. RST<b>7</b> of that slice points to the beginning of slice <b>2</b>, Region <b>0</b>. RST<b>0</b> of that slice points to the beginning of slice <b>2</b> of Region <b>1</b>. RST<b>1</b> of that slice points to the beginning of slice <b>2</b>, Region <b>2</b>. RST<b>2</b> of that slice points to the beginning of slice <b>2</b>, Region <b>3</b>. And RST<b>3</b> of that slice points to the beginning of the slice after slice <b>2</b>, Region <b>0</b>. The arrows showing the stitching of one slice to the next are identified by numerals <b>1</b> to <b>12</b>.

This single JPEG data presents the full 5M-pixel as if it were encoded in one single pass. There are many ways of doing stitching. It can be done either by the CPU <b>19</b> with an index table of RSTm locations, or by parsing the JPEG code to determine the existence of an RSTm. Alternative designs, including units such as a smart DMA or any other suitable processing units, can handle the stitching without the CPU's intervention.

Zooming

When the ISP <b>13</b> is required to perform zooming, the present Universal ISP Scheme also works. Before image data is transferred to the ISP <b>13</b>, the CPU <b>19</b> has to determine how the image is going to be divided and instruct the DMA, the ISP <b>13</b>, and the DCT <b>15</b> NVLC <b>17</b> accordingly. By using the correct width to divide the region, the line buffer (not shown) of the ISP is prevented from overflow when performing zooming. This practice is a general guideline in the present Universal ISP Scheme.

Advantages of the proposed invention are:

a) virtually no maximum limit in image size,

b) reduced power consumption over prior art designs,

c) perform large image processing in a small fixed-size line buffer,

d) economy of scale, since the same fixed-size line buffer can be used in many applications,

e) invention is applicable whenever the image size is larger than the ISP buffer size.

<figref idref="DRAWINGS">FIG. 4</figref> illustrates in block format the Universal-Pixel-Size ISP scheme <b>40</b> of a second preferred embodiment of the present invention. Blocks <b>11</b>, <b>12</b>, and <b>13</b> are the same as those of <figref idref="DRAWINGS">FIG. 1</figref> and perform the same function, however the output of ISP <b>13</b> is stored in memory <b>41</b> of Main Buffer <b>18</b>. Regions <b>0</b> to <b>3</b> (<b>120</b>-<b>123</b>) after processing by ISP <b>13</b> are stored sequentially in memory <b>41</b>. The image in memory <b>41</b> is an intact and complete representation of the image in Image Buffer <b>12</b>. Since there is neither compression, DCT, nor VLC there is no need for insertion of a restart marker, stitching or the like. Because the image in memory <b>41</b> is not compressed, the storage requirement is larger than that of Regions <b>0</b> to <b>3</b> (<b>180</b>-<b>183</b>) combined (of <figref idref="DRAWINGS">FIG. 1</figref>). If a large memory is available the second preferred embodiment of the present invention is the better implementation. Otherwise the first preferred embodiment of the present invention is better.

While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the invention.