Method and system for producing formatted data related to geometric distortions

The present invention relates to methods and systems for calculating transformed images from digital images and formatted information related to geometric transformation.

<p> The present invention relates to a method of calculating a transformed image from a digital image and formatted information related to geometric transformation, particularly from formatted information related to equipment chain distortion and / or chromatic aberration. This method involves calculating the transformed image from an approximation of the geometric transformation. Therefore, the memory resources, memory bandwidth, computational power, and thus power consumption required for the computation are reduced. Also, from this, the converted image has no visible or annoying defects for later use.</p>

<p>Digital images are composed of pixels, which are hereafter referred to as digital pixels. The transformed image is composed of pixels, which are hereafter referred to as transformed pixels. The transformed pixels are characterized by the transformed position and the transformed value. According to the present invention, the method preferably includes the step of calculating the value of the transformed pixel, for that calculation-in the digital image, from the formatted information, for each converted position. Process (a), which is the process of selecting one block of digital pixels, -Process (b), which is the process of calculating the digital position within the one block of digital pixels for each converted position from the formatted information. ),-For the converted position, the value of the converted pixel is one block.<u style="single">De</u>A general algorithm is used that includes process (c), which is the process of calculating as a function of the value of the digital pixel and the digital position.</p><p> Formatted information includes parameters. By using the parameters, at least one function related to the geometric transformation can be selected. By using one or more functions, it is possible to calculate blocks and digital positions of digital pixels from the converted positions.</p><p> According to the present invention, a general algorithmselects a converted pixel, hereafter referred to as an initial conversion pixelis a general algorithm for initial conversion pixels to obtain blocks of initial digital pixels and initial digital positions. It is preferably used in the procedure of applying processes (a), (b), and (c).</p><p> For each converted pixel other than the initial conversion pixel, --the process of selecting one block of digital pixels in the digital image from the initial digital block and / or each initial conversion position (d),- The process of calculating the digital position within a block of digital pixels from the initial digital block and / or each initial conversion position (e), --The converted pixel value is the digital pixel value of the digital pixel block and An optimized algorithm is applied that includes process (f), which is the process of computing as a function of digital position.</p><p> Applying a simple algorithm to other points from a combination of technical features, using formatted information that requires complex calculations for the initial digital points, while maintaining a good approximation of the geometric transformations. Therefore, the total calculation time can be shortened.</p><p> This method is used with hardware and / or software processing means. According to the present invention, it is preferable that the optimized algorithm exclusively uses integer data or fixed-point data. Due to the combination of technical features, even though processes (a) and (b) perform floating-point calculations, (a) and (b) are performed less frequently than (c) and (d). Also, even if floating-point arithmetic is used, it only needs to be emulated a little, so general and optimized algorithms can be executed without a floating-point processor or operator. The combination of technical features allows, for example, to minimize current consumption and operate as fast as possible while embedding algorithms in photographic equipment.</p><p> According to the present invention, the method further preferably comprises the step of quantizing the digital position to obtain the quantized digital position. Due to the combination of technical features, the coefficients can be represented in tabular form by using steps (c) and (f) with a limited number of inputs, thus significantly reducing the cache memory used. And the bandwidth of the main memory can be reduced.</p><p> According to the present invention, the method further preferably comprises the step of calculating the coefficients for a plurality of blocks. Processes (c) and (f)-select a block of coefficients using the quantized digital position and-calculate the value of the converted pixels from the blocks of coefficients and the blocks of digital pixels. Is executed by.</p><p> Due to the combination of technical features, the calculation of blocks of coefficients can be performed before compilation.</p><p> According to another aspect of the invention, the processes (c) and (f) of calculating the values of the transformed pixels can also be applied to transformations other than geometric transformations, especially attenuation of image smearing. .. Therefore, it is possible to apply a plurality of image conversions while reducing energy consumption and time.</p><p> Digital images are obtained from sensors with multiple channels. A color plane can be output by combining channels. Preferably, even in this case, the present invention allows the channels to be combined to obtain a color plane using the processes (c) and (f) of calculating the converted pixel values. From the combination of technical features, it can be seen that in the case of 3-color plane RGB, the calculation time and power consumption using the general algorithm and / or the optimized algorithm can be divided into about three. Furthermore, the accuracy is relatively good due to the combination of technical features. Moreover, due to the combination of technical features, there is little extra cost to add geometric transformation processing for equipment that includes channel coupling, especially for digital photographic equipment.</p><p> Digital images consist of color planes. In this case, according to the present invention, it is preferable that the method is such that different geometric transformations are applied to each color plane to correct chromatic aberration.</p><p> According to the present invention, the method further preferably comprises combining the geometric transformation with other geometric transformations that vary with the digital image, especially the zoom effect. Due to the combination of technical features, it may cost little extra time and energy to apply other geometric transformations, especially zoom effects, to a digital image at the same time as the geometric transformations. Moreover, due to the combination of technical features, it is possible to apply geometric transformations to other geometrically transformed digital images.</p><p>Formatted information may depend on variable characteristics that depend on the digital image, especially the size of the digital image. In this case, according to the present invention, the method further preferably comprises the step of determining the value of the variable property of the digital image. Processes (a) and (b) use formatted information about variable characteristics that depends on the values thus obtained.<u style="single">So these</u>Adopt a formatted information method that relies on variable characteristics from a combination of technical features<u style="single">Even if you did it</u>Adopts a formatted information method that does not depend on variable characteristics<u style="single">The result is the same as the method used.</u></p><p> According to the present invention, the formatted information is preferably related to distortion defects and / or chromatic aberrations in the equipment chain. The parameters relate to the measurement field.</p><p> The transformed image may show differences when compared to the image obtained by applying the geometric transformation to the digital image. In this case, according to the invention, the method further sets the general algorithm and / or the optimized algorithm and / or the initial conversion point so that the --threshold selection step, the difference is less than the threshold. It is preferable to include a step of selection.</p><p> The calculation time from the combination of technical features to reaching a certain level of image quality is the shortest.</p><p> The transformed image may show differences when compared to the image obtained by applying the geometric transformation to the digital image. In this case, according to the invention, the method is further--in the step of selecting the threshold,--the general algorithm and / or the optimized algorithm and / or the initial conversion point and / or so that the difference is less than the threshold. / Or preferably includes the step of selecting the quantization of the quantized digital position.</p><p> From the combination of technical features, the time to calculate to reach a certain level of image quality is the shortest.</p><p> According to the present invention, the method further sorts the transformed positions so that the blocks of digital pixels selected by processes (a) and / or (d) have a specified average number of common digital pixels. It is preferable to include a step of Due to the combination of technical features, even a small cache memory with a small number of registers can sufficiently accommodate most of the pixel values required for continuous iteration of processes (a) and / or (d). In addition, the combination of technical features can significantly reduce memory bandwidth. Also, due to the combination of technical features, it is not necessary to keep a complete digital image in memory. Also, due to the combination of technical features, it is not necessary to keep the fully converted image in memory. In addition, the combination of technical features reduces costs and power consumption. Systems The present invention relates to systems that calculate transformed images from digital images and formatted information related to geometric transformations, particularly formatting information related to equipment chain distortion and / or chromatic aberration. The system comprises a computational means of calculating the transformed image from an approximation of the geometric transformation.</p><p>Digital images are composed of pixels, which are hereafter referred to as digital pixels. The transformed image is composed of pixels, which are hereafter referred to as transformed pixels. The transformed pixels are characterized by the transformed position and the transformed value. According to the present invention, the system preferably comprises a computing means for calculating the value of the transformed pixels, for which-in the digital image, from the formatted information, for each transformed position. , The process of selecting one block of digital pixels (a),-The process of calculating the digital position within a block of digital pixels for each converted position from formatted information (b), --For the converted position, the value of the converted pixel is the value of the converted pixel for the above 1 block.<u style="single">De</u>Use data processing means with general algorithms, including process (c), which is the process of calculating as a function of digital pixel values and digital positions.</p><p> Formatted information includes parameters. By using the parameters, you can select at least one function related to the geometric transformation. By using one or more functions, it is possible to calculate blocks and digital positions of digital pixels from the converted positions.</p><p> According to the present invention, the computing means-selection of converted pixels, hereafter referred to as initial conversion pixels-is a process of a general algorithm of initial conversion pixels to obtain blocks of initial digital pixels and initial digital positions. It is preferable to use the general algorithm in the manner of applying (a), (b), and (c).</p><p> The calculation means is the process of selecting one block of digital pixels in a digital image from the initial digital block and / or each initial conversion position for each converted pixel other than the initial conversion pixel (the process). d), --The process of calculating the digital position within a block of digital pixels from the initial digital block and / or each initial conversion position (e), --The converted pixel value is digital in the block of digital pixels. Apply an optimized algorithm that includes process (f), which is the process of calculating as a function of pixel values and digital positions.</p><p> General algorithms or optimized algorithms are executed by hardware and / or software processing means. According to other convenient embodiments, the optimized algorithm exclusively uses integer or fixed-point data.</p><p> According to the present invention, it is preferable that the system further includes data processing means for quantizing the digital position and acquiring the quantized digital position.</p><p> According to the present invention, it is preferable that the system further includes a calculation means for calculating the coefficients for a plurality of blocks. Processes (c) and (f) are calculated by the calculator-using the quantized digital position to select a block of coefficients-the blocks of coefficients and the pixel values converted from the blocks of digital pixels. It is executed by calculating.</p><p> The processes (c) and (f) of calculating the values of the transformed pixels can also be applied to transformations other than geometric transformations, especially the attenuation of image smearing.</p><p> The digital image obtained from the sensor can have multiple channels. A color plane can be output by combining channels. According to the present invention, the processes (c) and (f) of calculating the converted pixel values are preferably processes such that channels can be combined to obtain a color plane.</p><p> According to the present invention, the digital image is preferably composed of a color plane. The system is a system in which different data processing means can apply different geometric transformations to each color plane to correct chromatic aberration.</p><p> According to the present invention, it is preferable that the system is such that the data processing means can combine the geometric transformation with other geometric transformations that change according to the digital image, especially the zoom effect.</p><p> Formatted information may depend on variable characteristics that depend on the digital image, especially the size of the digital image. In this case, according to the present invention, it is preferable that the system further includes data processing means for determining the value of the variable characteristic of the digital image of interest. Computational means that perform processes (a) and (b) use formatted information that depends on the values thus obtained for variable characteristics.</p><p> According to the present invention, the formatted information is preferably related to distortion defects and / or chromatic aberrations in the equipment chain. The parameters relate to the measurement field.</p><p> The transformed image may show differences when compared to the image obtained by applying the geometric transformation to the digital image. In this case, according to the invention, the system can further use general algorithms and / or optimized algorithms and / or initial transformation points such that the difference is less than the selected threshold. It is preferable to provide a processing means.</p><p> The transformed image may show differences when compared to the image obtained by applying the geometric transformation to the digital image. In this case, according to the invention, the system is further generalized and / or optimized algorithm and / or initial conversion point and / or quantized digital so that the difference is smaller than the selected threshold. It is preferable to have a data processing means capable of using position quantization.</p><p> According to the present invention, the system further sorts the transformed positions so that the blocks of digital pixels selected by processes (a) and / or (d) have a specified average number of common digital pixels. It is preferable to provide a data processing means for processing.</p>

Best mode to carry out the invention

Other features and advantages of the invention will become apparent when reading the description of other embodiments of the invention indicated and presented in non-limiting examples.

Equipment In particular, the concept of equipment P25 will be described with reference to FIG. Within the meaning of the present invention, the device P25 is specifically incorporated into a disposable photographic device, digital photographic device, reflective device, scanner, fax machine, endoscope, camcorder, surveillance camera, telephone, personal digital assistant, or computer. Image capture or image capture devices such as connected or connected cameras, thermal imaging cameras, or reverberant devices, --image recovery devices such as screens, projectors, TV sets, virtual reality goggles, or printers,- Humans with visual abnormalities such as turbulence, -devices that output images similar to those produced by Leica-branded devices, which are desired to be able to emulate,-have edge effects that add smearing, Image processing devices such as zoom software, --Multiple devices P25 equivalent virtual devices, scanners / faxes / printers, photo development minilabs, or more complex devices such as electronic conferencing devices P25 can be one device P25 or multiple devices machine can be regarded as a unit P25.

Device Chain The concept of device chain P3 will be explained with reference to FIG. 10. The device chain P3 is defined as a set of devices P25. The concept of equipment chain P3 can also include the concept of order.

The following example constitutes a device chain P3.

--Single device P25, --Image capture device and image restoration device, --For example, photo development minilab photo device, scanner, or printer, --For example, photo development minilab digital photo device or printer, --For example, computer scanner, screen, Or printers, --screens or projectors, and the human eye, --one and other devices that should be able to emulate, --photographic devices and scanners, --image capture devices and image processing software, --image processing software And image restoration equipment, --combination of the above examples, --other equipment set P25.

Defects The concept of defect P5 will be described with reference to FIG. Defects in equipment P25 P25 is defined as defects related to the characteristics of the optics and / or sensors and / or electronic units and / or software embedded in equipment P25, and as an example of defect P5, geometric distortion, There are smearing, vignetting, chromatic aberration, color play, flash uniformity, sensor noise, grain, astigmatism, and spherical aberration.

Digital Image The concept of digital image INUM will be described with reference to Figure 1 in particular. A digital image INUM is defined as an image captured, modified or restored by device P25. The digital image INUM is obtained from the device P25 of the device chain P3. The output destination of the digital image INUM can be the device P25 of the device chain P3. For animated images, such as video images that consist of a time series of still images, the digital image INUM is defined as the still image in the image sequence.

Formatted Information The concept of the formatted information IF is explained with particular reference to Figure 10. Formatted information is defined as data related to geometric transformations, for example, data related to defect P5 of one or more devices P25 in device chain P3 is such data, and using it, The transformed image ITR can be calculated by considering the defect P5 of the device P25. Defects P5 are particularly considered geometric distortion and / or chromatic aberration defects. The formatted information IF can be output using a variety of methods based on reference measurement and / or capture or restoration, and / or simulation.

To output formatted information IF, for example, a method filed on the same day as this application under the name Vision IQ and described in an international patent application entitled "Method and system for producing formatted information related to geometric distortions". It is possible to use. The application describes how to output a formatted information IF related to device P25 of device chain P3. The device chain P3 is, in particular, composed of at least one image capture device and / or at least one image recovery device. This method involves outputting a formatted information IF related to the geometric distortion of at least one instrument in the chain.

Device P25 can capture or restore images on media. Instrument P25 includes at least one fixed characteristic and / or one variable characteristic, depending on the image. Fixed and / or variable properties can be associated with the values of one or more properties, especially focal length and / or focusing, and related properties. This method involves outputting measurement formatted information related to the geometric distortion of the instrument from the measurement field. Formatted information IF contains measurement formatted information.

To output the formatted information IF, for example, "Method and system for reducing update frequency of image processing" was filed on the same day as this application under the name Vision IQ. It is possible to use the method described in the international patent application entitled "means". The application describes ways to reduce the frequency of image processing means, especially software and / or component updates. Image processing means can be used to modify the quality of digital images obtained from or destined for the device chain. The device chain consists of at least one image capture device and / or at least one image recovery device. The image processing means uses formatted information related to defects in at least one device P25 in the device chain P3. The formatted information IF depends on at least one variable. Formatted information makes it possible to define the correspondence between a part of a variable and a part of an identifier. The identifier can be used to determine the value of the variable corresponding to the identifier, taking into account the identifier and the image. From the combination of technical features, it is possible to determine the value of a variable, especially if the physical significance and / or the content of the variable is known only after the image processing means have been distributed. In addition, the combination of technical features allows the correction software to be updated twice at time intervals. In addition, from the combination of technical features, various economic activity organizations that create equipment and / or image processing means can update their products independently of other economic activity organizations, which is the latter. Even if it radically changes the characteristics of its products or it cannot force clients to update their products. Also, the combination of technical features allows new features to be rolled out gradually, starting with a limited number of economic activity organizations and pioneering users.

To output a formatted information IF, for example, in an international patent application entitled "Method and system for providing formatted information in a standard format to image-processing means" filed on the same day as this application under the name Vision IQ. It is possible to use the method described. The application describes how to supply standard-format formatted information IFs to image processing means, especially software and / or components. The formatted information IF is associated with a flaw in the device chain P3. The device chain P3 specifically includes at least one image capture device and / or one image recovery device. The image processing means uses the formatted information IF to modify the quality of at least one image obtained from or destined for the device chain P3. The formatted information IF includes data that characterize the defect P5 of the image capture device, in particular distortion characteristics, and / or data that characterize defects in the image recovery equipment, in particular distortion characteristics.

The method comprises writing a formatted information IF in at least one field of the standard format. The field is specified by the field name. A field contains at least one field value.

To search for formatted information IF, for example, an international filing under the name Vision IQ on the same day as this application and entitled "Method and system for modifying the quality of at least one image derived from or addressed to an appliance chain". It is possible to use the method described in the patent application. The application describes how to modify the quality of at least one image that is drawn from or destined for a designated device chain. The designated device chain comprises, in particular, at least one image capture device and / or at least one image recovery device. Image capture and / or image restoration devices that are being gradually introduced to the market by multiple economic organizations belong to an intermediate group of devices. Equipment P25 in this group of equipment shows defect P5 characterized by formatted information. For the image of interest, this method involves the following steps:

--The stage of compiling the directory of the source of the formatted information related to the devices of the device group, --Automatically specific formatted information related to the specified device chain between the formatted information compiled in this way. The stage of searching for digital images,-the stage of automatically processing digital image INUMs using image processing software and / or image processing components, taking into account the specific formatted information thus obtained.

Variable characteristics The concept of variable characteristics will be described. According to the present invention, the variable characteristic is defined as a measurable factor, which affects the defect P5 of the image captured, corrected or restored by the same device P25 and captured, corrected or restored by the device P25. Depending on the digital image INUM that it exerts, in particular, this is-the capture of images that are fixed for a given digital image INUM, such as global variables, such as those related to user adjustments or related to the automatic functions of the device P25. Characteristics of the device P25 at the time of restoration, --in-image, which is variable within a given digital image INUM, local variables, eg, different local processing can be applied depending on the zone of the digital image INUM, if necessary. Coordinates of, x, y, or ρ, θ.

A measurable factor that can be changed from one device P25 to the other, but fixed from one digital image INUM to the other image captured, modified, or restored by the same device P25, is generally not considered a variable characteristic. For example, the focal length of the device P25, which has a fixed focal length.

The formatted information IF depends on at least one variable characteristic.

From the variable characteristics, the following can be understood in particular.

--Optical length, --Resize settings applied to the image (digital zoom factor: magnifying part of the image, and / or sampling: reducing the number of pixels in the image), --Nonlinear brightness correction such as gamma correction, - To highlight contours, such as the level of smearing correction applied by equipment P25, --Sensor and electronic unit noise, --Optical system aperture, --Focal length, --Number of frames on film, --Unexposure or exposure Excess, --Film or sensor sensitivity, --The type of paper used in the printer, --The center position of the sensor in the image, --The rotation of the image relative to the sensor, --The position of the projector relative to the screen, - Other adjustments applied by the user of equipment P25, such as white balance used, --flash and / or its power activation, --exposure time, --sensor gain, --compression, --contrast, --operation mode, --equipment P25 Other automatic adjustments.

Variable characteristic value The concept of variable characteristic value VCV will be explained. The variable characteristic value VCV is defined as the variable characteristic value when capturing, modifying, or restoring the specified digital image INUM.

Parameterizable model Within the meaning of the present invention, a parameterizable model is defined as a mathematical model that relies on variable characteristics associated with one or more defects P5 of one or more devices P25. The formatted information IF associated with equipment defect P25 can be in the form of parameters in a parameterizable model that relies on variable characteristics.

Calculation of Transformed Image With reference to FIG. 1, a simplified embodiment of the method and system of the present invention will be described.

A digital image INUM contains a set of image elements defined as pixels PXnum.1 to PXnum.n that are regularly distributed on the surface of the image INUM. In Figure 1, the shape of these pixels is square, but other shapes such as circles are possible, depending on the style of the surface that is designed to transmit the image on the image capture and restore device. .. Further, in FIG. 1, the pixels are shown as concatenated, but in practice there is generally some spacing between the pixels.

The converted image ITR further contains a set of pixels defined as PXTR.n from the converted pixels PXTR.1. Each transformed pixel is characterized by a transformed position pxtr and a transformed value vxtr.

The transformed image is a corrected or modified image obtained by applying the transformation to the image. This transformation may be a geometric transformation, but when performed, it takes into account formatted information that takes into account the defects of the equipment used or the characteristics captured in the image.

Note that the formatted information is associated with a limited number of transformed pixels and / or can incorporate value VCVs of variable properties (focal length, focusing, aperture, etc.) that depend on the image. I want to. In this case, there may be auxiliary steps performed by interpolation to result in simple formatted information, such as formatted information for devices that do not have variable characteristics, and therefore devices with variable focal lengths. Case results in the case of a device with a fixed focal length.

In the example where the functions x', y'= f (x, y, t), t are variable characteristics, the formatted information has a limited number of values (xi, yi, ti, f (xi, yi, ti). )) Can be configured. Therefore, it is necessary to calculate approximate values for other values of x, y, and t. With the same format, t can be a vector and multiple variable characteristics can be included at the same time. In the case of distortion, the formatted information corresponds to, if necessary, a vector indicating the displacement that each point follows, or a set of discrete elements that represent measurement points used during the pre-calibration step, or an approximation of this discrete set. It can be configured with functions to reduce the amount of formatted information.

Exif can provide formatted information with data related to equipment used and designed in the preliminary phase, as well as details about equipment adjustments at the time of shooting (focal length, focusing, aperture, speed, flash, etc.) Or you can include all the information organized in other formats.

For example, the digital image INUM represents a capture of a rectangular image. In Figure 1, the pixels that correspond to the square traces are shown in black. Due to the distortion of the capture device, the rectangle is deformed as shown in the image INUM shown in Fig. 1. According to the present invention, in particular, by using the computational means CAPP that captures the approximation according to the desired final accuracy, the value vxtr of the transformed pixel with the position pxtr is obtained and the pixel on the transformed image ITR. It is possible to obtain a rectangle in which the position and value of are actually corrected to fit the approximate range.

Note that when the algorithm CAPP is applied, the deformed image becomes a complete or semi-perfect image in the case of distortion. It also uses the same algorithm to transform the deformed image into another image that has been deformed differently if necessary, much like an image of a known type (fisheye lens effect, reverse distortion, etc.). Can be output. The same algorithm also allows the transformed image to be reduced to an image that is not perfect (in the sense of a straight line, as shown in Figure 1) but looks optimal to the observer's eyes, which is perceived by the human eye. Geometric defects can be compensated as needed.

FIG. 2a is a diagram showing an example of the improved method and system according to the present invention. Only the pixels used in this method and system description are shown in the digital and converted images. The converted image shows the pixel PXTR.i for which the value must be determined. In the digital image, the position pxnum.i corresponds to the position of the converted pixel pxtr.i in the converted image, which in the case of distortion, for example, pxnum to lower back to pxtr.i. Obtained from formatted information (IF) containing the displacement vectors that need to be added to the .i.

The present invention includes the various steps described and shown in FIG.

First, the position pxtr.i of the converted pixel PXTR.i in the converted image is identified (step ET1). In the subsequent stages (ET2), if information about the location within the image ITR was obtained, it was transformed using a formatted information IF that conveys the characteristics of the device performing the image capture and / or restoration. Infer the position of the pixel block BPNUM.i of the digital image INUM that surrounds the position pxnum.i of the point corresponding to the pixel PXTR.i. According to the example in Figure 2a, one pixel block contains 5x5 pixels.

Then (step ET3), by using the transformed information, for the transformed position pxtr.i of the pixel PXTR.i, the digital position pxnum.i of the point of the digital pixel block corresponding to the pixel PXTR.i. calculate. As is clear from FIG. 2a, this point does not necessarily correspond to the position of the center of the pixel in the pixel block BPNUM.i. Therefore, you can see that the converted values of pixel PXTR.1 do not correspond to the values of digital pixels.

In the next step (ET4), if information about the position of the point pxnum.i in the block BPNUM.i is obtained, it will be placed in the block BPNUM.i and of the virtual pixel centered on the point pxnum.i. The value is calculated. This virtual pixel PXFIC has a value that needs to be considered for the value of the pixel surrounding the point pxnum.i in the block, and therefore the position of this point in the block. In a simple way, you can average the values of the pixels surrounding this point.

In addition, the pixel values of a block can be averaged by assigning the value of each pixel a coefficient that is a function of the distance of that pixel relative to the point position pxnum.i. This is the same as calculating the following sum for the entire pixel block.

Σvxnum.j × Cj (digital position pxnum.i) However, vxnum.j = the value of the pixel in the block, and Cj = the coefficient of the pixel as a function of the point position pxnum.i.

The coefficients of each pixel in a block can be calculated in various ways as a function of the point position pxnum.i. The first method uses analytical formulas to calculate the coefficients of each pixel of the block as a function of, for example, the order of the approximated surfaces, the accuracy of the computer, and the position pxnum.i within the block.

A simpler method is to use the method of quantization of position pxnum.i to limit the number of possible positions of points in the block. In such a case, the coefficient table shall be constructed for each possible position of one point in the block. For quantized values with different positions within the block, a plurality of coefficient series are calculated in each series together with the coefficient value for each pixel of the block.

Then, in step ET4 described above, the position pxnum.i of a point in the block is quantized so that the coefficient sequence can be accessed using one useful coefficient for each pixel of the block. It suffices to multiply this coefficient sequence by a sequence of values of the same pixel.

FIG. 4 is a schematic diagram of such a process. The position pxnum.i at a point is quantized as an explained value that can take a limited number of values from a to n.

There is a Cn from the coefficient table Ca for each of the described values of the point position. For the value "a", for example, table QU1 allows access to tables with coefficients a1 through an. For example, the table Ca can be accessed by using the value "a". The table Ca contains coefficients a1 to an, which are the same as the number of pixels contained in the pixel block BPNUM.i. Each of these coefficients is calculated as a function of the position pxnum.i of a point in the block, assigning a weight to the value of each pixel. It is easy to see that the pixel farthest from the point in the block at position pxnum.i has a low weight and the pixel closest to it has a high weight.

Using the set of pixel values in the coefficient table Ca and block BPNUM.i, the following calculation Σ (vxnum.j × Cj) is performed to obtain the value of the virtual pixel PXFIC, and the converted image ITR It is this value that is assigned to the converted pixel pxtr.i of.

In the embodiment, the position of the point pxnum.i obtained from the converted pixel position using the formatted information is determined by the integer part (or first address number) and the decimal part (or second address number). It is possible to think that it can be represented. For example, the quantization of the above-mentioned address pxnum.i can be uniquely associated with the decimal part.

The integer part can be the address of the pixel block BPNUM.i in the image INUM, or more precisely the address of the defined pixel of this block, such as the block's pixel PXnum.1.

In the decimal part, specify the address of a point in the block. For this address, it can be determined that the number of options is represented numerically and the address is represented by only a limited number of bits, thus limiting the number of Cn from the coefficient table Ca. For example, if the address in the block is represented by 3 bits, it is necessary to prepare Cn from the table Ca with 8 coefficients.

Therefore, even when performing calculations by floating-point arithmetic in stages ET2 and ET3, these stages tend to be executed less frequently than in stage ET4, and even if floating-point arithmetic is used, there are few operations that emulate. As conceivable, it is possible to carry out the methods of the invention without a floating point processor or operator, embedding the algorithm in, for example, a photographic device, while minimizing current consumption and running as fast as possible. Can be done. In this case, the system according to the invention comprises hardware and / or software processing means that do not use floating point processors or operators.

However, it is possible to use so-called floating-point processors or operators (eg Intel Pentium processors) instead of so-called fixed-point processors or operators (eg Texas Instruments signal processing processor TMS320C54xx).

The above method can be applied to all pixels of the transformed image and the value can be known from the value of the digital pixel.

In an example of this method, for example, a square block of digital pixels was taken. However, the block can also have other shapes (circular, hexagonal, etc.), as shown in Figures 2b, 2c, and 2d.

Therefore, an improvement of the method and system of the present invention for speeding up the processing will be described. It is assumed that for each converted pixel, the time required to calculate the position of a point in the digital image using the formatted information can be reduced.

FIG. 5a is an organizational diagram showing examples of other embodiments of the method for calculating the transformed image.

First, in (step ET0), select a certain number of pixels in the converted image and define them as the initial conversion pixels. For example, select 4 pixels PXINIT.1 to 4.

The above method is applied for each converted pixel. This is because FIG. 5a shows the same stages ET1 to ET4 as in FIG. After stages ET1 to ET4 are applied to the initial conversion pixels PXINIT.1 to 4, the system sends a query to itself (stage ET5) to see if all initial conversion pixels have been processed, and if not. , Repeat the process of steps ET1 through ET4 for another initial conversion pixel. Once all the initial conversion pixels have been processed, the system is ready to move on to the next stage of the method, ET6. The situation at the end of stage ET5 is shown in Figure 5b, and at the same time-four converted pixels PXINIT.1 to 4 in the converted image ITR,-point pninit.1 in the digital image INUM. Four blocks of early digital pixels, including BPINIT.1 to 4, are also shown. The positions of blocks BPINIT.1 to 4 in the image INUM are known (see step ET2 of the method). The positions of points pninit.1 to 4 within each block are also known (see step ET3 of the method).

Therefore, it is necessary to calculate the value of any converted pixel PXTR.i in the converted image. In Figure 5c, this pixel PXTR.i is placed between the initial transform pixels, but it does not have to be.

Therefore, in the process of step ET6, the pixel PXTR.i is selected and its position pxtr in the converted image is obtained.

In the process of step ET7, the relative position of pixel PXTR.i relative to the initial conversion pixels PXINIT.1 to 4 is calculated. For example, given the position information of all these pixels (px.1, px.2, etc.), the distance between the initial conversion pixels 11-14 as well as the pixels PXTR.i to the initial conversion pixels PXINIT.1-4 Distances to d1.1, d1.2, d2.1, ... d4.2 are calculated. Therefore, the position of the pixel PXTR.i is expressed as a relative position with respect to the initial conversion pixel and / or a relative position with respect to the distance separating the initial conversion pixels. In this way, this relative position can be expressed as a percentage of the distance. Also, this position is set to (d1.1) (d2.1) (px.4) + (1-d1.1) (d2.1) (px.3) + (d1.1) (1-d2. It can also be represented by some kind of bilinear relational expression such as 1) (px.2) + (1-d1.1) (1-d2.1) (px.1) or other higher-order relational expressions. Become.

In the process of step ET8, the digital pixel block BPNUM.i containing the points obtained from the mathematical projection of the converted pixel PXTR.i to the digital image is placed based on the proportional relationship already calculated.

In the process of step ET9, the same proportional rule is used to determine the position pxnum.i in the block BPNUM.i of points obtained from the mathematical projection of the converted pixel PXTR.i to a digital image.

At this stage, using a proportional rule that applies to a set of points in an image can be calculated in less time than using formatted information.

As mentioned above, it is possible to quantize the position of the point pxnum.i and represent it on a limited number of bits. It is possible to use the same quantization basis that was adopted when calculating the value of the initial transform pixel, and therefore the value of the pixel PXFIC using a table of the same coefficients as applied to the digital pixel block. And therefore the converted pixel PXTR.i value can be obtained (step ET10).

However, --the initial conversion pixels can be placed in a regular matrix, and the matrix spacing is 2 to take advantage of the processor's parallel execution instructions, especially Intel's MMX, SSE, or SSE2 instructions or AMD's 3DNow instructions. Note that it is possible to place a pixel block using bilinear interpolation (step ET8) and place points on this placed pixel block (step ET9).

According to another embodiment of the invention, the method of calculating pixel values is performed in multiple stages to reduce the block size and finalize the addition and / or multiplication performed by the computer. The condition can be to reduce the number of times. A general algorithm and / or an optimized algorithm for some values of the coefficient Cj, which represents what the trader calls a mathematically separable interpolation operator, first for horizontal geometric transformations, and then again for vertical. Applying geometric transformations to intermediate results, and thus theoretically, the number of multiplications and / or additions required to process a block can be divided by two.

Please note the following.

--Quantization of transformed and digital pixel positions can construct a table of constants and coefficients of limited dimensions, significantly reducing the amount of cache memory used and the bandwidth in main memory. This is possible because the calculations for steps ET3 and ET10 depend only on the digital position of the pixel block.

--The coefficient table can be configured only once and once in the system. Of course, it is possible to prepare a plurality of types of coefficient tables of different sizes corresponding to digital pixel blocks of different sizes. For example, it is possible to modify the quantization precision to modify the approximation of the processing operation and thus the quality of the resulting transformed image. In addition, in some cases, different types of tables of the same size, but for the same position of points within a digital pixel block, for example, to compensate for the selection of different types of prominent defects (distortion, vignetting, smearing, etc.). It is also possible to store multiple coefficient tables containing coefficients with different values in.

--A convenient dimension for the coefficient table is 4x4, but it doesn't have to be.

--The coefficients may be bicubic interpolation coefficients, and the calculations for steps ET3 and ET10 can be performed in the form of scalar products.

As explained, the approximation of the processing operations obtained as a result of the quantization of pixel positions for a limited number of bits is transformed, which may show the difference when compared to the image completely obtained by mathematical transformation. Image is output. According to these conditions, it can be determined in this way to limit this approximation by providing a step in selecting the threshold. This stage is then related to the selection of the quantized threshold by determining that the quantization does not fall below the specified threshold, and multiple digital points above this threshold within the pixel block. The position is obtained. Therefore, the calculation time can be optimized according to the specified quality of the converted image.

FIG. 8 shows an improvement of the invention that reduces memory cycle time and thus processing time. When calculating the values of two adjacent transformed pixels, one would use two digital pixel blocks with common pixels, that is, the values of these common pixels would be read twice (according to this example). ) It is easy to understand that it will be. An object of the improvement of the present invention is to avoid this double reading. Therefore, it is a prerequisite that the converted pixel positions are sorted so that they are processed in a certain order. For example, by tracing each line from left to right, it can be determined to process the transformed pixels of the transformed image line by line and within each line pixel by pixel. In FIG. 8, for example, pixel PXTR.1 is processed first, then pixel PXTR.2 is processed, and so on.

Once pixel PXTR.1 has been processed by the method described above, then --digital pixel block BPNUM1 is selected, --points are positioned within this pixel block, and --the converted pixel PXTR.1 value is this. Calculated as a function of point position and block pixel values.

According to the method and system improvements of the present invention, the value of BPNUM.1 pixel is retained in temporary memory. For practical purposes, the values of these pixels can be stored, for example, in cache memory.

When the pixel PXTR.2 is processed, it is subsequently selected, for example, the digital pixel block BPNUM.2, which has pixels in common with the block BPNUM.1 used in the above description. Since the values of these pixels are kept in temporary memory, the system only needs to search for the pixel values of BPNUM.2 that are not common to block BPNUM.1, and in Figure 8, these are the values of block BPNUM.2. The pixels in the right column. After pixel PXTR.2 is processed, the pixel value of block BPNUM.2 is kept in temporary memory in preparation for the next converted pixel PXTR.3.

According to another version of the invention, only the values of pixels common to two consecutively used pixel blocks need only be kept in temporary memory. In general, only the average number of pixels common to two consecutively used pixel blocks is stored in temporary memory.

Although the above description has been made within the range of distortion correction, the present invention can also be applied to smearing correction or attenuation, and even if a plurality of image conversions are performed, energy consumption is low and the execution time is short. ..

This same description is also applicable to device chains where one or more devices can exhibit defects such as distortion. Since the combination of formatted information is transmitted in the distortion space by a simple vector summation, it is possible to handle all the defects in the device chain in one pass.

In addition, the same transformation combines multiple types of corrections, such as embedding discrete RGB and / or CMYK values in a color image sensor, suppressing vignetting of an image, adding a magnifying or zooming effect, or changing perspective. You can also do it.

FIG. 6 outlines a calculation step (ET4 or ET10) in which an auxiliary table of coefficients can be prepared for the calculation of the value of each converted pixel. For example, in FIG. 6, the corrected pixel value can be calculated from the coefficient tables Ca to Cn after the distortion is corrected. The coefficient tables CA and CN can be used to correct for other defects such as image smearing and vignetting. Multiple corrections of different types can be combined by multiplying the tables against each other as shown in Figure 6 or by performing other operations that can combine the coefficients of the tables, thus processing time. Can be shortened and energy consumption can be reduced. With such an arrangement, one correction or the other correction can be applied arbitrarily, or all corrections can be applied if the user so desires.

A method of applying the above-mentioned method and an embodiment of the system to color image processing will be described.

Color images are considered to be obtained from a plurality of channels and provide a plurality of image planes or color planes.

As shown in FIG. 7, the color image is considered to be composed of a red image INUMR, a green image INUMG, and a blue image INUMB.

As far as the calculation stage of the converted pixel value is concerned, that is, as far as the stage ET3 of FIG. 3 or the ET9 of FIG. 5a is concerned, the above method is performed. Thereby, not only the positions of the digital pixel blocks (BPNUMR, BPNUMG, BPNUMB) in each image INUMR, INUMG, and INUMB, but also the positions of the points in these blocks corresponding to the converted pixels for which the values are calculated. known. In addition, the values of the various pixels that make up the blocks BPNUMR, BPNUMG, and BPNUMB are known.

By using the position of the point corresponding to the transformed pixel to be calculated, a table of coefficients such as table C1 in FIG. 7 can be accessed. As mentioned above, the calculation Σ (vxnum.i × Cj) is performed by taking the sum of the products of the pixel values of the block multiplied by the corresponding coefficients obtained from the coefficient table. This operation is performed first on the red image block BPNUMR, then on the green image block BPNUMG, and finally on the blue image block BNUMB. In this way, the converted pixel values of the red, green, and blue components are obtained.

Such methods and systems can be used to save time and energy consumption associated with color. Therefore, such a processing operation can be incorporated into a digital photographic device. Therefore, it can be assumed that geometric transformation is added to the processing operation.

The present invention can be applied to the processing of color images in which the number of pixels of different colors or the number of color channels (r, g, b) are not equal. For example, a three-color image IMrgb, as shown in Figure 9a, contains two green pixels, one for each red pixel and one for each blue pixel, but is typically the case with a color sensor whose eyes are very sensitive to the wavelength of green. The same is true. Further, from the viewpoint of software processing of images, a color image is considered to include the same number of images (or color planes) as the basic colors in the image. For example, the image IMrgb in FIG. 9a is considered to include the three color planes IMr, IMg, and IMb. In the methods and systems described above, each color plane is processed independently to obtain the three transformed images. In addition, it is convenient to assign a value to each pixel of each of these transformed images. FIG. 9b shows the three color image planes IMr, IMg, and the three transformed image planes ITRr, ITRg, and ITRb corresponding to IMb in FIG. 9a. Applying the method described above to calculate the value of each pixel in various transformed image planes, for each converted color pixel (eg red), the corresponding digital color image (plane in the example adopted). Extract the digital pixel block of IMr).

In addition, when applying this method and system to color images, the processing of the three color planes can be combined to save time and power.

In particular, distortion can be compensated, for example, by applying the same geometric transformation between the transformed color image plane and the digital color image plane.

It is also possible to apply different geometric transformations to different color planes to correct for chromatic aberration and / or distortion.

Note that for RGB images (using 3 channels), the transformed color plane can be obtained by extracting one of the three pixels.

In the above description, the transformed image and the digital image correspond to each other by geometric transformation. Especially from digital images and the fixed and known properties of equipment used with a second geometric transformation, such as variable geometric transformations that differ depending on the image capture conditions (eg, zoom effect). The first specified geometric transformations obtained can be combined, thereby reducing the extra cost of time and energy, as well as other geometric transformations, the zoom effect in the examples adopted, the first. At the same time as the geometric transformation of 1, it can be applied to the digital image, and the variable geometric transformation can be applied to the digital image that has undergone the first geometric transformation.

The second geometric transformations are, among other things: --Rotation, --90 or 180 degree rotation, --Translation, --Zoom effect, --Dimensioning effect, --Perspective change, --Reference frame change, --Projection, --Geometric transformation represented by a function obtained by calculating coordinates x'and y'from coordinates x and y, --uniform transformation, --combination of these examples, --linear or non-linear geometric transformation Can be done.

Combining two geometric transformations for different purposes-applying a second geometric transformation, especially the zoom effect, to a digital image at the same time as the first geometric transformation, geometrically Allows you to get a transformed image of the same size as a digital image when the transformation is related to distortion, for example a dimensional transformation effect where you need to get the linear edges of the image. , --Combine the distortions of multiple devices in the chain and correct it in one step, --For example, by applying a zoom effect, the digital photographic device outputs images of various sizes, and the geometric transformation is the device. When it is related to the geometric distortion of, the distortion can be corrected by combining the geometric transformation and the zoom effect, and the information related to the history of the processing operations applied to the digital image is used for this purpose. Can be useful for.

The combination of these two geometric transformations can be performed in different ways, for which to do-perform the calculations that apply to the parameters and-the calculations that apply to the function, for example, in the case of a vector field, of the vector. Addition, or a combination of polynomials in the case of polynomials, the function is a function that calculates the coordinates x'and y'from the coordinates x and y, which is calculated for each iteration of the process in steps ET2 and ET3. In this case, the extra computational time required to apply the second geometric transformation to the initial transformation points can be significantly reduced by adopting an optimized algorithm that applies the method of Figure 5a.

The present invention can be applied not only to image processing systems but also to photographic devices, video cameras, surveillance cameras, webcam type computer cameras and the like.

The embodiments of the invention thus described can be used in the form of software or hard wiring components.

FIG. 11 shows an embodiment of the system according to the present invention. The system comprises computing means MC. Computational means MC uses the methods according to the invention to relate to digital image INUM, characteristic variable values VCV, and formatted information IFs related to geometric transformations, especially distortion and / or chromatic aberration of equipment chain P3. Calculate the converted image ITR from the formatted information IF.

The system can be equipped with a data processing means MTI, which in particular uses the method according to the invention to quantify the digital position so that it can obtain the quantized digital position and / or the geometry. To combine the transformation with other geometric transformations that change with the digital image, especially the zoom effect, apply different geometric transformations to each color plane to correct the chromatic aberrationsfor example, digital image INUM. By using the data registered in Exif format in the included file, the value VCV of the variable characteristic is determined for the digital image of interest, and --the converted position is sorted.

Application to cost reduction of the present invention Cost reduction is defined as a method and system for reducing the cost of equipment P25 or equipment chain P3, especially the cost of equipment or equipment chain optics, and cost reduction is implemented by the following methods. To do.

--Reduce the number of lenses and / or --Simplify the shape of the lens and / or --Design an optical system with a defect P5 larger than the desired defect in the equipment or equipment chain, or with it from the catalog Choosing the same and / or-using materials, components, machining operations, or manufacturing methods that are low cost for equipment or equipment chains and add defect P5.

By using the methods and systems according to the invention, the cost of the equipment or equipment chain can be reduced, that is, the digital optical system is designed and the formatted information IF related to the defect P5 of the equipment or equipment chain is output. Using this formatted information, image processing means, whether embedded or not, modify the quality of images that are pulled out of the device or device chain, or sent to the device or device chain, and device or device chain. And image processing means can be combined to enable the capture, modification, or restoration of desired quality images at low cost.

<figref num="1">It is a figure of the simplified Example of this invention.</figref><figref num="2a">FIG. 5 is a diagram of an improved embodiment of a method for calculating a transformed image according to the present invention.</figref><figref num="2b">FIG. 5 is a diagram of an improved embodiment of a method for calculating a transformed image according to the present invention.</figref><figref num="2c">FIG. 5 is a diagram of an improved embodiment of a method for calculating a transformed image according to the present invention.</figref><figref num="2d">FIG. 5 is a diagram of an improved embodiment of a method for calculating a transformed image according to the present invention.</figref><figref num="3">It is an organization diagram which shows how the method of FIG. 2a functions.</figref><figref num="4">It is a figure which shows the stage of calculation of the value of the converted pixel.</figref><figref num="5a">FIG. 5 is an organizational chart of another embodiment of the method for calculating a transformed image according to the present invention.</figref><figref num="5b">It is explanatory drawing of FIG. 5a.</figref><figref num="5c">It is explanatory drawing of FIG. 5a.</figref><figref num="6">It is a figure which shows the stage of calculation of the value of the converted pixel which can use a plurality of kinds of corrections.</figref><figref num="7">It is a figure of this invention applied to a color image.</figref><figref num="8">It is a figure of the improvement of this invention which can perform the calculation of adjacent pixels.</figref><figref num="9a">It is a figure explaining the application of this invention to a GRGB type color image.</figref><figref num="9b">It is a figure explaining the application of this invention to a GRGB type color image.</figref><figref num="10">It is a figure of the formatted information IF related to the geometric distortion defect of the device P25 of the device chain P3.</figref><figref num="11">It is a figure which shows the Example of the system by this invention.</figref>