Method for detection and quantification of image modifications in a video image processed by a block-based data-reducing video-code

0001The invention relates to a method according to the preamble of the main claim.

0002In the transmission and storage of digital video images, data-reducing encoding methods are increasingly used. In particular, coding methods which perform a block-based discrete-cosine transform (DCT) as a basic coding step have found widespread use. Among them are the coding methods JPEG, MPEG-1 and MPEG-2. They exploit both the redundancy contained in an image and the characteristics of human visual perception in order to reduce the amount of data required for display (see, for example, ISO / IEC 11172 (1993) "Information technology: Coding of moving pictures- and associated audio for digital storage media at about 1.5 Mbps and ISO / IEC 13818-2 "Information technology: Generic coding of moving pictures and associated audio information: video").

0003In addition to the redundancy reduction and irrelevance reduction, such a coding and decoding process of a video image can result in visible changes compared to the original image, depending on the original, coder method and set encoding parameters in the decoded image.

0004From EP 0 723 375 A2 a post-processing method for reducing an image error is known, which arises due to the reconstruction of an image information from an image packed by means of block-based transformation. To minimize the artifacts at the block boundaries, a predetermined discrete cosine transformation (DCT) is performed, taking into account the information lost by quantization or inverse quantization in order to obtain the greatest possible continuity at the block boundaries in order to estimate the transformation coefficients. The inverse transformation takes place on the basis of the estimated transformation coefficients and supplements the image signal reconstructed by means of an inverse transformation process with the correction value thus obtained.

0005An influence on the resulting image signal, for example, have the following parameters:<ul id="ul0001" list-style="none"><li>the block-based transformation into the frequency domain (DCT) with subsequent quantization of the DCT coefficients</li><li>macroblock-based motion compensation (MPEG only) and</li><li>time differential encoding (MPEG only)</li></ul>

0006In the block-based transformation by means of the DCT, the representation of the image content after the transformation by coefficients takes place as a measure of the occurrence of a DCT basic pattern in the image content of the block. In natural images / image sequences occur after the transformation high spatial frequencies statistically average with a lower amplitude than low spatial frequencies. The quantization factor in the subsequent quantization is of the desired data rate or Image quality dependent. To reduce the amount of data resulting after encoding, the quantization factor is increased many times. This results in a coarser quantization. The desired effect of this technique is to reduce the number of bits needed to represent a coefficient. In addition, it is advantageous if, after the quantization, the coefficients with a low amplitude are reduced to zero and can thus be coded particularly efficiently in the subsequent variable-length coding. In the local area, this results in a damping of the high frequencies within the coded block. This effect is reinforced by the introduction of a quantization matrix whose coefficients can influence the quantization separately for each spatial frequency. In order to adapt to the characteristics of human visual perception, these relative quantization factors are usually chosen to increase at higher spatial frequencies, which enhances the effect of attenuation of high spatial frequencies within the coded blocks.

0007Statistically, the coding of natural images / image sequences with high probability results in a reduction or even elimination of fine structures and details in the local area of ​​the respective coded block. The degree of reduction depends on the image content, the chosen absolute quantization of the block, and the relative quantization of the different spatial frequencies set by the quantization matrix. In individual blocks, however, the opposite effect may also occur. For example, a block may contain an edge which, after the transformation into the frequency domain, will be described by a large coefficient number. After quantization, the number of coefficients is significantly reduced, which may result in overlapping of the edge with a fine, disturbing pattern in the local area. However, this effect is statistically much less likely than the one described first.

0008In the "intra-coding" described so far, each block consisting of 8 * 8 pixels in JPEG and MPEG is separately transformed and quantized. At the boundaries of neighboring blocks, coarse quantization can cause "discontinuities" in the image content. It can then be seen from the context of the image content that either the contents of the blocks are distinctly different, even though they originally had similar content, or that a relatively continuous content is split into distinctly distinct areas. In the latter, again statistically more probable case, the difference between the values ​​of two adjacent pixels across the block boundary is statistically greater than the difference of the values ​​of two adjacent pixels within a block.

0009The properties described so far apply to the coding of the luminance and chrominance components. When displaying the chrominance in the color format "4: 2: 2" or "4: 2: 0", the chrominance blocks are larger in size than the luminance blocks. The basic properties of the encoding remain unaffected.

0010The coding methods according to MPEG-1 and MPEG-2 offer in addition to the intra-coding also the possibility of exploiting similarities of temporally adjacent images by differential encoding.

0011For motion compensation in unidirectional and bidirectionally predicted pictures (P- or B-pictures), suitable picture contents in reference pictures are searched by suitable methods. The image contents of 16 * 16 pixel large areas, so-called macroblocks are compared. The content of the reference picture corresponds to the content that is also available to the decoder after decoding, so it is already quantized. The chrominance is usually not taken into account in the implemented search algorithms. If the search algorithm finds a suitable macroblock, it will be used as the basis for the code to be encoded. The remaining difference between the predicted and the image content to be encoded can additionally be encoded. As a result of exploiting the temporal correlation by motion compensation and differential encoding, the following effects can occur:

0012The block edges from the reference picture can occur at any position in the block to be coded, ie they do not necessarily occur in the block grid.

0013An additional coding of the differences can cause coarse quantization new block edges in the block grid. This is the effect that also occurs in I-pictures.

0014The image content at the macroblock edges is not considered separately from the remaining block content in the motion estimation. With only slightly matching image contents between reference and current content and no further coding of the differences (ie if the data rate is very short), clearly visible macroblock boundaries can occur.

0015By a prediction in half-pixel accuracy and the filtering required for it can lead to an attenuation of the high frequency components of the block.

0016Bidirectional prediction requires interpolation of the two reference blocks. High frequency components are again damped.

0017Since the chrominance components are mostly disregarded in the motion estimation, it is very likely that the contents of adjacent macroblocks do not "match" at the macroblock edges in terms of chrominance.

0018It follows from the considerations that a statistically very probable consequence of encoding at a relatively low data rate is a reduction of the energy content within the coding blocks. By contrast, at the block and macroblock boundaries, a statistically lower reduction than within the blocks or even an increase in the difference between the values ​​of adjacent pixels is to be expected.

0019It is an object of the invention to provide a simple method with which such image changes of a video image that has been processed with in a block-based, data-reducing video encoder can be detected and quantified without the original video image being known.

0020This object is achieved on the basis of a method according to the preamble of the main claim by its characterizing features. Advantageous developments emerge from the subclaims.

0021The method according to the invention makes use of the property of the known block coding methods for data reduction that the pixels of a block are processed jointly and independently of the pixels of the adjacent blocks. Thus, it is only necessary in each case the difference of the digital values ​​of the horizontally adjacent pixels or possibly additionally also still to determine the vertically adjacent pixels, and preferably each separately for the luminance and chrominance components. Relative to specific pixels, a value strongly dependent on the current image content results. However, if one summarizes the values ​​for all points of an image at the spacing of the block raster, then one obtains eight values ​​per component and direction, ie a statistical averaging over an entire image. The values ​​which describe the transitions at the block boundaries stand out from the other values ​​when the prerequisites are met. The difference between the values ​​within the blocks and the value at the block boundary provides an indication of the image characteristics and is a measure of the change by the coding.

0022The video image to be examined can either still be coded or already decoded, it is only necessary that the block pattern of the coding is known. Video image is understood to mean either an entire image or possibly only a part of an entire image.

0023For the inventive method no knowledge about the original video image is required, it is sufficient only the evaluation of the (former) block-coded video signal. By calculating several independent parameters, a statistically high independence compared to the original image is achieved. With the method according to the invention not only a detection of significant image properties is possible, but also a quantification of the coding result.

0024The value resulting from the comparison, which is a measure of the image change, can be displayed to the user in any manner, for example via a simple scale, as is customary in the meantime in the video sector for subjective test results for displaying image changes. The values ​​determined separately for luminance and chrominance can be displayed and displayed separately or together.

0025The method according to the invention is based on the following principle described in connection with FIGS. 1 and 2:

0026The coordinates of the image to be examined are shown in FIG. The origin is in the upper left corner. The luminance component of the image is horizontal<i>b</i>_<i>lum</i> Pixels with the coordinates <i>0 .. b</i>_<i>lum</i><i> -1</i> and in the vertical <i>H</i>_<i>lum</i> Pixels with the coordinates <i>0 .. H</i>_<i>lum</i><i> -1.</i>

0027The differences of adjacent pixels are combined and averaged depending on the position of the pixels to the block boundaries. For the luminance, 8 values ​​for the horizontal direction and 8 values ​​for the vertical direction, which describe a picture, are created. The corresponding parameter has the name<i>AD</i> For <i>Average difference.</i> The index <i>i</i> indicates the situation to the block boundary. <i>i</i>=<i>0</i> indicates the position at the block boundary. All other values ​​are differences within the blocks. For the calculation of the average of the difference in a grid position, the amounts of the individual differences are summed and then the resulting value is divided by the number of differences. Since there is no neighboring point with the position "-1" at the position "0", the respective first block row is left out in the direction of the evaluation. The following equations describe the calculations. They refer to a block size of 8 * 8 pixels.<i>H</i>_<i>lum</i> and <i>b</i>_<i>lum</i> are each integer multiples of the block dimension.

0028Luminance in the horizontal direction:<maths id="math0001" num=""><img file="EP0875857B1_D0001.tif" /></maths>

0029Luminance in the vertical direction:<maths id="math0002" num=""><img file="EP0875857B1_D0002.tif" /></maths> With:<ul id="ul0002" list-style="none" compact="compact"><li><i>lum</i>(<i>k</i>,<i>l</i>): Luminance value of the pixel with the coordinates <i>x = k, y</i>=<i>l</i>.</li></ul>

0030The reference value is the average of the 7 values ​​for the positions within the blocks:<maths id="math0003" num=""><img file="EP0875857B1_D0003.tif" /></maths><maths id="math0004" num=""><img file="EP0875857B1_D0004.tif" /></maths>

0031The difference between the parameters at the block boundary to this value eliminates the influence of the image content and is considered a measure of the change in the image content by the coding:<maths id="math0005" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, h</mtext></mrow></msub><mtext> = </mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, h</mtext></mrow></msub><mtext> (0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, h7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0005.tif" /></maths><maths id="math0006" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, v</mtext></mrow></msub><mtext> = </mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, v</mtext></mrow></msub><mtext>(0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum, v7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0006.tif" /></maths>

0032For the chrominance analog calculations result. In the table according to FIG. 2, the dimensions of the chrominance components are compared with different color formats. Any undersampling in the format "4: 2: 2" or "4: 2: 0" must be taken into account only in the dimension of the viewed image.

0033The block boundary is each with <i>i</i>=<i>0</i> described.

0034The following equations are used:<maths id="math0007" num=""><img file="EP0875857B1_D0007.tif" /></maths><maths id="math0008" num=""><img file="EP0875857B1_D0008.tif" /></maths> With:<ul id="ul0003" list-style="none" compact="compact"><li><i>cb</i>(<i>k</i>,<i>l</i>): Crominance value of the component cb of the pixel with the coordinates <i>x = k, y = l</i> in the matrix of chrominance.</li></ul><maths id="math0009" num=""><img file="EP0875857B1_D0009.tif" /></maths><maths id="math0010" num=""><img file="EP0875857B1_D0010.tif" /></maths> With:<ul id="ul0004" list-style="none" compact="compact"><li><i>cr</i>(<i>k</i>,<i>l</i>): Crominance value of the component cr of the pixel with the coordinates <i>x = k, y = l</i> in the matrix of chrominance.</li></ul>

0035The reference values ​​are calculated analogously to those of the luminance:<maths id="math0011" num=""><img file="EP0875857B1_D0011.tif" /></maths><maths id="math0012" num=""><img file="EP0875857B1_D0012.tif" /></maths><maths id="math0013" num=""><img file="EP0875857B1_D0013.tif" /></maths><maths id="math0014" num=""><img file="EP0875857B1_D0014.tif" /></maths>

0036In turn, the difference between the respective parameter at the block boundary and the corresponding reference value applies as a measure of the change in the image content through the coding:<maths id="math0015" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb h</mtext></mrow></msub><mtext> = </mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb h</mtext></mrow></msub><mtext>(0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb, h7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0015.tif" /></maths><maths id="math0016" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext></mrow></msub><msub><mrow><mtext>​</mtext></mrow><mrow><mtext>,</mtext></mrow></msub><msub><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">v</mtext></mrow></msub><mtext> = </mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext></mrow></msub><msub><mrow><mtext>​</mtext></mrow><mrow><mtext>,</mtext></mrow></msub><msub><mrow><mtext>​</mtext></mrow><mrow><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb, v7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0016.tif" /></maths><maths id="math0017" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, h</mtext></mrow></msub><msub><mrow><mtext mathvariant="italic"> = AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, h</mtext></mrow></msub><mtext>(0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, h7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0017.tif" /></maths><maths id="math0018" num=""><math display="block"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, v</mtext></mrow></msub><msub><mrow><mtext mathvariant="italic"> = AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, v</mtext></mrow></msub><mtext>(0) - </mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr, v7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></math><img file="EP0875857B1_D0018.tif" /></maths>

0037With the method can be calculated per color image 6 parameters that provide information about the image changes by the coding.

0038For a display of the calculation results, the parameters can be combined into one numerical value per image. Since each value indicates a change for itself, a maximum value determination makes sense. In terms of time, the low-pass filtering and the peak-value rectification are useful in order to give the parameter a smaller temporal variance.