Method for detection and quantification of picture modification in a video picture processed by block based data reduction of video-code

0001The invention relates to a method according to the preamble of the main claim.

0002Data-reducing coding methods are increasingly being used in the transmission and storage of digital video images. In particular, coding methods which perform a block-based discrete cosine transformation (DCT) as the basic coding step have found widespread use. They include the JPEG, MPEG-1 and MPEG-2 coding methods. They use both the redundancy contained in an image and the properties of human, visual perception to reduce the amount of data required for the display (see for example (according to ISO / IEC 11172 (1993) <img file="EP0875857A2_D0001.tif" />Information technology: Coding of moving pictures- and associated audio for digital storage media at up to about 1.5 Mbit / S "and ISO / IEC 13818-2 <img file="EP0875857A2_D0002.tif" />Information technology: Generic coding of moving pictures and associated audio information: video ").

0003In addition to the reduction in redundancy and irrelevance, such a coding and decoding process of a video picture can lead to visible changes in the decoded picture, depending on the template, coding method and set coding parameters in the decoded picture. The following parameters, for example, have an influence on the resulting image signal:<ul id="ul0001" list-style="none" compact="compact"><li>the block-based transformation into the frequency domain (DCT) with subsequent quantization of the DCT coefficients</li><li>macroblock based motion compensation (MPEG only) and</li><li>the temporally differential coding (only MPEG)</li></ul>

0004In the case of the block-based transformation using the DCT, the image content is displayed after the transformation using coefficients as a measure of the occurrence of a DCT basic pattern in the image content of the block. In natural images / image sequences, high spatial frequencies occur statistically on average with a lower amplitude than low spatial frequencies after the transformation. The quantization factor in the subsequent quantization is based on the desired data rate or Image quality dependent. In order to reduce the amount of data resulting after coding, the quantization factor is increased many times. This results in a coarser quantization. The desired effect of this technique is to reduce the number of bits required to represent a coefficient. It also has an advantageous effect if, after quantization, the coefficients with a small amplitude are reduced to zero and can thus be coded particularly efficiently in the subsequent variable-length coding. In the local area, this results in an attenuation of the high frequencies within the coded block. This effect is reinforced by the introduction of a quantization matrix, the coefficients of which can influence the quantization separately for each spatial frequency. In order to adapt to the properties of human visual perception, these relative quantization factors are usually chosen to increase towards higher spatial frequencies, which further enhances the effect of attenuating high spatial frequencies within the coded blocks.

0005Statistically, this technique results in a high probability of a reduction or even elimination of fine structures and details in the area of ​​the respective coded block when coding natural pictures / picture sequences. The degree of reduction depends on the image content, the chosen absolute quantization of the block and the relative quantization of the different spatial frequencies, which is determined by the quantization matrix. However, the opposite effect can also occur in individual blocks. For example, a block can contain an edge that is described by a large number of coefficients after the transformation into the frequency domain. After quantization, the number of coefficients is significantly reduced, which in the local area can result in an overlay of the edge with a fine, disturbing pattern. However, this effect is statistically much less likely than that described first.

0006With the previously described <img file="EP0875857A2_D0003.tif" />Intra-coding ", each block, which in JPEG and MPEG consists of 8 * 8 pixels each, is separately transformed and quantized. With rough quantization, adjacent blocks can be used <img file="EP0875857A2_D0004.tif" />Discontinuities "arise in the image content. From the context of the image content it can then be seen that either the contents of the blocks are clearly different, although they originally had a similar content, or that a relatively continuous content is separated into clearly separated areas. In the latter case, which is again statistically more likely, the difference between the values ​​of two neighboring pixels across the block boundary is statistically greater than the difference between the values ​​of two neighboring pixels within a block.

0007The properties described so far apply to the coding of the luminance and the chrominance components. When displaying the chrominance in color format<img file="EP0875857A2_D0005.tif" />4: 2: 2 "or <img file="EP0875857A2_D0006.tif" />4: 2: 0 ", the chrominance blocks have a larger size than the luminance blocks in comparison. The basic properties of the coding remain unaffected.

0008In addition to intra-coding, the coding methods according to MPEG-1 and MPEG-2 also offer the possibility of utilizing similarities of temporally adjacent pictures by differential coding.

0009For the motion compensation in unidirectionally and bidirectionally predicted images (P or B images), similar image contents are sought in reference images using suitable methods. The image contents of 16 * 16 pixel areas, so-called macro blocks, are compared. The content of the reference image corresponds to the content that is also available to the decoder after decoding, so it is already quantized. Chrominance is usually not taken into account in the implemented search algorithms. If the search algorithm finds a suitable macro block, this is used as the basis for the code to be coded. The remaining difference between the predicted and the image content to be encoded can also be encoded. The following effects can occur as a result of the utilization of the temporal correlation by motion compensation and differential coding:

0010The block edges from the reference picture can occur at any position in the block to be coded, ie they do not necessarily occur in the block grid.

0011Additional coding of the differences can result in new block edges in the block grid in the case of rough quantization. This is the effect that also occurs in I-pictures.

0012The image content at the macro block edges is not considered separately from the rest of the block content in the motion estimation. If the image content between the reference and current content is only moderately consistent and there is no further coding of the differences (i.e. if the data rate is very scarce), clearly visible macroblock boundaries can occur.

0013Prediction in half-pixel accuracy and the filtering required for this can result in an attenuation of the high frequency components of the block.

0014With bidirectional prediction, an interpolation of the two reference blocks is required. High frequency components are damped again.

0015Since the chrominance components are usually not taken into account in the motion estimation, it is very likely that the contents of neighboring macroblocks with regard to chrominance will not be at the macroblock edges <img file="EP0875857A2_D0007.tif" />match".

0016It follows from the considerations that a statistically very probable consequence of coding with a relatively low data rate is a reduction in the energy content within the coding blocks. At the block and macroblock boundaries, on the other hand, a statistically smaller reduction than within the blocks or even an increase in the difference between the values ​​of neighboring pixels can be expected.

0017It is an object of the invention to provide a simple method by means of which such changes in the image of a video image which has been processed in a block-based, data-reducing video encoder can be detected and quantified without the original video image being known.

0018This object is achieved on the basis of a method according to the preamble of the main claim by its characterizing features. Advantageous further developments result from the subclaims.

0019The method according to the invention takes advantage of the property of the known block coding methods for data reduction that the picture elements of a block are processed together and independently of the picture elements of the neighboring blocks. It is now only necessary to determine the difference between the digital values ​​of the horizontally adjacent pixels or possibly also to determine the vertically adjacent pixels, preferably separately for the luminance and chrominance components. In relation to specific pixels, this results in a value that is strongly dependent on the current image content. However, if you summarize the values ​​for all points of an image at a distance from the block grid, you get eight values ​​per component and direction, i.e. a statistical averaging over an entire image. The values ​​that describe the transitions at the block boundaries differ from the other values ​​when the requirements are met. The difference between the values ​​within the blocks and the value at the block boundary provides information about the image properties and is a measure of the change due to the coding.

0020The video image to be examined can either still be coded or already decoded, it is only necessary that the block grid of the coding is known. A video image is understood to mean either an entire image or possibly only a part of an entire image.

0021No knowledge of the original video image is required for the method according to the invention, only the evaluation of the (formerly) block-coded video signal is sufficient. By calculating several independent parameters, a statistically great independence from the original image is achieved. With the method according to the invention not only a detection of significant picture properties is possible, but also a quantification of the coding result.

0022The value resulting from the comparison, which is a measure of the image change, can be displayed to the user in any manner, for example via a simple measurement scale, which is meanwhile common in the video area for subjective test results for displaying image changes. The values ​​determined separately for luminance and chrominance can be displayed and displayed separately or in summary.

0023The method according to the invention is based on the following principle illustrated in connection with FIGS. 1 and 2:

0024The coordinates of the image to be examined are shown in Fig. 1. The origin is in the upper left corner. The luminance component of the image is horizontal<i>b</i>_<i>lum</i> Pixels with the coordinates <i>0 .. b</i>_<i>lum</i><i> -1</i> and in the vertical <i>H</i>_<i>lum</i> Pixels with the coordinates <i>0 .. H</i>_<i>lum</i><i> -1</i>.

0025The differences between adjacent pixels are combined and averaged depending on the position of the pixels to the block boundaries. For the luminance there are 8 values ​​for the horizontal and 8 values ​​for the vertical direction that describe an image. The corresponding parameter has the name<i>AD</i> For <i>Average difference</i>, The index<i>i</i> indicates the position to the block boundary. <i>i</i>=<i>0</i> denotes the position at the block boundary. All other values ​​are differences within the blocks. To calculate the average of the difference in a grid position, the amounts of the individual differences are summed and then the resulting value is divided by the number of differences. Because at the position<img file="EP0875857A2_D0008.tif" />0 "no neighboring point with the position <img file="EP0875857A2_D0009.tif" />-1 "is present, the first row of blocks is left out in the direction of the evaluation. The following equations describe the calculations. They relate to a block size of 8 * 8 pixels. <i>H</i>_<i>lum</i> and <i>b</i>_<i>lum</i> are integer multiples of the block dimension.

0026Luminance in the horizontal direction:<maths id="math0001" num=""><img file="EP0875857A2_D0010.tif" /></maths>

0027Luminance in the vertical direction:<maths id="math0002" num=""><img file="EP0875857A2_D0011.tif" /></maths> With:<ul id="ul0002" list-style="none" compact="compact"><li><i>lum</i>(<i>k</i>,<i>l</i>): Luminance value of the pixel with the coordinates <maths id="math0003" num=""><math display="inline"><mrow><mtext mathvariant="italic">x</mtext><mtext>=</mtext><mtext mathvariant="italic">k</mtext></mrow></math><img file="EP0875857A2_D0012.tif" /></maths>, <maths id="math0004" num=""><math display="inline"><mrow><mtext mathvariant="italic">y</mtext><mtext>=</mtext><mtext mathvariant="italic">l</mtext></mrow></math><img file="EP0875857A2_D0013.tif" /></maths>.</li></ul>

0028The average of the 7 values ​​for the positions within the blocks is averaged as a reference value:<maths id="math0005" num=""><math display="block"><mrow><mtable><mtr><mtd><mrow><mtable><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr></mtable></mrow></mtd></mtr></mtable></mrow></math><img file="EP0875857A2_D0014.tif" /></maths>

0029The difference between the parameters at the block boundary to this value eliminates the influence of the picture content and is a measure of the change in the picture content due to the coding:<maths id="math0006" num=""><math display="block"><mrow><mtable><mtr><mtd><mrow><mtable><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">lum</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr></mtable></mrow></mtd></mtr></mtable></mrow></math><img file="EP0875857A2_D0015.tif" /></maths>

0030Analog calculations are made for chrominance. The table according to FIG. 2 shows the dimensions of the chrominance components for different color formats. Any subsampling in format<img file="EP0875857A2_D0016.tif" />4: 2: 2 "or <img file="EP0875857A2_D0017.tif" />4: 2: 0 "only has to be taken into account in the dimension of the image under consideration.

0031The block boundary is shown with <i>i</i>=<i>0</i> described. The following equations are used:<maths id="math0007" num=""><img file="EP0875857A2_D0018.tif" /></maths> With:<ul id="ul0003" list-style="none" compact="compact"><li><i>cb</i>(<i>k</i>,<i>l</i>): Crominance value of the component cb of the pixel with the coordinates <maths id="math0008" num=""><math display="inline"><mrow><mtext mathvariant="italic">x</mtext><mtext>=</mtext><mtext mathvariant="italic">k</mtext></mrow></math><img file="EP0875857A2_D0019.tif" /></maths>, <maths id="math0009" num=""><math display="inline"><mrow><mtext mathvariant="italic">y</mtext><mtext>=</mtext><mtext mathvariant="italic">l</mtext></mrow></math><img file="EP0875857A2_D0020.tif" /></maths> in the matrix of chrominance.</li></ul><maths id="math0010" num=""><img file="EP0875857A2_D0021.tif" /></maths> With:<ul id="ul0004" list-style="none" compact="compact"><li><i>cr</i>(<i>k</i>,<i>l</i>): Crominance value of the component cr of the pixel with the coordinates <maths id="math0011" num=""><math display="inline"><mrow><mtext mathvariant="italic">x</mtext><mtext>=</mtext><mtext mathvariant="italic">k</mtext></mrow></math><img file="EP0875857A2_D0022.tif" /></maths>, <maths id="math0012" num=""><math display="inline"><mrow><mtext mathvariant="italic">y</mtext><mtext>=</mtext><mtext mathvariant="italic">l</mtext></mrow></math><img file="EP0875857A2_D0023.tif" /></maths> in the matrix of chrominance.</li></ul>

0032The reference values ​​are calculated analogously to those of the luminance:<maths id="math0013" num=""><math display="block"><mrow><mtable><mtr><mtd><mrow><mtable><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr><mtr><mtd><mrow><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover><mtext>=</mtext><mfrac><mrow><mtext>1</mtext></mrow><mrow><mtext>7</mtext></mrow></mfrac><mtext>·</mtext><apply><sum /><lowlimit><mtext mathvariant="italic">i</mtext><mtext>=1</mtext></lowlimit><uplimit><mtext>7</mtext></uplimit><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(</mtext><mtext mathvariant="italic">i</mtext><mtext>)</mtext></mrow></apply></mrow></mtd></mtr></mtable></mrow></mtd></mtr></mtable></mrow></math><img file="EP0875857A2_D0024.tif" /></maths>

0033The difference between the respective parameter at the block boundary and the corresponding reference value is again a measure of the change in the image content due to the coding:<maths id="math0014" num=""><math display="block"><mrow><mtable><mtr><mtd><mrow><mtable><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cb</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">H</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr><mtr><mtd><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>=</mtext><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext></mrow></msub><mtext>(0)-</mtext><mover accent="true"><mrow><msub><mrow><mtext mathvariant="italic">AD</mtext></mrow><mrow><mtext mathvariant="italic">cr</mtext><mtext>,</mtext><mtext mathvariant="italic">v</mtext><mtext>7</mtext></mrow></msub></mrow><mo>¯</mo></mover></mrow></mtd></mtr></mtable></mrow></mtd></mtr></mtable></mrow></math><img file="EP0875857A2_D0025.tif" /></maths>

0034The method can be used to calculate 6 parameters per color image, which provide information about the image changes caused by the coding.

0035To display the calculation results, the parameters can be combined into a numerical value per picture. Since each value indicates a change in itself, a maximum value determination makes sense. In terms of time, the low-pass filtering and the peak value rectification are appropriate in order to give the parameter a smaller temporal variance.