Method for processing and transmitting a picture sequence.

p0001The invention relates to a method for transmitter-side processing of an image sequence for transmission over a channel with limited data capacity and the receive-side reconstruction.

p0002For image information to be transmitted via a transmission channel with limited data capacity, there are numerous methods for redundancy and Irrelevanzverminderung. In Esprit '86 Results and Achievements, Commission of the European Communities, Directorate General XIII, 1987, North-Holland, pages 413-422, are suggested some such methods. When DPCM (Differential Pulse Code Modulation) method is transmitted, the difference between the current value and a prediction value instead of current sampling. The data compression is achieved by quantization. With adaptive DPCM prediction and the quantization is controlled based on a motion criterion. When ABTC (Adaptive Block Truncation Coding) process is controlled coding depending on the movement amount of image areas in 3 modes. The HPC (Hierarchical Predictive Coding) method based on the combination of predictive coding and interpolation. In transform coding, for example, the ADCT (Adaptive Cosine Transform) coding the local samples of an image block using a mathematical agreement are transformed to obtain decorrelated coefficients. The data compression is achieved by cutting off from the coefficients and non-linear quantization. In vector image into a large number of small cells is divided, for example, 2x2 or 4x4 pixels. Each cell of K pixels is then regarded as a vector in K-dimensional space. A number of representative vectors is selected for transmission. For the coding of the current image vectors are described by the representative vectors in its neighborhood.

p0003From DE-PS 37 04 777 it is known to determine the motion vectors for partial image regions and to encode this. In highly agitated areas pixel-dependent information for this partial image areas are transferred.

p0004Of SPIE (International Society for Optical Engineering), pages 119-128, it is known to determine for a whole moving object displacement vectors and transmitted. Prediction errors are coded in blocks, for example by means of a DCT (discrete cosine transform) transformation and transfer. Exuberant intermediate images are interpolated motion adaptive receiving end.

p0005Object of the invention is, starting from the preamble of claim 1 to provide another method, which in particular allows a good reception-side image reconstruction. This object is achieved by the features of claim 1. The sub-claims reveal advantageous developments of this method.

p0006The invention has the following advantages: There is achieved a high image quality, which is in particular made possible by the determination of a motion compensated estimate of the image and the activity-controlled coding of partial image regions.

p0007A motion-compensated receiver-side image Extra - (- inter) -polation a jerky image representation is avoided. The transmission channel is optimally utilized. In contrast to the implementation according to SPIE, pages 119 to 128, where an entire object is described by a displacement vector and thus in the peripheral areas of the object, a large range of uncertainty is created which can be kept small only by transfer of prediction errors in the entire range of uncertainty, must in the process be transferred according to the invention only a very few field fields / blocks original picture content or prediction. By choosing to be transmitted transformed partial image regions / blocks on the basis of differences (errors) to previous field areas / blocks can already reconstruct a relative of a movement secure image with little transmission data. The method of the invention or a subsequently executed codec allows the following modes: - Transfer of the head and shoulders view of a person with a temporal resolution of 25 (30) frames per second, - Transfer of still images with high spatial resolution and derived from it: - Still with frozen motion picture, - Still image with superimposed moving image.

p0008With reference to the drawings, an embodiment of the invention will now be explained in more detail. Show it:<ul><li>Fig. 1 is a block diagram of a videophone via ISDN, wherein the method used according to the invention,</li><li>Fig. 2 shows the principle of the motion compensated prediction, </li><li>Fig. 3 shows the principle of the block-oriented Bildextrapolation,</li><li>Fig. 4 is a video encoder encoding the source image sequence,</li><li>Fig. 5 shows a video coder encoding of the prediction error,</li><li>Fig. 6 is a bitmask to address the displacement vectors,</li><li>Fig. 7 is a characteristic curve for the attenuation of small differences,</li><li>Fig. 8 is a sorting table for the DFD</li><li>Fig. 9 shows a video decoder,</li><li>Fig. 10 shows a data frame for transmission,</li><li>Fig. 11, the vector and DCT mask for picture template with motion,</li><li>Fig. 12 is an overview of the DCT encoding,</li><li>Fig. 13, the DCT classification.</li></ul>

p0009The videophone according to FIG. 1 consists of separate coders and decoders for video and audio transmission. Thus, for example, a transmission over two B channels in an ISDN network is possible. A transmission of image and sound over a B-channel is to appropriate source coding of sound also possible, without changing anything on the principle of image coding. The input image sequence R, G, B is for example 601 scanned by the digital TV standard CCIR (A / D conversion and clock extraction). The audio information is also converted A / D and subjected to companding. The digital scanning is performed according to CCIR 601 as follows:<tables id="tabl0001" num="0001"><table frame="all"><tgroup cols="2" colsep="1" rowsep="0"><colspec colnum="1" colname="col1" colwidth="78.75mm" /><colspec colnum="2" colname="col2" colwidth="78.75mm" /><tbody valign="top"><row><entry namest="col1" nameend="col1" align="left">Luminance (Y) -Abtastfrequenz:</entry><entry namest="col2" nameend="col2" align="left">13.5 MHz 8-bit linear</entry></row><row><entry namest="col1" nameend="col1" align="left">Chrominance (U / V) - Sampling frequency:</entry><entry namest="col2" nameend="col2" align="left">6.75MHz 8-bit linear</entry></row><row><entry namest="col1" nameend="col1" align="left">Bits / pixel (color):</entry><entry namest="col2" nameend="col2" align="left">16</entry></row><row><entry namest="col1" nameend="col1" align="left">Active Picture content:</entry><entry namest="col2" nameend="col2" align="left">720 * 576 pixels</entry></row><row><entry namest="col1" nameend="col1" align="left">Image change:</entry><entry namest="col2" nameend="col2" align="left">50 (60) fields / s</entry></row><row><entry namest="col1" nameend="col2" align="justify">864 * 625 * 25 * 16 → 216 MBit / s</entry></row></tbody></tgroup></table></tables> The high quality of this sampling allows the storage of high-resolution still images, such as that provided for the aforementioned second mode.

p0010To align this source image sequence at the following video coder is this treated as follows (preprocessing):

p0011Halving the resolution of the image in horizontal and vertical directions.

p0012Reducing the chrominance resolution in vertical direction to 1/2.

p0013Use the blanking interval for transmission.

p0014Thus, 352 * 288 active pixels; 25 Hz frame rates.<tables id="tabl0002" num="0002"><table frame="all"><tgroup cols="2" colsep="1" rowsep="0"><colspec colnum="1" colname="col1" colwidth="78.75mm" /><colspec colnum="2" colname="col2" colwidth="78.75mm" /><tbody valign="top"><row><entry namest="col1" nameend="col1" align="left">Y sampling:</entry><entry namest="col2" nameend="col2" align="left">6.75MHz 8-bit linear</entry></row><row><entry namest="col1" nameend="col1" align="left">U / V sampling:</entry><entry namest="col2" nameend="col2" align="left">3.375 MHz 8-bit linear</entry></row><row><entry namest="col1" nameend="col1" align="left">Bits / pixel:</entry><entry namest="col2" nameend="col2" align="left">12</entry></row><row><entry namest="col1" nameend="col1" align="left">Image change:</entry><entry namest="col2" nameend="col2" align="left">25 (30) Hz non-interlaced</entry></row><row><entry namest="col1" nameend="col1" align="left">Data rate:</entry><entry namest="col2" nameend="col2" align="left">30.41 (36.49) Mbit / s</entry></row><row><entry namest="col1" nameend="col1" align="left">Reduction:</entry><entry namest="col2" nameend="col2" align="left">factor 7</entry></row></tbody></tgroup></table></tables>

p0015This measure reduces the source data rate of 216 Mbit / s to 30.41 (36.49) Mbit / s. The further reduction occurs in the video coder illustrated in FIG. 4 and FIG. 5 shows. With the transmitter and receiver side ISDN interface circuits transfer to 64 kbit / s ISDN channels is possible. On the receiver side, a corresponding to the transmitter-side processing decoding. 9 shows the corresponding video decoder. A signal post (post-processing) restores an under CCIR 601 standard-compliant image format that is suitable for playback on an RGB monitor by D / A conversion and addition of synchronizing information. The circuit blocks FECC (Forward Error Correction Control) and ECC (Error Correction Control) are units needed 'error correction. Transmission errors can thus be detected and corrected as far possible.

p0016The basic idea of ​​the codec will now be explained with reference to FIGS. 2 and 3 It consists in the image construction, only every third image (Fig. 2) encode. For this purpose from the image K + 3 in the coder a motion-compensated estimated image K + 3 * from image K is formed. The vectors with which K + 3 * is formed, are transferred, so that the decoder can generate the same estimated image. There are then the pixel-to-pixel differences between the estimated DFD Picture K + 3 * and the new Picture K + 3 blocks added (DFD = Displaced Frame Difference). The DFD within a block / partial image area of ​​8 * 8 pixels results to be DFD = ΣΣ | p_{K + 3}* (X, y) - p_{K + 3}(X, y) | with block coordinates: x = 1.8 y = 1.8 and pixel intensity p.

p0017For the partial image regions / blocks with large sums, the digits indicate significant divergences, or the difference image is either the original image content (Fig. 4) (Fig. 5) encodes a perceptually adaptive DCT (Discrete Cosine Transform). In the receiver, the inverse DCT is formed and inserted into the motion-compensated estimated image. The missing intermediate images are extrapolated motion compensated with the transmitted displacement vectors D or interpolated, so that again a soft sequence of motions (Fig. 3). As shown in FIG. 3 can be seen, the image K + 1 from the image K is extrapolated (displacement vector D + 1/3_K) And the image K + 2 from the image K + 3 (displacement vector - 1/3 D_{K + 3}).

p0018To ensure a fast refresh rate at the beginning of the connection, a planar by a factor of 4 low-pass filtered first image is transmitted first. Thereafter, the image content is updated, the planar low-pass filtered by a factor. 2 Finally unfiltered image content is updated. The measure for controlling these three modes is from the middle DFD - MDFD - won. The method ensures that the image content remains consistent even with large changes. That is an arbitrary "freeze" of image content with large changes is avoided. To determine this average DFD the pixel-to-pixel differences of all blocks each of 8 * 8 pixels are used. There is thus obtained: MDFD = ΣΣ DFD (i, j) i = 1.44 j = 1,36

p0019The function of the video coder is the example of Luminanzbildverarbeitung (Fig. 4 and Fig. 5) described, since it takes over from the Luminanzzweig in Chrominanzbildverarbeitung results. The video coder of FIG. 4, the source image sequence DCT is transformed, whereas a difference image is transformed DCT the video coder in FIG. 5. Since the construction of these two video coder versions is very similar, only one embodiment is described in detail and described for the other only the deviating realization.

p0020Each input of the video encoder supplied as Y-PCM signal new image K + 3 is processed in several ways:<ul><li>a) The image K + 3 is planar low-pass filtered in step TP1 by a factor of 2 and bisects the sampling frequency. At the output of an image (K + 3) appear '. This, reduced in the spatial resolution image content is used, if the average DFD a threshold TH2 exceeds (Spatiales Subsampling at major changes). The reduction is canceled again in the decoder and for the prediction by interpolation (step TP3).</li><li>b) The image K + 3 'is in the stage TP2 again by a factor of 2 planar lowpass filtered and halve the sampling frequency. the image (K + 3) ˝ appears at the output of stage TP2. This image is used in the case of the connection setup or a section and completely transferred using the DCT. The criterion of the most central DFD is also (as in a), but used with a threshold TH4> TH2. In order to obtain a fast refresh. The reduction is in the decoder and for the prediction by interpolation canceled (steps TP3 and TP4).</li><li>c) Using the data stored in an image memory FST2 image K 'and the image K + 3' is detected in the motion vector estimator ME a set of displacement vectors D. A vector applies to a range of 16 * 16 pixels.</li><li>d) The image K + 3 is compared with the data stored in the image memory FST1 image K. The image K is created by an inverse DCT transform DCT⁻¹ and possibly lifting the low-pass filtering (steps TP3, TP4). Using the frame difference level FD thereby blocks changed image areas are distinguished by unaltered. The formation of the frame difference is according to the relationship: FD = ΣΣ | p_K(X, y) - p_{K + 3}(X, y) | x = 1.8 y = 1.8</li></ul>

p0021For the changed image areas displacement vectors D motion compensation is transmitted to the decoder. In coder and decoder (Motion Compensated Frame Interpolator) a motion-compensated estimated image K + 3 * from image K is thus formed extrapolator for motion compensation MCFE. Addressing the displacement vectors is done using a bit mask (Fig. 6) Economics (variable word length) is encoded. The displacement vectors are statistically coded as differences from each previous value, wherein each image, only the first displacement vector is transmitted as an absolute value. In Noise limiter NL the difference between the new image K + 3 and the estimated image K + 3 * pixel by pixel is formed. This is subjected to non-linear characteristic (Fig. 7), the small differences and attenuates large leaves unchanged. Essential for the fact that small differences do not stand still in the picture, but can be broken down, is that the small differences only attenuated, but are not set to zero (Fig. 7). In this way, the noise floor of the new source image to a subdued and reduced to another small noise-like interference by an error in the motion-compensated prediction and DCT coding. This leads to a reduction of the bit need for the DCT encoding. The DFD is ranked formed in blocks of 8 * 8 pixels (level DFD) and with decreasing size in accordance with a table (Fig. 8). In addition, in this table for each of the DCT coding of the block (Block No.) is registered required number of bits. In addition, the chrominance is signaling (chromium. Flag) and the DCT-bit number DCT'COSTS registered '.

p0022All this information is processed by the control CONTROL to a transmission signal. This information is obtained from the DCT-coding the DFD image (prediction Fig. 5) or the source image (FIG. 4). The free channel bit rate is used for the transmission of DCT blocks. The blocks of the biggest DFD be transferred until the channel is filled. The bit map for addressing of the DCT blocks is VWL coded (FIG. 6).

p0023For the prediction in the coder, the inverse DCT transform DCT⁻¹ is formed. Furthermore, here are interpolation, the reduced image content maximize (TP3 and TP4). In image memory FST1 the reconstructed image K is stored. The data can be added as a transmission error protection additional information of an FEC codes (FEC Forward error correction). Control (Control) monitors all functions of the codec and turns it in accordance with the conditions in the corresponding mode. The ISDN interface connects the codec with the network. The selection of the transmitted image information is controlled by multiplexer MUX that received again from the controller (Control) its switching instructions.

p0024In deviation from the implementation of FIG. 4 5 is in Fig. Not done the image K + 3 via the multiplexer MUX to DCT transformation stage, but the difference between the image K + 3 and the estimated image K + 3 * (Level Σ 1) , This only the prediction error DCT are transformed. The 'Noise Limier' NL is connected here to the input of the multiplexer MUX. Get back to have for the prediction, a source image is available, the difference has to be reversed after the inverse DCT DCT⁻¹ (stage Σ 2).

p0025In the video decoder (Fig. 9) - only the luminance portion is shown - are obtained by the ISDN interface, the control information from the data stream (SYNC CONTROL '). The optional error correction EC recognizes transmission errors and corrects them as far as possible. Demultiplexer DEMUX separates the displacement vectors, DCT coefficients and the chrominance component. The displacement vectors are stored, the bit masks are decoded. Using the decoded vectors D 'and stored in the image memory FST image K a motion-compensated estimated image K + 3 * of the current image is formed using the extrapolator for motion compensation MCFE1. The transmitted image blocks from the image K + 3 are subjected to an inverse DCT transform DCT⁻¹ and * used in the estimation image K + 3 means the summing Σ. 3 Another extrapolator for motion compensation MCFE2 extrapolates the dropped frames K + 1 and K + 2 motion-compensated according to FIG. 3.

p0026According to FIG. 3, the image K is + 1 of image K extrapolated by converting the vectors, which have led to Picture K. Picture K + 2 is from Picture K + 3, extrapolated by converting the vectors, which have led to Picture K +. 3 In this way the original frame rate of 25 (30) frames / s is restored. The image signal can be freed with a recursive temporal filter of small movement defects (not shown).

p0027The value of a pixel p (x, y) is calculated according to the formula: p (x, y) = p-1 (x, y) + 1/2 (p (x, y) - p-1 (x, y)).

p0028Rapid changes are delayed to multiple images; there arises a transition effect that, for example, in the eyes and mouth movements, which are not described by vectors, beneficial effect.

p0029Figure 10 shows a data frame for transmission. Data is transferred serially in seven different time slots: synchronous word (Sync.), Vector mask (Vector Mask.) Control information, vector information, luminance and chrominance DCT (Lum DCT (Contr Inform..) (Vector Inform..). , chrome. DCT) and update information (update Mask).

p0030In Figure 11, the vector mask is (outer contours hatched) and the DCT mask (flat shaded) for a moving image template shown. As seen in Figure 11, only strongly moving areas of the face and shoulder area by DCT have to be transferred.

p0031For the motion estimator ME may be used 236 519 A1 a gradient in particular according to EP, which requires only three iterations. The starting value of the estimate is expected by the co-located block from the previous vector field. Fractional vectors may be interpolated bilinear. A complete picture can be described by vectors 396.

p0032As extrapolation MCFE which also known from EP 236 519 A1 FIR can be used (finite impulse response) -Filterstrukturen. A planar interpolation nonintegral vectors with such filters following possible: Luminance: 2x oversampling with seven coefficients chrominance: 4x oversampling with eleven coefficients.

p0033For the DCT transform a fixed block size of 8 * 8 pixels is used as already described. Depending on the operating state processes the DCT image blocks of 8 * 8 pixels * full resolution (normal mode) according to Table 1 (Figure 8) or * Half resolution in both directions in case of imminent buffer overflow (equivalent to image section of 16 × 16 points). * With a quarter of the resolution in screen layout and scene cut (corresponding to an image section of 32 * 32 points).

p0034For all modes of same DCT algorithm. The DCT coder achieved an average data rate of &lt;1 bit / pixel.

p0035A two dimensional DCT transform of 8 * 8 pixels is achieved by two successive one-dimensional DCT transforms of each eight pixels. For Return transformation, the same algorithm is used. An overview of the entire DCT encoding is Figure 12, which is described in detail in DE 37 09 094.1. Here only a brief description of the function blocks according to Figure 12:

Edge detection and adaptive spectral thresholds

p0036By edge detection a local adaptation of the spectral visibility thresholds is reached. If diagonal edges detected in a block, as well as coefficients are considered that reach their visibility threshold only halfway.

classification

p0037The coefficients that exceed the visibility threshold shall be divided into classes and transfer (Figure 13). The class specifies which coefficients are transmitted in the order (coefficient address). The class information is transmitted as additional information entropy.

Luminance block

p0038* Suppress singular coefficients. Select * Relevant coefficient groups. A set of coefficients is taken into account only if at least half of their coefficients exceeds the visibility threshold. All coefficients of a relevant set of coefficients are transmitted (eg 18 sets of coefficients). * All relevant coefficient groups are combined to form a class, for example, 16 classes (Figure 13).

Chrominance block

p0039Since only a few coefficients must be transmitted, the coefficients can be combined directly to classes (eg 8 classes at max. 7 coefficients).

quantization

p0040Only the selected by the classification of a block coefficients are quantized. Each coefficient is quantized according to its visibility threshold. The Wahrnehmbarkeitsschwellen for the individual coefficients were determined by visual tests.

entropy

p0041All data and control information is entropy:

p0042* The selected and quantized AC (alternating component) coefficients. * The prediction for the DC (direct component) coefficients. * The class information.

p0043The most frequent values ​​of the AC and DC coefficients are optimally encoded, the rarely occurring values ​​are encoded with a prefix / postfix binary code.

p0044The following variations of the codec are possible: - Use another transform coding instead of DCT. - Use a vector instead of the DCT. At higher bit rates also DPCM. - Motion estimator: It can Vedas applied instead of a Gradientenschätzverfahrens even a block matching search method. (Eg Hierarchical Block Matching Displacementschätzverfahren to EP 236 519 A1. - In the decoder can be done instead of extrapolating skipped frames interpolation. - The temporal low-pass filtering of the reconstructed images is possible with the aid of a FIR structure as well as means of IIR (infinite impulse response).