Improvements relating to video data transmission

(74) Agent

Tandbergs Patentkontor AS, Oslo.

(30) Priority Requested 23.07.82, G8, No. 8221407.

(54) Name of the Invention Video Transfer Device.

<td>(57) Summary</td><td>A conditional refill video data transfer system comprises the transfer of digital values ​​that are representative. displays the brightness changes in image elements of a raster scanned image since the previous image. Since nothing is transmitted for the pixels that are without a significant brightness change, the amount of data to be transmitted will depend on the degree of motion in the image, and buffer stores (103, 207) are arranged to allow the use of a constant transfer rate. If the degree of motion in the image is very high, the buffer storage in the receiver will tend to fill up and may overflow. To prevent this from occurring, reduction of the amount of data to be transmitted is utilized by element sub-sampling whereby, in a bundle of alternating elements, data concerning only some, such as alternating, of the elements is transmitted and interpolation is used in the receiver to calculate the brightness change data for the omitted items. The blurring of vertical edges in the image resulting from the interpolation is avoided by setting the sub-sample where vertical edges are detected in the image. In order to avoid having to mark the normally omitted elements for which data is transmitted in the sub-sampling mode, the receiver is arranged to detect the presence of a vertical comb and respond by treating the data as if it relates to all elements near the edge and not just the sub-sampled elements. , Transfer</td>

(56) Published Publications United States (US) Patent Nos. 3715483, 3824590.

errors can result in the receiver being incorrect in the detection of edges, and to avoid this, the number of quantization levels during subsampling is reduced from 6 to 4, one set of four codes being used for the normally present elements and the other set of four codes being used for the normally omitted elements. Since a vertical edge will usually appear in more than one line, the receiver may be arranged to check the previous line for a vertical edge and predict the inclusion of normally omitted elements if such edge has

<img file="NO158708B_D0001.tif" />

The invention relates to video data transfer and is particularly useful in transmitting video information using conditional loading.

A television screen image typically contains approx. 200,000 pixels, and if the data is to be transmitted digitally, it is desirable that 8 bits be provided to transmit the brightness information in each element. With a partial repetition rate of 25 pr. per second for a moving image, digital transmission on the above basis would require a transmission channel capacity of 40 M bit / s. This requirement can be greatly reduced by making use of the high correlation present between one frame and the next, which arises because usually only a small portion of the frame will be moving at any given moment. Conditional loading involves transferring the changes only with an accuracy of, for example, only 16 non-linearly distributed quantization levels from one frame to the next, and the resulting data reduction is usually greater than the additional data that must be sent to identify exactly the particular areas of the frame as far as the changes are concerned. The areas are identified by the line number and the pixel address along the line. Although 9 bits are needed to fully identify the line number, a saving is made by using only 3 bits that give the number with module 8. An image field or field synchronization code is sent to identify the first line of a field, and this is done for all line numbers. whether it's a change in the line or not. The data transfer rate required when using conditional refill varies considerably because it depends on the portion of the moving image at that particular time, whereas it is much more convenient to have a constant data transfer rate. To overcome this difficulty, buffer bearings are provided at both ends of the transfer channel, and it is important that these buffer bearings are neither empty nor flooded. Using this technique it has been found possible to transmit a moving image satisfactorily over a 2 M bit / s channel.

1Ξ 87 03. If (t) is the number of bits stored in the encoder buffer (referred to as the encoder state) at the channel transfer end at a time t, and B_D(t) is the number of bits stored in the decoder buffer at the receiving end of the channel at time t, it can be shown that

B_E(t - At) + B_D(t) = V_R.To where V_R is the channel's transmission capacity (in bits / s) and Iantas to be constant, and At is the time delay between data entering the encoder buffer storage and those leaving the decoder buffer storage. Normally, an optimum value is selected for At which makes the total mass of the data stored in the buffers equal to half the total available buffer memory, and the control of the decoder buffer is based on the state of the encoder buffer at a time At earlier. This means that the emptying and overfilling of the decoder buffer can be predicted and appropriate

precautions are taken, either to temporarily suspend decoding or discard data, so that the distortion or_; The distortion of the rendered image that results from 20 data suddenly being unavailable or in the process of being lost can be avoided.

Then the fill data is derived from an image scanned using a conventional television grid and the rendered image is updated based on the fill data line by line 25 _{by e}t similar raster, it goes without saying that the readout of data from the decoder buffer must keep up to date with the setting of data in the encoder buffer if changes in the first image are to appear exactly in the rendered image. This forms a further coercive means on the readout from the decoder buffer θ '³ and handled by transmitting the encoder buffer state B_E (t) to the decoder so that the decoder buffer state B (it) can be predicted on the basis of the above equation.

In addition, four techniques are used to reduce the extremely high video data rate that may otherwise occur when there is a lot of image movement. These four techniques are as follows:

(1) The sensitivity of the motion detector in the encoder decreases as the encoder buffer fill increases, reducing the number of detected image change areas.

(2) Field sub-sampling is introduced on a timely basis. By this technique, alternating fields of information are discarded and the decoder is operated to interpolate the missing information from the two adjacent transmitted fields.

(3) Subsampling on an element basis is introduced so that certain moving area elements are discarded by the encoder lo and interpolated by the decoder.

(4) If the above techniques fail to prevent threatening overfilling of the encoder buffer, all movement of the transmitter image is ignored until the encoder buffer state has returned to a safe value.

The reduction of the sensitivity of the motion detector in the encoder has the effect of freezing change information with low contrast, resulting in an effect known as dirty window. This can be accepted for short periods. Field subsampling produces a motion jerk across the entire field of view, and the provision of an interpolator in the decoder can be used to reduce jerk by assuming that all movement takes place at uniform speed. Element sub-sampling, which involves transferring the changes relating to, for example, alternating elements, allows the amount of data transmitted when there is a lot of image motion to be reduced to a minimum without the introduction of the stop motion technique mentioned in (4) above. A disadvantage of element sub-sampling is that interpolation in areas of image detail can be reprehensible because the interpolation assumes that the brightness of an interpolated point is the average of the two adjacent points on the same line.

This is clearly not correct when the interpolated point lies on a vertical border between light and dark areas.

It is an object of the invention to avoid this disadvantage of element subsampling.

According to the invention there is provided a video transfer device for transmitting an image represented by a sequence of sub frames each comprising digi158708 speech values ​​corresponding to the brightness of elements along parallel lines across the image which comprises a conditional filling coding device which is arranged to generate filler data for transmission 5 with respect to the elements which change in brightness from, one sub-frame to the next, for updating a digital representation of the image stored in a receiver, and a sub-sampling device adapted to reduce the data transfer rate, if necessary, to suppress the transfer of loading data with respect to certain isolated elements , which device is characterized by a device for detecting the presence of a vertically extending, moving edge in the image, and to block the sub-sampling device for image elements located near the edge so that filler data is transmitted for all elements, near this edge. ^!

According to the invention there is also provided a receiver for use with the specified transmission device, comprising an image storage device, a device for receiving refill data and updating a stored representation of the image in the image storage device, a device for detecting the presence of sub-sampled data where the refill data for certain isolated image elements are missing, and an interpolation means for calculating η E ° interpolated values ​​for the missing pixels, from the sampled data, which.

the receiver is characterized by a device which, in the presence of sub-sampled data, is arranged to recognize that sub-sampling has been blocked and, in response, further, received data in image interpolated values. ;

the difference between the brightness obstructions of adjacent or nearly contiguous elements in a line depends on whether or not there is a vertically extending, moving edge in the image. A large difference will indicate the presence of such an edge, while a small difference will indicate its absence. An effect of element sub-sampling, ie. not transmitting the brightness changes for certain isolated elements, but calculating them based on the brightness changes for the adjacent elements, ie.

on this - to introduce the warehouse instead

The size • interpolation, can degrade high-contrast vertical edges. This effect is avoided by the invention because undersampling is not used where there are vertical details.

The element sub-sampling may involve transmitting the brightness changes for only alternating elements, giving the maximum data rate reduction that can be achieved because all the omitted elements, i.e. the elements whose brightness changes are not transmitted, must have a transmitted element on each page so that the sub-sampling does not cause significant horizontal obscuration. The omitted elements should preferably lie diagonally instead of vertical lines.

The line or field sync passwords may indicate that the video data has been subjected to element sub-sampling, and the receiver may be adapted to test the received data to determine whether a moving vertical edge is likely to be detected by the encoder generating the data for transmission, and thus whether or not the brightness change for a particular element is included in the received data. Therefore, no special indication is required that a normally omitted element of reality has been transmitted, because this information is inherent in the video data.

Although the system just described is very satisfactory with error-free data transmission, it unfortunately only takes one error in the brightness change. a single element for the recipient to decide incorrectly as to whether or not an element in the next line has been omitted, which in turn will result in the represented changes on being assigned to the incorrect elements, and consequent errors in the later ones. lines. To overcome this problem, the codes used to transmit the brightness changes for normally omitted elements may contain an indication that the element is an element that would normally be omitted. One way of achieving this is to reduce the number of quantization levels used when working in the element sampling mode compared to normal transmission, to use the codes representing the reduced set of quantization levels, to the normally transmitted elements, and to use. the remaining codes for any normally omitted items being transferred. Preferably, the codes used for normally transmitted elements and for normally omitted elements to be transmitted must be sufficiently different that a transmission error should not cause a code for one type of element to be corrupted into the code for the other type of element. The receiver may still test the elements in the previous line to check for the expected transmission of normally omitted elements, and be on the lookout for the codes for such elements.

IN

BRIEF DESCRIPTION OF THE DRAWINGS The invention will be described in greater detail in the following with reference to embodiments with reference to the drawings, in which: FIG. 1 is a diagram to be used to explain the conditional filling and generation of signals 15 representing the changes in different pixels; FIG. 2 represents the framing structure to be used in transmitting the data; FIG. 3 shows an example of quantization with 8 levels and 4 levels; FIG. 4 shows a diagram to be used in explaining the criterion for determining whether or not to transmit the information regarding the change of a particular image element; 5 is a block diagram of a video data encoder; and FIG. 6 shows a block diagram of a video data decoder.

In FIG. 1, the rectangle 1 represents the image area of ​​a television image, and 2 represents a group of eight consecutive lines of the image. The eight lines are given line numbers with module 8, so that the number can be represented by a group of 3 bits from 000 to 111 as shown. Along each line, the image elements are given addresses depending on their position along the line, with 256 elements in each line. A bundle 3 of elements in line 011 is shown / and it is believed that these elements have undergone a change since the previous section. The elements forming the bundle 3 are identified by the line number 011 and the address of the start_av the bundle. Once this information has been provided, the changes to the elements are presented continuously using variable length Huffman codes. At the end of the bundle, a bundle end code appears.

If the transmitter and receiver have image memories that digitally record the brightness of the elements, which are scanned in conventional television mode in synchronization with the other image memory, the data on changes in element brightness can be transmitted from one image memory to another, so that the receiving image memory reproduces the information in transmitter. This data will, of course, require line and field sync passwords to ensure that the scanning of the two images is in sync. The line sync passwords also contain the 3-bit line number. The field sync passwords are of two types that identify the different or equal fields in the normal line jump scan.

The data to be transmitted from the transmitter to the receiver is included in a conventional CCITT G732 framing or alignment structure. As shown in FIG. 2, this structure consists of 32 serially arranged 8-bit time slots forming a route or frame. Sixteen such frames form a multi-frame. Time slot 0 in different frames contains a frame synchronization word for the frame structure in the form of a Barker sequence, together with one bit 'which together with the individual bits in other different 0 time slots form a Barker sequence over a multi frame so that the timing of the framing structure can be recognized by receiver. Alarm and control information is transmitted in the same 0-time slots. Audio data and additional sync words appear in time slots 1 and 2. The remaining 29 time slots in each frame are filled with video data comprising the line and field synchronization passwords representing the changes to be transmitted from the transmitter image store to the receiver image store.

Since the proportion of an image moving at any point in time is variable, the amount of data to be transferred from the transmitter image store to the receiver image store will vary, but this data must be sent to maintain the data in the receiver image store substantially similar to the data in the transmitter image store. In other words, the amount of data that must be transmitted in one field scan period varies with the degree of motion in the image, and consequently the video data is asynchronous with the framing structure with G732 format signal in which they are fed.

The brightness of an image element is represented by an 8-bit binary number in the incoming PCM data, so that the end58708 ring in the brightness of an element can be in the range -255 to +255. A representation of the changes on this basis would require 9 bits, and to reduce this, the changes are quantized to 16 levels. Fig. 3 shows the characteristic of an 8-bit quantizer for positive output values ​​as a function of

In positive input values. With a corresponding characteristic of negative values, this would provide a 16-level quantizer. An embodiment requires the use of an 8-level quantizer instead of a 16-level when the normally omitted elements of a sub-sampled field are transmitted, due to the presence of moving vertical edges. The dashed characteristic of FIG. 3 represents the relationship between a positive output value and a positive input value for a 4-level quantization. Where the dotted characteristic is above a solid line of the 8-level quantization characteristic, the dotted line is not shown. This is especially noticeable at very low levels where the lines are superimposed over the entire first step of the characteristic. As with the 16-level quantization, the characteristic of the 8-level quantization for negative values ​​is the same as for positive values, but reversed.

When element subsampling is in use, the line synchronization signal will contain a code indicating this fact. Usually, the use of element subsampling will mean that the codes representing the brightness differences for alternating elements are omitted, provided that each omitted element on both sides is flanked by a transmitted element. If this is not the case, this item is sent which would otherwise be omitted, for example at the end of a bundle. Where an image change region extends over multiple lines, the omitted elements are usually selected in a diagonal format or arranged as five eyes on a cube. One effect of element sub-sampling is to reduce the data rate, and it is introduced if, the encoder buffer is filled over a certain threshold value and the techniques (1) and (2), i.e., reduction of the motion detector's sensitivity and field sub-sampling, are insufficient or incapable of handling problem. One effect of this reduction in data rate is to reduce the spatial resolution in areas where it is used, and the present invention seeks to avoid the introduction of this reduction in resolution in areas of high vertical detail, so that only minimal image degradation acts as a result of its application.

It is proposed that the decoder be able to reconstruct for itself information about the way in which the groups of data concerning the elements have been encoded. In this case, this means that the decoder must be able to determine which elements have been sub-sampled and which^is been transmitted, as it is undesirable to transmit additional signaling information.

An adaptive process that relies on the information already coded in the previous line is proposed as the criterion by which an element that would normally be omitted during subsampling is in fact transmitted. If the encoder is operated on this basis, the decoder may operate on the same basis and determine for itself whether or not a normally omitted element is sent.

Referring now to FIG. 4, showing nine elements in three successive lines of an image, the element X in the center being the element under consideration as to whether or not to transmit it. It is believed that elements A and E would be sent and element X omitted in the usual way in element sub-sampling mode. One test proposed to determine whether the element X should be transmitted or not is to bring about the sheer magnitude of the difference between the brightness of e.g. the elements B and D, being in line n-1 which precedes line nine on which element X lies. If this difference exceeds some threshold value which has been found to give good results by experiment, the element X is sent because there is a probability that a substantial vertical edge or sharpness passes through the array of nine elements. On the other hand, if the difference is less than the threshold value, the element X is omitted and the value of the element's brightness difference is determined by interpolation in the receiver.

It will be appreciated that the elements B and D may not be sent, in which case the values ​​obtained by inter

Ξ 8703 polishing between the values ​​being sent could be used] or the elements outside B and D could be used instead.

Unfortunately, when this process is used exactly as described above, it is prone to breakdowns where one or more brightness differences are distorted or distorted due to a transmission error or due to noise. For example, if the brightness changes at elements B and D in the encoder are different by an amount exceeding the threshold value, but the values ​​of the changes received at the receiver do not exceed the threshold, the encoder will correctly transmit the change at element X, but the receiver will interpret this change as if it belongs to the element · Ε because the criterion for transmitting the change at element X was not met in the receiver. This mistake will cause a noticeable image distortion that could persist until the end of the special field. in

To overcome this difficulty when the brightness change of an element normally omitted is transmitted, the coding of the brightness changes of all the affected elements, and also of the elements in the same lines and near the affected elements, is subjected to 8-level quantization in instead of 16-level quantization, and in addition, the elements that would normally be omitted are transmitted using codes different from those used in the normally transmitted elements. The 16-level quantization requires 16 different codes that can be numbered 1 to 16.

IN

For example, when normally omitted elements are transmitted, codes 1 to 8 are used for normally transmitted elements, and, codes 9-16 are used for normally omitted elements.

The coding selected for the 8-level quantization levels for the normally transmitted elements and for the normally omitted elements is preferably such that the change in the number of quantization levels from 16 to 8 will be apparent to the decoder, for example, using Huffman codes , and the chance that a code for one type of element due to noise will be converted to the code for the other type element is also kept to a minimum.

IN

FIG. 5 is a block diagram of one example of an encoder for use in the present invention. 8-bit, PCM video data is applied to the encoder via a contact terminal 100 which is connected to the input of a DPCM encoder 102. via a space / time filter 101. The filter 101 processes the image by means of spatial, non-linear filtering and noise reduction. to improve the performance of the later motion detector. As a second input signal, encoder 102 receives video data representing the foregoing image of the image from an image memory 103, and from a motion detector 104, encoder 102 provides data enabling the encoder to detect the elements of the incoming video data that have undergone a brightness change which is greater than a threshold level determined by a sensitivity control signal applied to a terminal 105. The output of encoder 102 is applied to a port 106 which is controlled by a sub-sampling control circuit 108 which responds to a signal supplied to a terminal 107 and indicates whether or not element sub-sampling is in use. When element sub-sampling is not used, the output of encoder 102 is applied to a 4-bit quantizer 109, and the codes representing the various levels are sent to a multiplexer 110 where they are combined with framing signals applied to a terminal III, and with signal indicating whether element sub-sampling has become used or not, to produce a combined output of a terminal 112 for supply to an encoder buffer. When element sub-sampling is in use, alternating codes in the output signal of encoder 102 of port 106 are transmitted to 4-bit quantizer 109 and thence to terminal 112 as before, with control circuit 108 controlling gate 106. A detector 113 is switched on to respond to the input signal of port 106 to produce an output signal when the video 30 data indicates the presence of a wandering vertical edge. The output signal of the detector 113 is applied to the control circuit 108 and to a port 116. As long as no such edges are detected, the subsampled signals are supplied to the 4-bit quantizer 109 as described above. When a wandering vertical edge is detected, the output of port 106 is switched to the input of a port 116 which is controlled by the output of detector 113 so that normally transmitted elements are applied to a 3-bit quantizer 114 and normally omitted elements are applied to a 3-bit. quantizer 115. Quantizers

<img file="NO158708B_D0002.tif" />

<img file="NO158708B_D0003.tif" />

114 and 115 outputs are also connected to multiplexer 110.

From a view of FIG. 5, it will be obvious how sub-sampled signals as described above are generated.

FIG. 6 is a block diagram of one example of a decoder for signals of the type that can be generated by the encoder of FIG. 5. The received signals are received via a terminal

200 and stored in an update data store 201, and also applied to a signaling decoder 202 which detects the presence of an indication of sub-sampling. If the sub-sampling is present, an indication is sent to a sub-sampling control circuit 203 and it controls a port 204 for routing the data from memory 201 either via a 4-bit de-quantizer 205 to input circuits 206 for an image gate 207, or via one or the other. of two 3-bit quantizers 208 and 209, depending on the control of a port 210 to which the output of port 204 is connected. A detector 211 coupled to the outputs of update data store 201 and image store 207 looks for signals indicating the presence of a wandering vertical edge as described above, and if such edge is detected, the detector generates signals to control gate 210 and also supplies a signal to the sub-sampling control circuit 203. An interpolator 212 is provided to interpolate between the output signal of the quantizer 205 and an output of the image memory 207, to produce the brightness change signal for a normally omitted element. Address circuits 213 perform the scanning of image storage 207 to produce the video output signal on a terminal.214, and also addresses the update data store

201 so that any update data for the item currently being scanned in the image storage 207 is read out for updating the image storage 207.

During operation of the decoder of FIG. 6, if an element sub-sampling is not in use, the output signal of the data store 201 is applied via the 4-bit de-quantizer 205 to update the data in the image storage 207. When element sub-sampling is used, an output of the control circuit 203 activates the interpolator 212, so that the update data for the output elements is extracted. based on the data of the transmitted elements15708 ter. If detector 211 detects a wandering vertical edge by utilizing the criterion referred to above, it supplies signals to the sub-sampling control circuit 203 and port 210, so that the update data is applied alternately via quantizers 208 and 209 to the input circuits 206 of the image storage 207.