Video conference terminal and image and sound reproducing method used for the same

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Technical field to which the invention belongs]

The present invention relates to a video conferencing terminal and an image / audio reproduction method used therein, and more particularly to an image / audio reproduction method in a video conferencing terminal.

【0002】

[Conventional technology]

When transmitting images and audio using a video conferencing terminal, the time required for compression and decompression processing of each signal is different, so the timing of the image and audio is the same after the compression / decompression processing. It will be different from the signal.

【0003】

In general, since the amount of processing of an image is larger than that of audio, the processing of the image signal is delayed from that of the audio signal on the receiving side. As a result, unnatural reproduction such that the video is delayed with respect to the audio is performed.

【0004】

Conventionally, in order to avoid this problem, a video conferencing terminal having a function of selecting a delay amount at the time of audio reproduction from several types of fixed values predetermined on the receiving side has been developed. If a video conferencing terminal having this function is used, the reproduction timings of the image and the sound can be matched to some extent.

【0005】

Another technology is the MPEG-2 (Moving Picture Experts Group phase 2) image / audio synchronization method, which is one of the video compression methods.

【0006】

In MPEG-1 (Moving Picture Experts Group phase 1) and MPEG-2 method (hereinafter referred to as MPEG method), image packets and audio packets each have a time stamp called PTS (Presentation Time Stamp).

【0007】

The PTS is added to the beginning (packet header) of the video packet and the audio packet at the time of transmission, and is transmitted to the receiving side. Figure 6 shows a state in which images and sounds are multiplexed into packets using MPEG-2. There is a packet header at the beginning of the image packet and audio packet, and the PTS value is stored in this packet header.

【0008】

On the other hand, the receiving side has a counter (STC: System Time Clock) that is accurately synchronized with the transmitting side, and the decoder makes the PTS stored in the packet header of the received image and audio equal to the STC of the receiver. At that time, the image and sound are reproduced.

【0009】

That is, if the PTS value (reproduction time) is set at the time of transmission so that the images and sounds are reproduced at the same time on the receiving side, an output in which the image and the sound are synchronized can be obtained. This MPEG method is described in detail in "Latest MPEG Textbook" (published by ASCII Publishing, 1995).

【0010】

[Problems to be Solved by the Invention]

In the conventional video conferencing terminal described above, when transmitting an image and an audio, the time required for the compression process and the decompression process is not constant depending on the content of the input signal, and the image and the audio are fixed in the conventional method in which the delay amount is fixed. There is a problem that it is not possible to always match the playback timing with.

【0011】

Further, in this image / sound synchronization method adopted by the MPEG method, the reproduction timings of the image and the sound can be matched, but on the other hand, there are the following problems.

【0012】

First, the above-mentioned image / audio synchronization method has a problem that the amount of information for synchronous reproduction is large. The PTS value has a long setting range on the receiving side (24 hours or more), so the PTS data width is as large as 44 bits. Therefore, as the circuit scale increases, the header length increases because the PTS is stored in the header.

【0013】

Further, the above-mentioned image / audio synchronization method has a problem that it is an MPEG-only method. Since PTS is stored in the header of the packet specific to the MPEG method, PTS can be used only in the system using the MPEG method, and is not compatible with other video coding methods.

【0014】

Therefore, an object of the present invention is to solve the above-mentioned problems, and it is natural that the image and the sound are reproduced at the same timing as the transmitting side without increasing the header information and without depending on the compression / decompression method. It is an object of the present invention to provide a video conferencing terminal capable of easily realizing a video conferencing and an image / audio reproduction method used therefor.

【0015】

[Means for solving problems]

The television conference terminal according to the present invention has a quantization means for quantizing each analog signal of input image and audio, and a marking signal preset for the image and audio signals quantized by the quantization means. The marking signal adding means for simultaneously and periodically adding the marking signal, the compressing means for compressing and transmitting the signal to which the marking signal is added by the marking signal adding means, the decompressing means for decompressing the received signal, and the decompression means. The marking signal detecting means for detecting the marking signal from the signal extended by the means and detecting the arrival time difference of the marking signal added to each of the image and audio signals, and the signal extended by the extending means for the image and the signal. It is provided with a reproduction means for reproducing each analog signal of voice and a delay means for delaying the output of one of the analog signals of the image and audio reproduced by the reproduction means according to the detection result of the marking signal detection means. ing.

【0016】

In the image / audio reproduction method of the video conference terminal according to the present invention, each analog signal of the input image and audio is quantized on the transmitting side, and the quantized image and audio signals are marked in advance. The signal is added simultaneously and periodically, the signal to which the marking signal is added is compressed and transmitted, the received signal is extended on the receiving side, and the marking signal is detected from the extended signal to detect the marking signal. The arrival time difference of the marking signal added to each of the image and sound signals is detected, and the output of one of the image and sound analog signals reproduced from the extended signal is delayed according to the detection result. There is.

【0017】

That is, the video conferencing terminal of the present invention can reproduce the video and audio at the same timing as the transmission on the receiving terminal by always and accurately matching the reproduction timing of the image and the audio on the transmitting side. It is characterized by.

【0018】

More specifically, in the video conferencing terminal of the present invention, an analog image input signal from a camera or the like is quantized by an image A / D converter, and then compressed according to the speed of a transmission line by an image compression circuit through a marking signal addition circuit. After receiving the processing, it is multiplexed with the audio signal by the multiplexing circuit and sent to the transmission line.

【0019】

On the other hand, the analog audio input signal from the microphone or the like is quantized by the audio A / D converter, compressed by the audio compression circuit, multiplexed with the image signal by the multiplexing circuit, and then sent to the transmission line. The image compression circuit and the audio compression circuit use a reversible coding algorithm to compress the quantized image and audio signals.

【0020】

On the receiving side, the signal received from the communication partner through the transmission line is separated into an image signal and an audio signal by a separation circuit, and the image signal is an analog image output signal through an image extension circuit, a marking signal detection circuit, and a D / A converter. Play as.

【0021】

Similarly, the audio signal is also reproduced as an analog audio output signal by the audio extension circuit and the audio D / A converter. The image decompression circuit and the audio decompression circuit decompress the signal compressed by the algorithm opposite to that of the transmitting side.

【0022】

In the video conferencing terminal of the present invention, each of the input image and audio analog signals is quantized immediately after being quantized by the A / D converter, that is, before being compressed, respectively. A marking signal is simultaneously and periodically added to the signal by the marking signal addition circuit. The marking signal is added by replacing some bits of the signal with the marking signal.

【0023】

Here, by using a reversible algorithm for the compression and decompression processing of the signal, the signal before compression is completely reproduced on the receiving side. Therefore, the marking signal added on the transmitting side can be detected by the marking signal detection circuit on the receiving side, and the arrival time difference t between the marking signal added to the image signal and the marking signal added to the audio signal can be known. ..

【0024】

As a result, by delaying the output of the signal that arrived earlier among the image signal and the audio signal by the arrival time difference t in the delay circuit, the image and the audio can be reproduced at the same timing as at the time of transmission. That is, even if there is a difference in arrival time between the image and the audio signal, the receiving side video conferencing terminal can always reproduce the image and the audio at the same timing as at the time of transmission.

【0025】

In the method of the present invention, the marking signal is processed immediately after the image signal and the sound signal are quantized or immediately before being converted into analog. Therefore, if the algorithm is lossless, the sound and image are independent of the compression / decompression method. It is possible to provide a synchronous playback function with.

【0026】

In the video conferencing terminal described in the present invention, instead of using a long time stamp, a short marking signal is directly embedded in the image signal and the audio signal, so that it is possible to perform synchronous reproduction of the image and the audio with a smaller amount of information. It becomes.

【0027】

BEST MODE FOR CARRYING OUT THE INVENTION

Next, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a video conferencing terminal according to an embodiment of the present invention. In FIG. 1, the video conferencing terminal 1 is multiplexed with an image A / D (analog / digital) converter 11, an audio A / D converter 12, a marking signal addition circuit 13, an image compression circuit 14, and an audio compression circuit 15. Conversion circuit 16, separation circuit 17, image extension circuit 18, audio extension circuit 19, marking signal detection circuit 20, image D / A (digital / analog) converter 21, audio D / A converter 22, It is composed of a delay circuit 23.

【0028】

On the transmitting side, the analog image input signal 101 from a camera or the like (not shown) is quantized by the image A / D converter 11, the marking signal is added by the marking signal addition circuit 13, and the speed of the transmission line 100 is added by the image compression circuit 14. After receiving the compression processing according to the above, the multiplexing circuit 16 multiplexes the audio signal and sends the signal to the transmission line 100.

【0029】

On the other hand, the analog audio input signal 102 from a microphone or the like (not shown) is quantized by the audio A / D converter 12, the marking signal is added by the marking signal addition circuit 13, compressed by the audio compression circuit 15, and then multiplexed. It is multiplexed with the image signal by the circuit 16 and sent to the transmission line 100.

【0030】

The marking signal addition circuit 13 converts the image and sound into the quantized signals immediately after being quantized by the image A / D converter 11 and the voice A / D converter 12, that is, before being compressed. On the other hand, marking signals are added simultaneously and periodically. The marking signal is added by replacing some bits of the signal with the marking signal.

【0031】

The image compression circuit 14 and the audio compression circuit 15 compress the signal to which the marking signal is added by the marking signal addition circuit 13, that is, the quantized image and audio signals using a lossless coding algorithm.

【0032】

On the receiving side, the signal received from the communication partner through the transmission line 100 is separated into an image signal and an audio signal by the separation circuit 17. The image signal, that is, the compressed signal is decompressed by the image decompression circuit 18 by the algorithm opposite to that on the transmitting side, input to the image D / A converter 21 via the marking signal detection circuit 20, and analog by the image D / A converter 21. It is reproduced as an image output signal 103.

【0033】

Like the image signal, the audio signal is decompressed by the audio decompression circuit 19 by the algorithm opposite to that of the transmitting side, input to the audio D / A converter 22 via the marking signal detection circuit 20, and is input by the audio D / A converter 22. It is reproduced as an analog audio output signal 104.

【0034】

In the video conferencing terminal 1 according to the embodiment of the present invention, the input image and audio analog signals are immediately after being quantized by the image A / D converter 11 and the audio A / D converter 12, that is, before being compressed. At this stage, marking signals are simultaneously and periodically added to the quantized signals of the image and the sound by the marking signal addition circuit 13.

【0035】

The marking signal is added by replacing some bits of the signal with the marking signal. Here, by using a reversible algorithm for the compression and decompression processing of the signal, the signal before compression is completely reproduced on the receiving side.

【0036】

Therefore, the marking signal added on the transmitting side is detected by the marking signal detection circuit 20 on the receiving side, and the arrival time difference t between the marking signal added to the image signal and the marking signal added to the audio signal is known. Can be done. As a result, among the image signal and the audio signal, the output of the signal arriving earlier is delayed by the arrival time difference t in the delay circuit 23, so that the image and the audio can be reproduced at the same timing as at the time of transmission.

【0037】

That is, even if there is a difference in arrival time between the image and the audio signal, the video conferencing terminal 1 on the receiving side can always reproduce the image and the audio at the same timing as at the time of transmission.

【0038】

In the video conferencing terminal 1 according to the embodiment of the present invention, the marking signal is processed immediately after the image signal and the audio signal are quantized or immediately before being analogized. It is possible to provide a synchronous playback function of audio and image without dependence.

【0039】

In the video conferencing terminal 1 according to the embodiment of the present invention, instead of using a long time stamp, a short marking signal is directly embedded in the image signal and the audio signal, so that the image and the audio are synchronizedly reproduced with a smaller amount of information. be able to.

【0040】

Subsequently, examples of the present invention will be described with reference to the drawings. FIG. 2 is a block diagram showing a configuration of a video conferencing terminal according to an embodiment of the present invention. In FIG. 2, the video conferencing terminal 3 is composed of an image codec 31, an audio codec 32, a marking circuit 33, a multiplexing circuit 34, a separation circuit 35, and an audio delay circuit 36.

【0041】

The image codec 31 quantizes and compresses the image input 201, decompresses and analogizes the received image signal, and outputs it as the image output 202. The voice codec 32 quantizes and compresses the voice input 203, decompresses and analogizes the received voice signal, and outputs it as the voice output 204.

【0042】

The marking circuit 33 sends a marking signal to the encoded image and audio signals, and processes the marking signal detected from the received image signal and audio signal. The multiplexing circuit 34 multiplexes the outputs of the image codec 31 and the audio codec 32 and sends them to the transmission line 200. The separation circuit 35 separates the signal received from the transmission line 200 into an image signal and an audio signal. The voice delay circuit 36 delays the output of the voice signal according to the instruction of the marking circuit 33.

【0043】

FIG. 3 is a block diagram showing the configurations of the image codec 31 and the audio codec 32 of FIG. FIG. 3 shows the internal configurations of the image codec 31 and the audio codec 32 having a common circuit configuration.

【0044】

The codec 4 is composed of an A / D converter 41, a marking signal addition circuit 42, a compression circuit 43, an extension circuit 44, a marking signal detection circuit 45, and a D / A converter 46.

【0045】

The compression circuit 43 and the decompression circuit 44 process by a reversible algorithm. The marking signal addition circuit 42 and the marking signal detection circuit 45 are connected to the marking circuit 33 shown in FIG. The output of the marking circuit 33 controls the voice delay circuit 36 connected to the output of the voice codec 32.

【0046】

FIG. 4 is a diagram showing a configuration of signals sent and received on the transmitting side and the receiving side in one embodiment of the present invention. In FIG. 4, the signals sent and received on the transmitting side and the receiving side are composed of digitized data and marking signals, and 0 is normally added to the marking signals and 1 is added every time T.

【0047】

The operation of the video conferencing terminal 3 according to the embodiment of the present invention will be described with reference to FIGS. 2 to 4. First, the processing on the transmitting side will be described. The analog image input signal 201 and the analog audio input signal 203 input to the video conference terminal 3 from the camera and the microphone (not shown) are quantized by the A / D converter 41 in the image codec 31 and the audio codec 32, respectively. A marking signal is added by the marking circuit 33.

【0048】

The marking signal is used to measure the delay of the image signal with respect to the audio signal during reproduction. In this embodiment, the marking signal is a 1-bit digital signal, and normally "0" is inserted, but "1" is inserted every T (for example, 50 milliseconds) for a certain period of time.

【0049】

In the marking signal addition circuit 42, a part of the image signal and audio signal data quantized by the A / D converter 41 is replaced with this marking signal. The bits to be replaced are the lower bits of the quantized signal so that the information of the original image signal and the audio signal is not significantly impaired by this replacement process.

【0050】

For example, as shown in FIG. 4, for data quantized with 8 bits, the lowest 8th bit is replaced with this marking signal, and the data of the upper 7 bits is transmitted as it is.

【0051】

The image signal and the audio signal to which the marking signal is added are compressed by a lossless algorithm inside the image codec 31 and the audio codec 32. Examples of reversible algorithms include a "variable length coding method" that uses a Huffman code or the like, and a "run length coding method" that uses a modified Huffman code or the like.

【0052】

Next, the processing on the receiving side will be described. The signal received from the transmission line 200 is separated into an image signal and an audio signal by the separation circuit 35, and then subjected to decompression processing inside the image codec 31 and the audio codec 32, respectively.

【0053】

As described above, since the reversible compression algorithm is used, the bit string before compression including the marking signal is completely restored after the decompression processing. After that, the marking signal detection circuit 45 monitors the lower 1 bit of the image and audio data before being decoded into an analog signal by the D / A converter 46 inside each codec, and detects the marking signal 1. ..

【0054】

In general, the time required for signal compression and decompression processing is longer for an image signal than for an audio signal. Therefore, it can be considered that the image signal lags behind the audio signal. Therefore, if the timer (not shown) is started when the marking signal 1 is detected from the audio signal and then the value of the timer when the marking signal 1 is detected from the image signal is t, the image signal It can be seen that was transmitted with a delay of t from the audio signal.

【0055】

Therefore, by controlling the audio delay circuit 36 connected to the output on the playback side of the audio codec 32 so that the audio output is delayed by t from the marking circuit 33, the image is output with respect to the audio. Since it is output without delay, it is possible to realize a highly realistic video conference.

【0056】

In this way, by measuring the arrival time difference t of the marking signal added immediately after the quantization of the image signal and the audio signal on the receiving side, the arrival time difference t between both the image signal and the audio signal can be grasped on the receiving side. , The image signal and the audio signal can be reproduced at the same timing as at the time of transmission. This makes it possible to hold a video conference with a high sense of presence.

【0057】

FIG. 5 is a diagram showing a configuration of signals sent and received on the transmitting side and the receiving side in another embodiment of the present invention. FIG. 5 shows an example in which the replacement pattern of the marking signal for the signal sent and received on the transmitting side and the receiving side is further devised as compared with the embodiment of the present invention.

【0058】

The video conferencing terminal used in the other embodiment of the present invention has the same configuration as the video conferencing terminal 3 according to the embodiment of the present invention shown in FIG. 2, and only the method of adding the marking signal is different.

【0059】

The marking circuit 33 usually has "00" in the lower 2 bits of the output of the A / D converter 41 for image and audio signals at the time of transmission, every T for a certain period of time, that is, "01" at time T1, "10" at T1 + T, and T1. Insert "11" in order at + 2T. At time T1 + 3T, the marking signal becomes 01 again and the same insertion cycle is repeated. This is shown in Fig. 5.

【0060】

The marking circuit 33 detects patterns of "01", "10", and "11" from the audio signal and the image signal at the time of reception, and calculates the delay time between them in the same manner as described above.

【0061】

As described above, by preparing three types of marking signals, it is possible to make the same marking signal appear only every three cycles. When only one type of marking signal is used, the same marking signal appears after one cycle, so it is not possible to accurately measure the delay beyond the insertion cycle.

【0062】

On the other hand, if the insertion cycle of the marking signal is lengthened, the interval at which the delay time T is updated also becomes long, and it becomes difficult to follow the change in the delay time in a short time. However, in this method using a plurality of marking signals, the delay amount of the image signal can be accurately measured even if the image signal is delayed more than the insertion cycle of the marking signal without impairing the followability by extending the delay time measurement cycle. You can know.

【0063】

Further, as a further extension of the above embodiment, the marking signal may be used with 3 bits or more. Generally, when n bits (n is a positive integer) are assigned to the marking signal, (2)ⁿ -1) Marking signals can be used. Here, if the insertion interval of the marking signal is T, (2)ⁿ -1) It is possible to detect up to a delay of × T.

【0064】

If a large number of bits are assigned to the marking signal, the information of the original signal is lost accordingly. Therefore, the number of bits of the marking signal is the image quality by replacing the quantization bit number of the image and audio signals and a part of the data with the marking signal. It should be determined in consideration of the deterioration allowance and the required delay detection performance.

【0065】

[Effect of the invention]

As described above, according to the present invention, on the transmitting side, each analog signal of the input image and sound is quantized, and a preset marking signal is simultaneously applied to the quantized image and sound signals. In addition, the signal is periodically added, and the signal to which the marking signal is added is compressed and transmitted, and the received signal is extended on the receiving side, and the marking signal is detected from the extended signal to detect an image and audio signal. The head information is increased by detecting the arrival time difference of the marking signal added to each and delaying the output of one of the image and audio analog signals reproduced from the extended signal according to this detection result. There is an effect that it is possible to easily realize a natural video conference in which the image and the sound are reproduced at the same timing as the transmitting side without depending on the compression / decompression method.

[Simple explanation of drawings]

[Figure 1]

It is a block diagram which shows the structure of the video conference terminal by embodiment of this invention.

[Figure 2]

It is a block diagram which shows the structure of the video conference terminal by one Example of this invention.

[Fig. 3]

It is a block diagram which shows the structure of the image codec and the audio codec of FIG.

[Fig. 4]

It is a figure which shows the structure of the signal sent and received on the transmitting side and the receiving side in one Example of this invention.

[Fig. 5]

It is a figure which shows the structure of the signal sent and received on the transmitting side and the receiving side in another Example of this invention.

[Fig. 6]

It is a figure which shows the structure of the signal sent and received on the transmitting side and the receiving side in the conventional example.

[Explanation of symbols]

1,3 Video conferencing terminal 4 codec 11 Image A / D converter 12 Audio A / D converter 13,42 Marking signal addition circuit 14 Image compression circuit 15 Audio compression circuit 16,34 Multiplexing circuit 17,35 Separation circuit 18 Image expansion circuit 19 Audio extension circuit 20,45 Marking signal detection circuit 21 Image D / A converter 22 Audio D / A converter 23 Delay circuit 31 image codec 32 voice codec 33 Marking circuit 36 Voice delay circuit 41 A / D converter 43 compression circuit 44 Extension circuit 46 D / A converter 100,200 transmission line