Apparatus and method for synchronization of audio and video streams.

Apparatus and method for synchronizing clover frequency and video stream

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the rights of the U.S. Provisional Application with serial number 60/374269 filed on April 19, 2002, the entire content of which is incorporated herein by reference.

[0003] Background of the Invention Field of Invention

[0004] The present invention relates to the field of multimedia communication systems, and more specifically to minimizing lip synchronization errors caused by variable delay transmission networks.

Background technique

[0005] The "lip sync" (or lip sync) problem is widely known. Briefly, instantaneous errors in the presentation of audio and video streams through the presentation device may lead to a situation in which audio information is displayed before (leading) or after (lagging) the presentation of the corresponding video information, resulting in, for example, the inter-frequency of the speakers voice. The synchronization between the representation and the video representation of the speakers lips is poor.

[0006] The existing technologies for solving the so-called lip synchronization problem are relatively complicated, and sometimes cause degradation of audio and/or video information. For example, everyone knows that the loss of video frames causes the time of the video image to advance, so it is necessary to correct the advanced audio signal.

[0007] Many reasons can cause lip synchronization errors. Of particular relevance is the use of variable delay networks, such as the Internet and other packet-switched networks. In such a network, audio and video information are transmitted as separate and independent streams. During the transmission process before introducing these streams to the variable delay network, a transport layer header containing time stamps and other metadata (such as encoder sampling rate, packet order, etc.) is added to some or all of the transport packets. The time stamp of audio and video information typically comes from a common source, such as a real-time clock. Unfortunately, when audio and video data packets pass through a variable delay network, due to network conditions, transient abnormalities occur, data packets are lost, the order of the data packets is not preserved, and the delay time of the data packets changes. The end result is lip synchronization errors in the received audio and video streams transmitted over the variable delay network.

Summary of the invention

[0008] The present invention includes a method and device for reducing the lip synchronization problem of the corresponding audio and video streams by adjusting the statistical distribution of the instantaneous error below the error range that the listener can notice and dislike.

[0009] In particular, the method for synchronizing audio and video streams according to an embodiment of the present invention includes the following steps: receiving a video access unit and a corresponding audio access unit for representing the video and audiovisual information. The corresponding audio access unit helps to display the instantaneous audio-visual synchronization error described by the first probability distribution function (pdf); and instantaneously delays one of the received audio and video access units by a delay amount, The undelayed and corresponding delayed access unit for representing audiovisual information helps to display the instantaneous audio-visual synchronization error described by the second pdf, which uses asymmetric synchronization capacity than the first pdf. A larger share of the limited window.

[0010] In another embodiment, a method for generating an encoded video and a corresponding audio stream includes the following steps

CN 1745526 Β

Step: instantaneously encode corresponding video and audio information to generate encoded video and audio streams containing corresponding video and audio access units; and instantaneously delay the encoding by a delay amount corresponding to an asymmetric synchronization error tolerance model One of the video and audio streams; the error tolerance model defines a synchronization tolerance window; and the instantaneous delay results in a probability distribution function describing the synchronization error between the audio stream and the corresponding video stream pdf, this function is shifted to a more beneficial direction with the corresponding synchronization tolerance window.

[0011] In another embodiment, a device for synchronizing audio and video streams includes: a delay element for transmitting an instantaneous delay to at least one of an audio signal and a corresponding video signal in response to An asymmetric error tolerance model; and an encoder for encoding audio and video signals to produce encoded audio and video streams; the error tolerance model defines a synchronization tolerance window; and the instantaneous delay causes A probability distribution function pdf describing the synchronization error between the audio signal and the corresponding video signal, the function being shifted in a direction that is more beneficial to the corresponding synchronization tolerance window.

Description of the drawings

[0012] The teachings of the present invention can be easily understood by considering the detailed description and drawings hereinafter, in which:

[0013] Figure 1 depicts a high-level block diagram of a communication system;

[0014] Figure 2 depicts a high-level block diagram of a controller;

[0015] FIG. 3 depicts a graphical representation of the probability density function p(e) of the lip synchronization error e that is helpful for understanding the present invention;

[0016] FIG. 4 depicts a graphical representation of a lip sync error tolerance (LSET) window that is helpful for understanding the present invention;

[0017] FIG. 5 depicts a graphical representation of the pdf offset in the tolerance window;

[0018] FIG. 6 depicts a method for processing audio and/or video data packets according to the present invention;

[0019] FIG. 7 depicts a high-level block diagram of a communication system according to an alternate embodiment of the present invention; and

[0020] FIG. 8 illustrates a high-level block diagram of an embodiment of the present invention, in which a pdf estimator is executed on a receiver side.

[0021] For ease of understanding, wherever possible, the same reference numbers have been used to designate the same elements in the drawings.

[0022] Detailed description of the invention

[0023] The present invention will be discussed in the context of variable delay networks such as the Internet, where variable delay networks tend to produce instantaneous errors when video and/or audio data packets pass, so that lip synchronization errors may occur. . However, the method of the present invention can be easily adapted to various instantaneous error sources. The present invention runs on video and/or audio presentation units (e.g., video and audio frames), which can be packaged to be suitable for transmission over a network such as a variable delay network.

[0024] Moreover, although the standard communication definition of "lip synchronization" makes the synchronization (or synchronization processing) of voice or singing be related to the video, so that the movement of the lips in the video appears to be naturally consistent with the sound; but for the present invention, this The interpretation of the definition cannot be so restricted. On the contrary, "lip synchronization" refers to the synchronization of any action represented in the video with the corresponding audio track or stream, so that the sound produced by the action is appropriately matched with the video that produced the sound intentionally. In other words, for the present invention, "lip sync" refers to the synchronization between the sound represented by the audio information signal and the corresponding video represented by the video information signal; regardless of the content of the corresponding audio and video. Therefore, the reference to "lip synchronization error" is essentially comprehensive and can be interpreted as various "audio-visual time synchronization errors".

[0025] FIG. 1 depicts a high-level block diagram of a communication system including the present invention. In particular, the communication system 100 includes an audiovisual source 110, such as a mass storage device, a video camera, a microphone, a network input, or other audiovisual information sources. The audiovisual source 110 provides a video stream V to a video encoder 120V and a corresponding audio stream A to the audio encoder 120A, respectively. The encoders 120V and 120A (for example, they constitute MPEG or other compression encoders) respectively encode the video stream V and the audio stream A to generate an encoded video stream VE and an encoded audio stream AE, and an encoded video VE and audio AE. The stream is adapted to the variable delay network 140 (for example, is Ethernet, ATM, or other transport stream encoder, which is suitable for the video VE and audio AE stream according to the specific transmission format suitable for the variable delay network 140 The specific transmission frame for encoding) is processed by the transmission processor 130.

[0026] The transport stream T is propagated to the destination through the variable delay network 140, such as the Internet, Intranet, ATM, Ethernet, LAN, WAN, public switched telephone network (PSTN), satellite, or Other networks, where it is received as a transport stream transport stream T, include the original transport stream T with some delays or other errors introduced by the variable delay network 140 transmission.

[0027] The synthesized transport stream T is received by the transport processor 150 (for example, an Ethernet, ATM, or other transport stream decoder), and the processor extracts one from the received transport stream. The encoded video stream VE, and the corresponding encoded audio stream AE'. The encoded video VE' and audio AE' streams include original encoded video VE and audio AE streams with some instantaneous errors caused by, for example, the transmission processor 130, the variable delay network 140, and/or the transmission processor 150. The received encoded video VE, and audio AE, streams are decoded by the decoder 160 to generate a synthesized video V, and audio A, streams. The generated video V and audio A streams are presented by a presentation device 170 (such as a television or other display device), and 170V has associated an audio presentation device (such as a speaker 170A) with it.

[0028] FIG. 2 depicts a high-level block diagram of a controller suitable for use in systems and devices in accordance with the principles of the present invention. In particular, the controller 200 in FIG. 2 may be used to execute one or more of the functional elements in the above description with reference to FIG. 1 and various functional elements in the following description with reference to FIGS. 7 and 8.

[0029] The exemplary controller 200 in FIG. 2 includes a processor 230 and a memory 240 for storing various programs 245. The processor 230 cooperates with a conventional support circuit 220 such as a power supply, a clock circuit, a cache memory, etc., and a circuit that assists in executing the software program stored in the memory 240. Likewise, it is foreseen that some of the processing steps discussed herein, such as software processing, may be executed in hardware, for example, like a circuit in cooperation with the processor 230 to perform various steps. The controller 200 also includes an input/output (I/O) circuit 210, which is formed at an interface of various functional elements that communicate with the functional elements of the controller 200.

[0030] Although the controller 200 in FIG. 2 is described as a general-purpose computer for performing various instantaneous corrections of audio and/or video streams according to the present invention, the present invention can be implemented in hardware, such as a Application-specific integrated circuit (ASIC). Likewise, the processing steps described herein can be broadly interpreted as being implemented equally by software, hardware, or a combination thereof.

[0031] Lip sync error (LSE) can be defined as follows according to Equation 1:

[0032] corpse (t; one dagger) one (t: -t;) (formula 1)

[0033] In Formula 1, and t/ are the times when the relevant audio and video frames arrive at the reproduction device 170 at the receiver respectively; and and are the times when the audio and video frames arrive at the audio and video encoders, respectively.

[0034] FIG. 3 depicts a graphical representation of the probability density function p(e) of the lip synchronization error e, which is helpful for understanding the present invention. Due to, for example, the random delay introduced by the variable delay network 140, the delay between the audio data packet at the receiver and its corresponding video data packet is a random variable. The random variable is represented by its probability density function (pdf), p(e)

The limit is shown as the solid line 310 in FIG. 3. In particular, the graphical representation of FIG. 3 describes a horizontal axis that defines the temporal relationship between video data and corresponding audio data. Time zero is selected as the time representing the video data content to which the synchronized audio data has been combined. When this distribution is described as a Gaussian distribution, other symmetric or asymmetric pdf curves can be used, depending on the specific simulated error source and the number of simulated error sources (that is, one can be used for composite video and audio Source compound symmetric or asymmetric Pdf curve).

[0035] As time increases from zero to the positive direction, the audio data lags further behind the video data (that is, the audio data packet is more delayed than the corresponding video data packet). As the time increases from zero to the negative direction, the audio data is more advanced than the video data (that is, the video data packet is more delayed than the corresponding audio data packet).

[0036] FIG. 4 depicts a diagrammatic representation of a lip sync error tolerance (LSET) window 410 that is helpful in understanding the present invention. In particular, the LSET window is defined by the function formula (2) as follows, where a and b are the lower and upper limits of the tolerance of the LSET window.

/, fl (a <e <b)

[0037] W (e) = £ (other) (Formula 2)

[0038] The inventor has noticed the asymmetric error tolerance of audio and video data packets and a large number of problems that arise when the audio data packet is received before the corresponding video data packet. Typical value range, for example [a, b]= [-20ms,40ms].

[0039] FIG. 5 depicts a graphical representation of the pdf offset in the tolerance window. In particular, the graphical representation of FIG. 5 describes a horizontal axis that defines the temporal relationship between video data and corresponding audio data. Time zero is selected in the manner described above with reference to FIGS. 3 and 4. The delay tolerance window 410 represents a delay tolerance or instantaneous error combined with lip synchronization that is not annoying by the viewer. It can be noted that the delay tolerance window 410 in Figure 5 extends from -20 milliseconds (that is, the audio data packet is 20 milliseconds ahead of the video data packet) to +40 milliseconds (that is, the audio data packet lags the video data packet by 40 milliseconds) . It can be noticed that the lip synchronization error of audio information ahead of video information is often more annoying than the lip error of audio information lagging video information (for example, it is easier to make the viewer notice and/or distract attention), so the delay in Figure 5 The tolerance window 410 is asymmetrical.

[0040] Referring to FIG. 5, the "tail" part on the left side of the pdf curve 510 falls into an area 540 that exceeds the lower delay tolerance window range. It can be noted that the "tail" part on the right side of the pdf curve 510 is fully zeroed before the upper limit of the delay window tolerance range. The error window tolerance range is defined as the range in which instantaneous errors such as lip sync errors are considered less annoying. Therefore, either positive or negative delays that exceed the delay tolerance range include these delays that are regarded as annoying or highly annoying for the general audience.

[0041] The shifted pdf curve 520 represents the original probability distribution curve 510 that is shifted in time, so that a larger area under the pdf curve is within the error tolerance window 410. Therefore, the original or first pdf has been shifted in time so that the increased area (preferably the largest area) below the final or second pdf is included in the error tolerance window 410. The offset on the pdf is caused by adaptive timing parameters combined with video and/or audio information, such as the display time stamp of the video and/or audio access unit. Therefore, if the audio and/or video instantaneous information is suitable to cause such an offset on the corresponding pdf, then the possibility of annoying lip synchronization errors will be minimized, or at least a reduction in the curve caused by the offset The Pdf under the error is reduced by the corresponding amount. Therefore, the best solution to maximize the area under the lip LSE curve in the lip LSET is to maximize the objective function given by Equation 3 as follows:

[0042] J = p(e -QW(e)de

[0043] =fp (e-how many) Ne (formula 3)

[0044] =Ρ(ά-Ζ₀) -Ρ(α-/₀）

[0045] In formula 3, p(e) is the pdf of LSE, P(e) is the cumulative distribution function and W(e) is the LSET window function defined in (2). The optimization process is used to maximize the area enclosed by the pdf curve within the bounds of [a, b]. This is equivalent to minimizing the "tail" area beyond the window. The optimization problem can be solved by finding J vs. t₀And solve the following formula 4 to get t₀To solve:

5J __Λ

[0046] ---0 (Formula 4)

[0047] It can be proved that the t of the symmetric Gaussian LSE pdf shown in FIG. 2₀The best solution is the average of the lower and upper limits of the LSET window:

[0048]% = learn (formula 5)

[0049] For other LSE pdfs, the best may be a positive or negative value, depending on the relative positional relationship between the pdf and the error tolerance window. A positive t₀Means a delay in the audio frame, and a negative t₀This means delaying in the video frame to offset the LSE and maximize Equation 4.

[0050] FIG. 6 illustrates a method for processing audio and/or video data packets according to the present invention. In particular, FIG. 6 describes a method for adapting corresponding video and/or audio frames or access unit data packets to minimize lip synchronization errors and especially advanced audio type lip synchronization errors. In the context of the method of FIG. 6, the lip synchronization error is caused by an error source including one or more of a variable delay network, an encoder, a transmission processor, or other error sources according to block 605.

[0051] In step 610, the instantaneous error that may be generated by the error source is expressed as a probability density function (pdf). For example, as described above with reference to FIG. 5, a pdf that incorporates the instantaneous errors that may be generated by the variable delay network is shown. The pdf includes, for example, a random number distribution with a Gaussian shape (which may or may not be centered on a zero point), where the zero point indicates that there is no lip sync error (that is, instantaneous correction of video and audio data).

[0052] In step 620, an error tolerance window associated with the pdf is defined. As shown in block 615, the error tolerance window may be defined in relation to lip sync error or other errors. As shown in FIG. 5, a delay tolerance window associated with lip synchronization errors is defined as a delay between -20 milliseconds and +40 milliseconds, for example. That is to say, an asymmetric audio delay tolerance (relative to the zero time point) of an audio access unit with a corresponding video access unit ahead of 20 milliseconds or a video data packet lagging 40 milliseconds is considered acceptable of. Other tolerance windows can be defined based on factors related to the communication system using the present invention.

[0053] In step 630, the method adapts timing parameters, such as time stamps, related to at least one of the video and audio frames constituting the content stream. Optionally, one or all of the uncompressed audio and video streams are delayed before encoding. Such modification is performed in a manner that attempts to cause the Pdf related to the error source to shift from the initial position (for example, centered at zero) to the position of the maximum utilization delay tolerance window. It may be noted that the above-mentioned adaptation in block 625 may occur during encoding processing, transmission processing, or other processing. Referring back to Figure 5, an appropriate pdf offset is shown to increase the number of areas under the probability distribution curve within the range determined by the delay tolerance window.

[0054] FIG. 7 depicts a high-level block diagram of a communication system according to an alternate embodiment of the present invention. In particular, the communication system 700 in FIG. 7 and the communication system 100 in FIG. 1 are substantially the same. The main difference is that the delay element 710A is used to delay the audio stream A before the audio encoder 120A encodes the original audio stream. As discussed earlier

Lip sync error tolerance (LSET) model, the delay element 710A generates a delay time t° to the audio stream to offset the corresponding pdfo. It can be noted that the communication system 700 in FIG. 7 can be modified to include a corresponding The video delay element 710V (not shown) is used to delay the video source signal V before it is encoded by the video encoder 120V. One or all of the audio 710A and video 710V delay elements can be used.

[0055] In this embodiment of the present invention, an error tolerance window 410 as shown in FIG. 5 is used. For example, each audio frame or access unit is delayed by about 5 relative to each video frame before encoding. millisecond. By shifting each audio frame back in time by t₀In milliseconds, the pdf associated with the error generated by the variable delay network is shifted in the manner described above with respect to FIG. 5. That is, the forward or backward shift of the pdf depends on Mark 5, from the trend of lip synchronization error with leading audio packets to the trend of no lip synchronization error or lagging audio packet lip synchronization error (it is more than leading Audio data packet lip sync errors are less annoying). Therefore, the probability that any audio packet delay will still remain within the error tolerance established by the error tolerance window 410 increases.

[0056] In an embodiment of the present invention, a symmetric Gaussian pdf as shown in FIG. 2 is assumed, and the time stamp of the audio or video frame is modified to minimize the timing mismatch. For a constant bit rate audio encoder, the video time stamp can optionally be modified in a way to increase the probability of timing mismatches maintained in the LSET at the decoder. In this embodiment, for example, the video mark is rounded to a lower tens of milliseconds, as indicated in Equation 6, as follows (wherein and f: are the original and rounded time mark in milliseconds for the video frame ):

[0057] ?: =/: -(/; modulo 10) (formula 6)

[0058] The above technique introduces a uniformly distributed delay in the range of 0 to 9 milliseconds in audio data packets. Other ranges can also be selected (such as mod 15, mod 20, etc.), and audio data packets can also be processed in this way.

[0059] In the previously described embodiment, the LSE of pdf is known and considered to be relatively stable. As a result, the predetermined time shift is performed on all audio (or video) access units. In a more advanced embodiment, the LSE pdf may not be known or unstable, the LSE pdf is monitored and estimated, and the time offset is not predetermined.

[0060] FIG. 8 illustrates the LSE in the embodiment, in which the pdf estimator is executed at the receiving end. In particular, for example, the receiving end device described in Figs. 1 and 7 above is modified to include the LSE pdf estimator 810 and the audio delay element 820Ao. Although not shown, a video delay element 820V can also be used. The LSE pdf estimator 810 receives the decoded audio A, and video V, and generates a delay indicator signal t in response to the LSET model information_0OIn the embodiment of FIG. 8, the delay indication signal is processed by the audio delay element 820A to the decoded audio stream A, and the corresponding delay amount is transmitted to generate a delayed audio stream A". The estimator 810 continuously Collect the display time stamps of the audio and video access units. Each LSEe is calculated using Equation 1. All LSEs are used to form the LSE (1) by using the LSET model, the best time offset t. Can be solved by Equation 4 finds the time offset t₀Formula to get. Delay in audio frame (5> 0) or video frame (t° <0) is added to offset LSEpdfo

[0061] In one embodiment, a certain optical time shift is transmitted from the receiver to the encoder so that at least one of the audio and video streams to be encoded and transmitted is before encoding, before transmission processing, and/or transmission It is delayed before reaching the receiver.

[0062] Although various embodiments including the teachings of the present invention have been described and shown in detail herein, those skilled in the art can easily design many other various embodiments that also include these teachings. .

CN 1745526 Β