Method and apparatus for lipsync measurement and correction

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of U.S. Provisional Patent Application No. 60/409,346 entitled “METHOD AND APPARATUS FOR LIPSYNC MEASUREMENT AND CORRECTION,” by Leon J. Stanger, filed Sep. 9, 2002, which application is hereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

[0002] 1. Field of the Invention

[0003] The present invention relates to systems and methods for transmitting data, and in particular to a system and method for measuring and correcting for audio to video synchronization errors.

[0004] 2. Description of the Related Art

[0005] In presenting media programs such as films and television programs, it is important that the video portion of the program and the audio portion of the program be adequately synchronized. While digital processes have generally improved both the audio and video portions of media programs, such processes have not improved the synchronization of the video and audio portions of such programs. In fact, many such processes have contributed to additional synchronization errors over and above those typically experienced with analog systems. Although each segment of the system can be properly timed, the end-to-end accumulation of synchronization errors remains a difficult problem to control.

[0006] While systems have been devised to synchronize video and audio portions of media programs, such systems are either not universally effective, or require substantial modification of existing equipment.

[0007] What is needed is a simple, yet effective method and system for synchronizing the video and audio portions of media programs. What is further needed is that this system be backwards compatible and not require substantial modification to existing equipment. The present invention satisfies that need.

SUMMARY OF THE INVENTION

[0008] To address the requirements described above, the present invention discloses a method and apparatus for synchronizing audio and video portions of a media program signal. The method comprises the steps of detecting an audio event in the audio portion of the media program signal, measuring the timing interval from the audio event to a subsequent video synchronization pulse in the video portion of the media program signal, and storing the timing interval in a third portion of the media program signal. The apparatus comprises an audio event detector for detecting an audio event in the audio portion of the media program signal, a timer for measuring the timing interval from the audio event to a subsequent video synchronization pulse in the video portion of the media program signal, and a processor for storing the timing interval in a third portion of the media program signal.

[0009] The present invention identifies naturally occurring timing points within the audio program and marks the timing relationship to the video program in a third portion of the media program signal such as the portion used for closed captioning. After the video has been marked, it is transmitted to a receiver, where it is read and compared to the original timing marks. In one embodiment, the present invention uses the closed caption standard described in EIA 608/708, (e.g. ANSI/EIA/CEA-608-B, which is hereby incorporated by reference herein). Since the closed caption standard, EIA 608 is widely used, it is possible to measure, and correct errors at any point in the distribution chain, even in consumer equipment.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Referring now to the drawings in which like reference numbers represent corresponding parts throughout:

[0011]FIG. 1 is a diagram depicting a media distribution system;

[0012]FIG. 2 is a block diagram depicting one embodiment of a synchronization encoder <b>108</b>;

[0013]FIGS. 3A and 3B are flow charts depicting the synchronization encoding of the media program signal;

[0014]FIG. 4 is a diagram depicting one example of how an audio event can be defined; and

[0015]FIG. 5 is a flow chart illustrating exemplary method steps that can be used to decode and use the synchronization information.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0016] In the following description, reference is made to the accompanying drawings which form a part hereof, and which is shown, by way of illustration, several embodiments of the present invention. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

[0017] In the following description, reference is made to the accompanying drawings which form a part hereof, and which show, by way of illustration, several embodiments of the present invention. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

Video Distribution System

[0018]FIG. 1 is a diagram depicting a media distribution system <b>100</b> employing one embodiment of the present invention. A media program signal <b>102</b> comprising an audio portion or channel <b>102</b>A and a video portion or channel <b>102</b>B is provided to a synchronization encoder <b>108</b>. The video portion <b>102</b>B of the media program signal <b>102</b> includes video segments <b>104</b> having picture information such as video frames and non-video segments <b>106</b>. The non-video segments <b>106</b> include, for example, the closed captioning information in line <b>21</b> of the media program signal, information in the vertical blanking interval (VBI), and/or other information. The synchronization encoder <b>108</b> adds synchronization information in the to the media program signal <b>102</b> to generate a sync-encoded media program signal <b>120</b>. In the illustrated embodiment, the synchronization information (conceptually illustrated by the asterisks <b>122</b> in FIG. 1) is added to the non-video segments <b>106</b> of the video portion <b>102</b>B of the media program signal <b>102</b>. However, the synchronization information can also be added to the video segments <b>104</b> in an unobtrusive manner, for example, by adding the information in the first line of active video in each frame of the video signal <b>102</b>B. Because the overwhelming majority of television sets are not aligned with sufficient precision to show the first video line, the synchronization information is virtually invisible to virtually all viewers.

[0019] The sync-encoded media program signal <b>120</b> is transmitted by transmitter <b>110</b> and received by receiver <b>112</b>. The received sync modified media program signal <b>120</b> is provided to a synchronization decoder <b>114</b>. A synchronization detector <b>115</b> detects the synchronization information in the sync modified media program signal <b>120</b> and uses that information to control a variable delay element <b>118</b> to synchronize the audio <b>124</b>A and video <b>124</b>B channels of the media program signal.

[0020] The sync-encoded media program signal <b>120</b> can be transmitted and received via satellite, cable, the Internet, land line, or other means. For example, the media program signal <b>102</b> may be a live broadcast such as a basketball game. The audio portion <b>102</b>A and video portions <b>102</b>B of the media program <b>102</b> are provided to the synchronization encoder <b>108</b>, and the sync-encoded media program signal <b>120</b> provided via satellite or cable to a signal distributor, where additional program material (e.g. advertisements and other information) is added or the media program is edited, and thenceforth transmitted (again, via satellite, cable, or other means) to media program distributors such as satellite or cable distribution systems. In another embodiment, the media program signal <b>102</b> is a signal provided by a media program distributor. In this embodiment, synchronization information is added to the media program signal as described above to produce the sync-encoded media program signal <b>120</b>, and this signal is transmitted to customers having satellite, cable or broadcast television receivers. The television receivers decode the sync-encoded media program signal <b>120</b> to recover the synchronization information, and correct for audio/video synchronization errors before presenting the media program to the viewer.

[0021] In the illustrated embodiment, the audio portion <b>102</b>A is provided by a microphone, and the video portion <b>102</b>B is provided by a video camera <b>112</b>. However, the synchronization encoder <b>108</b> may also accept a composite signal including both the audio portion <b>102</b>A and video portion <b>102</b>B. In this case, the synchronization decoder decodes the audio and video portions <b>102</b>A, <b>102</b>B of the media program signal <b>102</b> before encoding the synchronization information.

[0022] The functionality implemented in the blocks depicted in FIG. 1 can be implemented by one or more hardware modules, one or more software modules defining instructions performed by a processor, or a combination of both.

Synchronization Encoding

[0023]FIG. 2 is a block diagram depicting one embodiment of a synchronization encoder <b>108</b>. FIG. 2 will be discussed with reference to FIGS. 3A and 3B, which present flowcharts describing exemplary process steps that can be used to practice one embodiment of the invention.

[0024] Referring to FIG. 3A, the audio portion of the incoming media program signal is examined to detect an audio event. This is shown in block <b>302</b>. This can be performed, for example, by the audio event detector <b>202</b> shown in FIG. 2. The audio event can be any identifiable pattern in the audio portion <b>102</b>A or the media program signal <b>102</b>. In one embodiment, the audio event comprises a period of silence followed by an abrupt increase in audio signal intensity. This characteristic is hereinafter termed a “start of sentence” or SoS, but need not be in fact, the start of a sentence.

[0025] In another embodiment, the audio event comprises a significant increase in the signal intensity of the low frequency portion of the audio spectrum, such as a drum beat. In another embodiment, the audio event comprises a significant increase in the signal intensity of a high-frequency portion of the audio spectrum, such as a cymbal crash. The audio event can also be determined from more than one measure. For example, two audio sub-events occurring at the same time, such as a musical chord having multiple notes played at the same time. The audio event can also be determined using speech processing software algorithms which identify words or phrases, particularly those associated with the start of a sentence.

[0026] In one embodiment, the audio event detector <b>202</b> comprises a voice detector which detects the start of a sentence in the audio portion of the signal. Next, synchronization information such as a timing interval from the audio event to a datum is measured, as shown in block <b>304</b>. This can be performed, for example, by the timer <b>204</b> shown in FIG. 2. In one embodiment, the datum is the video synchronization pulse in the video portion <b>102</b>B of the media program signal <b>102</b>. The timing interval is then stored in a third portion of the media program signal, as shown in block <b>306</b>.

[0027] In one embodiment, the synchronization information is stored in a portion of the non-video segment <b>106</b> of the video portion <b>102</b>B of the media program signal <b>102</b>. For example, the synchronization information may be stored in the portion of the media program signal <b>102</b> ordinarily dedicated to transmitting closed captioning information (e.g. line <b>21</b>), as described further below. The synchronization information may also be transmitted in other lines (e.g. line 20), or even in part of a video segment <b>104</b>. For example, the vast majority of televisions are aligned so that many of the video lines around the edges of the picture are not displayed (this prevents alignment errors from causing black stripes on the periphery of the video image). The synchronization information can therefore be stored in one of the video lines near the periphery of the image without disturbing the presented video image in the overwhelming majority of television receivers.

[0028]FIG. 3B is a flow chart illustrating exemplary process steps that can be used to detect the audio event in the audio portion of the media program signal <b>102</b>. The audio portion of the media program signal may be a stereo signal with right and left components or a multi-channel coded signal. To minimize the impact of multi-channel effects on the determination of the audio event, a monaural version of the audio portion of the media program signal may be generated, as shown in block <b>308</b>. While not necessary, this can improve the accuracy of the audio event determination. In surround sound applications, the back channel signals are mixed to LT and RT, and a monaural channel is created by summing the LT and RT signals.

[0029] Next, the audio component of the media program signal is bandpass-filtered to a range. In embodiments in which the audio event is based on vocal information, the bandpass filter approximates that of the vocal bandpass range for human being. This is shown in block <b>308</b> and can be performed, for example, by the bandpass filter <b>208</b> shown in FIG. 2. In one embodiment, this is performed by a Gaussian bandpass filter having—3 dB points at 500 and 2200 Hz. The selection of this bandpass region passes most of the human voice, but rejects most system noise.

[0030] The audio level of the audio portion of the media program signal is then normalized, as shown in block <b>312</b>. This can be accomplished, for example, by the automatic gain control (AGC) module <b>210</b> shown in FIG. 2. The attack and decay time of the AGC can be varied to optimize performance. The signal is then rectified and low pass filtered, as shown in blocks <b>314</b> and <b>316</b>, respectively. These operations can be performed by the rectifier <b>212</b> and low pass filter (LPF) <b>214</b> shown in FIG. 2. In embodiments using vocal audio events (including one using the start of sentence as the audio event), the resulting pre-processed signal is applied to a silence interval detector (SID) <b>216</b> which determines a silence interval t_s, during which the pre-processed audio signal has an amplitude below the AGC threshold and identifies the end of the silence interval as the start of sentence time if the silence interval ts exceeds a silence interval threshold t_s_{<sub2>thresh</sub2>}.

[0031]FIG. 4 is a diagram illustrating one example of how a start of sentence (SoS) period can be defined. In this embodiment, a SoS period is defined as a period of silence followed by a syllable (or abrupt increase in audio level).

[0032] To reduce errors, the start of the sentence period is defined according to the set of rules. These rules make use of characterizations of the processed signal as being (1) silent, or a (2) syllable. The processed signal can be characterized as “silent” when the amplitude of the processed signal is below a silence threshold T_silence. In one embodiment, the silence threshold T_silenceis 26 dB below the AGC threshold. The processed signal can be characterized as a syllable when the processed signal magnitude is above a syllable threshold T_syllable.

[0033] It is also useful to define a silence interval Δt_silence<b>404</b> of the processed signal as a time interval when the processed signal remains silent for a minimum time period min Δt_silence. For example, in the embodiment illustrated in FIG. 4D, the signal is deemed to be silent when the processed signal remains at least 20 dB below the AGC threshold for at least 20 milliseconds. Time interval <b>404</b> represents an exemplary silent period.

[0034] From these definitions of “silence,” a “syllable” and a “silent period,” the rules for determining a SoS can be defined. In the exemplary embodiment shown in FIG. 4, a SoS is identified when the processed signal is at or below the silence threshold T_silencefor a minimum time period Δt_silence<b>404</b>, followed by an increase in the amplitude of the processed signal to at least the syllable threshold T_syllablewithin a syllable rise time Δt_syllable<b>406</b>. In addition, a valid SoS must have no false SoS designations in the within a period Δt_false<b>402</b> of the onset of the silence period Δt_silence. A false SoS designation is one in which there is Δt_silenceof silence followed by a syllable, but the processed signal does not violate other timing constraints. An examples of one such timing constraint is a constraint which requires that there are no similarly identified events within a preceding time period (e.g. no SoS designations within a preceding period). Finally, once a valid SoS has been identified and an SoS timing reference <b>410</b> marked, another SoS is not identified for a minimum period of Δt_{{overscore (SoS)}}<b>408</b>. In the embodiment illustrated in FIG. 4D, representative values for Δt_false<b>402</b>, Δt_silence<b>404</b>, Δt_syllable<b>406</b>, and Δt_{{overscore (SoS)}}<b>408</b> are &gt;250 milliseconds, &gt;20 milliseconds, &lt;2 milliseconds, and &gt;250 milliseconds, respectively.

[0035] After the SoS timing reference <b>410</b> is found, a timing interval Δt_SoS-sync<b>414</b> from the SoS timing reference <b>410</b> to the vertical sync pulse t_v_{<sub2>sync </sub2>}<b>412</b> is determined. Vertical sync pulses are described in “Digital Television Fundamentals, by Michael Robin and Michel Poulin, 1998, which is hereby incorporated by reference herein.

[0036] In one embodiment, the reference timing value is determined by rounding to an integer number of milliseconds. Typically, a valid range for the timing interval Δt_SoS-sync<b>414</b> is approximately zero to 17 milliseconds for 30 frame systems, and zero to 20 milliseconds for 25 frame systems. It is noted that the foregoing describes only one embodiment of the present invention. Given the teaching above, it is possible to define an operational system using different values for the parameters above. It should also be noted that the steps of generating a monaural signal, bandpass filtering to a vocal bandpass range, normalizing the signal with AGC, rectifying and further low pass filtering the signal are operations which improve the accuracy of the determination of the SoS, but may not be required to practice the present invention in some embodiments. For example, the start of the sentence can be determined by digital algorithmic techniques that do not involve low pass filtering. Further, while the combination of the multi-channel audio signal to a single monaural signal simplifies the design and operation of the system, the information on such multiple channels may instead be used to improve the start of sentence determination.

[0037] After the reference timing value is determined and rounded, it is stored in a third portion of the media program signal. In one embodiment this third portion is the captioning portion of the media program signal.

[0038] In the NTSC format, line <b>21</b> of the vertical blanking interval (VBI) is allocated to carry two bytes of closed captioning information. Closed captioning information can be included on line <b>21</b> of the first field of each video frame and/or on line <b>21</b> of the second field of the video frame. The first field of the video frame includes CC1, CC2, Text 1 and Text 2 channels and the second field includes CC3, CC4, Text 3 and Text 4, and XDS channels.

[0039] In one embodiment, a plurality of special non-printing characters can be used to represent the timing values. For example, a timing interval Δt_SoS-syncof 0 msec can be represented by a first non-printing character, a timing interval Δt_SoS-syncof 10 msec can be represented by a second non-printing character, and so on.

[0040] In this embodiment, the maximum rate at which the timing interval values Δt_SoS-sync<b>414</b> can be stored is approximately one every 500 milliseconds. It is preferable to retain any existing closed-captioning information in the signal. Hence, in a preferred embodiment, the timing interval value Δt_SoS-sync<b>414</b> is transmitted in the form of two special characters. If there is CC1, CC3, Text 1 or Text 2 data already present on line <b>21</b>, the timing interval value Δt_SoS-sync<b>414</b> can be discarded, and the CC1 or CC3 data already present on line <b>21</b> is left undisturbed. If line <b>21</b> of frame <b>2</b> already includes closed captioning information (CC2, CC4, Text 3, Text 4, XDS data), this data may be delayed until the next frame, and the timing interval value Δt_SoS-sync<b>414</b> inserted in its place. Alternatively, the closed captioning information may be simply overwritten.

[0041] In another embodiment, the synchronization information is written using printing characters, but is written to closed captioning portions that are not ordinarily viewed by the user. For example, if the closed captioning information is presented in the CC1 channel, any characters can be used to store the synchronization information in the CC2, CC3, CC4, Text 1 or Text 2 channels, even by simply writing the synchronization information values directly. Of course, in this embodiment, if the viewer were to command the television receiver to display CC2 closed-captioning information, this information would be displayed in a closed captioning window, and may appear as random characters or numbers. However, the original closed captioning information is still available and viewable, so long as the user commands the television receiver to display CC1 closed-captioning information.

[0042] The estimate of when the SoS occurs can be improved by statistical analysis of the audio and/or video waveforms. Such techniques reduce the probability of errors in the SoS determination caused by imperfect audio measuring equipment and audio distortion. Such statistical analysis can be accomplished by generating a histogram of the results. Correct SoS data points are likely to be closely clustered together, while erroneous results are likely to be widely scattered. Using such analysis, such widely scattered timing references can be disregarded.

[0043] The foregoing describes a system and method wherein the third portion of the media program signal is the closed captioning information that is transmitted in the VBI interval in an NTSC signal. However, the present invention can also be practiced with current and evolving digital television standards such as EIA-708. EIA-708 offers an enhanced character set with more accented letters and non-English letters, and more special symbols, and higher bandwidth, as well as other improvements. It is also noted that while the closed captioning may be available in earlier embodiments of MPEG encoding of NTSC signals using captioning encoders, and that current and planned MPEG encoding make specific allowances to transmit closed captioning information. Hence, the present invention may be practiced by storing the timing reference information in the closed captioning portions of the media program signal supported by these formats.

[0044] Lipsync timing errors tend to be fixed or to move very slowly. By accumulating data over a period of time and discarding erroneous samples, the timing error can be determined and the synchronization between the video and the audio portions of the media program can be re-established and maintained.

Synchronization Decoding

[0045]FIG. 5 is a flow chart illustrating exemplary method steps that can be used to decode and use the synchronization information. Synchronization information such as a timing interval disposed in a third portion of a media program signal is read, as shown in block <b>502</b>. The audio and video portions of the media program can then be synchronized using the timing interval, as shown in block <b>504</b>.

[0046] In one embodiment, the foregoing steps are performed by the synchronizing detector <b>116</b> and the synchronizer <b>118</b> of the synchronizing decoder <b>114</b> shown in FIG. 1. The sync-encoded media program signal <b>120</b> is provided to the synchronizing detector <b>116</b>, which reads and processes the timing interval in a third portion of the media program signal. A communicatively coupled synchronizer <b>118</b> accepts the timing interval uses the timing interval to synchronize the audio and video portions of the media program. Typically, this synchronization is performed by delaying or advancing the audio portion of the media program relative to the video portion of the media program using audio delay element <b>118</b>A. However, the video portion of the media program can be delayed or advanced using video delay element <b>118</b>B, while leaving the audio portion timing untouched, if desired. Further, both the audio and video portions of the media program can be adjusted, if desired.

[0047] In one embodiment, the synchronizing detector <b>116</b> processes the timing interval in the third portion of the media program signal by generating a second timing reference, using techniques analogous to those outlined above. A valid value for the second timing reference is between −200 milliseconds and +217 milliseconds for 30 frame systems, and between −200 and +220 milliseconds for 25 frame systems. The second timing reference is compared to the timing reference Δt_SoS-sync<b>474</b> read from the third portion of the media program signal. The difference between the first and second timing references represents a lipsync error. Typically, the lipsync error is within ±200 milliseconds. The synchronizer <b>552</b> adjusts the timing of the video or audio portions of the media program signal (or both) to account for this lipsync error.

Conclusion

[0048] This concludes the description of the preferred embodiments of the present invention. The foregoing description of the preferred embodiment of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. For example, it is noted that the functions depicted and described in the foregoing disclosure can be implemented by one or more hardware modules, one or more software modules defining instructions performed by a processor, or a combination of both as desired.

[0049] It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto. The above specification, examples and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.