Method of audio-video synchronization

FIELD OF THE INVENTION

0001The present invention relates to audio and video generally and, more particularly, to a method for audio-video synchronization.

BACKGROUND OF THE INVENTION

0002Audio-visual (A/V) synchronization errors as short as plus or minus one-half film frame can be detected by most film editors. Since film is projected at 24 frames per second (fps) in the U.S. and <b>25</b> fps in Europe, one-half film frame equates to approximately +/−20 msec. Similarly, plus or minus one video frame corresponds to +/−33-40 msec.

0003The acceptable range in which audio-video synchronization can vary is non-symmetrical because human perception of audio-video synchronization is weighted more in one direction than the other. Because light travels much faster than sound, events are usually seen before the accompanying sound is heard. For example, simultaneously seeing and hearing a basketball hitting the court in a large sports venue appears relatively correct to the first few rows. However, the further back a viewer is located, the more the sound of the ball hitting the floor lags behind the sight of the ball hitting the floor. Even though the amount the sound lags increases the further back the viewer is located, the difference is perceived as natural.

0004However, if the audio-video timing was reversed, a spectator watching a basketball game would hear the sound of the ball hitting the floor before the ball appeared to make contact with the floor. The arrival of the sound of the ball hitting the floor before the ball appears to have hit the floor would be a very unnatural experience. The discrepancy would seem incorrect even to spectators in the first few rows where the amount of audio-visual synchronization error would be small because the error is in the “wrong” direction. In particular, human perception is much more forgiving for sound lagging behind sight.

0005International Telecommunications Union (ITU) recommendation ITU-R BT.1359-1 (1998) was based on research that showed the reliable detection of audio-video synchronization errors fell between 45 msec for audio leading video and 125 msec for audio lagging behind video. The recommendation states that the tolerance from the point of capture to the viewer and or listener should be no more than 90 msec for audio leading video to 185 msec for audio lagging behind video. The Advanced Television Systems Committee (ATSC) Implementation Subcommittee (<b>1</b>S) issued a finding (Doc. IS-191 (Jun. 23, 2003)) recommending a tolerance of +/−15 msec.

0006Conventional solutions synchronize audio and video by setting the audio as master and dropping or repeating frames of video to synchronize the two signals. However, dropping and repeating video frames can affect the quality of the video image presented.

0007A method of synchronizing audio and video without affecting the quality of the video presented would be desirable. It would also be desirable to switch seamlessly between trick play modes (e.g., x1.5 playback) and normal (e.g., x1) modes without stopping the playback and/or going through a full handshake procedure to avoid a gap in either or both the audio and the video.

SUMMARY OF THE INVENTION

0008The present invention concerns a method for synchronization of an audio stream and a video stream comprising the steps of (A) determining a first presentation time stamp from the video stream and generating a first handshake signal when the video stream is ready to be transmitted, (B) repeating and dropping one or more audio frames of the audio stream in response to the first handshake signal and a plurality of first predetermined threshold values until a second presentation time stamp from the audio stream matches the first presentation time stamp and generating a second handshake signal when the audio stream is ready to be transmitted and (C) transmitting the video stream and the audio stream in response to the second handshake signal.

0009The objects, features and advantages of the present invention include providing a method and/or apparatus for audio-video synchronization that may (i) switch between modes without stopping playback (ii) use video as a master during synchronization, (iii) synchronize multiple audio streams to a video stream, (iv) synchronize an analog audio stream to a digital audio stream, (v) provide hysteresis in the synchronization of audio and video, (vi) maintain alignment of audio and video by use of a multi-level synchronization correction feedback loop with hysteresis on switching between levels and/or (vii) provide hysteresis on switching between unmute and mute states.

BRIEF DESCRIPTION OF THE DRAWINGS

0010These and other objects, features and advantages of the present invention will be apparent from the following detailed description and the appended claims and drawings in which:

0011<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram illustrating an audio-video system in accordance with a preferred embodiment of the present invention;

0012<figref idref="DRAWINGS">FIG. 2</figref> is a more detailed block diagram illustrating a decoder block of <figref idref="DRAWINGS">FIG. 1</figref>;

0013<figref idref="DRAWINGS">FIG. 3</figref> is a timing diagram illustrating an audio-video bitstream;

0014<figref idref="DRAWINGS">FIG. 4</figref> is a timing diagram illustrating an example audio frame;

0015<figref idref="DRAWINGS">FIG. 5</figref> is a state diagram illustrating an example synchronization operation in accordance with a preferred embodiment of the present invention;

0016<figref idref="DRAWINGS">FIG. 6</figref> is a more detailed block diagram illustrating a preferred embodiment of the present invention;

0017<figref idref="DRAWINGS">FIG. 7</figref> is a timing diagram illustrating example thresholds for providing hysteresis in a synchronization decision;

0018<figref idref="DRAWINGS">FIG. 8</figref> is a timing diagram illustrating a plurality of hysteresis thresholds in accordance with a preferred embodiment of the present invention;

0019<figref idref="DRAWINGS">FIG. 9</figref> is a state diagram illustrating a mute decision in accordance with the present invention;

0020<figref idref="DRAWINGS">FIG. 10</figref> is a block diagram illustrating an alternative embodiment of the present invention; and

0021<figref idref="DRAWINGS">FIG. 11</figref> is a block diagram illustrating another alternative embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

0022The present invention generally provides an audio and video decoder that may use video as master and audio as slave. The use of video as master generally maintains the quality of the video. In some cases, however, using video as master may not be easy because of difficulty in maintaining accurate points on both audio and video streams. For example, accurate points may be difficult to maintain in a DVD video playback system operating at 1.5 times normal speed. For example, with audio as slave, if the sync offset is large enough to require frame add/drop, the drops may be audible. The present invention may be configured to provide modes that may operate as x1 playback where the video is the master and modes, such as x1.5, where the video is the slave.

0023The present invention may use an audio master operating at x1.5 for a rough synchronization and go straight into the x1 audio slave fine synchronization state without muting the audio. The present invention may use a process directly analogous to the process for fine synchronization after a rough synchronization in a handshake mode (described below in connection with <figref idref="DRAWINGS">FIG. 5</figref>).

0024Referring to <figref idref="DRAWINGS">FIG. 1</figref>, a block diagram of a circuit <b>100</b> in accordance with a preferred embodiment of the present invention is shown in the context of an audio-video transmission system. The circuit <b>100</b> may be implemented as a receiver/decoder. The circuit <b>100</b> may be configured to maintain audio-video synchronization by changing a rate of data consumption. In one example, the rate of data consumption may be changed even though the audio is unmuted. The circuit <b>100</b> may be further configured to fine tune the audio-video synchronization by repeating and/or dropping audio frames and/or audio samples. The circuit <b>100</b> may be further configured to implement hysteresis in the decisions to repeat and/or drop audio frames and/or samples. The hysteresis may be implemented with a plurality of thresholds that control when audio frames and/or samples are repeated and/or dropped.

0025The circuit <b>100</b> may have an input <b>102</b> that may receive an encoded bitstream (e.g., BITSTREAM), an output <b>104</b> that may present a signal (e.g., VIDEO_OUT) and an output <b>106</b> that may present a signal (e.g., AUDIO_OUT). The signal BITSTREAM may be received from any of a number of transmission media or sources <b>108</b> (e.g., DVD player, satellite box, set-top-box, network, over-the-air broadcast, etc.). The signal BITSTREAM may be encoded in accordance with any appropriate encoding and/or compression standard (e.g., MPEG-2, MPEG-4, H.264, etc.).

0026The signal VIDEO_OUT may comprise one or more video signals. For example, the signal VIDEO_OUT may comprise in one embodiment an analog video signal. In another example, the signal VIDEO_OUT may comprise an analog video signal and a digital video signal. Alternatively, the signal VIDEO_OUT may comprise a main picture signal and a sub-picture signal. However, other video signals may be implemented as part of the signal VIDEO_OUT to meet the design criteria of a particular application.

0027The signal AUDIO_OUT may comprise one or more audio signals. In one example, the signal AUDIO_OUT may be implemented as an analog audio signal. In another example, the signal AUDIO_OUT may be implemented as a multichannel audio signal. For example, the signal AUDIO_OUT may be implemented as a 5.1 channel audio output. The signal AUDIO_OUT may comprise, for example, both analog audio and digital audio signals (e.g., an IEC60958 signal, IEC61937 data streams, etc.). The IEC61937 data streams may comprise multichannel sound (e.g., MPEG-2, AC-3, DTS, etc.). When the signal AUDIO_OUT includes an IEC61937 digital audio signal, the bits carrying audio samples may be replaced with data bits from the datastream and headers of, for example, a Sony/Philips digital interface format (S/PDIF) signal.

0028In one example, the signal BITSTREAM may be generated by an encoder circuit <b>110</b>. The encoder circuit <b>110</b> may have an input <b>112</b> that may receive a signal (e.g., VIDEO_IN), an input <b>114</b> may receive a signal (e.g., AUDIO_IN), and an output <b>116</b> that may present the signal BITSTREAM. The signal VIDEO_IN may comprise one or more analog video signals. The signal AUDIO_IN may comprise one or more analog audio signals. Alternatively, the signals VIDEO_IN and AUDIO_IN may comprise digital signals.

0029In one example, the encoder <b>110</b> may comprise a block <b>120</b>, a block <b>122</b>, a block <b>124</b>, a block <b>126</b>, a block <b>128</b>, a block <b>130</b>, a block <b>132</b>, and a block <b>134</b>. The block <b>120</b> may be implemented as a video digitizer. The block <b>122</b> may be implemented as an audio digitizer. The block <b>124</b> may be implemented as a video frame buffer. The block <b>128</b> may be implemented as an audio frame buffer. The block <b>126</b> may be implemented as a presentation (or system) clock source. The block <b>126</b> may be configured, for example, to generate Society of Motion Picture and Television Engineers (SMPTE) time codes. The block <b>130</b> may be implemented as a video compression block. The block <b>132</b> may be implemented as an audio compression block. The block <b>134</b> may be implemented as a multiplexing block.

0030The block <b>120</b> may be configured to generate a digital (e.g., sampled) video signal (e.g., DV) in response to the signal VIDEO_IN. The block <b>122</b> may be configured to generate a digitized (e.g., sampled) audio signal (e.g., DA) in response to the signal AUDIO_IN. However, when one or more of the signals VIDEO_IN and AUDIO_IN are implemented as digital signals, one or both of the circuits <b>120</b> and <b>122</b> may be omitted.

0031The digitized video signal DV may be presented to an input of the block <b>124</b>. The block <b>124</b> may have a second input that may receive a presentation time-stamp (e.g., PTS_V) from the circuit <b>126</b>. The digitized audio signal DA may be presented to an input of the audio frame block <b>128</b>. The audio frame block <b>128</b> may have a second input that may receive a presentation time-stamp (e.g., PTS_A) from the circuit <b>126</b>. In general, the circuits <b>124</b> and <b>128</b> may be configured to associate a presentation time-stamp with each video frame and each audio frame, respectively. For example, each video frame and each audio frame may be configured to have a header portion containing an associated presentation time-stamp.

0032The circuit <b>124</b> may present one or more video frames and the associated presentation time-stamps (e.g., PTS_V) to an input of the video compression block <b>130</b>. The video compression block <b>130</b> may present a compressed video signal (e.g., CV) to a first input of the circuit <b>134</b>. The circuit <b>128</b> may present one or more audio frames and the associated presentation time-stamps (e.g., PTS_A) to an input of the audio compression block <b>132</b>. The audio compression block <b>132</b> may present a compressed audio signal (e.g., CA) to a second input of the circuit <b>134</b>. The audio and video signals may be compressed using any appropriate compression scheme (e.g., MPEG-2, MPEG-4, H.264, etc.).

0033The circuit <b>134</b> is generally configured to generate the signal BITSTREAM in response to the compressed video signal CV and the compressed audio signal CA. In one example, the circuit <b>134</b> may have another input configured to receive a compressed sub-picture signal (e.g., ENCODED SUBPICTURE). The compressed subpicture signal may be generated similarly to the compressed video and audio signals CV and CA. In one example, the compressed video signal, the compressed audio signal and the compressed subpicture signal may be time division multiplexed (TDM) in the signal BITSTREAM. The signal BITSTREAM may be transmitted or stored via the medium <b>108</b> for subsequent presentation.

0034Referring to <figref idref="DRAWINGS">FIG. 2</figref>, a more detailed block diagram of the circuit <b>100</b> is shown. In one example, the circuit <b>100</b> may comprise a circuit (or block) <b>136</b> and a circuit (or block) <b>138</b>. The circuit <b>136</b> may be configured to receive the signal BITSTREAM. The circuit <b>136</b> may be further configured to present audio and video signals and associated time stamps recovered from the signal BITSTREAM. In one example, the circuit <b>136</b> may be configured to present a first signal (e.g., VIDEO), a second signal (e.g., PTS_V), a third signal (e.g., SUBPICTURE), a fourth signal (e.g., PTS_SP), a fifth signal (e.g., AUDIO) and a sixth signal (e.g., PTS_A). The signals VIDEO, SUBPICTURE and AUDIO generally comprise digital signals. The signals PTS_V, PTS_SP AND PTS_A generally comprise presentation time-stamps associated with the signals VIDEO, SUBPICTURE and AUDIO, respectively. The circuit <b>136</b> may be implemented using conventional approaches.

0035The block <b>138</b> may be implemented in one example, as an audio-video (A/V) synchronization block. The block <b>138</b> may be further configured to composite sub-picture and video (e.g., main picture) information into a single (e.g., composite or blended) video output signal. The block <b>138</b> may be configured to synchronize the video and audio signals in response to the video and audio presentation time-stamps (e.g., PTS_V, and PTS_A, respectively) from the block <b>136</b>. However, in an alternative embodiment, the block <b>138</b> may be further configured to recover the presentation time stamps PTS_V and PTS_A from the decoded audio and video signals. The block <b>138</b> is generally configured to use the video signal as master and adjust the audio signal until synchronization is obtained. However, the block <b>138</b> may be configured to use other signals (e.g., a digital audio signal) as master instead of the video signal. In one example, the block <b>138</b> may be configured to repeat and/or drop audio frames and/or audio samples during the synchronization process.

0036The block <b>136</b> may comprise a block <b>140</b>, a block <b>142</b>, a block <b>144</b>, a block <b>146</b>. The block <b>140</b> may be implemented, in one example, as a demultiplexer block. The block <b>142</b> may be implemented, in one example, as a video decoder block. The block <b>144</b> may be implemented, in one example, as a subpicture decoder block. The block <b>146</b> may be implemented, in one example, as an audio decoder block.

0037The signal BITSTREAM may be presented to an input of the block <b>140</b>. The block <b>140</b> may have a first output that may present a video bitstream to an input of the block <b>142</b>, a second output that may present a subpicture bitstream to an input of block <b>144</b> and a third output that may present an audio bitstream to an input of the block <b>146</b>. The block <b>142</b> may be configured to decode the video bitstream into the signal VIDEO and a presentation time-stamp PTS_V. The block <b>144</b> may be configured to generate the signal SUBPICTURE and the presentation time-stamp PTS_SP in response to the compressed subpicture bitstream. The block <b>146</b> may have a first output that may present the decoded audio stream AUDIO and a second output that may present the presentation time-stamp PTS_A for the audio frames of the decoded audio stream.

0038The circuit <b>138</b> may comprise a circuit <b>148</b> and a circuit <b>150</b>. In one example, the circuit <b>148</b> may be implemented as a composition and synchronization circuit (or block). The circuit <b>150</b> may be implemented, in one example, as an audio-video (A/V) synchronization block. The signals VIDEO, PTS_V, SUBPICTURE and PTS_SP may be presented to the block <b>148</b>. The block <b>148</b> may be configured to generate a blended or composite video signal (e.g., COMP. VIDEO) in response to the signals VIDEO, PTS_V, SUBPICTURE and PTS_SP. The circuit <b>148</b> generally synchronizes the signals VIDEO and SUBPICTURE using the signals PTS_V and PTS_SP. The signal COMP. VIDEO is generally synchronized to the signal VIDEO as master. The circuit <b>148</b> may be configured to present the signal COMP. VIDEO and the presentation time stamp PTS_V to the circuit <b>150</b>.

0039The circuit <b>150</b> may be further configured to receive the signal AUDIO and the signal PTS_A. The circuit <b>150</b> may be configured to generate the signals VIDEO_OUT and AUDIO_OUT in response to the signals COMP. VIDEO, PTS_V, AUDIO, and PTS_A. The circuit <b>150</b> generally synchronizes the signals COMP. VIDEO and AUDIO using COMP. VIDEO as master. The circuits <b>148</b> and <b>150</b> generally implement a handshake scheme between the signals being synchronized. The circuits <b>148</b> and <b>150</b> may implement hysteresis with a number of thresholds in the synchronization process. The block <b>148</b> may be omitted when a single video signal is implemented.

0040Referring to <figref idref="DRAWINGS">FIG. 3</figref>, a timing diagram <b>160</b> is shown illustrating an example portion of the signal BITSTREAM. The signal BITSTREAM may comprise audio information (e.g., a compressed or uncompressed version of the original analog audio), video information (e.g., a compressed, or uncompressed for high-end professional applications, version of the original analog video), subpicture information (e.g., a data stream that may include subtitles and/or similar information), timing information, copyright information, navigation information, program information, etc. The timing information generally comprises a time-stamp in each audio frame and each video frame that represents the time at which the encoder generated the video and audio data. Audio and video frames that are substantially simultaneously presented by, for example, a camera are generally tagged with time-stamps (e.g., PTS_A and PTS_V, respectively) having the same value. For example, time-stamps PTS_A<b>1</b> and PTS_V<b>1</b> have the same value. Similarly, time-stamps PTS_A<b>2</b> and PTS_V<b>2</b> have the same value.

0041However, when the audio and video frames are multiplexed in the signal BITSTREAM, the audio and video information is not presented simultaneously. Rather, each of the audio and video frames are presented serially. The timing information associated with the audio and video data is generally inserted in-band (e.g., on the same channel or path as the data). For example, the signal BITSTREAM may comprise a concatenation of packets, each packet starting with a header that includes (i) a startcode to enable the demultiplexer to synchronize with the bitstream, (ii) the type of the packet payload, (iii) the time-stamp, if any, associated with the payload and (iv) the size of the payload in the particular packet. The decoder is generally configured to ensure that the frames in the signal BITSTREAM that are tagged with the same time-stamp values are presented simultaneously.

0042Referring to <figref idref="DRAWINGS">FIG. 4</figref>, a timing diagram <b>170</b> illustrating an example audio frame is shown. An audio frame generally comprises a number of audio samples. In one example, the audio frame (e.g., an AC-3 syncframe) may contain 1536 audio samples, as well as partially represent 1792 samples.

0043Referring to <figref idref="DRAWINGS">FIG. 5</figref>, a state diagram <b>200</b> is shown illustrating various states of an audio-video synchronization process in accordance with a preferred embodiment of the present invention. The synchronization process may comprise a state <b>202</b>, a state <b>204</b>, a state <b>206</b>, a state <b>208</b>, a state <b>210</b>, a state <b>212</b>, a state <b>214</b>, and a state <b>216</b>. The states <b>202</b>-<b>206</b> generally comprise video processing states. The states <b>208</b>-<b>216</b> generally comprise audio processing states. In one example, the interrelationship of the states <b>202</b>-<b>206</b> and <b>208</b>-<b>216</b> may implement a handshake scheme between audio processing and video processing.

0044The state <b>202</b> may represent an idle video state. The state <b>204</b> may represent a ready video state. The state <b>206</b> may represent a running video state. The state <b>208</b> may represent a state in which the audio is waiting for the video as master. The state <b>210</b> may represent a state in which the audio is waiting for coarse synchronization with the video. The state <b>212</b> may represent a state in which an audio buffer is being filled. The state <b>214</b> may represent a state in which the audio is running. In the state <b>214</b>, audio samples may be repeated and/or dropped to bring the audio signal into fine synchronization with the video signal. The state <b>216</b> generally represents a state in which audio and video are considered synchronized within a predetermined tolerance. Although the synchronization process is shown with video as master, another signal (e.g., a digital audio signal) may be used as master with the process.

0045The video portion of the synchronization process generally moves from the state <b>202</b> to the state <b>204</b> in response to a start presentation command (e.g., START_PRESENTATION_MESSAGE). In the state <b>204</b>, the video is generally indicated as being ready (e.g., a first handshake signal may be generated) and a time-stamp variable (e.g., START_PTS) may be set to a pre-determined value for the time-stamp (e.g., PTS_V) of the video signal. The synchronization process may move from the state <b>204</b> to the state <b>206</b> in response to the audio becoming ready (e.g., roughly synchronized to the video). For example, a second handshake signal may be implemented indicating when audio is ready.

0046The audio portion of the synchronization process may move from the state <b>208</b> to the state <b>210</b> in response to a video frame being ready (e.g., receipt of the first handshake signal). The synchronization process may move from the state <b>210</b> to the state <b>212</b> in response to a value of the audio presentation time-stamp (e.g., PTS_A) matching the value of the video presentation time-stamp (e.g., START_PTS). The synchronization process may move from the state <b>212</b> to the state <b>214</b> when an audio buffer has reached a predetermined fill point (e.g., half full, etc.). The second handshake signal may be generated (or asserted) when the synchronization process is in the state <b>214</b>. However, other conditions may be implemented accordingly to meet the design criteria of a particular implementation. The synchronization process may monitor a difference between the synchronization of the video and the audio signals. When the synchronization difference (e.g., |Δ|, or D) is less than a predetermined threshold (e.g., THRESHOLD), the synchronization process may move from the state <b>214</b> to the state <b>216</b>.

0047Referring to <figref idref="DRAWINGS">FIG. 6</figref>, a block diagram of a circuit (or block) <b>300</b> is shown illustrating an example synchronization block in accordance with the present invention. The block <b>300</b> may be implemented as part of the block <b>148</b> and/or the block <b>150</b>, described above in connection with <figref idref="DRAWINGS">FIG. 2</figref>. The block <b>300</b> may be implemented in hardware, software (or firmware) or a combination of hardware and software. In one embodiment, the circuit <b>300</b> may comprise a circuit (or block) <b>301</b>, a circuit (or block) <b>302</b>, a circuit (or block) <b>304</b> and a circuit (or block) <b>306</b>. The block <b>301</b> may be implemented, for example, as a sequencer (or state machine or process). The block <b>302</b> may be implemented, for example, as a flow clock controller (or process). The block <b>304</b> may be implemented, in one example, as a video process (or processor) The block <b>306</b> may be implemented, in one example, as an audio process (or processor). In one example, the flow clock controller <b>302</b> may represent an interface used by the video process <b>304</b> and the audio process <b>306</b> to communicate with each other. The flow clock controller <b>302</b> generally does not imply a call-back mechanism or a separate thread of control.

0048Initially, audio and video flows may be stopped. Both audio and video generally (i) hold until the video output has received a first new frame, (ii) the audio output has dropped/repeated one or more frames to approximately match the audio PTS (e.g., PTS_A) with the video PTS (e.g., PTS_V) of the first new video frame (e.g., coarse synchronization) and (iii) the audio output has filled, for example, half of a buffer with audio data. When both audio and video are ready, both are enabled to start transmitting data. In one example, a detailed synchronization sequence may be described by the following psuedo code.

0049A flow clock control structure S may be defined with elements {START_PTS (PTS64 format), VIDEO READY, AUDIO READY}. At initialization and after dvdplayback stop( ) {START_PTS=INVALID, VIDEO READY=TRUE, AUDIO READY=TRUE}. The set of parameters above generally ensures the previously implemented audio-video synchronization behavior may be used for loopthrough, etc. At dvdplayback start( ), the flow clock controller <b>302</b> may be configured to set S={START_PTS=INVALID, VIDEO READY=FALSE, AUDIO READY=FALSE}. At start, the video process <b>304</b> and the audio process <b>306</b> are generally in following states: the video process <b>304</b> (i) does not free data, (ii) outputs a blank screen or the last picture (e.g., frame repeat) and (iii) is configured to respond to commands; the audio process <b>306</b> (i) does not free data, (ii) outputs zeros and (iii) is configured to respond to commands.

0050The flow clock controller (FCC) <b>302</b> generally maintains information regarding which process (e.g., audio or video) is master and which process (or processes) is (are) slave. In general, the sequencer <b>301</b> generally tells the flow clock controller <b>302</b> the respective status of the video and audio processes <b>304</b> and <b>306</b>. In one example, the block <b>301</b> may be configured to convert top-level commands (e.g., “stop”, “pause”, etc.) into instructions or signals that underlying software and/or hardware components may understand. For example, the sequencer <b>301</b> may tell the flow clock controller that the audio process <b>306</b> is slave and the video process <b>304</b> is master. The video process <b>304</b> and the audio process <b>306</b> are generally configured to check the flow clock controller to determine whether the respective process is master or slave.

0051The flow clock controller may be implemented as a set of synchronization functions that (i) may be called by the video process <b>304</b> and the audio process <b>306</b> and (ii) may hold synchronization variables that may be written to and/or read from by the video and audio processes. The flow clock controller generally does not, for example, imply call-back notification of a state change. For example, the video process <b>304</b> may call the flow clock controller to set VIDEO READY to TRUE. Subsequently, the audio process may use a relevant flow clock controller function to determine that VIDEO READY is TRUE.

0052When the video process <b>304</b> receives a start command (e.g., START_PRESENTATION_MESSAGE) from the sequencer <b>301</b>, the video process <b>304</b> generally sends the presentation time-stamp value of the data that is ready to be transmitted (e.g., PTS_V). The flow clock controller generally sets the variable START_PTS to the value of the presentation time-stamp value of the data that is ready to be transmitted (e.g., PTS_V). The video process <b>304</b> sets VIDEO READY to TRUE, and continues to present the video frame presented at the start of the synchronization procedure.

0053The sequencer <b>301</b> generally sends a start command to the audio process <b>306</b>. When the START_PTS is valid, the audio process <b>306</b> holds or drops frames until the presentation time-stamp of the audio data (e.g., PTS_A) matches the value of the video time stamp (e.g., PTS_V) in the variable START_PTS. The comparison of the audio presentation time-stamp with the variable START_PTS may be performed by the audio process <b>306</b> or through a function implemented as part of the flow clock controller <b>302</b> (e.g., the flow clock controller may comprise functions and data that may be called in the audio task context). When the audio time-stamp matches the video time-stamp, the audio process <b>306</b> may be configured to fill a buffer to a predetermined fullness. Zeros are generally output during the above steps. When the buffer fullness reaches a predetermined threshold (e.g., half-full, etc.), the audio process <b>306</b> generally sets AUDIO READY to TRUE and audio data starts to be transmitted. If the audio process <b>306</b> receives a signal or message (e.g., NO_AUDIO) from the demultiplexing circuit (e.g., the circuit <b>136</b>) or times out, the audio process <b>306</b> generally sets AUDIO READY to TRUE.

0054When the video process <b>304</b> detects that AUDIO READY is TRUE, the video process <b>304</b> generally starts to transmit video data. The audio process <b>306</b> may be configured to repeat/drop samples while transmitting unmuted in order to achieve a better synchronization.

0055In one example, the block <b>304</b> may comprise a block (or process) <b>310</b>, a block (or process) <b>312</b> and a block (or process) <b>314</b>. The block <b>310</b> may be implemented as a video frame block. The block <b>312</b> may be implemented as a control block. The block <b>314</b> may be implemented as a frame repeater block. The signal VIDEO (or COMP. VIDEO) and the presentation time-stamp PTS_V may be presented to an input of the block <b>310</b>. The block <b>310</b> may be configured to present the presentation time-stamp PTS_V to the block <b>312</b>. The block <b>312</b> may be configured to receive the signal START_PRESENTATION_MESSAGE from the sequencer <b>301</b>. The block <b>312</b> may present a signal to or set a variable of (e.g., VIDEO READY) the flow clock controller <b>302</b> and present the presentation time-stamp PTS_V to the flow clock controller <b>302</b>. The block <b>312</b> may be configured to determine (e.g., via the flow clock controller <b>302</b>) when audio is ready (e.g., the signal AUDIO READY). The block <b>312</b> may be configured to generate a signal (e.g., HOLD) that may be used to select between an output of the block <b>310</b> and an output of the block <b>314</b> for presentation as the signal VIDEO_OUT.

0056The block <b>306</b> may comprise in one example, a block (or process) <b>320</b>, a block (or process) <b>322</b>, a block (or process) <b>324</b>, a block (or process) <b>326</b> and a block (or process) <b>328</b>. The block <b>320</b> may be configured to manipulate audio frames of the signal AUDIO. The block <b>322</b> may be configured to manipulate audio samples of the signal AUDIO. The block <b>324</b> may be implemented as an audio buffer. The block <b>326</b> may be implemented as a control block. The block <b>328</b> may be implemented as a zero generator.

0057The block <b>320</b> generally receives the signals AUDIO and PTS_A. The block <b>320</b> may be configured, in one example, to present the signal PTS_A to the control block <b>326</b>. The block <b>320</b> may be configured to repeat and/or drop one or more audio frames in response to commands (or information) received from the block <b>326</b> (e.g., via a signal FRAME REPEAT/DROP). The block <b>320</b> may present the audio frames or audio samples to the block <b>322</b>.

0058The block <b>322</b> may be configured to repeat or drop one or more audio samples of the audio frames in response to commands (or information) received from the control block <b>326</b> (e.g., via a signal SAMPLE REPEAT/DROP). An output of the block <b>322</b> is generally presented to an input of the buffer <b>324</b>. The buffer <b>324</b> may be configured to present a signal (e.g., FILL LEVEL) to the control block <b>326</b>. The signal FILL LEVEL generally is configured to indicate a level of fullness of the buffer <b>324</b>.

0059The control block <b>326</b> may be configured (i) to receive a start instruction (e.g., START) from the sequencer <b>301</b>, (ii) to receive the signals VIDEO READY and START_PTS from the flow clock controller <b>302</b> and (iii) to present a control signal to or set a variable in (e.g., AUDIO READY) the flow clock controller <b>302</b>. The control block <b>326</b> may be configured to generate a control signal (e.g., HOLD) that may be used to select between an output of the buffer <b>324</b> and an output of the zero generator block <b>328</b> for presentation as the signal AUDIO_OUT.

0060In general, the functions performed by the blocks <b>301</b>, <b>302</b>, <b>304</b> and <b>306</b> may be implemented using conventional techniques according to the teachings of the present specification, as will be apparent to those skilled in the relevant art(s).

0061A variable (e.g., D) may be implemented to represent the synchronization offset between the signals VIDEO_OUT and AUDIO_OUT. Synchronization may be adjusted to reduce a value of the variable D below a predetermined threshold. In one example, the threshold may be set to 500 samples (e.g., 10.4 msec at 48 kHz). However, other thresholds may be implemented accordingly to meet design criteria of a particular implementation.

0062Audio-video synchronization may be lost while streaming if, for example, there is an underflow. Audio-video synchronization may also be lost during Pause/Step/Resume transitions. The present invention may provide a process for recovering audio-video synchronization while streaming that may be implemented, in one example, by following two steps. In a first step, the current error timing error corrected for latency until current data is “on air” may be calculated according to the following equation: <br /><i>D</i>=flow clock−audio PTS−isrPtsCorrection+twoISR+latency, <br /> where flow clock represents time set by the flow clock master, audio PTS represents the PTS read from the audio header, isrPtsCorrection represents a current value of a hardware PTS (e.g., a value stamped at last Audio ISR), twoISR (converted into PTS units) represents the length of two Audio ISR DMA transfers in time (e.g., one ISR length is because of the hardware queue and the other ISR length is because the ISR reader refers to the next ISR while the isrPtsCorrection is referred to the previous ISR) and latency (converted into PTS units) is generally equal to final buffer write (AO task)—Final buffer read pointer (ISR). D is generally converted into units of time where one unit is the length of 1 audio sample (e.g., at 48 kHz D′=D*48,000/90,000). In a second step, D′ may be smoothed with a finite impulse response (FIR) filter as summarized by the following equation: <br />Δ′ frame smooth=filter (latest <i>D</i>′). <br /> The filter generally contains the PTS from N packets up to and including the last packet. Each PTS is generally normalized by multiplying by 1/N and the filter returns the sum of the normalized PTSs. In one example, N may be set to 4.

0063Referring to <figref idref="DRAWINGS">FIG. 7</figref>, a timing diagram illustrating example hysteresis thresholds is shown. In one example, a pair of thresholds (e.g., THRESHOLD 1 and THRESHOLD 2) may be implemented to determine when audio frames are dropped or repeated. By implementing a pair of thresholds, the present system may provide hysteresis. As used herein, hysteresis generally refers to a process or property where an input threshold changes as a function of the input (or output) level. More specifically, in a system with hysteresis, an output signal generally changes (i) from a first state to a second state when an input signal crosses a first threshold and (ii) changes from the second state to the first state when the input signal crosses a second threshold. The amount of hysteresis is generally determined by the difference between the first threshold and the second threshold. In general, the hysteresis filter may be configured to determine whether to play, drop or repeat frames.

0064Hysteresis is generally implemented to (i) reduce the probability of a small fluctuation in Δ′ frame smooth during unmuted play causing frames to be repeated or dropped and (ii) ensure that if a fluctuation occurs and synchronization is lost, the synchronization may be brought back into a narrow synchronization window. In general, hysteresis determined by the thresholds THRESHOLD 1 and THRESHOLD 2 may be used to decide when to repeat frames when D′ frame smooth is greater than 0. For example, while Δ′ frame smooth is less than THRESHOLD 2 a hysteresis flag may be set (e.g., de-asserted) and no frames are repeated (or dropped). When Δ′ frame smooth exceeds THRESHOLD 1, the hysteresis flag may be set TRUE (e.g., asserted) and frames are always repeated as long as Δ′ frame smooth exceeds THRESHOLD 1. When Δ′ frame smooth is less than THRESHOLD 1 and greater than THRESHOLD 2, frames are repeated when the hysteresis flag is set TRUE. Otherwise, frames are not repeated.

0065Since the hysteresis flag is set when Δ′ frame smooth is greater than or equal to THRESHOLD 1 and reset when Δ′ frame smooth is less than or equal to THRESHOLD 2, hysteresis is provided that may reduce sensitivity to small variations in Δ′ frame smooth. The amount of hysteresis (e.g., the size of a fluctuation in Δ′ frame smooth that results in change in frame repeat/drop behavior) is generally controlled by selection of THRESHOLD 1 and THRESHOLD 2. Another pair of thresholds (not shown) may be implemented symmetrically about the Δ′ frame smooth=0 axis to control the decision whether to drop frames.

0066Referring to <figref idref="DRAWINGS">FIG. 8</figref>, a timing diagram illustrating example thresholds for controlling frame and sample repeat/drop decisions in accordance with the present invention are shown. In one example, a number of thresholds (e.g., T1-T8) may be implemented to determine whether audio frames are dropped and/or repeated and/or audio samples are dropped and/or repeated. The determination of whether to repeat or drop samples may be implemented in a similar way to the frame repeat or drop decision. In one example, smoothing of D may be performed with a filter length of 8 rather than the filter length of 4 used for frame repeat/drop determinations. In one example, the thresholds for repeating and dropping samples and frames may set for positive and negative values of D (or A) to the following values: <ul id="ul0001" list-style="none"><li id="ul0001-0001" num="0000"><ul id="ul0002" list-style="none"><li id="ul0002-0001" num="0067"> T1=2048 samples (43 msec at 48 KHz) </li><li id="ul0002-0002" num="0068"> T2=T6=1024 samples (21 msec at 48 KHz) </li><li id="ul0002-0003" num="0069"> T3=T7=−1024 samples (−21 msec at 48 KHz) </li><li id="ul0002-0004" num="0070"> T4=−2048 samples (−43 msec at 48 KHz) </li><li id="ul0002-0005" num="0071"> T5=T6+128 samples (T6+2.3 msec at 48 kHz) </li><li id="ul0002-0006" num="0072"> T8=T7-128 samples (T7-2.3 msec at 48 kHz) <br /> However, other values may be implemented to meet the design criteria of a particular implementation. </li></ul></li></ul>

0073Referring to <figref idref="DRAWINGS">FIG. 9</figref>, a state diagram <b>400</b> is shown illustrating a control decision for controlling a mute parameter during synchronization. In one example, the mute/unmute decision process may be coupled to the A/V synchronization. The mute/unmute decision process may have a state <b>402</b> in which mute is on during synchronization and a state <b>404</b> in which mute is off during synchronization. The process may be configured to move from the state <b>402</b> to the state <b>404</b> in response to a number of good frames counted being greater than a first threshold (e.g., N1). The process may be further configured to move from the state <b>404</b> to the state <b>402</b> in response to a number of bad frames counted being greater than a second threshold (e.g., N2). In general, the state <b>402</b> represents a state in which a determination of bad synchronization results in the good frame count being set to zero and a determination of a good synchronization increments a good frame count value. In the state <b>404</b>, the process generally zeros a bad frame count variable in response to detection of good synchronization and increments the bad frame count variable in response to detection of bad synchronization. The thresholds N1 and N2 generally provide hysteresis to the mute/unmute decision process.

0074When frames are repeatedly dropped or repeated because of a loss of A/V synchronization, the audio is generally muted. When A/V synchronization is re-acquired, the audio is generally unmuted. If the mute decision is made on whether a single frame is played, dropped or repeated, however, repeated muting and unmuting may occur due to transient A/V synchronization behavior. The present invention generally mitigates repeated muting and unmuting by applying hysteresis to the decision when to mute and unmute. A process for muting and unmuting in accordance with the present invention may be summarized as follows:

0075When slave audio can exist in one of two states: <tables id="TABLE-US-00001" num="1"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="77PT" align="left" /><colspec colname="2" colwidth="140PT" align="left" /><thead><row><entry /></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>SYNC MUTE ON</entry><entry>(mute audio because of bad synchronization);</entry></row><row><entry>SYNC MUTE OFF</entry><entry>(synchronization good so no sync mute).</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables><br /> Thresholds are generally used to determine whether the synchronization is good or not. For example, <ul id="ul0003" list-style="none"><li id="ul0003-0001" num="0000"><ul id="ul0004" list-style="none"><li id="ul0004-0001" num="0076"> IF mod(D′frame smooth)>Threshold THEN BAD SYNC ELSE GOOD SYNC <br /> Thresholds on the good and bad frame counts may be used to determine the transitions between the two states <b>402</b> and <b>404</b>. </li></ul></li></ul>

0077The present invention generally uses video as master and audio as slave. For analog audio and IEC 958 signals the mechanism for A/V synchronization is generally for audio to drop or repeat audio frames and/or drop or repeat audio samples. The present invention may provide a mode to maintain A/V synchronization by changing the rate of data consumption even though the audio is unmuted. For example, the present invention may be used with a digital video (DV) input. Another example is after a playback sequencer start when audio is unmuted after an initial synchronization has been achieved even though synchronization may not have achieved the final target accuracy. The present invention allows fine tuning of the A/V synchronization.

0078In general, repeating or dropping audio frames when unmuted may be unacceptable because clearly discernable audible artifacts may be generated. The present invention may repeat or drop M in N audio samples, where M and N are integers. In a preferred embodiment, M may be fixed at 1 and N may be fixed at 128. A “sampling slip” audio filter may be used with the present invention to smooth over the repeated or dropped sample(s). For example, the sampling slip filter generally works using “timeshift, ramp and add” as follows:

0000Define X(m: m=0: m=L) as a vector with first index m=0 and last index m=L.

0000To remove (or drop) a sample:

0000<ul id="ul0005" list-style="none"><li id="ul0005-0001" num="0000"><ul id="ul0006" list-style="none"><li id="ul0006-0001" num="0079"> 1. Let A(m: m=0: m=N−2)=I(t_i: i=0: i=N−2) be one set of input samples; </li><li id="ul0006-0002" num="0080"> 2. Apply linear ramp down on A with scale factor 1 to 0; </li><li id="ul0006-0003" num="0081"> 3. Let B(m: m=0: m=N−2)=I(t_i, i=1: I=N−1) be a time-shifted set of samples; </li><li id="ul0006-0004" num="0082"> 4. Apply linear ramp up on B with scale factor 0 to 1; </li><li id="ul0006-0005" num="0083"> 5. Define filter output C(n:n=0:n=N−2)=A(n)+B(n). <br /> To repeat a sample: </li><li id="ul0006-0006" num="0084"> 1. Let A(m: m=0: m=N−1)=I(t_i: i=0: i=N−1) be one set of input samples; </li><li id="ul0006-0007" num="0085"> 2. Define A(N)=0; </li><li id="ul0006-0008" num="0086"> 3. Apply linear ramp down on A with scale factor 1 to 0; </li><li id="ul0006-0009" num="0087"> 4. Let B(m: m=1: m=N)=I(t_i, i=0: I=N-1) be the same time-shifted set of samples; </li><li id="ul0006-0010" num="0088"> 5. Define B(0)=0; </li><li id="ul0006-0011" num="0089"> 6. Apply linear ramp up on B with scale factor 0 to 1; </li><li id="ul0006-0012" num="0090"> 7. Define filter output C(n:n=0:n=N)=A(n)+B(n). </li></ul></li></ul>

0091Sample repeat/drop generally produces a slight pitch change. However, the change in pitch is generally only noticeable with a pure sine-wave output. Sample repeat/drop may also be relatively slow in adjusting synchronization. For example, to achieve a 20 msec synchronization adjustment, sample repeat/drop may take 20 msec×128=2.56 s.

0092For certain trick plays, maintaining approximate A/V synchronization may be necessary in order to minimize data loss and the length of time for achieving A/V synchronization when normal play conditions are resumed. Examples may include certain play speed changes or DVD Video pause/resume. In order to minimize the chance of glitches happening during the transition, the audio is generally internally muted during the trick play (for example slow play). When the playback sequencer sets the play back to normal (for example normal speed) an audio sequencer api may be called to put the audio in the SYNC MUTE ON state with goodFramesCount=0.

0093Referring to <figref idref="DRAWINGS">FIG. 10</figref>, a block diagram of a circuit <b>138</b>′ is shown illustrating another embodiment of the present invention. The circuit <b>138</b>′ may be configured to present a plurality of audio output signals. In one example, the circuit <b>138</b>′ may be configured to present the signal AUDIO_OUT comprising a number of analog audio output signals. In another example, the circuit <b>138</b>′ may be configured to present the signal AUDIO_OUT as an analog output signal and as a digital signal. For example, the digital signal generally contains the same information as the analog signal, but in a digital format. In one example, the digital audio output signal may be implemented as an IEC61937 compliant signal.

0094In general, when the circuit <b>138</b>′ is configured to output a plurality of analog audio signals, each of the audio signals are slaved to the video signal and synchronization may be performed as described above. However, the video signal and the plurality of audio signals wait to be transmitted until all of the audio signals are ready.

0095When the circuit <b>138</b>′ is configured to output an analog audio signal and a digital audio signal (e.g., an IEC61937 signal), the synchronization process may be modified. The analog audio signal and the digital audio signal are generally synchronized coarsely to the video signal as master according to the process described above in connection with <figref idref="DRAWINGS">FIG. 5</figref>. However, for the digital audio signal, particularly an IEC61937 type signal, fine synchronization to the video signal may not be possible. Instead, once coarse synchronization is obtained between the video and the audio signals, the analog audio signal may be fine synchronized with the digital audio signal as master. In general, the precision of the synchronization between the analog and digital audio signals is more critical than the precision of the synchronization of the audio relative to the video.

0096Once the audio signals (e.g., both analog and digital) and the video signal have been coarsely synchronized, the audio may be unmuted. The fine synchronization of the analog audio with the digital audio as master may be performed with audio unmuted.

0097Referring to <figref idref="DRAWINGS">FIG. 11</figref>, a block diagram illustrating an alternative embodiment in accordance with the present invention is shown. A/V synchronization may also be implemented on a record/loopthrough side of a Sony/Philips digital interface format (S/PDIF). In one example, a circuit <b>500</b> may be implemented. The circuit <b>500</b> may have a first input S<b>1</b> that may receive an analog audio input, a second input S<b>2</b>, that may receive a S/PDIF (IEC958) digital audio input and a third input S<b>3</b> that may receive a signal VIDEO_IN. The circuit <b>500</b> may have a first output D<b>1</b> that may present a loopthrough analog AUDIO_OUT signal, an output D<b>2</b> that may present a loopthrough digital audio output signal, an output D<b>3</b> that may present a video output signal and an output E<b>1</b> that may present a bitstream signal. The circuit <b>500</b> may comprise a loopthrough A/V synchronization block <b>502</b>, a number of encoders <b>504</b>-<b>508</b>, and a multiplexer circuit <b>510</b>. The encoders <b>504</b>-<b>508</b> and the multiplexer <b>510</b> may operate in parallel with the loopthrough circuit <b>502</b>. The bitstream signal presented at the output E<b>1</b> may be presented to, in one example, a DVD recorder <b>512</b>.

0098In one example, the loopthrough digital audio output may be used as the master and the loopthrough analog audio output and video output may be slaved to the digital audio output. In general, the synchronization process may be implemented similarly to the process described above except that the loopthrough digital audio output takes the role that video had in the above discussion. For example, the handshaking process involves the loopthrough digital audio output setting the initial presentation time stamp and waiting for both the loopthrough analog audio output and the video output to be ready before all three begin transmitting.

0099As used herein, the terms “simultaneous” and “simultaneously” are meant to describe events that share some common time period but the term is not meant to be limited to events that begin at the same point in time, end at the same point in time, or have the same duration.

0100The function(s) performed by the present invention may be implemented using a conventional general purpose digital computer programmed according to the teachings of the present specification, as will be apparent to those skilled in the relevant art (s). Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure, as will also be apparent to those skilled in the relevant art(s).

0101The present invention may also be implemented by the preparation of ASICs, FPGAs, or by interconnecting an appropriate network of conventional component circuits, as is described herein, modifications of which will be readily apparent to those skilled in the art(s).

0102The present invention thus may also include a computer product which may be a storage medium including instructions which can be used to program a computer to perform a process in accordance with the present invention. The storage medium can include, but is not limited to, any type of disk including floppy disk, optical disk, CD-ROM, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMS, Flash memory, magnetic or optical cards, or any type of media suitable for storing electronic instructions.

0103While the invention has been particularly shown and described with reference to the preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the invention.