Audio and video data streaming for media effects

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No. 14/498,778, filed Sep. 26, 2014, which is a continuation of U.S. patent application Ser. No. 13/099,312, filed May 2, 2011, now U.S. Pat. No. 8,861,926 which issued on Oct. 14, 2014. The subject matter of these related applications is hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION

Field of the Invention

Embodiments of the present invention relate generally to digital media and, more specifically, to streaming media effects using audio data and video data.

Description of the Related Art

Digital media content distribution systems conventionally include a content server, a content player, and a communications network connecting the content server to the content player. The content server is configured to store digital media content files, which can be downloaded from the content server to the content player. Each digital media content file corresponds to a specific identifying title, such as “Gone with the Wind,” which is familiar to a user. The digital media content file typically includes sequential content data, organized according to playback chronology, and may comprise audio data, video data, or a combination thereof.

The content player is configured to download and play a digital media content file, in response to a user request selecting the title for playback. The process of playing the digital media content file includes decoding audio and video data into a synchronized audio signal and video signal, which may drive a display system having a speaker subsystem and a video subsystem. Playback typically involves a technique known in the art as “streaming,” whereby the content server sequentially transmits the digital media content file to the content player, and the content player plays the digital media content file while content data is received that comprises the digital media content file. To account for variable latency and bandwidth within the communications network, a content buffer queues the incoming content data ahead of the content data actually being played. During moments of network congestion, which leads to lower available bandwidth, less content data is added to the content buffer, which may drain down as content data is being de-queued to support playback at a certain playback bit rate. However, during moments of high network bandwidth, the content buffer is replenished and additional buffer time is added until the content buffer is generally full again. In practical systems, the content buffer may queue content data corresponding to a time span ranging from seconds to more than a minute.

Each digital media content file stored on the content server typically includes video data that is processed before being output for display. The processing may include decoding the video data to raw frames of PCM streams. The processed video data is then output for display. In contrast, the audio data is passed from the content server to an output device, e.g., speakers, without modification. Consequently, playback volume is entirely controlled by a volume setting for the output device and the decoded audio signal is modulated using simple calculations based on the volume setting.

For some digital media content, a preview clip may be designated corresponding to a pivotal moment in a movie. In some cases, the audio data is very loud, e.g., a battle scene, and the output volume control of the content player may also be set at a high level based on the previous content playback. When playback of the preview clip starts, the resulting output volume level of the audio signal may be unpleasantly high. Because the audio data is passed form the content server to the content player without modification, it is not possible to decrease the volume level by modulating the audio data before the audio data is provided to the content player.

As the foregoing illustrates, what is needed in the art is an ability to modulate the audio data without requiring the viewer to adjust the volume control of the content player.

SUMMARY OF THE INVENTION

One embodiment of the present invention sets forth a method for streaming digital media content with transition effects. The method comprises the steps of receiving a playback location within the digital media content which includes an original data stream that is encoded as a sequence of intervals and determining a nearest interval in the original data stream relative to the playback location. A first interval of an effect data stream that comprises a modulated version of the original data stream is retrieved, where the first interval corresponds to the nearest interval in the original data stream. One or more additional intervals of the original data stream are retrieved, where the one or more additional intervals are subsequent to the nearest interval in the sequence of intervals. The first interval of the effect data stream is played and while playing the first interval of the effect data stream, it is determined that an interval boundary is reached. In response, the one or more additional intervals of the original data stream are played in sequence.

One advantage of the disclosed technique is that audio data may be modulated and received by the streaming device for an audio fade-in effect. Similarly, video data may be modulated to perform various visual effects. Such a technique allows for fading-in of audio and/or video, fading-out of audio and/or video, and effects that combine audio from different content and video from different content during a transition between from one content to another. A consumer may personalize the video and/or audio effects that are applied during transitions. The modulated audio and video data to generate the effect may be precomputed or computed in real-time and transmitted to the streaming device.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.

<figref idref="DRAWINGS">FIG. 1</figref> illustrates a content distribution system configured to implement one or more aspects of the present invention;

<figref idref="DRAWINGS">FIG. 2</figref> is a more detailed view of the streaming device of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 3A</figref> is an illustration of an original audio stream generated by the streaming server of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 3B</figref> is an illustration of a fade-in audio stream generated by the audio effect generator of <figref idref="DRAWINGS">FIG. 2</figref>, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 3C</figref> is an illustration of another fade-in audio stream generated by the audio effect generator of <figref idref="DRAWINGS">FIG. 2</figref>, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 4A</figref> is an illustration of the original audio stream and the fade-in audio stream encoded using fixed rate audio intervals, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 4B</figref> is an illustration of the original audio stream and the fade-in audio stream encoded using variable rate audio intervals, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 5A</figref> is a more detailed view of the streaming device of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 5B</figref> is a flow diagram of method steps for playing the fade-in audio stream and transitioning to the original audio stream, according to one embodiment of the invention;

<figref idref="DRAWINGS">FIG. 6A</figref> is a flow diagram of method steps for playing fade-in and fade-out audio and video streams and transitioning to the original audio and video streams, according to one embodiment of the invention; and

<figref idref="DRAWINGS">FIG. 6B</figref> is a flow diagram of method steps for playing a switch event stream and during playback of audio and video streams, according to one embodiment of the invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to one of skill in the art that the present invention may be practiced without one or more of these specific details. In other instances, well-known features have not been described in order to avoid obscuring the present invention.

<figref idref="DRAWINGS">FIG. 1</figref> illustrates a content distribution system <b>100</b> configured to implement one or more aspects of the invention. As shown, the content distribution system <b>100</b> includes a streaming server <b>102</b>, a communications network <b>106</b>, a streaming device <b>108</b>, and a output device(s) <b>104</b>. The content distribution system <b>100</b> may include a plurality of communications networks <b>106</b>, such as routers and switches, configured to facilitate data communication between the streaming server <b>102</b> and the streaming device <b>108</b>. The output device(s) <b>104</b> is configured to produce a display image and associated sound and is typically directly coupled to the streaming device <b>108</b> by a wired or wireless connection. Persons skilled in the art will recognize that many technically feasible techniques exist for transmitting data between the streaming server <b>102</b>, the streaming device <b>108</b> and the output device(s) <b>104</b>, including technologies practiced in deploying the well-known internet communications network.

The streaming server <b>102</b> is a computer system configured to encode video and/or audio streams associated with digital media content files for streaming. The content distribution system <b>100</b> maybe include one or more streaming servers <b>102</b>, where each streaming server <b>102</b> is configured to perform all the functions needed to encode the video and/or audio streams or where each streaming server <b>102</b> is configured to perform a particular function needed to encode the video and/or audio streams. The digital media content files including the encoded video and audio streams are retrieved by the streaming device <b>108</b> via the communications networks <b>106</b> for output to the output device(s) <b>104</b>.

As shown in <figref idref="DRAWINGS">FIG. 1</figref>, audio data <b>103</b> and video data <b>101</b> represent the encoded audio and video streams that are transmitted from the streaming server <b>102</b> to the streaming device <b>108</b>. The streaming device <b>108</b> passes the audio data <b>103</b> through to the output device <b>104</b> unchanged. The video data <b>101</b> is uncompressed (if in a compressed format) or decoded into raw frames or PCM (pulse code modulated) intervals and output by the streaming device <b>108</b> to the output device(s) <b>104</b> as video signal <b>105</b>. The output device(s) <b>104</b> may include a display device and speaker device for presenting video image frames, and generating acoustic output, respectively.

The streaming server <b>102</b> comprises one or more computer systems configured to serve download requests for digital media content files from the streaming device <b>108</b>. The digital media content files may reside on a mass storage system accessible to the computer system. The mass storage system may include, without limitation, direct attached storage, network attached file storage, or network attached block-level storage. The digital media content files may be formatted and stored on the mass storage system using any technically feasible technique. A data transfer protocol, such as the well-known hyper-text transfer protocol (HTTP), may be used to download digital media content files from wherever the digital media content files are stored to the streaming device <b>108</b>.

The streaming device <b>108</b> may comprise a computer system, a set top box, a mobile device such as a mobile phone, or any other technically feasible computing platform that has network connectivity and is coupled to or includes the output device(s) <b>104</b>. The streaming device <b>108</b> is configured for streaming, i.e., to download units of a video stream encoded to a specific playback bit rate. In one embodiment, the streaming device <b>108</b> is configured to switch to downloading subsequent units of a video stream encoded to a different playback bit rate based on prevailing bandwidth conditions within the communications network <b>106</b>. As bandwidth available within the communications network <b>106</b> becomes limited, the streaming device <b>108</b> may select a video stream encoded to a lower playback bit rate. As the bandwidth increases, a video stream encoded to a higher playback bit rate may be selected. The audio stream is typically a much lower playback bit rate than the corresponding video stream and is therefore not typically encoded at different playback bit rates.

Although, in the above description, the content distribution system <b>100</b> is shown with one streaming device <b>108</b>, persons skilled in the art will recognize that the architecture of <figref idref="DRAWINGS">FIG. 1</figref> contemplates only an exemplary embodiment of the invention. Other embodiments may include any number of streaming device <b>108</b>. Thus, <figref idref="DRAWINGS">FIG. 1</figref> is in no way intended to limit the scope of the present invention in any way.

<figref idref="DRAWINGS">FIG. 2</figref> is a more detailed view of the streaming server <b>102</b> of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention. As shown, the streaming server <b>102</b> includes a central processing unit (CPU) <b>202</b>, a system disk <b>204</b>, an input/output (I/O) devices interface <b>206</b>, a network interface <b>208</b>, an interconnect <b>210</b> and a system memory <b>212</b>.

The CPU <b>202</b> is configured to retrieve and execute programming instructions stored in the system memory <b>212</b>. Similarly, the CPU <b>202</b> is configured to store application data and retrieve application data from the system memory <b>212</b>. The interconnect <b>210</b> is configured to facilitate transmission of data, such as programming instructions and application data, between the CPU <b>202</b>, the system disk <b>204</b>, I/O devices interface <b>206</b>, the network interface <b>208</b>, and the system memory <b>212</b>. The I/O devices interface <b>206</b> is configured to receive input data from I/O devices <b>222</b> and transmit the input data to the CPU <b>202</b> via the interconnect <b>210</b>. For example, I/O devices <b>222</b> may comprise one or more buttons, a keyboard, and a mouse or other pointing device. The I/O devices interface <b>206</b> is also configured to receive output data from the CPU <b>202</b> via the interconnect <b>210</b> and transmit the output data to the I/O devices <b>222</b>. The system disk <b>204</b>, such as a hard disk drive or flash memory storage drive or the like, is configured to store non-volatile data such as encoded video streams. The encoded video streams can then be retrieved by the streaming device <b>108</b> via the communications network <b>104</b>. The network interface <b>218</b> is coupled to the CPU <b>202</b> via the interconnect <b>210</b> and is configured to transmit and receive packets of data via the communications network <b>104</b>. In one embodiment, the network interface <b>208</b> is configured to operate in compliance with the well-known Ethernet standard.

The system memory <b>212</b> includes software components that include instructions for encoding one or more audio and video streams associated with a specific content title for streaming. As shown, these software components include an audio effect generator <b>214</b>, a video stream encoder <b>216</b>, an audio stream encoder <b>224</b>, a sequence header index (SHI) generator <b>218</b>, and a video effect generator <b>220</b>.

The video stream encoder <b>216</b> executes encoding operations for encoding a video stream to a specific playback bit rate such that the encoded video stream complies with a particular video codec standard, such as VC<b>1</b>, and is configured for streaming. In an alternative embodiment, the video stream can be encoded to comply with a different video codec standard such as MPEG or H.264. In operation, for a particular video stream, the video stream encoder <b>216</b> encodes the video stream to different constant bitrates to generate multiple encoded video streams, each encoded video stream associated with a different constant bitrate and, thus, having a different quality. An encoded video stream generated by the video stream encoder <b>216</b> includes a sequence of groups of pictures (GOPs), each GOP comprising multiple image frames of video data. In practice, a GOP may include multiple scenes or portions of a scene. A GOP typical corresponds to 2.5 seconds or 10 seconds of playback time, although other durations may also be used. A GOP is specific to video data and one or more GOPs are included in an interval. For each interval of video data, there may be a corresponding interval of audio data. The video and audio streams each include a sequence of intervals.

The SHI generator <b>218</b> generates a sequence header index associated with each encoded video stream. To generate the sequence header index, the SHI generator <b>218</b> first searches the encoded video stream for the key frames associated with the different intervals included in the encoded video stream. The key frames can be located by the SHI generator <b>218</b> based on the sequence start codes specified in the sequence headers included in the key frames. For the interval associated with each of the identified key frames, the SHI generator <b>218</b> defines a switch point within the sequence header index that stores (i) a data packet number that identifies the data packet that includes the key frame associated with the interval and (ii) the playback offset associated with the interval. Again, the playback offset associated with the interval is determined based on the location of the interval in the sequence of intervals included in the encoded video stream.

The audio stream encoder <b>224</b> executes encoding operations for encoding an audio stream to a specific playback bit rate such that the encoded audio stream is configured for streaming and synchronization with the video stream. The sequence header indexes associated with each encoded video stream that are generated by the SHI generator <b>218</b> are also associated with the encoded audio stream. The switch points defined by the SHI generator <b>218</b> within the sequence header index stores (i) a data packet numbers that identifies the data packet for the audio data corresponding to each interval of the audio and video data and (ii) the playback offset in the audio data associated with each interval of the audio data.

The audio effect generator <b>214</b> and the video effect generator <b>220</b> are configured to generate encoded audio and video streams, respectively, based on a specified effect, e.g., fade-in, fade-out, dissolve, zoom-in, zoom-out, three-dimensional rotation, spiraling, particle burst, page turn, page roll, page peel, distortion, and the like. In sum, any function may be applied to an audio or video data to modulate the audio or video data, respectively. In one embodiment, a fade-in audio stream may be generated by the audio effect generator <b>214</b> for a preview clip corresponding to a pivotal moment in a movie. When a preview clip is selected for playback, the fade-in audio stream may be retrieved by the streaming device <b>108</b> so that the resulting output volume of the audio signal starts at a low level and increases to the full level of the original audio during a fade-in time envelope. The fade-in ensures that the audio volume level is not unpleasantly high at the start of the preview clip. Because the audio data <b>103</b> is passed from the streaming server <b>102</b> to the streaming device <b>108</b> without modification, the fade-in audio stream is precomputed by the streaming server <b>102</b> or may be generated on-the-fly, i.e., in real-time, by the streaming server <b>102</b> when adequate computation resources are available. The streaming device <b>108</b> typically is not configured with enough computation resources to perform the modulation of the audio stream on-the-fly.

<figref idref="DRAWINGS">FIG. 3A</figref> is an illustration of an original audio stream <b>300</b> that is encoded by the streaming server <b>102</b> of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention.

<figref idref="DRAWINGS">FIG. 3B</figref> is an illustration of a fade-in audio stream <b>302</b> generated by the audio effect generator <b>214</b> of <figref idref="DRAWINGS">FIG. 2</figref>, according to one embodiment of the invention. During a fade-in envelope <b>305</b> the original audio stream <b>300</b> is modulated to start at the lowest volume level and then increase across the fade-in envelope to match the volume level of the original audio stream <b>300</b> at the same point in time. After the fade-in envelope <b>305</b> the fade-in audio stream <b>302</b> matches the original audio stream <b>300</b>.

In one embodiment, the audio effect generator <b>214</b> precomputes the fade-in audio stream <b>302</b> and the entire fade-in audio stream <b>302</b> is provided to the streaming device <b>108</b> instead of the original audio stream <b>300</b>. In another embodiment, the audio effect generator <b>214</b> computes the fade-in audio stream <b>302</b> on-the-fly when the corresponding content is requested by the streaming device <b>108</b>. The audio stream <b>300</b> or <b>302</b> provides the clock track for playback of the audio and video streams. Therefore, switching from one audio stream to another audio stream, such as switching from the fade-in audio stream <b>302</b> to the original audio stream <b>300</b>, is only possible when the different audio streams are encoded to have the same playback time intervals and the same playback offsets. The streaming device <b>108</b> may be configured to switch between different encoded audio streams and between different encoded video streams.

<figref idref="DRAWINGS">FIG. 3C</figref> is an illustration of an interval fade-in audio stream <b>310</b> generated by the audio effect generator <b>214</b> of <figref idref="DRAWINGS">FIG. 2</figref>, according to one embodiment of the invention. For each interval of the original audio stream <b>300</b>, a fade-in interval <b>311</b>, <b>312</b>, <b>313</b>, <b>314</b>, and <b>315</b> is generated to produce the interval fade-in audio stream <b>310</b>. Fade-in intervals <b>312</b>, <b>313</b>, <b>314</b>, and <b>315</b> are subsequent to fade-in interval <b>311</b> in the interval fade-in audio stream <b>310</b>. Interval fade-in audio stream <b>310</b> provides a fade-in audio signal at multiple start locations of the content, not just starting from the beginning as is the case for the fade-in audio stream <b>302</b>.

During each fade-in interval <b>311</b>, <b>312</b>, <b>313</b>, <b>314</b>, and <b>315</b>, the original audio stream <b>300</b> is modulated to start at the lowest volume level and then increase to match the volume level of the original audio stream <b>300</b> at the same point in time. The streaming device <b>108</b> may be configured to start playing the content at the start of an interval. The streaming device <b>108</b> retrieves the fade-in interval and switches from the interval fade-in audio stream <b>310</b> to the original audio stream <b>300</b> after retrieving one fade-in interval from the interval fade-in audio stream <b>310</b>. The streaming device <b>108</b> may fade-in the audio stream following a trick play, e.g., fast-forward or rewind activity, at any interval boundary.

The video effect generator <b>220</b> of <figref idref="DRAWINGS">FIG. 2</figref> may be configured to generate intervals of video data based on specific video effects. Video streams may be generated with one or more intervals generated according to a video effect. Different video streams may be generated for each different video effect. The video streams are encoded and may be requested by the streaming device <b>108</b> to display the video effect at transitions between different content, at the start of play, following a trick play, or at the end of play. Similarly, the audio effects may be output at transitions between different content, at the start of play, following a trick play, or at the end of play.

Additionally, at transitions, a video transition stream that encodes a blended version of the ending video stream and the starting video stream may be retrieved by the streaming device <b>108</b> and output to the output device(s) <b>104</b>. Similarly, an audio transition stream that encodes a blended version of the ending audio stream and the starting audio stream may be retrieved by the streaming device <b>108</b> and output to the output device(s) <b>104</b>. Various audio and/or video effects may be applied to generate the video or audio transition streams that are a blended version of two different video or audio streams for a transition interval. The two different video or audio streams maybe from different locations in the same digital media content or from different digital media content. The video and audio transition streams may be precomputed or generated on-the-fly and may be considered as a particular type of fade-in stream.

In addition to generating multiple encoded video and audio streams for specified effects, the streaming server <b>102</b> may generate multiple encoded video streams associated with the same content title and encoded to different playback bit rates. The encoding process implemented by the streaming server <b>102</b> ensures that, across the different encoded video and audio streams the intervals are associated with the same playback time interval and that corresponding intervals across the different encoded video and audio streams are associated with the same playback offsets. Therefore, each switch point defined in a sequence header included in one of the encoded video stream associated with a specific content title has a corresponding switch point defined in a sequence header included in each of the other encoded video stream associated with the same content title. Similarly, when multiple encoded audio streams are generated, the audio data corresponding to the interval are associated with the same playback time interval and the same playback offsets. The streaming device <b>108</b> may switch between different encoded video streams based on the interval boundaries defined by the corresponding sequence header indices. Importantly, in order to properly switch between the different audio streams, the switch points defined by the SHI generator <b>218</b> within the sequence header index for the audio streams are matching in terms of time duration, bytes, and indices.

<figref idref="DRAWINGS">FIG. 4A</figref> is an illustration of the original audio stream <b>300</b> and the fade-in audio stream <b>302</b> encoded using fixed rate audio intervals, according to one embodiment of the invention. Each audio interval in the audio stream corresponds to an interval of the video stream and is associated with the same digital media content. The original audio stream <b>300</b> encoding generates audio intervals <b>300</b>(<b>0</b>), <b>300</b>(<b>1</b>), <b>300</b>(<b>2</b>), <b>300</b>(<b>3</b>), and <b>300</b>(<b>4</b>). The fade-in audio stream <b>302</b> encoding generates audio intervals <b>302</b>(<b>0</b>), <b>300</b>(<b>1</b>), <b>300</b>(<b>2</b>), <b>300</b>(<b>3</b>), and <b>300</b>(<b>4</b>). Audio interval <b>302</b>(<b>0</b>) is the modulated audio signal for the fade-in effect during the fade-in envelope <b>305</b>. Because a fixed rate encoding is performed, each interval is of equal and constant length in terms of bytes and playback duration. Video data may also be encoded using constant bit rates to generate video streams having different effects and having different constant bit rates for the same content.

When the fade-in audio stream <b>302</b> is retrieved by the streaming device <b>108</b>, for viewing a preview clip, there is no need to switch between the fade-in audio stream <b>302</b> and another audio stream. However, after a trick play, the audio stream resumes playback at an interval boundary, not necessarily at the start of the fade-in audio stream <b>302</b>. Therefore, the interval fade-in audio stream <b>305</b> may be retrieved for one of the intervals nearest to the playback point. After one interval is retrieved from the interval fade-in audio stream <b>305</b>, the streaming device <b>108</b> switches to the original audio stream <b>300</b>.

The streaming device <b>108</b> can efficiently switch between the encoded video streams and/or audio streams by identifying the appropriate switch points in the sequence header indices. When switching between a currently playing encoded audio stream and a different encoded audio stream, the streaming device <b>108</b> searches the sequence header index included in the different encoded audio stream to locate the particular switch point specifying the playback offset associated with the next interval to be played. The streaming device <b>108</b> can then switch to the new encoded audio stream and download the interval stored in the data packet specified at the particular switch point for playback. For example, for encoded video streams where each interval was associated with a playback time interval of three seconds, if the first interval associated with the playback offset of zero seconds were currently being played, then the next interval to be played would be associated with the playback offset of three seconds. In such a scenario, the streaming device <b>108</b> searches the sequence header associated with the new encoded stream for the particular switch point specifying a playback offset of three seconds. Once locating the particular switch point, the streaming device <b>108</b> would download the interval stored in the data packet specified in the switch point for playback.

<figref idref="DRAWINGS">FIG. 4B</figref> is an illustration of the original audio stream <b>300</b> and the interval fade-in audio stream <b>310</b> encoded using variable bit rate (VBR) audio intervals, according to one embodiment of the invention. Rather than encoding the video and audio streams at a fixed bit rate, each interval is encoded based on the content for the respective interval. For example, interval for a scene of low complexity is encoded to a lower bit rate to “save” bits for scenes having a higher complexity. The average bit rate across a VBR video stream is, thus, not reflective of the bit rate of a particular interval within the VBR video stream.

The VBR encoded original audio stream <b>400</b> includes intervals <b>304</b>(<b>0</b>), <b>304</b>(<b>1</b>), <b>304</b>(<b>2</b>), <b>304</b>(<b>3</b>), and <b>304</b>(<b>4</b>) corresponding to intervals <b>300</b>(<b>0</b>), <b>300</b>(<b>1</b>), <b>300</b>(<b>2</b>), <b>300</b>(<b>3</b>), and <b>300</b>(<b>4</b>) of the original audio stream <b>300</b>, respectively. The interval fade-in audio stream <b>406</b> is the VBR encoding of the interval fade-in audio stream <b>310</b>. Intervals <b>401</b>, <b>402</b>, <b>403</b>, <b>404</b>, and <b>405</b> are the encoded fade-in intervals <b>311</b>, <b>312</b>, <b>313</b>, <b>314</b>, and <b>315</b>, respectively. Note that the interval boundaries are not aligned between the original audio stream <b>400</b> and the interval fade-in audio stream <b>406</b> due to the VBR encoding. Therefore, the switch points for the different audio streams are different and the streaming device <b>108</b> cannot easily locate corresponding intervals in the different audio streams. In order to easily switch between the different audio streams, the switch points defined by the SHI generator <b>218</b> within the sequence header index for the audio streams match in terms of time duration, bytes, and indices.

The interval fade-in audio stream <b>410</b> includes VBR encoded intervals that match the intervals in the VBR encoded original audio stream <b>400</b> in terms of time duration, bytes, and indices. Intervals <b>411</b>, <b>412</b>, <b>413</b>, <b>414</b>, and <b>415</b> are the encoded fade-in intervals <b>311</b>, <b>312</b>, <b>313</b>, <b>314</b>, and <b>315</b>, respectively, and intervals <b>411</b>, <b>412</b>, <b>413</b>, <b>414</b>, and <b>415</b> correspond to intervals <b>304</b>(<b>0</b>), <b>304</b>(<b>1</b>), <b>304</b>(<b>2</b>), <b>304</b>(<b>3</b>), and <b>304</b>(<b>4</b>) of the VBR encoded original audio stream <b>400</b>, respectively. The intervals <b>411</b>, <b>412</b>, <b>413</b>, <b>414</b>, and <b>415</b> may be generated by including padding <b>408</b> in one or more of the intervals <b>401</b>, <b>402</b>, <b>403</b>, <b>404</b>, and <b>405</b> to match the length in bytes of the corresponding intervals of the original audio stream <b>400</b>, e.g., intervals <b>304</b>(<b>0</b>), <b>304</b>(<b>1</b>), <b>304</b>(<b>2</b>), <b>304</b>(<b>3</b>), and <b>304</b>(<b>4</b>). The streaming device <b>108</b> may easily locate corresponding intervals in the original audio stream <b>400</b> and the interval fade-in audio stream <b>410</b> in order to switch between the two audio streams at any interval boundary. Video data may be similarly encoded using a fixed bit rate or variable bit rate and the streaming device <b>108</b> may switch between different video streams at interval boundaries.

Prior to initiating playback, the streaming device <b>108</b> may measure available bandwidth from the content server and select a digital media content file having a bit rate that can be supported by the measured available bandwidth. To maximize playback quality, a digital media content file with the highest bit rate not exceeding the measured bandwidth is conventionally selected. To the extent the communications network <b>106</b> can provide adequate bandwidth to download the selected digital media content file while satisfying bit rate requirements, playback proceeds satisfactorily. In practice, however, available bandwidth in the communications network <b>106</b> is constantly changing as different devices connected to the communications network <b>106</b> perform independent tasks.

To counter the variability of network conditions, adaptive streaming may be implemented where, for each title, multiple video streams having different fixed bit rates exist. As the network conditions vary, the streaming device <b>108</b> may switch between video streams according to the network conditions. For example, video data may be downloaded from video streams encoded to higher fixed bit rates when the network conditions are good, and, when the network conditions deteriorate, subsequent video data may be downloaded from video streams encoded to lower fixed bit rates. The bit rate of the audio stream is typically much lower than the bit rate of the video stream, so the audio stream is typically only encoded for a single fixed bit rate. Because the bit rate for a particular interval of a VBR encoded video stream is not fixed, adaptive stream is best suited for use with fixed bit rate streams.

<figref idref="DRAWINGS">FIG. 5B</figref> is a more detailed view of the streaming device <b>108</b> of <figref idref="DRAWINGS">FIG. 1</figref>, according to one embodiment of the invention. As shown, the streaming device <b>108</b> includes, without limitation, a central processing unit (CPU) <b>510</b>, a graphics subsystem <b>512</b>, an input/output (I/O) device interface <b>514</b>, a network interface <b>518</b>, an interconnect <b>520</b>, and a memory subsystem <b>530</b>. The streaming device <b>108</b> may also include a mass storage unit <b>516</b>.

The CPU <b>510</b> is configured to retrieve and execute programming instructions stored in the memory subsystem <b>530</b>. Similarly, the CPU <b>510</b> is configured to store and retrieve application data residing in the memory subsystem <b>530</b>. The interconnect <b>520</b> is configured to facilitate transmission of data, such as programming instructions and application data, between the CPU <b>510</b>, graphics subsystem <b>512</b>, I/O devices interface <b>514</b>, mass storage <b>516</b>, network interface <b>518</b>, and memory subsystem <b>530</b>.

The graphics subsystem <b>512</b> is configured to generate image frames of video data and transmit the frames of video data to display device <b>550</b>. In one embodiment, the graphics subsystem <b>512</b> may be integrated into an integrated circuit, along with the CPU <b>510</b>. The display device <b>550</b> may comprise any technically feasible means for generating an image for display. For example, the display device <b>550</b> may be fabricated using liquid crystal display (LCD) technology, cathode-ray technology, and light-emitting diode (LED) display technology (either organic or inorganic). An input/output (I/O) device interface <b>514</b> is configured to receive input data from user I/O devices <b>552</b> and transmit the input data to the CPU <b>510</b> via the interconnect <b>520</b>. For example, user I/O devices <b>552</b> may comprise one of more buttons, a keyboard, and a mouse or other pointing device. The I/O device interface <b>514</b> also includes an audio output unit configured to generate an electrical audio output signal. User I/O devices <b>552</b> includes a speaker configured to generate an acoustic output in response to the electrical audio output signal. In alternative embodiments, the display device <b>550</b> may include the speaker. A television is an example of a device known in the art that can display video frames and generate an acoustic output. A mass storage unit <b>516</b>, such as a hard disk drive or flash memory storage drive, is configured to store non-volatile data. A network interface <b>518</b> is configured to transmit and receive packets of data via the communications network <b>106</b>. In one embodiment, the network interface <b>518</b> is configured to communicate using the well-known Ethernet standard. The network interface <b>518</b> is coupled to the CPU <b>510</b> via the interconnect <b>520</b>.

The memory subsystem <b>530</b> includes programming instructions and data that comprise an operating system <b>532</b>, user interface <b>534</b>, and playback application <b>536</b>. The operating system <b>532</b> performs system management functions such as managing hardware devices including the network interface <b>518</b>, mass storage unit <b>516</b>, I/O device interface <b>514</b>, and graphics subsystem <b>512</b>. The operating system <b>532</b> also provides process and memory management models for the user interface <b>534</b> and the playback application <b>536</b>. The user interface <b>534</b> provides a specific structure, such as a window and object metaphor, for user interaction with streaming device <b>108</b>. Persons skilled in the art will recognize the various operating systems and user interfaces that are well-known in the art and suitable for incorporation into the streaming device <b>108</b>.

The playback application <b>536</b> is configured to retrieve digital media content, e.g., audio and video streams, from the streaming server <b>102</b> via the network interface <b>518</b> and play the digital media content through the graphics subsystem <b>512</b>. The graphics subsystem <b>512</b> is configured to transmit a rendered video signal to the display device <b>550</b>. In normal operation, the playback application <b>536</b> receives a request from a user to play a specific title. The playback application <b>536</b> then identifies the different encoded video streams associated with the requested title, wherein each encoded video stream is encoded to a different playback bit rate. A preview clip may be encoded separately from the requested title or may be indicated by an index into the video and audio streams encoded for the requested title.

After the playback application <b>536</b> has located the encoded video streams associated with the requested title, the playback application <b>536</b> downloads sequence header indices associated with each encoded video stream associated with the requested title from the streaming server <b>102</b>. As previously described herein, a sequence header index associated with an encoded video stream includes information related to the encoded sequence included in the digital media content file.

In one embodiment, the playback application <b>536</b> begins downloading the digital media content file associated with the requested title comprising the encoded sequence encoded to the lowest playback bit rate to minimize startup time for playback. For the purposes of discussion only, the digital media content file is associated with the requested title and comprises the encoded sequence encoded to the lowest playback bit rate. The requested digital media content file is downloaded into the content buffer <b>543</b>, configured to serve as a first-in, first-out queue. In one embodiment, each unit of downloaded data comprises a unit of video data or a unit of audio data. As units of video data associated with the requested digital media content file are downloaded to the streaming device <b>108</b>, the units of video data are pushed into the content buffer <b>543</b>. Similarly, as units of audio data associated with the requested digital media content file are downloaded to the streaming device <b>108</b>, the units of audio data are pushed into the content buffer <b>543</b>. In one embodiment the units of video data are stored in video buffer <b>546</b> within the content buffer <b>543</b>, and units of audio data are stored in audio buffer <b>544</b>, also within the content buffer <b>543</b>.

A video decoder <b>548</b> reads units of video data from the video buffer <b>546</b>, and renders the units of video data into a sequence of video frames corresponding in duration to the fixed span of playback time. Reading a unit of video data from the video buffer <b>546</b> effectively de-queues the unit of video data from the video buffer <b>546</b> (and from the content buffer <b>543</b>). The sequence of video frames is processed by graphics subsystem <b>512</b> and transmitted to the display device <b>550</b>.

An audio decoder <b>542</b> reads units of audio data from the audio buffer <b>544</b>, and processes the units of audio data into a sequence of audio samples, generally synchronized in time with the sequence of video frames. In one embodiment, the sequence of audio samples is transmitted to the I/O device interface <b>514</b>, which converts the sequence of audio samples into the electrical audio signal. The electrical audio signal is transmitted to the speaker within the user I/O devices <b>552</b>, which, in response, generates an acoustic output.

Given the bandwidth limitations of the communications network <b>106</b>, the playback application <b>536</b> may download consecutive portions of video data from different constant bit rate encoded video streams based on available bandwidth. Other performance factors that may influence the specific encoded stream from which to download the portion of video data include the buffer size of the video buffer <b>546</b>, the behavior of the end-user viewing the video content, the type of display being generated (high-definition, standard-definition, etc) and the available lead time. These factors combined with the bandwidth limitations of the communications network <b>106</b> may be used to determine a specific encoded video stream from which to download each interval of the video data.

The transition component <b>304</b> receives content playback information including the content title and playback starting point. The transition component <b>304</b> determines the nearest interval in the digital media content at which to retrieve the audio and video streams and controls transitions between different streams. The sequence header indexes <b>538</b>-<b>1</b>, <b>538</b>-<b>2</b>, and <b>538</b>-<b>3</b> are each associated with a respective video or audio stream and are used by the transition component <b>304</b> to locate switch points defined by the SHI generator <b>218</b> within each stream. The transition component <b>304</b> may switch from playing a first audio and/or video stream at an interval boundary to playing a second audio and/or video stream.

Any of the video or audio streams may be a stream that represents a video or audio effect, including an effect that blends two different streams to produce a transition. For example, a fade-out audio stream may be blended with a fade-in audio stream for the same interval to generate an audio stream for transitions between different content titles. A fade-out audio stream is generated as a version of the original data stream that starts at a full volume level and decreases in volume level across the first interval of the second effect data stream. Similarly, a dissolve video stream or fade-out video stream may be blended with a fade-in video stream for the same interval to generate a video stream for transitions between the different content titles. A fade-in video stream is generated as a version of the original data stream that starts at a low visibility level and increases the visibility level across the first interval of the effect data stream. A fade-out video stream is generated as a version of the original data stream that starts at a full visibility level and decreases the visibility level across the first interval of the second effect data stream.

<figref idref="DRAWINGS">FIG. 5B</figref> is a flow diagram <b>560</b> of method steps for playing the fade-in audio stream <b>302</b> or the interval fade-in audio stream <b>310</b> and transitioning to the original audio stream <b>300</b>, according to one embodiment of the invention. Although the method steps are described in conjunction with the systems for <figref idref="DRAWINGS">FIGS. 1, 2 and 5A</figref>, persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the invention.

At step <b>562</b>, the playback application <b>536</b> receives a playback location associated with the digital media content for which playback has been initiated. The playback location may be at the start of the digital media content when the digital media content is first selected or at any timestep of the digital media content, including the start, following a trick play. At step <b>564</b>, the playback application <b>536</b> rounds down from the playback location to the nearest interval boundary. When the playback location coincides with an interval boundary, the playback application <b>536</b> uses the coinciding interval. In other embodiments, the playback application <b>536</b> may be configured to select the nearest interval boundary, rounding upward or downward as needed.

At step <b>566</b>, the playback application <b>536</b> retrieves one interval from the effect audio stream, e.g., interval fade-in audio stream <b>310</b> or interval fade-in audio stream <b>410</b>. When the playback location is the start of the digital media content, the playback application <b>536</b> may retrieve interval <b>302</b>(<b>0</b>) from the fade-in audio stream <b>302</b>. When the playback application <b>536</b> is configured to perform a transition and a transition stream is available for the one interval, the playback application <b>536</b> retrieves the one interval of the transition stream as the effect audio and/or video stream. At step <b>568</b>, the playback application <b>536</b> initiates retrieval of additional intervals that follow the one interval in sequence, from the original audio stream <b>300</b>.

At step <b>570</b>, the playback application <b>536</b> plays the one interval from the effect audio stream. At step <b>572</b> the playback application <b>536</b> determines if an interval boundary has been reached during playback, and, if not, then the playback application <b>536</b> repeats step <b>570</b> to continue playing the one interval. Otherwise, at step <b>574</b>, an interval boundary is reached and the playback application <b>546</b> transitions to play from the original audio stream <b>300</b>.

The technique shown in <figref idref="DRAWINGS">FIG. 5B</figref> may also be used to transition between video streams at the interval boundaries. In this manner audio and/or video effects may be implemented when the playback application <b>536</b> transitions from one digital media content to a different digital media content. A transition may also include beginning playback of a first digital media content when there is no previous digital media content or ending playback of a last digital media content without playing another digital media content. Finally, audio and/or video effects may be implemented when the playback application <b>536</b> transitions from one location in a digital media content to a different location in the same digital media content, i.e., following a trick play.

In order to select a specific encoded video stream from a set of fixed bit rate encoded video streams representing the same video data, the playback application <b>536</b> executing on the streaming device <b>108</b> may be configured to dynamically determine the encoding level (high, medium, or low bit rate) of the video stream for the next portion of the video data to be downloaded during playback of a different (previous) portion of the digital media content.

<figref idref="DRAWINGS">FIG. 6A</figref> is a flow diagram <b>600</b> of method steps for playing effect audio and video streams and transitioning to the original audio and video streams and transitioning between streams of different digital media content, according to one embodiment of the invention. Although the method steps are described in conjunction with the systems for <figref idref="DRAWINGS">FIGS. 1, 2 and 5A</figref>, persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the invention.

At step <b>602</b>, the playback application <b>536</b> receives a playback location associated with the digital media content for which playback has been initiated. The playback location may be at the start of the digital media content when the digital media content is first selected or at any timestep of the digital media content, including the start, following a trick play. At step <b>604</b>, the playback application <b>536</b> determines the interval boundary closest to the playback location, i.e., by rounding down or rounding up. When the playback location coincides with an interval boundary, the playback application <b>536</b> uses the coinciding interval.

At step <b>606</b>, the playback application <b>536</b> retrieves one interval from the interval audio and/or video effect streams, e.g., fade-in streams. At step <b>608</b>, the playback application <b>536</b> initiates retrieval of additional intervals from the original audio and video streams.

At step <b>610</b>, the playback application <b>536</b> plays the one interval from the effect audio and video streams, e.g., fade-in or transition streams. At step <b>612</b> the playback application <b>536</b> determines if an interval boundary has been reached during playback, and, if not, then the playback application <b>536</b> repeats step <b>610</b> to continue playing the one interval. Otherwise, at step <b>614</b>, an interval boundary is reached and the playback application <b>546</b> transitions to play from the original audio and video streams.

When digital media content is near a transition boundary, such as the end of the digital media content, the playback application <b>536</b> may transition from the original audio and video streams to different effect audio and video streams, e.g., fade-out effect streams. In some cases, the end of the digital media content may not be reached because the user selects a new playback location using a trick play or because the user selects different digital media content for playback.

At step <b>616</b> the playback application <b>536</b> determines if a transition boundary has occurred during playback, and, if not, then the playback application <b>536</b> repeats step <b>614</b> to continue playing the digital media content. Otherwise, at step <b>616</b>, a transition boundary is reached and at step <b>620</b> the playback application <b>536</b> determines if the next digital media content (if any) has been selected. When the next digital media content has not been selected, then at step <b>522</b> the playback application <b>536</b> plays the one interval from the fade-out audio and video streams. Alternatively, at step <b>622</b> the playback application <b>536</b> may retrieve and play different effect streams or the original streams for the last interval.

Otherwise, at step <b>618</b> when the next digital media content has been selected, and at step <b>620</b> the playback application <b>536</b> determines if a transition stream is available. When a transition stream is available for the interval and digital media content(s), then at step <b>622</b> the playback application <b>536</b> retrieves one interval from the transition streams before returning to step <b>608</b>. When a transition stream is not available the interval and digital media content(s), then the playback application <b>536</b> returns to step <b>606</b> and retrieves one interval from the effect streams of the selected digital video content.

The technique described in conjunction with <figref idref="DRAWINGS">FIG. 6A</figref> may be used to fade out a preview clip or to dynamically assemble a sequence of clips from different digital media content, transitioning between the different digital media content by retrieving and playing transition intervals that blend an interval from the previously played digital media content with the same interval from the next digital media content or a different location in the same digital media content.

<figref idref="DRAWINGS">FIG. 6B</figref> is a flow diagram <b>650</b> of method steps for playing a switch event stream and during playback of audio and video streams, according to one embodiment of the invention. Although the method steps are described in conjunction with the systems for <figref idref="DRAWINGS">FIGS. 1, 2 and 5A</figref>, persons skilled in the art will understand that any system configured to perform the method steps, in any order, is within the scope of the invention.

A switch event occurs when supplementary audio and/or video data is overlaid during playback of digital media content. The supplementary audio and video data may include an out-of-band signal such as a chime or beep for an alarm-clock, indication that a new message, email, voicemail, tweet, or other communication is received. The supplementary audio or video data may be unrelated to the digital media content that is being streamed for playback.

At step <b>655</b>, the playback application <b>536</b> plays the audio and video streams for the digital media content. At step <b>660</b> the playback application <b>536</b> determines if a switch event has occurred during playback, and, if not, then the playback application <b>536</b> repeats step <b>655</b> to continue playing the digital media content. Otherwise, at step <b>665</b>, the playback application <b>536</b> identifies the next interval in the audio and video streams.

At step <b>670</b>, the playback application <b>536</b> retrieves one interval from the event streams that corresponds to the next interval. The audio event stream may be a modulated version of the original audio stream combined with the supplementary audio signal. Similarly, the video event stream may be a modulated version of the original video stream combined with the supplementary video signal.

At step <b>675</b>, the playback application <b>536</b> initiates retrieval of additional intervals from the original audio and/or video streams. At step <b>680</b>, the playback application <b>536</b> plays the one interval from the event streams. At step <b>685</b> the playback application <b>536</b> determines if an interval boundary is reached during playback of the event streams, and, if not, then the playback application <b>536</b> repeats step <b>680</b> to continue playing the event streams. Otherwise, at step <b>685</b>, an interval boundary is reached and the playback application <b>546</b> returns to step <b>655</b> to play from the original streams.

One advantage of the disclosed technique is that audio data may be modulated and received by the streaming device for an audio fade-in effect. Similarly, video data may be modulated to perform various visual effects. Such a technique allows for fading-in of audio and/or video, fading-out of audio and/or video, and effects that blend audio from different content and video from different content during a transition from one digital media content to a different digital media content or within the same digital media content. A user may personalize the video and/or audio effects that are applied during transitions. The modulated audio and video data to generate the audio or video effect during a transition may be precomputed or computed on-the-fly, i.e., in real-time, and transmitted to the streaming device.

In one embodiment of the invention the streaming device <b>108</b> comprises an embedded computer platform such as a set top box. An alternative embodiment of the invention may be implemented as a program product that is downloaded to a memory within a computer system, for example as executable instructions embedded within an internet web site. In this embodiment, the streaming device <b>108</b> comprises the computer system.

While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof. For example, aspects of the present invention may be implemented in hardware or software or in a combination of hardware and software. One embodiment of the invention may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored. Such computer-readable storage media, when carrying computer-readable instructions that direct the functions of the present invention, are embodiments of the present invention.

In view of the foregoing, the scope of the present invention is determined by the claims that follow.