RTP Payload Format

(74) Proxy (54) Designation (56) Cited publications (57) Summary

Microsoft Technology Licensing, LLC, One Microsoft Way, US-WA98052 REDMOND, USA James M Alkove, 15906 210th Avenue NE, US-WA98 072 WOODINVILLE, USA Anders E Klements, 10834 Redmond-Woodinville Road NE, US-WA98052 REDMOND, USA Bryn Aarflot AS, PO Box 449 Sentrum, 0104 OSLO, Norway

Utility data format under RTP (Real-time Transport Protocol)

NAFAA A. et al .: RTP4mux: A Novel MPEG-4 RTP Payload for Multicast Video Communications over Wireless IP, IEEE-PV 2003, 13th International Packet Video Workshop, April 28, 2003, [Online],

EP 1041823 A2

A.KIemets: Common Generic RTP Payload Format, Internet Engineering Task Force, INTERNET DRAFT, draft-clemets-generic-rtp-OO, March 13, 1998.

A. Clamps: RTP Playload Format for ASF Streams, IETF, Microsoft Corporation, INTERNET DRAFT, draft clamps-asf-rtp-OO, Oct. 8.1997.

A. PERIYANNAN ET AL .: Delivering Media Generically over RTP, IETF, draft-periyannan-generic-rtp-OO, March 13,1998.

A data stream is encrypted to form cryptographic devices that are packaged into RTP packets. Each RTP p acket comprises an RTP packet header, one or more sets of useful data from a common or the same type of data stream and an RTP useful data format header for each set of useful data, which comprises, for the associated cryptographic units, a limit for the useful data. The utility data can be one or more cryptographic units or a proportion of one of the cryptographic units. The cryptographic units are reassembled using the utility data in the RTP packets and the respective boundaries provided by the respective RTP utility data format header. The reassembled cryptographic units are decrypted for rendering. Each RTP utility data format header can contain properties of the associated utility data that can be used for or when rendering the utility data. The RTP packets can be sent from server to client or between identical data devices.

<img file="NO339940B1_D0001.tif" />

[0001] The present invention relates to the RTP (Real-time Transport Protocol), and more specifically to a transmission format under RTP for streaming media (e.g. audio and video) over a network, such as the Internet.

[0002] The following description assumes that the reader is familiar with the standards IETF RFC 1889 - RTP: A Transport Protocol for Real-Time Applications and IETF RFC 1890 - RTP Profile for Audio and Video Conferences with Minimal Control.

[0003] RTP, as defined in the RFC 1889 standard, provides end-to-end network transport features suitable for applications that transmit real-time data, such as audio, video, or simulation data, over multicast or unicast-based network services. These transport features provide end-to-end delivery services for real-time data, such as interactive audio and video. Such services include identification of the type of utility data, sequence numbering, time stamping and delivery control. RTP supports data transmission to multiple destinations using multicast transmission if supported by the underlying network.

[0004] The RFC 1889 standard does not provide a mechanism to ensure timely delivery or provide other quality of service guarantees, but assumes that lower tier services do so. It neither guarantees delivery nor prevents delivery in the wrong order, nor does it assume that the underlying network is reliable and delivers the data packets in the correct order. The sequence numbers incorporated in RTP allow the receiver to reconstruct the sender's packet sequence, but sequence numbers can also be used to determine the correct position of a data packet, for example when decoding video, without the data packets necessarily being decoded in the correct order.

A. PERIYANNAN ET AL: Delivering Media Generically over RTP, IETF,

March 13, 1998 sets out a method for delivering generic media streams over the Realtime Transport Protocol (RTP). This proposal is intended for media or codec types that have not already been addressed by other RTP payload specifications. Three packaging devices are defined for media data carriers. Session Description Protocol (SDP) is used to communicate to recipients which packing device is being used, and media data encoding format and parameters for the media encoding format.

A. Klemets: RTP Playload Format for ASF Streams, IETF, Microsoft Corporation, INTERNET DRAFT, Oct. 8, 1997 describes a payload format specification for encapsulating "Advanced Streaming Format (ASF)" streaming in a real-time transport protocol (RTP). This specification is primarily intended for ASF media types or codecs that have not already been addressed by other RTP payload specifications. Each ASF stream is sent with a separate RTP synchronization source ID and streams are synchronized using standard RTP techniques. A special encapsulation scheme for ASF streams is described, where each RTP package contains an ASF payload header and an ASF payload. This specification is primarily intended for the flow of ASF currents that do not require reliable transmission.

A.KIemets: Common Generic RTP Payload Format, Internet Engineering Task Force, INTERNAL ET-DRAFT, March 13, 1998 describes a generic payload format for encapsulating arbitrary data in RTP packages. The payload format implements a minimal set of features that are expected to be useful for most applications, while limiting overhead to as little as one byte. An extension mechanism makes it possible to use the common generic payload format as a basis for more complex payload formats. This specification is primarily intended for compaction devices that are not covered by other RTP payload formats. It is expected that this specification will be suitable for, but not limited to, flow data stored in a file format that supports several media types, such as QuickTime, ASF, and MPEG-4 Intermedia file format.

EP1041823 A1 describes a content distribution apparatus for implementing copy in Protect Ise when digital content is distributed as a real-time stream on the Internet. T he device encrypts the content and distributes it to a receiving device via the Internet, and performs an authentication procedure and a key exchange procedure between itself and the receiving device. The encoded content is encoded by a prescribed encoding system and is encrypted (S401), an encryption extension tip is generated which includes at least one attribute information of attribute information indicating whether or not the content is encrypted, and attribute information indicating the encryption system used (S403). transport protocol the processing required to transfer the content is performed and a basic transport header is generated (S407), a package is sent that includes basic transport header, encryption extension header, and encrypted content (S409).

NAFAA A. et al describe in the article: RTP4mux: A Novel MPEG-4 RTP Payload for Multicast Video Communications over Wireless IP, IEEE-PV 2003, 13th International Packet Video Workshop, April 28, 2003, on admini stration and distribution in a framework that is is expected to become an important part of many upcoming Wireless IP Multimedia services. The current solution for transporting MPEG-4 elementary streams over IP networks is not optimized for wireless group communication. To address this issue, a new (RTP) RealTime Transport Protocol payload is proposed, called RTP4mux, which provides better data multiplexing and flow aggregation over shared wireless IP connections. RTP4mux offers the following transport features: (1) an MPEG-4 elementary flow retrieval mechanism that minimizes time code dependency between adjacent RTP packets and then improves shed packet loss tolerance; (2) a configurable two-level access device multiplexing device, which optimizes wireless utilization of bandwidth and reduces end-to-end transmission delays with a lower data control overhead and a shorter packing latency.

[0005] A typical application of RTP involves the flow of data, where packets of ASF (Advanced Systems Format) formatted audiovisual data are transmitted in RTP packets over a network from a server to a client or between equivalent computers. The ASF-formatted audio and video data can be stored together in one ASF-formatted data packet, or ASF packet. As such, an RTP packet can contain both audio and video data.

[0006] RTP, as defined by the RFC 1889 standard, lacks the flexibility to combine multiple payloads, or sets of payload data, into a single RTP packet, or to split a set of payload data across multiple RTP packets. Nor does RFC 1889 define any format in which metadata can be transmitted with each set of utility data in an RTP packet. Another disadvantage of RFC 1889 is that it does not include any mechanism for flowing encrypted data blocks over a network that simultaneously maintains a defined boundary of each encrypted block so that its receiver has the ability to decrypt the encrypted data blocks. It will be an advance in technology to provide this flexibility as an improvement in RTP-based flow. Accordingly, there is a need for improved methods, computer readable media, data structures, apparatus and data devices capable of providing or supporting such functionality.

[0007] In one embodiment, packets of ASF-formatted audiovisual data are repackaged into RTP-formatted data packets and transmitted over a network from a server to a client, or through peer-to-peer network communication, in response to a request to stream the audiovisual data. The audiovisual data is encrypted to form cryptographic units. The repackaging process involves wrapping the cryptographic units in the RTP packets, each of which includes an RTP packet header, one or more sets of payload data from a common data stream, and an RTP-PF (Payload Format) header for each set of new data. The RTP-PF header includes, for the associated crypto-raw fisheries units, a limit for the utility data. The utility data in the RTP package can be one or more cryptographic raw fish units or a proportion of a cryptographic unit. After the RTP packets have been sent over a network, the cryptographic raw fish units contained in the received RTP packets are reassembled. The re-assembly process uses the utility data in the RTP packets and the associated boundaries in the respective RTP-PF headers. The re-assembled crypto-raw fishing units can then be decrypted for rendering. Each RTP-PF header can contain information regarding its associated utility data that can be used in or for rendering the utility data.

[0008] In a variation of the above embodiment, data in a format other than ASF may be used to form the RTP packets. In yet another variation of the above embodiment, the RTP packets are formed by utility data that is not encrypted.

[0009] In yet another embodiment, a transmission format is provided for streaming encrypted data blocks protected by WM DRM (Windows® Media).

Digital Rights Management) over a network in RTP packets (eg for streaming

WM DRM protected content). Each RTP packet contains header data that delimits each encrypted block so that each cryptographic raw device can be decrypted by the receiver. After decryption using the WM DRM protocol, the data stream can be rendered by the receiver.

[0010] Figure 1 illustrates an example method according to an embodiment of the invention for converting two (2) packets of ASF-formatted audiovisual data into four (4) RTP packets, where the audio data and the video data are packaged separately in the resulting The RTP packets and where the block boundaries of each set of utility data are preserved so that original samples of audiovisual data that was encrypted and wrapped in the two ASF packets can be reconstructed by a decryption mechanism.

[0011] Figure 2 illustrates alternative examples of methods according to various embodiments of the invention for converting two (2) packets of ASF formatted video data into one (1) RTP packet, where one alternative method takes the useful data from the ASF packets and adds enter them as separate sets of utility data in the RTP packet and the second alternative method compiles the utility data from the ASF packets into a single set of utility data in the RTP packet, and where block boundaries for each set of utility data are preserved so that an original video clip that was encrypted and wrapped in the two ASF packets can be reconstructed by a decryption mechanism.

[0012] Figures 3a-3b show respective structures of data structures, in accordance with an embodiment of the present invention, for an RTP header and an associated utility data header.

[0013] Figure 4 is a block diagram, in accordance with an embodiment of the present invention, illustrating a network-based client / server system where data flow may occur from server to client or between equivalent computers.

[0014] Figure 5 is a block diagram, in accordance with an embodiment of the present invention, illustrating communication between a server (or client) and a client, where the server (or client) communicates to the client a requested stream of audiovisual data that the client is able to showcase.

[0015] Figure 6 is a block diagram, in accordance with an embodiment of the present invention, illustrating a networked computer that may be used to realize either a server or a client.

[0016] Embodiments described herein define transmission formats for the delivery of uniform or mixed data streams, such as Windows® media data, under RTP. The transfer can take place between a server and a client or between equivalent entities (for example, in a Windows® Messenger for software environment for audiovisual conferences).

[0017] A transmission format, in various embodiments, extends the IETF RFC 1889 standard to provide more flexibility for transmission under RTP.

Embodiments provide a mechanism for streaming audio data in RTP packets separately from video data in RTP packets. Embodiments also provide a transmission format with which metadata can be transmitted along with each set of utility data in an RTP packet, where the metadata provides rich information describing the utility data. Still other embodiments provide a mechanism for streaming encrypted data blocks over a network that simultaneously maintains a block boundary for each encrypted block so that the recipient thereof can decrypt the encrypted data blocks. In another embodiment, a transmission format provides the delivery of data protected with WM DRM (Windows® Media Digital Rights Management), so that the delivery thereof can be unencrypted / decrypted for display.

[0018] Various embodiments described herein include packets of data from a series of media packets incorporated into a system layer bitstream. This data is repackaged into RTP packets as follows, but still extends the RFC 1889 standard, so that the system layer bitstream is mapped to RTP. In this image, each media packet contains one or more sets of utility data. In some system layer bitstreams, there may be mixed media packets that contain data such as audio data, video data, program data, JPEG formatted data, HTML formatted data, MIDI formatted data, etc. A mixed media packet is a media packet where two or several contained sets of utility data belong to different media streams, or streams of different media.

[0019] Different embodiments are applicable to system-layer bitstreams where all media packets are uniform, or comprise a single medium. In a uniform media package, all sets of utility data belong to the same media stream. Other embodiments are applicable to system layer bitstreams where each media packet always contains only one (1) set of utility data. In still other embodiments, the size of the utility data header in the media packet is zero - which is common if each media packet contains only one set of utility data, but may also occur if multiple sets of utility data exist and the media packet header contains information about the size of each set of utility data.

[00020] Figures 1-2 show examples of an embodiment where the system layer bitstreams comprise a series of ASF shapes. data packets each containing data. This data is repackaged into RTP packets as follows, but still extends the standard RFC 1889. Here, the system layer bitstreams comprise a series of ASF-formatted media packets, and the utility data in each ASF packet is ASF utility data. Although ASF packets are used here for illustrative purposes, the generation of RTP packets in other embodiments described herein is not limited to the use of ASF formatted data, but may instead use other formats for storing data to be streamed. Such other formats, as well as the ASF format, are generally described herein as system-layer bitstreams that include a number of media packets that all contain data, and this data is converted to the RTP format in various embodiments.

[0021] Figure 1 shows ASF formatted audiovisual data 100. The data 100, which includes audio data 102 and video data 104, is wrapped in an ASF packet A 106 and an ASF packet B 108. ASF packet A 106 comprises a first ASF header, an ASF utility data header, audio data 102, a second ASF utility data header and a portion A of video data 104. ASF packet B 108 comprises an ASF header, an ASF utility data header and a portion B of video data 104.

[0022] The audiovisual ASF data 100, as shown in the form of ASF packet A 106 and ASF packet B 108, can in one embodiment be wrapped in several RTP packets. As can be seen in Figure 1, these RTP packets include A 110, RTP packets 112 (1) to 112 (N) and RTP packet D 116. Each RTP packet, in accordance with RFC 1889, has an RTP packet header , payload data and an RTP-PF (Payload Format) header. As used herein, the RTP-PF header is a payload data header in the RTP packet. Only one (1) type of medium is included in the RTP package. In other words, the RTP package does not contain payloads that include mixed media. In the embodiment illustrated in Figure 1, the video data A in the ASF packet A 106 is too large to fit in a single RTP packet. As a result, the video data A in ASF packet A 106 is distributed on the RTP packets 112 (1) to 112 (N). The size of the RTP packets may depend on a physical property of an underlying network over which the RTP packets are to be transmitted, on an administrative provision for packet size, which may, for example, be implemented by the person responsible for the underlying network, or on an assessment of the available bandwidth for transmission in the underlying network.

[0023] After repackaging to RTP packets as illustrated in Figure 1, the audio data 102 is incorporated in RTP packet A 110 and the video data B from ASF packet B is incorporated in RTP packet D 116. Each RTP-PF header in each RTP packet may contain information about the division of the audio and video data into respective separate RTP packets. The audiovisual data 124 can thus be reconstructed from the audio data in RTP packet A 110, the portions 1 to N of the video data A in the respective RTP packets 112 (1) to 112 (N) and the video data B in RTP packet D 116. When the reconstruction of the audiovisual data 124 is completed, the audio clip data 120 and the video clip data A + B 122 therein can be displayed in a data stream context. In summary, Figure 1 illustrates a transfer format according to which smaller RTP packets are generated from larger ASF packets, where the repackage writes utility data from different data streams to separate data packets each having its own RTP-PF header. Figure 1 also illustrates an embodiment of a transmission format according to which block boundaries for each set of utility data are preserved, so that original audio and video clips that were encrypted and packaged in ASF packets can be reconstructed by a decryption mechanism used on RTP. packages.

[0024] Figure 2 illustrates ASF formatted audiovisual data 200. The data 200, which includes video data 202, is wrapped in an ASF packet A 208 and an ASF packet B 210. ASF packet A 208 contains an ASF header, a ASF utility data header and video data A 204. ASF packet B 210 contains an ASF header, a

ASF utility data header and video data B 206. Figure 2 illustrates two (2) options for repackaging the data 200 into RTP packets as follows, yet extending the RFC 1889 standard.

[0025] In the first alternative, in the direction indicated by the arrow 250, video data A 204 and video data B 206 are wrapped in a single RTP packet A 212 having an RTP header. Each of the video data A 204 and the video data B 206 is initialized by an RTP-PF header. The RTP package A 212, in accordance with RFC 1889, has an RTP header, several sets of utility data and associated RTP-PF headers.

[0026] In the second alternative, also in the direction indicated by the arrow 250, video data A 204 and video data B 206 from respective ASF packets are wrapped in an RTP packet B 214 having an RTP header. The video data A 204 and the video data B 206 are composed and constitute the useful data in RTP package B 214. The useful data is initiated by an RTP-PF header. The RTP package B 214, in accordance with RFC 1889, has one RTP header, utility data and one RTP-PF header.

[0027] After repackaging into RTP formatted data packets as illustrated in Figure 2, the video data A and B (204, 206) are incorporated into either RTP packet A 212 or RTP packet B 214. Each RTP-PF header may contain information about the associated utility data. Each of the alternative RTP packets 212, 214 contains sufficient information to reconstruct ASF packet A 208 and ASF packet B 210 to fill them with video data A and B (204, 206). When the reconstruction is complete, the video clip data 222 can be displayed in a data stream context. In summary, Figure 2 illustrates a transmission format under RTP according to which larger RTP packets are generated from smaller ASF packets, and where block boundaries for each set of utility data are preserved, so that original video clips that were encrypted and packed in the two ASF packets can be reconstructed by a decryption mechanism applied to the RTP packets.

[328] Figure 3a illustrates a data structure for fields in an RTP header. The RTP header is described in more detail in RFC 1889. The timestamp field in the RTP header should be set to the time for presentation of the data samples contained in the RTP package. In one embodiment, the clock frequency is 1kHz unless otherwise specified by means of means independent of RTP.

[0029] Bit number eight from the beginning of the RTP header is interpreted as a marker bit (M-bit). The M-bit is set to zero, but will be set to one (1) if the associated RTP packet contains utility data that is not a fragment of a data sample, that contains the last fragment of a data sample, or that is one of several complete data samples in the RTP package. The M-bit can be used by a receiver to detect reception of a complete data clip for decoding and presentation. The M-bit in the RTP header can thus be used to mark significant events in a stream of data packets (e.g. frames of frames in a video clip).

[0030] Figure 3b illustrates one embodiment of an RTP-PF header or utility data header. The RTP-PF header comprises a fixed portion of sixteen (16) bit lengths, followed by a variable length portion. The fields in the RTP-PF header shown in Figure 3b comprise an 8 bit long data string indicated by the data character fields SGLRTDXZ, a length / offset field, a relative time stamp field, a decompression time field, a duration field and a field indicating the length to the Payload Extension (PE) data with an associated field for the PE data, all of which are explained below.

[0031] The S field comprises one (1) bit and is set to one (1) if the associated useful data (e.g. a data sample, a portion of a data sample or a compilation of several data samples) is a key sample , ie an internally coded data sample or an I-frame. Otherwise it is set to zero. The S-bit in all RTP-PFheaders that initiate shares of the same data sample must be set to the same value.

[0032] The G field comprises one (1) bit and is used to group sub-samples into an associated set of utility data which constitutes a single data sample. Windows® Media Digital Rights Management encrypts content based on ASF utility data boundaries. To enable proper decryption of this content, the boundaries of the sub-samples in the utility data can be communicated to the client who is to receive the utility data. For example, a cryptographic unit may be packaged in such a way that it is divided into several transmission units (e.g. placed in separate data packets) to be sent. Before the divided collection of transmission units can be decrypted by a receiving client, they must be returned to the original, encrypted form. As with other decryption techniques and mechanisms, the client can use the boundaries to reconstruct the encrypted cryptographic units in preparation for decrypting the encrypted content. For this reason, each set of ASF utility data should be preceded by this RTP-PF header.

[0033] The value in the G field is set to zero (0) to indicate that an encrypted device is split. If the ASF format is used, the crude phishing unit will be ASF utility data and the bit value will be set to zero (0) in all divided sets of ASF utility data, except the last set of ASF utility data. In this case, it does not matter if a data sample is split or not. If the ASF format is not used, the crude fishing unit is a media sample, in which case the Gbit is set to zero (0) in all split media samples except the last sample. In this latter case, the issue of whether ASF utility data is divided or not is not relevant, as ASF has not been used.

[0034] The L field comprises one (1) bit and is set to one (1) whose length / offset field contains a length. Otherwise it is set to zero (0), which indicates that the length / offset field contains an offset. The L-bit must be set to one (1) in all RTP-PF headers that initiate utility data that contains a complete (unfragmented) data sample and must be set to zero in all RTP-PF headers that initiate a set of utility data that contains a fragmented data sample.

[0035] The R field comprises one (1) bit and is set to one (1) if the RTP-PF header contains a relative time stamp. Otherwise it is set to zero. The R-bit in all headers that initiate shares of the same data sample must be set to the same value.

[0036] The T field comprises one (1) bit and is set to one (1) if the RTP-PF header contains a decompression time. Otherwise it is set to zero. The T-bit in all RTPPF headers that initiate utility data that contains a portion of the same data sample must be set to the same value.

[0037] The D field comprises one (1) bit and is set to one (1) if the RTP-PF header contains the duration of a data sample. Otherwise it is set to zero. The D-bit in all

RTP-PF headers that initiate utility data that contain shares of the same data sample must be set to the same value.

[0038] The X field comprises one (1) bit and is for optional or unspecified use. A sender of an RTP packet should do this to zero, and the recipient of the RTP packet does not have to worry about this field.

[0039] The Z field comprises one (1) bit and is set to one (1) if the RTP-PF header contains payload expansion data, which may be metadata associated with the associated payload data. Otherwise, the Z field is set to zero. The value in the Z field can be zero for all RTP-PF headers if M-bit is zero, but should be set in all RTP-PF headers if M-bit is set to one (1) if the associated utility data has associated payload extension data.

[0040] The length / offset field comprises twenty-four (24) bits and indicates the length or offset of a single data sample which is divided into several RTP packets. The L-bit is set to zero and the length / offset field contains the offset value in bit octets of the first bit octet in this fragment from the beginning of the associated utility data (eg a data sample or a portion of such) . If one or more complete data samples are contained in the RTP packet, set the L-bit to one (1) in each RTP-PF header, and the length / offset field contains the length of the data sample (including the RTP-PF header).

[0041] The relative timestamp field comprises thirty-two (32) bits and is provided only if the above-mentioned R-bit is set to one (1). It contains the relative timestamp of the associated data sample relative to the timestamp in the associated RTP header. The time scale used is the same as that used in the time stamp in the RTP header. The relative timestamp field is specified as a sign-fixed 32-bit number to enable negative offset values ​​relative to the timestamp in the RTP header. When the relative timestamp field is not provided, a default value of zero can be used.

[0042] The decompression time field comprises thirty-two (32) bits and is provided only if the above-mentioned T-bit is set to one (1). It contains the decompression time relative to the time stamp in the RTP header. The time scale used is the same as that used for the time stamp in the RTP header. This field is specified as a sign-fixed 32-bit number to enable negative offset values ​​relative to the time stamp in the RTP header.

[0043] The duration field comprises thirty-two (32) bits and is provided only if the above-mentioned D-bit is set to one (1). It indicates the duration of the associated data sample. The time scale used is the same as that used for the time stamp in the RTP header. The duration field should be set to the same value in all RTP-PF headers that initiate fragments of the same data sample. When this field is not provided, the default duration is obtained indirectly or directly from the data sample. If this is not appropriate, the default value is equal to the difference between the time stamp of this data sample and the time stamp of the next data sample.

[0044] The field indicating the length of the utility data expansion data comprises sixteen (16) bits and is provided only if the above-mentioned Z-bit is set to one (1). It indicates the number of bit octets of utility data expansion data (PE data) contained after the fixed part of the RTP-PF header. The PE data has variable length and contains one or more attributes that describe the associated utility data that it initiates. The field indicating the length of the PE data immediately follows the fixed part of the utility data header, and will indicate the number of bit octets containing the PE data. The structure of the PE data is communicated between the server and the client (or between equivalent entities), for example through an SDP description. In one embodiment using content protected with WM DRM, there may be at least 4 bit octets with DUE data representing the WM DRM utility data identifier associated with each data sample.

[0045] Although Figures 3a-3b illustrate different fields in a given order in an RTP header and an RTP-PF header, not all fields are mandatory and the order of the fields can be changed. In some embodiments, the mandatory fields and their order may follow, but still increase the flexibility of the RFC 1889 standard. Although ASF packets are used for illustration purposes in Figures 3a-3b, generation of RTP packets, RTP-PF headers and associated utility data in other embodiments set forth herein is not limited to the use of ASF formatted data, but may instead be used. other formats for storing data to be streamed.

[0046] Figure 4 shows a client / server-based network system 400 and an environment in accordance with the invention. Generally, the system 400 includes one or more (m) multimedia servers and one or more (k) clients 404. The computers communicate with each other over a data communication network, which in Figure 4 comprises a wired / wireless network 406.

The data communication network 406 may include the Internet or local area networks and private regional networks. Servers 402 and clients 404 communicate with each other using any of a variety of known protocols, such as TCP (Transmission Control Protocol) or UDP (User Datagram Protocol).

[0047] Multimedia servers / clients 402/404 have access to streaming media content in the form of streams of various media. These media streams may be streams of individual media (eg audio, video, graphics, simulation data, etc.), or alternatively streams of composite media comprising several such uniform data streams. Some media streams may be stored as records or files 408 in a database (e.g. ASF files) or another file storage system, while other media streams 410 may be delivered to the multimedia server 402 or client 404 live from other data sources through reserved communication channels or directly over the Internet.

[0048] The media streams received from servers 402 or from clients 404 are displayed by the client 404 as a multimedia presentation, which may include media streams from one or more of the servers / clients 402/404. These different media streams may comprise one or more of the same or different types of media streams. For example, a multimedia presentation may include two video streams, one audio stream and one graphic image stream. A user interface at client 404 can provide the user with a set of controls, which, for example, allow the user to either increase or decrease the speed of reproduction of the multimedia presentation.

[0049] In the description that follows, the invention will be described in the general context of computer-executable instructions, such as program modules, which are executed by one or more traditional personal computers. In general, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will appreciate that the invention may be practiced in other computer system designs, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, networked personal computers, minicomputers, mainframes, and the like. In a distributed computing environment, application modules can be located in both local and remote storage devices. Alternatively, the invention may be implemented in hardware or in a combination of hardware, software and / or firmware. For example, one or more application-specific integrated circuits may be programmed to realize the invention.

[0050] As can be seen in Figure 4, a network system in accordance with the present invention comprises one or more network servers and clients 402/404 from which a number of media streams are available. In some cases, the media streams are stored by server (s) 402 and / or client (s) 402,404.

In other cases, server (s) and client (s) 402,404 may provide the media streams from other network sources or devices. In general, network kl in ente r 404 responds to user input requesting media streams corresponding to the selected multimedia content. In response to a request for a media stream corresponding to the desired multimedia content, server (s) 402 and / or client (s) 404 send the requested media streams to the querying network client 404 according to a transmission format under RTP. The client 404 decrypts the utility data in the respective RTP packets and displays the resulting unencrypted data streams to produce the requested multimedia content.

[0051] Figure 5 illustrates the input and storage of audiovisual data streams by a server 402 or a client 404. Figure 5 also illustrates communication between server and client (402-404) or between equivalent computers (404404) according to different embodiments. In summary, the server 402 or client 404 receives input of a stream of audiovisual data from an input device 530. The server or client 402, 404 encodes the input using a recoder in a codec. The coding can, but need not, be performed on ASF formatted data. If ASF formatted data is used, the coding is performed on ASF packets, each of which includes an ASF header, an ASF utility data header, and audiovisual utility data (in the form of audio and / or video). The encoding may include encryption, for example if WM DRM is used. The ASF packets are stored by the server / client 402,404 to execute future requests for this data.

[0052] Some time later, the client requests the current stream of audiovisual data from the server / client. The server / client retrieves and sends to the client the current audiovisual data stream that the server / client has previously stored. Upon receipt, the client decodes the audiovisual data stream and reconstructs and decrypts encrypted, divided streams of audiovisual data sets using block boundaries communicated in the associated RTP-PFheaders. The client can then display the streamed audiovisual data.

[0053] Figure 5 illustrates the flow of data between and within blocks / steps 504-530. In step 504, an input device 502 provides an input to server / client 402/404 which includes a stream of audiovisual data. As an example, the audiovisual data stream may be provided live to server / client 402/404 by the input device 502 over reserved communication channels or over the Internet. The audiovisual data stream is supplied to an encoder in step 504 for inserting the data into ASF packets. In step 506, any WM DRM encryption is performed, and the ASF packets are stored at server / client 402/404. One result of WM DRM encryption and packaging may be that a cryptographic device is divided into a number of separate data packets. Before the split transfer units can be decrypted by a receiving client, it must reconstruct the original cryptographic units. For this purpose, the boundaries of the divided transmission units are stored in ASF utility data headers in step 506.

[0054] In step 508, client 404 requests the stream of audiovisual data sent to server / client 402/404, as indicated by arrow 510 in Figure 5. In step 512, server / client 402/404 receives the request. The current

The ASF packets containing the requested audiovisual data stream are retrieved. In step 514, audio and video data from the ASF packets are logically separated for separate wrapping in RTP packets. Boundaries are identified for each logically separated set of audio and video data.

[0055] The available bandwidth in the network over which the RTP packets are to be transmitted is determined. This is done to calculate a predetermined RTP packet size. If the ASF packets are smaller than the predetermined RTP packet size, utility data of the same type can be combined into a single RTP packet. If the ASF packets are larger than the predetermined RTP packet size, the utility data in the ASF packets can be split for insertion as utility data in several RTP packets. Limits for each set of RTP utility data are determined using the corresponding, logically separated audio and video data from the ASF packets.

[0056] In step 516, the RTP header, the RTP-PF header and the associated utility data for each RTP packet are assembled. A number of RTP packets are now generated representing a number of ASF packets, the ASF packets containing the audiovisual data stream requested by client 404. The RTP packets are streamed in step 518 using a server / client transfer function 402/404 for showing at client 404.

[0057] An arrow 520 in Figure 5 indicates the transfer of the RTP packets from server / client 402/404 to client 404. In step 522, client 404 receives the RTP packets. In step 524, an RTP decoder at client 404 decodes each received RTP packet, including the RTP header and the RTP-PF header. In step 526, a process performs defragmentation and reconstruction of the ASF packets containing the requested audiovisual data stream. The defragmentation and reconstruction process uses limits specified in the RTP-PF header for each associated set of utility data, for example containing a data sample or a portion of such.

[0058] In step 528, the reconstructed ASF packets are decrypted for display in step 530. The RTP-PF header in an RTP packet may contain PE data describing the corresponding utility data. The PE data can thus provide metadata that can be used to display the utility data in the corresponding RTP packet in step 530. Steps 522-530 are repeated for each RTP packet received at client 404, thereby completing the flow of the audio / video data from server / client 402/404 for display.

[0059] Figure 6 shows a general example of a computer 642 which may be used in accordance with the invention. The computer 642 is shown as an example of a computer capable of performing the functions of any of the clients 402 or servers 404 in Figures 4 and 5. The computer 642 comprises one or more processors or processing units 644, a system memory 646 and a system bus 648 which connects various system components including the system memory 646 to the processing unit (s) 644.

[0060] Bus 648 represents one or more of any of many available bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus, which uses any of a variety of alternative bus architectures. . The system memory includes read only memory (ROM) 650 and direct access memory (RAM) 652. A cache 675 with levels L1, L2 and L3 may be created in RAM 652. A BIOS (Basic Input / Output System) 654, which contains the basic routines that help to transfer information between elements of the computer 642, for example during startup, is stored in ROM 650. The computer 642 further includes a hard disk drive 656 for reading from and writing to a hard disk (not shown), a magnetic disk drive 658 for reading from and writing to a removable magnetic disk 660, and an optical disk drive 662 for reading from or writing to. a removable optical disk 664, such as a CD-ROM or other optical medium.

[0061] Any of the hard disk (not shown), the magnetic disk drive 658, the optical disk drive 662 or the removable optical disk 664 may be an information medium in which information is stored. The information medium has a data area for storing data streams using data stream packets, each containing a packet area containing one or more data packets. As an example, each data packet is encoded and decoded by a codec in the application programs 672 running on the processing unit 644. As such, the encoder distributes the data stream to the data packet areas in the data stream packets so that the distributed data streams are stored in the data packet areas using an encoding algorithm. Alternatively, encoding and decoding of data packets may be performed by a function in or dependent on the operating system 670 running on the processing unit 644.

[0062] The hard disk drive 656, the magnetic disk drive 658 and the optical disk drive 662 are connected to the system bus via a SCSI interface 666 or other suitable interface. The drives and their associated computer-readable media provide non-volatile storage of processor-readable instructions, data structures, program modules, and other data for the computer 642. For example, although the environment described herein includes a hard disk, a removable magnetic disk 660 and a removable optical disk 664, those skilled in the art will appreciate that other types of computer readable media capable of storing data that can be accessed by a computer, such as magnetic cassettes, flash memory cards, DVDs, RAM, ROM and the like can also be used.

[0063] A number of program modules may be stored in the hard disk, the magnetic disk 660, the optical disk 664, ROM 650 or RAM 652, comprising an operating system 670, one or more application programs 672 (which may comprise said codec), other program modules 674 and program data 676. A user may enter commands and / or information into the computer 642 via input devices such as a keyboard 678 and a pointing device 680. Other input devices (not shown) may include a microphone, a joystick, a game controller, a satellite dish, a scanner or the like. These and other input devices are connected to the processing unit 644 via an interface 682 connected to the system bus. A monitor 684 or other type of display device is also connected to the system bus 648 via an interface, such as a video adapter 686. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers.

[0064] The computer 642 runs in a network environment that uses logical connections to one or more remote computers, such as a remote computer

688. The remote computer 688 may be another personal computer, a server, a router, a network PC, a peer device or another common network node, and typically includes many or all of the elements described in connection with the computer 642. although only one memory storage device 690 is shown in Figure 6. The logical connections shown in Figure 6 comprise a local area network (LAN) 692 and a regional network (WAN) 694. Such network environments are common in offices, enterprise-wide computer networks, intranets and the Internet. In the described embodiment of the invention, the remote computer 688 runs a browser program such as the Internet Explorer® browser, manufactured and distributed by Microsoft Corporation in Redmond, Washington.

[0065] When used in a LAN network environment, the computer 642 is connected to the local network 692 via a network interface or adapter 696. When used in a WAN network environment, the computer 642 typically includes a modem 698 or other mechanisms for establishing communication over the regional network 694, such as the Internet. The modem 698, which may be internal or external, is connected to the system bus 648 via a serial port interface 668. In a network environment, program modules shown in connection with the computer 642, or parts thereof, may be stored in the remote storage device. It will be appreciated that the illustrated network connections are examples only, and that other mechanisms for establishing a communication connection between the computers may be used.

[0066] In general, the data processors in the computer 642 are programmed by means of instructions which are at different times in the various computer-readable storage media in the computer. Programs and operating systems are typically distributed via floppy disks or CD-ROMs. From these, they are installed on or loaded into a computer's secondary storage device. When running, they are at least partially loaded into the computer's primary electronic memory. The invention as described herein includes these and various other types of computer-readable storage media when such storage media contain instructions or programs for performing the steps described below in combination with a microprocessor or other data processing unit. The invention also encompasses the computer itself, when programmed in accordance with the methods and techniques described below. Furthermore, certain components of the computer may be programmed to perform the functions and steps described below. The invention includes such sub-components when programmed as described. In addition, the invention described herein comprises data structures, as described, incorporated in various memory or storage media.

[0067] For purposes of illustration, programs and other executable program components, such as the operating system, are shown here in the form of separate blocks, although it is known that such programs and components are located at different storage units in the computer at different times, and are executed by the processing unit. ) in the computer.

[0068] Embodiments described herein define a transmission format that can be used in the transmission of multimedia data between servers and clients and between equivalent computers under RTP. The transmission format is more flexible than the IETF RFC 1889 standards currently used for transmission under RTP. Embodiments of the transmission format enable the flow of encrypted data, provide a mechanism for transmitting metadata for each data sample during RTP, and enable the flow of data protected using WM DRM.

[0069] Although the invention has been described in connection with specific structural features and / or actions in methods, it is to be understood that the invention as defined in the appended claims is not necessarily limited to the specific features and actions described. Rather, the specific features and actions are described as examples of carrying out the required invention.