Delivery of information over a communication channel

(54) Title: SYNCHRONIZATION OF AUDIO AND VIDEO DATA IN A WIRELESS COMMUNICATION SYSTEM (30) Unionist Priority: 13/05/2004 us 60 / 571,673 (71) Depositor (s): Qualcomm Incorporated (US) (72) Inventor (s): Harinath Garudadri, Phoom Sagetong, Sanjiv Nanda (74) Attorney: Montaury Pimenta, Machado & Lioce (86) International order: pct US2005 / 016839 of 13/05/2005 (87) International Publication: wo 2005/115009 of 01/12/2005 (57) Summary: SYNCHRONIZATION OF AUDIO AND VIDEO DATA IN A WIRELESS COMMUNICATION SYSTEM Techniques are described for encoding an audio and video stream that is transmitted over a network, for example, an IP or wireless network, such that a frame whole audio and one entire video frame are transmitted simultaneously within a period required to render the audio and video stream frames by an application on a receiver. Aspects of the techniques include receiving RTP audio and video streams and assigning an entire frame of RTP video data to communication channel packets that occupy the same period, or less, than the video frame rate. In addition, an entire frame of RTP audio data is assigned to communication channel packets that occupy the same period, or less, than the audio frame rate. The video and audio communication channel packets are transmitted simultaneously. Reception and assignment of RTP streams can be performed at a remote station, or at a base station.

AUDIO DATA

CLE VIDEO TABLES

RTP PACKAGES

VIDEO DAOOS COMMUNICATION CHANNEL PACKAGES

<img file="BRPI0510953A_D0001.tif" />

AUDIO DATA COMMUNICATION CHANNEL PACKAGES

<td>FRAME</td><td>FRAME</td><td>FRAME</td><td>FRAME</td><td colspan="2">TABLE) TABLE 1 H * d 'me 1</td>

<td>PACKAGE «</td><td>PACKAGE</td><td>PACKAGE 11 * 2</td><td>PACKAGE II *}</td><td>PACKAGE 11 * 4</td><td>.322 PACKAGE 11 + 5</td>

<td> , 324</td><td></td><td></td><td></td><td></td><td>- Bytes</td>

<td>⁷rrrr</td><td>τπ r</td><td>ΓΓΤΓ</td><td>INI</td><td> 1 1 1 1</td><td>ί 1 í 1 1.</td>

<td> 326</td><td> 323</td><td> 330</td><td> 332</td><td> 334</td><td>TEMPü 336</td>

SYNCHRONIZATION OF AUDIO AND VIDEO DATA IN A WIRELESS COMMUNICATION SYSTEM

Priority Claim under 35 USC § 119

This Patent Application claims priority for Provisional Application US 60 / 571,673, entitled Multimedia Packets Carried by CDMA Physical Layer Products, filed on May 13, 2004, and assigned to the assignee of the present and expressly incorporated herein by reference.

REFERENCE TO CO-PENDING PATENT APPLICATIONS

This Patent Application relates to the following co-pending US Patent Applications:

Delivery Of Information Over A Communication Channel, Attorney Document No. 030166U1, deposited simultaneously with the present, attributed to its assignee, and here expressly incorporated, in full, as a reference;

Method And Apparatus For Allocation Of Information To Channels Of A Communication System, Attorney Document No. 030166U2, filed simultaneously with the present, attributed to its assignee, and here expressly incorporated, in full, as a reference; and

Header Compression Of Multimedia Data Transmitted Over A Wireless Communication System, Attorney Document No. 030166U3, filed simultaneously with the present, assigned 'to its assignee, and here expressly incorporated, in full, as a reference.

BACKGROUND

I. Field

The present invention generally relates to the

2/32 delivery of information through a wireless communication system, and more specifically to the synchronization of audio and video data transmitted through a wireless communication system.

II. Foundations

Several techniques for transmitting multimedia data or data in real time, such as audio or video data, through different communication networks have been developed. One such technique is the real-time transport protocol (RTP). RTP provides end-to-end network transport functions, suitable for applications transmitting data in real time through multicast or unicast network services. RTP does not deal with resource reservation and does not guarantee quality of service for services in real time. The data transport is augmented by a control protocol (RTCP) to allow monitoring of data delivery in a scalable way for large multicast networks, and to provide minimal identification and control functionality. RTP and RTCP are designed to be independent of the underlying network and transport layers. The protocol supports the use of RTP-level converters and mixers. Additional details about RTP can be found at RTP: A Transport Protocol for Real-Time Applications, H. Schulzrinne [Columbia University], S. Casner [Packet Design], R. Frederick [Blue Coat Systems Inc.], V. Jacobson [ Packet Design], RFC-3550 draft standard, Internet Engineeríng Steering Group, July 2003, incorporated in its entirety here as a reference.

An example illustrating aspects of RTP is an audio conference where RTP is carried out over Internet Protocol (IP) services from the Internet for voice communications. Through an allocation mechanism, a conference originator obtains a multicast group address and a

3/32 pair of doors. One port is used for audio data, and the other is used for control packets (RTCP). This address and port information is distributed to the intended participants. The audio conferencing application used by each conference participant sends audio data on small partitions, for example, partitions of 20 ms in length. Each audio data partition is preceded by an RTP header; and the RTP header and the combined data are encapsulated in a UDP packet. The RTP header includes information about the data, for example, it indicates what type of audio encoding, such as PCM, ADPCM or LPC, is contained in each packet, Time stamp (TS) of the time in which the RTP packet must be rendered, Sequence Number (SN) a sequential package number that can be used to detect missing / duplicate packages, etc. This allows senders to change the type of encoding used during a conference, for example, to accommodate a new participant who is connected via a low bandwidth link or to react to network congestion indications.

According to the RTP standard, if both audio and video media are used in an RTP conference, they are transmitted as separate RTP sessions. That is, separate RTP and RTCP packets are transmitted to each medium using two different pairs of UDP ports and / or multicast addresses. There is no direct coupling at the RTP level between the audio and video sessions, except that a user participating in both sessions must use the same name in RTCP packets for both so that sessions can be associated.

One motivation for transmitting audio and video as separate RTP sessions is to allow some .1 »

4/32 conference participants receive only one medium if they so choose. Despite the separation, synchronized reproduction of an audio and a source video can be achieved using timing information carried in the RTP / RTCP packets for both sessions.

Packet networks, such as the Internet, can occasionally lose, or reorder, packets. In addition, individual packages may experience varying amounts of delay in their respective transmission times. To deal with this damage, the RTP header contains timing information and a sequence number that allows a receiver to reconstruct the timing produced by the source. This timing reconstruction is performed separately for each source of RTP packets in a session.

Although the RTP header includes timing information and a sequence number, due to the fact that audio and video are delivered in separate RTP streams, there is a potential time slip, also referred to as lip sync or AV sync, between the streams. An application on a receiver will have to re-synchronize these streams before rendering audio and video. In addition, in applications where RTP streams, such as audio and video, are transmitted over wireless networks there is a greater likelihood that packets may be lost, thereby making it more difficult to re-synchronize streams.

There is, therefore, a need in the art to improve the synchronization of RTP audio and video streams that are transmitted over networks.

SUMMARY

Modalities described here deal with the needs mentioned above by encoding data streams, such as an audio / video stream, which is

5/32 transmitted over a network, for example, an IP or wireless network, such that data flows are synchronized. For example, an entire audio frame and an entire video frame are transmitted within the frame period required to render the audio and video frames by an application on the receiver. For example, a data stream synchronizer may include a first decoder configured to receive a first stream of encoded data and to output a first stream of decoded data, wherein the first stream of encoded data has a first bit rate during a information interval. The synchronized data may also include a second decoder configured to receive a second stream of encoded data and output a second stream of decoded data, wherein the second stream of encoded data has a second bit rate during the information interval. A first buffer (buffer) is configured to accumulate the first stream of decoded data over at least one interval of information and output a frame of the first stream of decoded data at each interval period. A second store configured to accumulate the second stream of decoded data over at least one interval of information and to output a frame of the second stream of decoded data at each interval period. Then, a combiner that is configured to receive the frame of the first decoded data stream and the frame of the second decoded data stream emits a synchronized frame of the first and second decoded data streams. The first stream of encoded data can be video data, and the second stream of encoded data can be audio data.

One aspect of this technique includes receiving flows

6/32

Audio and video RTP and assigning an entire frame of RTP video data to communication channel packets that occupy the same or less period than the video frame rate. In addition, an entire frame of RTP audio data is assigned to communication channel packets that occupy the same period, or less, than the audio frame rate. The video and audio communication channel packets are transmitted simultaneously. Reception and assignment of RTP streams can be performed at a remote station, or a base station.

Another aspect is that of receiving communication channel packets that include audio and video data. Decode the audio and video data and accumulate the data for a period equal to the frame period of the audio and video data. At the end of the frame period, a video frame and an audio frame are combined. Because the frame, audio and video frame are transmitted at the same time, and each transmission occurs within a frame period, the audio and video frames are synchronized. Decoding and accumulation can be performed at a remote station or at a base station.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is an illustration of parts of a communication system constructed in accordance with the present invention.

Figure 2 is a block diagram illustrating an exemplary packet data network and several air interface options for delivering packet data over a wireless network in the system in Figure 1.

Figure 3 is a graph illustrating synchronization difficulties in a conventional technique for transmitting separate RTP streams through a wireless communication channel.

7/32

Figure 4 is a graph illustrating a technique for transmitting separate RTP streams through a wireless communication channel according to the invention.

Figure 5 is a block diagram of a part of a wireless audio / video receiver configured to receive communication channel packets.

Figure 6 is a block diagram of a part of a wireless audio / video transmitter configured to transmit communication channel packets.

Figure 7 is a flow chart of transmission of independent RTP flows over a wireless communication link.

Figure 8 is a flowchart of incoming audio and video data over a wireless communication channel.

Figure 9 is a block diagram of a wireless communication device, or mobile station (MS), constructed in accordance with an exemplary embodiment of the present invention.

DETAILED DESCRIPTION

The term exemplary is used here to mean serving as an example, instance, or illustration. Any modality described here as an example should not necessarily be considered as preferred or advantageous over other modalities.

The term streaming is used here referring to the real-time delivery of multimedia data of a continuous nature, such as video, audio or speech information through dedicated and shared channels in chat, unicast and broadcast applications. The phrase multimedia frame, for video, is used here to mean video frame that can be displayed / rendered on a display device after decoding. a

8/32 video frame can be further divided into independently decodable units. In video language, these are called slices. In the case of audio and speech, the term multimedia board is used here to mean that information in a time window through which speech or audio is compressed for transport and decoding at the receiver. The phrase unit information interval is used here representing the time duration of the multimedia board described above. For example, in the case of video, the unit information interval is 100 milliseconds in the case of video at 10 frames per second. Additionally, as an example, in the case of speech, the information unit interval is typically 20 milliseconds in cdma2000, GSM and WCDMA. From that description, it should be evident that, typically, audio / speech frames are not further divided into independently decodable units and, typically, video frames are further divided into two slices that are independently decodable. This should be evident from the context when the multimedia frame phrases, information unit range, etc. refer to video, audio and speech multimedia data.

Techniques for synchronizing RTP streams transmitted over a set of constant bit rate communication channels are described. Techniques include dividing units of information that are transmitted in RTP streams into data packets where the size of the data packets is selected to match the sizes of the physical layer data packets of a communication channel. For example, audio and video data that are synchronized with each other can be encoded. The encoder can be limited such that it encodes the data in sizes that match the physical layer packet sizes available from the

9/32 communication channel. Limiting data packet sizes to match one or more of the available physical layer packet sizes supports the transmission of multiple RTP streams that are synchronized because the RTP streams are transmitted simultaneously, or serially, but within the time frame in which audio and video packages must be rendered with synchronization. For example, if RTP audio and video streams are transmitted, and the data packets are limited so that their sizes match the available physical layer packets, then the audio and video data are transmitted within the display time and are synchronized. As the amount of data needed to represent the RTP flow varies, the capacity of the communication channel varies by selecting different sizes of physical layer packets as described in the co-pending orders listed in REFERENCE TO CO-PENDING PATENT APPLICATIONS , above.

Examples of information units, such as RTP streams, include variable bit rate data streams, multimedia data, video data, and audio data. Information units can occur at a constant repetition rate. For example, information units can be frames of audio / video data.

Different domestic and international standards have been established to support the various air interfaces including, for example, Advanced Mobile Telephone Service (AMPS), Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), GSM Data Environment Enhanced (EDGE), Interim Standard 95 (IS95) and its derivatives, IS-95A, IS-95B, ANSI J-STD-008 (often referred to here collectively as IS-95), and emerging high rate data systems such as cdma2000, Universal Mobile Telecommunications Service

10/32 (UMTS), broadband CDMA, WCDMA, and others. These standards are promulgated by the Telecommunication Industry Association (TIA), 3^The Generation (3GPP), European Telecommunications Standards Institute (ETSI), and other well-known standards corporations.

Figure 1 shows a communication system 100 constructed in accordance with the present invention. Communication system 100 includes infrastructure 101, multiple wireless communication devices (WCD) 104 and 105, and land line communication devices 122 and 124.

WCDs will also be referred to as mobile (MS) or mobile stations. In general, WCDs can be mobile or fixed. Landline communication devices 122 and 124 may include, for example, server nodes, or content servers, which provide various types of multimedia data such as streaming multimedia data. In addition, MSs can transmit streaming data, such as multimedia data.

Infrastructure 101 can also include other components, such as base stations 102, base station controllers 106, mobile switching centers 108, a switching network 120, and the like. In one embodiment, the base station 102 is integrated with the base station controller 106, and in other embodiments, the base station 102 and the base station controller 106 are separate components.

Different types of switching networks 120 can be used to route the signals in the communication system 100, for example, IP networks, or the public switched telephone network (PSTN).

The term direct link or downlink refers to the signal path from infrastructure 101 to an MS, and the term reverse link or uplink refers to the signal path from an MS to infrastructure 11/32. As shown in Figure 1, MSs 104 and 105 receive signals 132 and 136 on the forward link and transmit signals 134 and 138 on the reverse link. In general, signals transmitted from an MS 104 and 105 are intended for reception on another communication device, such as another remote unit, or a landline communication device 122 and 124, and are routed through the network. switching 120. For example, if the signal 134 transmitted to an initiator WCD 104 must be received by a destination MS 105, the signal is routed through infrastructure 101 and a signal 136 is transmitted on the direct link to destination MS 105. Similarly, signals initiated on infrastructure 101 may be broadcast to an MS 105. For example, a content provider may send multimedia data, such as streaming multimedia data, to an MS 105. Typically, a communication device, such as an MS or a landline communication device, can be both a signal initiator and a destination for the signals.

Examples of an MS 104 include cell phones, personal computers enabled for wireless communication, and personal digital assistant (PDA), and other wireless devices. The communication system 100 can be designed to support one or more wireless standards. For example, standards may include standards referred to as Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Enhanced Data GSM Environment (EDGE), TIA / EIA-95-B (IS-95) , ΤΙΆ / ΕΙΑ98-C (IS-98), IS2000, HRPD, cdma2000, Broadband CDMA (WCDMA), and others.

Figure 2 is a block diagram illustrating an exemplary packet data network and several air interface options for delivering packet data over

12/32 a wireless network. The techniques described can be implemented on a packet-switched data network 200 such as that illustrated in Figure 2. As shown in the example in Figure 2, the packet-switched data network system may include a wireless channel 202, a plurality of receiving nodes or MS 204, a sending node or content server 206, a server node 208, and a controller 210. 0 sending node 206 can be coupled to server node 208 via a network 212 such as the Internet.

The server node 208 can comprise, for example, a packet data server node (PDSN) or a GPRS Support Services Node (SGSN) or a GPRS Support Gateway Node (GGSN). Server node 208 can receive packet data from sending node 206, and serve information packets to controller 210. Controller 210 may comprise, for example, Base Station Controller / Packet Control Function (BSC / PCF ) or Radio Network Controller (RNC). In one embodiment, controller 210 communicates with server node 208 through a Radio Access Network (RAN). Controller 210 communicates with server node 208 and transmits information packets over wireless channel 202 to at least one of the receiving nodes 204, such as an MS.

In one embodiment, server node 208 or sending node 206, or both, may also include an encoder to encode a data stream, or a decoder to decode a data stream, or both. For example, the encoder could encode an audio / video stream and thereby output data frames, and the decoder could receive data frames and decode them. Similarly, an MS can include an encoder to encode a data stream, or a decoder to decode a received data stream, or both. The term

13/32 codec is used to describe the combination of an encoder and a decoder.

In an example illustrated in Figure 2, data, such as multimedia data, from sending node 206 which is connected to the network, or Internet 212, can be sent to a receiving node, or MS 204, via the server node , or Packet Data Server Node (PDSN) 206, and a Controller, or Base Station Controller / Packet Control Function (BSC / PCF) 208. The wireless channel interface 202 between the MS 204 and the BSC / PCF 210 is an overhead interface and, typically, can use many channels for signaling and bearer, or payload data.

The air interface 202 can operate according to any of several wireless standards. For example, standards may include TDMA-based standards, such as Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Enhanced GSM Data Environment (EDGE), or CDMA-based standards such as TIA / EIA-95-B (IS-95), TIA / EIA-98-C (IS-98), IS2000, HRPD, cdma2000, Broadband CDMA (WCDMA), and others.

Figure 3 is a graph illustrating synchronization difficulties in a conventional technique for transmitting separate RTP streams through a wireless communication channel. In the example shown in Figure 3, video and audio data frames are encoded in RTP streams and then assigned to the communication channel packets. Figure 3 illustrates a stream of 302 video frames. Typically, video frames occur at a constant rate. For example, video frames can occur at a rate of 10 Hz, that is, a new frame occurs every 100 milliseconds.

As shown in Figure 3, individual video frames can contain different amounts of data,

14/32 as indicated by the height of the bar representing each frame. For example, if the video data is encoded as Motion Picture Expert Group (MPEG) data then the video stream is composed of intra frames (I frames), and predictive frames (P frames). An I frame is self-contained, that is, it includes all the information needed to render, or display, a complete video frame. A P frame is not self-sufficient and will typically contain differential information relative to the previous frame, such as motion vectors and differential texture information. Typically, I-frames can be up to 8 to 10 times larger than a P-frame, depending on the content and settings of the encoder. Although video frames can have different amounts of data, they still occur at a constant rate. Frames I and P can be further divided into multiple video slices. A video slice represents a smaller region on the display screen and can be individually decoded by the decoder.

In Figure 3, video frames N and N + 4 could represent frames I, and video frames N + 1, N + 2, N + 3 and N + 5 could represent frames P. As shown, frames I include a greater amount of data, indicated by the height of the bar representing the frame, than the P frames. The video frames are then packaged in packages in an RTP 304 stream. As shown in Figure 3, RTP N and N + 4 packets, corresponding to video frames I and N + 4, are larger, as indicated by their width, than RTP packets N + 1, N + 2 and N + 3, corresponding to the video frames P in N + 1, N + 2 and N + 3.

The RTP video packets are allocated to the 306 communication channel packets. In a conventional communication channel, such as CDMA or GSM, the data packets of

15/32 communication channels 306 are in a constant size, and are transmitted at a constant rate. For example, communication channel data packets 306 can be transmitted at a rate of 50 Hz, that is, a new data packet is transmitted every 20 milliseconds. Because the communication channel packets are of a constant size, a larger number of communication channel packets is required to transmit the larger RTP packets. Thus, more 306 communication channel packets are required to transmit the RTP packets corresponding to the N and N + 4 video I frames, than the communication channel packets required to transmit the smaller RTP packets corresponding to the video P frames N + 1, N + 2 and N + 3. In the example illustrated in Figure 3, video frame N occupies a block 308 of nine communication channel packets 306. The video frames N + 1, N + 2, and N + 3 occupy blocks 310, 312, and 314 respectively, each with four packets of communication channel 306. The video frame N + 4 occupies a block 316 of nine 306 communication channel packages.

For each video data frame there is an audio data correspondent. Figure 2 illustrates a stream of audio frames 320. Each audio frame N, N + 1, N + 2, N + 3, N + 4, and N + 5 corresponds to the respective video frame and occurs at a rate of 10 Hz, that is, a new audio frame starts every 100 milliseconds. In general, the audio data is less complex, so that it can be represented by fewer bits, than the associated video data and is typically encoded in such a way that the RTP 322 packets are of a size that can be transmitted through the communication channel within the frame period. In addition, typical audio frames are generated once every 20 milliseconds in

16/32

CDMA, GSM, WDCMA, etc. Multiple audio frames are grouped in such cases, so that the audio and video packages represent the same length of time for RTP packaging. For example, RTP packets N, N + 1, N + 2, N + 3, N + 4 and N + 5 are of such a size that each RTP packet can be assigned to communication channel packets 324 such that each RTP packet can be transmitted over the communication channel within a 100 millisecond frame period.

As shown 'in Figure 3, audio frame packs N, N + 1, N2, N + 3, N4 and N + 5 each occupy blocks 326, 328, 330, 332, 334 and 336 respectively, each of them with five 324 communication channel packages.

Comparison between the assignment of video frames and audio frames with their respective communication channel packages illustrates the loss of synchronization between the audio and video frames. In the example illustrated in Figure 3, a block 308 of nine communication channel packets 306 is required to transmit video frames N. Audio frames N associated with video frames N were transmitted in block 326 of the five communication channel packets. 324. Because the video and audio communication channel packets are transmitted at the same time, during the transmission of the video frame Ν, the audio frame N, as well as four of the five communication channel packages in block 328 of the video frame N + 1 audio is transmitted.

For example, in Figure 3, if the associated video and audio frame rate is 10 Hz and the communication channel packet rate is 50 Hz, then, during the 100 millisecond N frame period, all data audio are transmitted, but only a portion of the video data is transmitted. In this example, all video data for

17/32 the quadro frame is not transmitted until another four 306 communication channel packets have been transmitted resulting in the full N video frame requiring 180 milliseconds for transmission compared to the 100 milliseconds for full transmission of the N audio frame. Because the RTP audio and video streams are independent, a portion of the N + 1 audio frame data is transmitted during the time that the N video frame data is transmitted. This loss of synchronization between the video and audio streams can result in a slip between the video and the audio in a receiver of the communication channel.

Due to the fact that video encoders such as H.263, AVC / H.264, MPEG-4, etc. are inherently variable rate in nature due to predictive encoding and also due to the use of variable length encoding (VLC) of many parameters, real-time delivery of variable rate bit streams across circuit switched networks and switched networks per packet is usually performed by formatting traffic with stores at the sender and receiver. Traffic format stores introduce additional delay that is typically undesirable. For example, additional delay can be a source of annoyance during a conference call when there is a delay between the moment a person speaks and the moment when another person hears the speech.

For example, because video on a communication channel receiver is played at the same rate as the original video frame rate, delays on the communication channel can cause playback pauses. In Figure 3, video frame N cannot be played until data from the entire frame has been received. Due to the fact that full-frame data is not

18/32 received during the frame period, playback has to be paused until all video data for frame N is received. In addition, all data from audio frame N needs to be stored until all video data for frame N is received so that audio and video playback is synchronized. It is also noted that the audio data from the N + 1 frame that is received while the video data from the N frame is still being received, must be stored until all video data from the N + frame l be received. Due to the variable size of the video frames, large traffic format stores are required to perform the synchronization.

Figure 4 is a graph illustrating a technique for transmitting separate RTP streams through a wireless communication channel according to the invention. Figure 4, similarly to Figure 3, illustrates a stream of video frames 302 of varying size, and a stream of audio frames 320 that are encoded in independent RTP streams 304 and 322 respectively. The video and audio frames occur at a constant rate, for example, a rate of 10 Hz.

In Figure 4, as in Figure 3, video frames N and N + 4 could represent frames I, and video frames N + 1, N + 2, N + 3 and N + 5 could represent frames

Q. The video frames are packaged in packets in an RTP 304 stream. As shown in Figure 4, RTP packets N and N + 4, corresponding to video frames I, N and N + 4, are larger, as indicated by their width, than packages

RTP N + l, N + 2 and N + 3, corresponding to video frames P

N + 1, N + 2 and N + 2.

RTP video packets are allocated to 406 communication channel packets. Using techniques

19/32 as described in the co-pending application listed in REFERENCE TO CO-PENDING PATENT APPLICATIONS above, the capacity of the communication channel is variable. Due to the variable capacity of the communication channel packets 406, the video frame N can be transmitted in a block 408 containing five communication channel packets 406.

In a conventional communication channel, such as CDMA-based standards such as TIA / EIA-95-B (IS-95), TIA-EIA-98-C (IS-98), IS2000, HRPD, cdma2000, and CDMA of Broadband (WCDMA), 406 communication channel data packets can be transmitted at a rate of 50 Hz, that is, a new data packet is transmitted every 20 milliseconds. Because the capacity of the communication channel packets 406 can be varied, the encoding of the N video frame can be limited such that the entire N video frame can be transmitted over a frame period. As shown in Figure 4, the capacity of the communication channel packets 406 is increased by transmitting the RTP packet N, corresponding to the video frame N, so that the entire packet can be transmitted during the frame period. The techniques described can also be applied to communication channels based on GSM, GPRS, or EDGE.

As shown in Figure 4, video frames N, N + 1, N + 2, N + 3, N + 4 and N + 5 are encoded in RTP packets and assigned to communication channel blocks 408, 410, 412, 414 , 416 and 418, respectively. It is also observed that by varying the communication channel capacity the entire video frame is transmitted within a frame period. For example, if the video frame rate is 10 Hz then an entire frame of video data is transmitted over a 100 millisecond frame period.

For each frame of video data 302 there is a

20/32 corresponding audio frame 320. Each N, N + l, N + 2, N + 3, N + 4 and N + 5 audio frame corresponds to the respective video frame and occurs at a rate of 10 Hz, this is a new audio frame starts every 100 milliseconds. As discussed in relation to Figure 3, audio data is generally less complex, so that it can be represented by fewer bits than associated video data and is typically encoded in such a way that

<td>the packages</td><td>RTP 322 which are</td><td>in</td><td>one</td><td>size</td><td>can</td><td>to be</td>

<td>transmitted</td><td>through the channel</td><td>in</td><td colspan="2">Communication</td><td>during</td><td>one</td>

<td>period of</td><td>100 milliseconds</td><td>in</td><td>one</td><td>frame.</td><td>This is,</td><td>the</td>

<td>RTP packets</td><td colspan="2">audio N, N + l, N + 2,</td><td>N + 3</td><td colspan="2">, N + 4 and N + 5 are</td><td>one</td>

<td>size that</td><td colspan="2">each RTP packet can</td><td>to be</td><td>assigned</td><td colspan="2">to the blocks</td>

<td> 326, 328,</td><td colspan="2">330, 332, 334 and 336</td><td>in</td><td colspan="2">channel packages</td><td>in</td>

<td>Communication</td><td>, respectively.</td><td colspan="2">Of this</td><td>mode if</td><td>the rate</td><td>in</td>

video frame is 10 Hz so each video frame can be transmitted over the communication channel within a 100 millisecond frame period. Similarly to video, if the audio packet size is large, the capacity of the communication channel can also be varied to support the transmission of an entire audio frame over a frame period.

In Figure 4, a comparison between the assignment of video frames and audio frames to their respective communication channel packages illustrates that the video and audio frames remain synchronized. In other words, in each frame period, an entire video frame and audio frame are transmitted. Due to the fact that an entire frame of video and audio is transmitted every frame period, there is no need for additional buffering. The video and audio data received need only be accumulated during a frame period and then they can be played. Due to the fact that

21/32 there is no delay introduced by the communication channel, the video and audio frames remain synchronized.

It can be seen that, as illustrated in Figure 3, the video frames N + 1, N + 2 and N + 3 only required four video communication channel packets 306 to transmit the entire frame of video data. As illustrated in Figure 4, video communication channel packets 406 can be reduced in size so that video data is accommodated in five packets, or blank packets can be transmitted. Similarly, blank packets can be transmitted if there is excess capacity available on the audio communication channel. In this way, the video and audio data are encoded so that an entire frame of audio and video data is assigned to the communication channel packets that occupy the same period, or less, or the respective frame rate.

As described below, depending on aspects of the communication network, different techniques can be used to synchronize RTP flows. For example, the communication network can be over-supplied, that is, it has excess capacity, or the communication network can have a guaranteed Quality of Service. In addition, RTP flows can be modified in order to maintain synchronization when transmitted over a communication network. Each of these techniques will be discussed below.

Over-supplied Communication Network

In the scenario when a communication link between PDSN 208 and sender 206 is over-supplied, that is, there is excess capacity available for data transmission over the wired Internet, then there is no delay due to congestion. Due to the fact that there is excess capacity on the communication link, there is no need to

22/32 delay a transmission so that the transmission can be accommodated by the communication link. With no transmission delay, there is no time lag between voice and video packets when they arrive at the infrastructure, such as at a PDSN. In other words, the audio and video data remain synchronized with each other until the PDSN and synchronization is maintained between the PDSN and the MS; as described in this invention.

In the over-supply scenario, audiovisual synchronization is easily accomplished. For example, video data can have a frame rate of 10 frames per second (fps), based on a frame of 100 milliseconds, and the associated audio can have a frame rate of 50 fps, based on a frame of speaks of 20 milliseconds. In this example, five frames of received audio data would be stored, so that they would be synchronized with the video frame rate. That is, five frames of audio data would be stored, corresponding to 100 milliseconds of audio data, so that it would be synchronized with the 100 millisecond video frame.

Communication networks with guaranteed QoS at maximum delay

By storing an appropriate number of speech frames at a higher frame rate, it is possible to combine a lower frame rate video frame. In general, if video packages are delivered with a guaranteed quality of service (QoS) delay guarantee:

QoS_delay = nT ms Eq.l where n is the delay in frames; e T = 1000 / frames _ per _sec ond

So, a storage sized to store nT / w speech frames where w is the duration of the

23/32 speech in milliseconds, it is necessary to store enough speech frames to ensure that speech and video can be synchronized. In cdma2000 UMTS, the duration of a speech frame, w, is 20 milliseconds, in other communication channels the duration of a speech frame may be different, or vary.

Another technique for synchronizing audio and video data includes storing both data streams. For example, if a communication system has a guaranteed maximum delay of D_Q milliseconds, meaning that D_O it is the maximum delay that can be experienced when transmitting audio and video streams, so an appropriately sized storage can be employed to maintain synchronization.

For example, with a guaranteed maximum delay of D_q, then the storage of D_{Q /}T video frames (T is the duration of the video frames in milliseconds) and DqiW speech frames (w is the duration of the speech frames in milliseconds) will guarantee audio / video synchronization (AV-synchronization). These additional storage spaces are commonly referred to as a dejitter storage.

The techniques described synchronization of audio and video data streams. The techniques can be used with any data streams that need to be synchronized. If there are two data streams, a first higher bit rate data stream and a second lower bit rate data stream that have the same information range and need to be synchronized, then the storage of the data rate data upper bit allows them to be synchronized with lower bit rate data. The size of the storage can be determined, depending on the QoS as described above. Similarly, both flows of

24/32 upper and lower bit rate data can be stored and synchronized as described above.

The techniques described can be performed by a data stream synchronizer that includes a first decoder configured to receive a first encoded data stream and output a first decoded data stream, wherein the first encoded data stream has a first data rate. bit during an information interval. And a second decoder configured to receive a second stream of encoded data and output a second stream of decoded data, wherein the second stream of encoded data has a second bit rate during the information gap. The synchronized data stream also includes a first storage configured to accumulate the first data stream decoded by at least₍s an information interval and output a frame of the first decoded data stream at each interval period, and a second store configured to accumulate the second decoded data stream for at least one information interval and output a frame of the second data stream decoded at each interval period. Then, a combiner configured to receive the frame of the first decoded data stream and the frame of the second decoded data stream and output a synchronized frame of the first and second decoded data streams. In one example, the first stream of encoded data can be video data and the second stream of encoded data can be audio data, such that the first bit rate is higher than the second bit rate.

Single RTP Stream with Multiplexed Audio and Video

Another modality is to carry audio and video in a single RTP stream. As noted, it is not common practice

25/32 over IP networks transmit audio and video as a single RTP stream. RTP was designed to allow participants with different resources, for example, terminals suitable for video and audio, and terminals suitable only for audio, to communicate in the same multimedia conference.

The restriction for transmitting audio and video as separate RTP streams may not be applicable on a wireless network for video services. In this case, a new RTP profile can be designed to carry specific speech and video codec payloads. Combining audio and video in a common RTP stream eliminates any time slips between audio and video data without requiring an over-supplied communication network. Therefore, audio and video synchronization can be performed using techniques described in connection with an over-supplied network as described above.

Figure 5 is a block diagram of a part of a wireless audio / video receiver 500 configured to receive communication channel packets. As shown in Figure 5, the audio / video receiver 500 includes a communication channel interface 502 configured to receive communication channel packets. The communication channel interface 502 outputs a video communication channel packet to a 504 video decoder and audio communication channel packets to a 506 audio decoder. The video decoder 504 decodes the video communication channel packets. video and output the video data to a 508 video store. The audio decoder 506 decodes the audio communication channel packets and outputs the audio data to a 510 audio store. The video store 508 and the audio store accumulate video and audio data respectively for a frame period. The storage

26/32 video 508 and audio store 510 output a video frame and an audio frame respectively to a 512 combiner. Combiner 512 is configured to combine the video and audio frames and to output a synchronized audio / video signal. . Operation of the video store 508, audio store 510 and combiner 512 can be controlled by a controller 514.

Figure 6 is a block diagram of a part of a wireless audio / video transmitter 600 configured to transmit communication channel packets. As shown in Figure 6, the audio / video transmitter 600 includes a video communication channel interface 602 configured to receive an RTP stream of video data. The video communication channel interface assigns RTP packets to communication channel packets. As noted, the capacity of the communication channel packets can vary in order to assign an entire frame worth of RTP video data to the communication channel packets that occupy the same period as the video frame. The audio / video transmitter 600 also includes an audio communication channel interface 604 configured to receive an RTP stream of audio data. The audio communication channel interface 604 assigns the RTP packets to the communication channel packets. As noted, in general, the capacity of the communication channel packets will be sufficient to assign an entire frame of RTP audio data to the communication channel packets that occupy the same period as the audio frame. If the channel capacity is not sufficient then it can be varied, similarly to video communication channel packets so that there will be enough capacity to assign an entire frame of RTP audio data to the communication channel packets that occupy the same period than the audio frame.

27/32

The video and audio communication channel packets are emitted by the video and audio communication channel interfaces 602 and 604 respectively, and communicated to a 606 combiner. The 606 combiner is configured to accept the video and communication channel packets. audio and combine them and output a composite signal. The output of combiner 606 is communicated to a transmitter 608 that transmits that composite signal to the wireless channel. The operation of the video communication channel interface 602, the audio communication channel interface 604 and the combiner 606 can be controlled by a controller 614.

Figure 7 is a flow chart of transmission of independent RTP flows over a wireless communication link. The stream starts at block 702 where the video and audio RTP data streams are received. The flow continues

<td>so stop</td><td>the block</td><td colspan="2">704 where the</td><td>RTP stream</td><td>in</td><td>video</td><td>is</td>

<td colspan="2">assigned to packages</td><td>in</td><td>channel</td><td>Communication.</td><td>At the</td><td>block</td><td> 706</td>

<td>the RTP stream</td><td>audio</td><td>is</td><td>assigned</td><td>to packages</td><td>in</td><td>channel</td><td>in</td>

<td>Communication.</td><td colspan="2">In the block</td><td>708 os</td><td colspan="2">packages of</td><td>channel</td><td>in</td>

Video and audio communication are combined and transmitted over a wireless channel.

Figure 8 is a flowchart of incoming audio and video data over a wireless communication channel. The stream starts at block 802 where video and audio data are received over a wireless communication channel. The stream continues to block 804 and the video and audio data are decoded. In block 806, the decoded video and audio data are mounted on the respective video and audio frames. In block 810 the video and audio data are combined into a synchronized video / audio frame. In block 810, the synchronized video / audio frame is output.

Figure 9 is a block diagram of a

28/32 wireless communication device, or a mobile station (MS), constructed in accordance with an exemplary embodiment of the present invention. Communication device 902 includes a network interface 906, codec 908, a host processor 910, a memory device 912, a program product 914, and a user interface 916.

Signals from the infrastructure are received by the network interface 906 and sent to the host processor 910. The host processor 910 receives the signals and, depending on the content of the signal, responds with appropriate actions. For example, the host processor 910 can decode the received signal itself, or it can route the received signal to codec 908 for decoding. In another embodiment, the received signal is sent directly to codec 908 from network interface 906.

In one embodiment, the network interface 906 can be a transceiver and an antenna to interface with the infrastructure over a wireless channel. In another embodiment, the network interface 906 may be a network interface card used to interface with the infrastructure over land lines. Codec 908 can be implemented as a digital signal processor (DSP), or a general processor such as a central processing unit (CPU).

The host processor 910 and codec 908 are connected to a memory device 912. The memory device 812 can be used to store data during WCD operation, as well as store program code that will be executed by the host processor 910 or by DSP 908. For example, the host processor, codec, or both, can operate under the control of programming instructions that are temporarily stored in the

29/32 memory device 912. The host processor 910 and codec 908 may also include program storage memory of their own. When programming instructions are executed, the host processor 910 or codec 908, or both, perform their functions, for example, decoding or encoding multimedia streams, such as audio / video data and assembling the audio and video frames. In this way, the programming steps implement the functionality of the respective host processor 910 and codec 908, so that the host processor and codec can be individually carried out to perform the functions of decoding or encoding content streams and frame assembly, as desired. Programming steps can be received from a 914 program product. The program product 914 can store, and transfer the programming steps to memory 912 for execution by the host processor, codec, or both.

The program product 914 can be semiconductor memory chips, such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, as well as other storage devices such as a hard disk, a removable disk, a CD -ROM, or any other form of storage medium known in the art that can store computer-readable instructions. Additionally, the program product 914 can be the source file including the program steps that are received from the network and stored in memory and then executed. In this way, the processing steps necessary for operation according to the invention can be incorporated into the program product 914. In Figure 9, the exemplary storage medium is shown coupled to the host processor 910 in such a way that the host processor can read information from,

30/32 and record information on the storage medium. Alternatively, the storage medium can be integral to the host processor 910.

The 916 user interface is connected to the 910 host processor as well as to the 908 codec. For example, the 916 user interface may include a display and speaker used to output multimedia data to the user.

Those skilled in the art will recognize that the step of a method described in connection with a modality can be interchanged without departing from the scope of the invention.

Those skilled in the art would also understand that information and signals can be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may have been cited throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

Those skilled in the art would further consider that several illustrative logic blocks, modules, circuits and steps of algorithms described in connection with the modalities disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, several illustrative components, blocks, modules, circuits and steps have been described above generally in terms of their functionality. Whether such functionality is implemented with hardware or software depends on the specific application and design limitations imposed on the total system. Those knowledgeable can implement the functionality described in several ways

31/32 for each specific application, but such application decisions should not be interpreted as departing from the scope of the present invention.

The various logic blocks. illustrations, modules and circuits written in connection with the modalities disclosed here can be implemented or carried out with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an arrangement of programmed ports in field (FPGA) or other programmable logic device, transistor logic or discrete gate, discrete hardware components, or any combination of them designed to perform the functions described here. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the modalities disclosed here can be incorporated directly into hardware, a software module executed by a processor, or a combination of the two. A software module can reside in RAM, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor in such a way that the processor can read information from, and write information to, the media.

32/32 storage. Alternatively, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside on a user terminal. Alternatively, the processor and the storage medium can reside as discrete components in a user terminal.

The foregoing description of the disclosed modalities is provided to allow anyone skilled in the art to make or use the present invention. Several changes in these modalities will be easily evident to those skilled in the art, and the general principles defined herein can be applied to other modalities without departing from the spirit or scope of the invention. Thus, it is not intended that the present invention be limited to the modalities shown here, but it should be granted the broadest scope consistent with the principles, and with the innovative features, which are described here.