System and method for audio multicast

FIELD OF INVENTION

0001The present invention relates generally to systems and methods for audio multicast and particularly, for audio multicast in a multi-party teleconference.

BACKGROUND OF THE INVENTION

0002In an N-party peer-to-peer teleconferencing implementation, each participating device transmits unicast audio to all the other conference devices, i.e., N-1 unicast transmissions. The receiving device mixes all the unicast audio streams and plays back the mixed audio. An annoying effect called echo can occur if the participating device receives its own audio signal. To avoid this, the peer-to-peer devices transmit their own audio but do not receive self-generated audio.

0003Endpoint devices are typically embedded systems with limited processing resources and cannot handle a large number of incoming audio streams simultaneously. This limitation is acceptable for very small conferences; however, as more peers are added to the conference, the endpoint is unable to process all the audio streams. Thus, in the peer-to-peer situation, teleconferencing is available for only a small number of conference devices.

0004Multicast standards, such as RFC 3550, discuss a modified peer-to-peer teleconferencing that utilizes IP multicast to reduce system bandwidth utilization. For instance, instead of establishing a unicast link with each of the conference devices, the participating device sends only one multicast transmission to deliver its audio to all other conference devices. This technique avoids the audio echo problems because the participating devices do not receive their own self-generated audio. However, the participating endpoint is required to process and decode each of the incoming audio streams and therefore, this technique has the same limitation on conference size as the unmodified peer-to-peer unicast approach.

0005Thus, a system and method is needed for audio multi-party teleconferencing which permits small or large scale conferences. Additionally, a bandwidth efficient multicast system is desirable.

BRIEF DESCRIPTION OF THE DRAWINGS

0006These and other features, aspects, and advantages of the present invention may be best understood by reference to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals indicate similar elements:

0007<figref idref="DRAWINGS">FIG. 1</figref> illustrates an exemplary system for audio multicast;

0008<figref idref="DRAWINGS">FIG. 2</figref> illustrates an exemplary server system in accordance with the various embodiments of an audio multicast system;

0009<figref idref="DRAWINGS">FIG. 3</figref> illustrates an exemplary endpoint system in accordance with the various embodiments of an audio multicast system; and

0010<figref idref="DRAWINGS">FIGS. 4A and 4B</figref> illustrate exemplary-audio packets in accordance with the various embodiments of an audio multicast system.

DETAILED DESCRIPTION

0011The present invention provides an improved, bandwidth-conserving, system and method for audio multicast in a multi-party teleconference. The present disclosure is particularly useful for a multimedia teleconferencing system capable of processing both audio and video information; however, the systems and methods disclosed are proposed for only the audio portion of the conference. In general, an audio multicast system according to the various embodiments can support small scale or large scale conferences having a flexible number of active and passive endpoint devices communicating with a teleconference server. The server receives unicast packets having audio information from each of the participating endpoints and mixes the audio to create a multicast stream transmission. The multicast is sent back to all the associated endpoints, irregardless of participation in the conference. The endpoint devices receive the multicast packets and determine if the received stream contains any information that was self-generated as an active participant. In other words, the endpoint determines if the received mixed audio includes audio that was contributed from the endpoint. The endpoint isolates its own audio contribution and removes that portion from the multicast stream. In this manner, although the multicast may include audio information from the receiving endpoint, the endpoint plays only the mixed audio from the other participants and none of the audio originating from itself.

0012<figref idref="DRAWINGS">FIG. 1</figref> illustrates an exemplary audio multicast system <b>10</b> in accordance with the various embodiments. System <b>10</b> generally includes a plurality of endpoint devices <b>30</b> in communication with a server <b>20</b> via a packet network <b>12</b>. Endpoint device <b>30</b> may include a telephone (stationary and portable), keyset, personal computer, computing device, personal digital assistant, pager, wireless remote client, messaging device, and any other communication device capable of transmitting and receiving communications such as during a teleconference. In the particular embodiment depicted in <figref idref="DRAWINGS">FIG. 1</figref>, endpoints <b>30</b> include desktop keysets as well as keysets coupled to personal computing devices. It should be appreciated that the architecture illustrated in <figref idref="DRAWINGS">FIG. 1</figref> is only one example of suitable endpoints and not intended to be limiting in any manner. In particular embodiments, some or all of the endpoints may include a processor, memory, network interface, user I/O and power conversion, as needed, to establish the device as an operational unit or other real-time communicating device connected to packet network <b>12</b>. Additionally, endpoint <b>30</b> includes particular hardware and/or software for determining if a multicast stream contains audio that was self-generated and for removing the self-generated audio from the stream prior to play-back. These particular elements and features will be described in more detail below.

0013Server <b>20</b> may include one or more computing servers connected to the network backbone to provide teleconferencing services. Server <b>20</b> may include hardware and/or software for performing the various functions of receiving and transmitting audio packet streams within system <b>10</b>. The particular features and functions of server <b>20</b> will be described in more detail below.

0014Packet network <b>12</b> includes any suitable networking system capable of routing digital packets between endpoints <b>30</b> and server <b>20</b>. In this particular embodiment, a plurality of data routers <b>15</b> are utilized for processing both multicast and unicast data exchanges. Data routers <b>15</b> and their functionality are well known in the telecom industry and may comprise both hardware and software components to perform packet routing. There may be various other elements present in network <b>12</b> that are not depicted on <figref idref="DRAWINGS">FIG. 1</figref> or described herein but are well understood in the industry as common actions within a communications system.

0015Used herein, “participating” or “participating device” refers to an endpoint that is actively participating in the conference by sending an audio stream. This contrasts with a passive or non-participating device that is merely listening to the conference. At any point in time, an endpoint can change its status by sending an audio stream or by stopping the transmission, such as when the endpoint user is done speaking. In the current example of <figref idref="DRAWINGS">FIG. 1</figref>, there are four endpoints <b>30</b> in the conference. Of the four, only three of the endpoints are actively participating in the conference, i.e., transmitting an audio stream to server <b>20</b>. Thus, there are currently three “active” endpoints and one “passive” endpoint. Of course, this number can change at any time and they could all be active. Moreover, it should be appreciated that while four endpoints are depicted, this is not intended to be limiting in any manner. The systems and methods of audio multicast are useful for any number of endpoints (active or passive) and are limited only by the bandwidth restrictions of the network and the processing capacity of server <b>20</b>.

0016In a low cost implementation of the system and method for audio multicast, only some of the conferencing endpoints are eligible to actively participate and the remaining endpoints are passive. In this particular environment, the passive endpoints may not be equipped with the hardware and/or software necessary to actively participate in the conference but are merely listening to the conference and receiving a multicast from the server. The active endpoints are able to participate by sending audio data to the server and receive the multicast from the server.

0017In the various embodiments of a system and method for audio multicast, server <b>20</b> receives the unicast audio stream from each participating endpoint. In the exemplary system <b>10</b>, server <b>20</b> receives audio streams from three participating endpoints <b>30</b>. The endpoints send audio packets in a unicast manner which are routed within network <b>12</b> to server <b>20</b>. The three audio streams are shown as “C<b>1</b>” “C<b>2</b>” and “C<b>3</b>” which are received at server <b>20</b>. Server <b>20</b> generates only one mixed output audio from the three audio streams, which is shown on <figref idref="DRAWINGS">FIG. 1</figref> as audio stream “M”. A single transmission is multicast from server <b>20</b> to all the endpoints on the conference regardless of their status of participation.

0018<figref idref="DRAWINGS">FIG. 2</figref> illustrates an exemplary server system <b>20</b> in accordance with the various embodiments of an audio multicast system. In general, server system <b>20</b> includes processing elements configured as jitter processor <b>22</b>, media processor <b>24</b>, emphasis selector <b>25</b>, scaler <b>26</b>, mixer <b>27</b>, audio encoder <b>28</b>, and multicast generator <b>29</b>. Server <b>20</b> receives the audio input packets from the participating endpoints via packet network <b>12</b>. In our example, three endpoints are currently participating in the conference and thus server <b>20</b> receives packets from three endpoints, i.e., C<b>1</b>, C<b>2</b>, C<b>3</b>. Jitter processor <b>22</b> includes jitter handling techniques applied to the data from each participant. Jitter processing is used, for example, to compensate for variable network delays.

0019Media processor <b>24</b> applies appropriate algorithms to decode and convert each packet to a linear digital format, e.g., 16-bit. The decoding is quite flexible and is able to decode encrypted packets as well as a wide variety of standard audio encoding formats. As will be described in more detail below, each packet includes tag information, media information and encoded audio. This information, in part, remains with the packet and will be used by the server to keep track of the origin of data for each conferee. Server <b>20</b> will use the data to accurately associate the packets that are being processed with the multicast output to be generated by multicast generator <b>29</b>.

0020In packet networks, it is inevitable that audio packets will arrive late due to large network delays or disappear altogether. In either situation, the audio data is deemed lost and certain packet loss concealment (PLC) techniques may be used to synthesize the lost audio. The endpoint is provided information on what PLC technique was used at server <b>20</b> so the endpoint can synthesize the same audio data used in the mixing process.

0021Separate data streams each representing the active participants are processed by emphasis selector <b>25</b> to add audio power to the stream that appears to be the strongest. In one particular embodiment, emphasis selector <b>25</b> uses the comparative acoustic energy of the streams as an indicator of which stream is the strongest. The emphasis selector may continue to increase the gain for the most active channel and decrease the gain for the other channels until a predetermined maximum skew value is reached. The settings may change as soon as more energy is detected at one of the other channels. The level of emphasis to the channels is used later at the endpoints during processing, thus this information is retained for transmission by multicast generator <b>29</b>.

0022Next the signals are scaled <b>26</b> in preparation for mixing <b>27</b>. Because mixing often introduces the possibility of data overflow, scaler <b>26</b> is used to prevent overflow or clipping. Overflow is generally a mixer output error where the result exceeds the most-positive or most-negative value so that the system cannot represent the signal correctly. Scaler <b>26</b> may function as a compressor/expander to increase the clarity of the mixed output. The scaling adjustment is retained for transmission by multicast generator <b>29</b> and will used during signal processing at the endpoints.

0023Audio encoder <b>28</b> receives the mixed signal and applies a preselected audio encoding and/or encryption algorithm to the signal that is to be multicast.

0024Multicast generator <b>29</b> prepares the signal for packet transmission on packet network <b>12</b>. The encoded mixer output combined with the data regarding the participant identities, the tag that identifies each audio segment, emphasis factors, and other processing factors is applied for each channel and transmitted out to all the endpoints via network <b>12</b> in a multicast stream.

0025<figref idref="DRAWINGS">FIG. 3</figref> illustrates an exemplary endpoint system <b>30</b> in accordance with the various embodiments of an audio multicast system. Endpoint system <b>30</b> includes features for encoding audio information for unicast transmission to server <b>20</b>, as well as features for processing multicast transmissions received from server <b>20</b> in preparation for endpoint playback. For discussion purposes it is assumed that <figref idref="DRAWINGS">FIG. 3</figref> is an exemplary block diagram of a participating endpoint <b>30</b>. In other words, endpoint <b>30</b> is actively participating in a current conference and therefore packets of audio information are generated by the endpoint for transmission to server <b>20</b>. However, it should be realized that regardless of the level of participation (active or passive), each endpoint is preferably equipped with the described features. In general, endpoint system <b>30</b> includes data packet extractor <b>31</b>, audio data linearizer <b>32</b>, D/A converter <b>33</b>, A/D converter <b>43</b>, media processor <b>34</b>, emphasis compensator <b>35</b>, scaler compensator <b>36</b>, audio reconstruction <b>37</b>, audio encoder <b>38</b>, tag generator <b>41</b>, history buffer <b>42</b>, and composite generator <b>39</b>.

0026Encoding audio information in preparation for unicast transmission generally begins with an audio signal from a microphone, such as when the user is talking into the endpoint device. A/D converter <b>43</b> converts the analog speech to a digital signal as bytes of data. Audio encoder <b>38</b> applies an encoding and/or encryption algorithm to the audio data. Preferably, the encoding scheme is extremely flexible and capable of changing to another scheme if needed to accommodate the network conditions, user selection, etc. The encoding scheme may change during the course of a conference and any changes are preferably transmitted to the server. Additionally, a record of the encoding scheme used at the endpoint may be retained in the history buffer for future use by the endpoint. The encoding scheme may be used by server <b>20</b> in the decoding of the packets and thus will be included in the packet generated by composite generator <b>39</b>. The encoding scheme may also be used to encrypt the information with a key that is transmitted along with the corresponding media-type data byte. It should be appreciated that A/D converter <b>43</b> and audio encoder <b>38</b> maybe combined to a single hardware/software or may include a hardware or software stand-alone product.

0027Tag generator <b>41</b> assigns a unique identifier, e.g., packet tag, to each data packet before leaving endpoint <b>30</b>. The tag will be used by both server <b>20</b> and the issuing endpoint <b>30</b> to facilitate correct processing of the audio data. In one particular embodiment, the tag is a combination of the endpoint device code and a timestamp code.

0028Composite generator <b>39</b> compiles the data packets and readies each packet for transmission on packet network <b>12</b>. As will be discussed in more detail below, prior to transmission a representation of each packet is stored in history buffer <b>42</b> for later use. In this particular embodiment, C<b>1</b> packets are transmitted from endpoint <b>30</b>.

0029<figref idref="DRAWINGS">FIG. 4B</figref> illustrates an exemplary C<b>1</b> packet according to the various embodiments of an audio multicast system and method. Typically each digital packet generated by endpoint <b>30</b> includes the identification tag, encoded audio initiated at the endpoint, and processing or media information that allows server <b>20</b> to correctly decode the audio. The tag may include, for example, timestamp information and a unique endpoint identifier. Media information may include, for example, code to indicate how each participant stream has been encoded and can vary depending on, for example, the endpoint's capabilities and transmission link used. The encoded audio is the audio data transmitted from the endpoint.

0030<figref idref="DRAWINGS">FIG. 4A</figref> illustrates an exemplary M packet according to the various embodiments of an audio multicast system and method. Typically each digital packet generated by server <b>20</b> includes mixed audio output (shown on <figref idref="DRAWINGS">FIG. 4A</figref> as “E<b>1</b>(C<b>1</b>+C<b>2</b>+C<b>3</b>)”). Along with the mixed audio, each M packet includes information to identify whose audio is mixed into the mixed audio, information that the receiving endpoints need to identify the segment of the audio history that is used in the mixed audio, and any additional information to allow the receiving endpoints to process the mixed audio correctly. The media information may further include a selection mechanism used by the server to select a few active participants to participate in the mixed audio. Since different participants may be involved in different segments of the mixed audio, the server can disclose the active participant's information in every segment of the mixed audio. If the server modifies or replaces the source audio used in the mixing process or modifies the mixed audio output, the server discloses such information to the endpoints. The endpoints use this information to modify or replace the stored audio data history so that the participating endpoints can use the correct audio data to properly remove their own audio data from the mixed audio. Since each endpoint may have different tags, server <b>20</b> preferably associates the audio of each participating endpoint with its own tag information. For example, if audio from C<b>1</b>, C<b>2</b>, and C<b>3</b> are used in the mixed audio, server <b>20</b> may transmit the illustrative M packet of <figref idref="DRAWINGS">FIG. 4A</figref>.

0031With continued reference to <figref idref="DRAWINGS">FIG. 3</figref>, endpoint <b>30</b> receives multicast packets from server <b>20</b>. In the current example, the receiving packet is identified as “Packet M” and is received at data packet extractor <b>31</b>. The data packet extractor <b>31</b> isolates the read tag index from the multicast stream and uses it to access the sample audio portion from history buffer <b>42</b>. Recall that server <b>20</b> included processing information in the multicast stream which would be used by the endpoint. For instance, in accordance with the various embodiments of the system, the encoding adjustment, level of emphasis, and scaling adjustment applied to the signal at the server may be used by the endpoint. This information was transmitted to the endpoint in the multicast and is extracted and forwarded to emphasis compensator <b>35</b> and scaler compensator <b>36</b> for signal processing.

0032As previously mentioned, endpoint <b>30</b> retains a sample of the composite stream prior to transmission. History buffer <b>42</b> saves the data which is later used to support the removal of the endpoint's own audio contribution from the received multicast data. History buffer <b>42</b> is preferably capable of holding all the audio samples that are transmitted from endpoint <b>30</b> during a particular time period. In one particular embodiment, the time period is one second or about 8000 samples for a standard telephone audio quality sample rate of 8K samples/second using G.711 encoding. As history buffer <b>42</b> is written with data, the information is placed into the next available memory location as indicated by a rotating pointer. Once the pointer completes each addressing cycle, the oldest data is overwritten with new updated history bytes.

0033The samples stored in history buffer <b>42</b> are accessed using a tag index, such as the tag associated with the packet by tag generator <b>41</b>. In one particular embodiment, each tag index is based on a unique timestamp code therefore making each memory storage location uniquely identifiable. The tag on the stored information is compared with the multicast stream tag index and the matching sample, if any, is taken out of the buffer. Media processor <b>34</b> uses the media encoding adjustment factor received from server <b>20</b> and, in conjunction with emphasis compensator <b>35</b>, and scaler compensator <b>36</b>, processes the sample to closely approximate the endpoint's original contribution.

0034The multicast stream is linearized by audio data linearizer <b>32</b>. Audio reconstruction <b>37</b> receives the two audio streams, i.e., the multicast and the endpoint's historic sample, and subtracts the historic sample from the multicast audio data. In other words, a single stream leaves audio reconstruction <b>37</b> that is the multicast stream having the endpoint's originally transmitted audio signal removed. A D/A converter <b>33</b> returns the digital signal to analog format suitable for a speaker, earphone or whatever play-back equipment the endpoint employs. By removing the endpoint's past contribution from the mixed multicast, the play-back retains the natural audio of a multiparty conference without the loop feedback problems.

0035It should be realized that if no match is found in history buffer <b>42</b>, then the endpoint will not have a history audio sample stream and audio reconstruct <b>37</b> will only be presented with one stream, i.e., the multicast stream. With only one input, the audio reconstruct <b>37</b> removal function does not occur so the playback is the entire multicast stream. This situation occurs if the endpoint is passive.

0036In the event of lost or late arriving packets to the server, certain PLC techniques may have been used to synthesize the lost data. Typically, the synthesized audio is different from the original history audio data stored in history buffer <b>42</b>. Therefore, the endpoint cannot use the stored history audio data to remove its own audio from the mixed audio. The endpoint is provided information on what PLC technique was used at server <b>20</b> so the endpoint can synthesize the same audio data used in the mixing process. The synthesized audio can be used in the same methods as previously described to remove the endpoint's contribution from the mixed signal.

0037Presented herein are various systems, methods and techniques for audio multicast, including the best mode. Having read this disclosure, one skilled in the industry may contemplate other similar techniques, modifications of structure, arrangements, proportions, elements, materials, and components for audio multicast, and particularly in a teleconference, that fall within the scope of the present invention. These and other changes or modifications are intended to be included within the scope of the present invention, as expressed in the following claims.