Spatialized audio in a three-dimensional computer-based scene

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates generally to audio conferencing, and more particularly to spatial audio in a computer-based scene.

[0003] 2. Related Art

[0004] An audio conference consists of an environment shared by viewers using the same application over an interactive TV network. In a typical application, viewers move graphic representations (sometimes referred to as personas or avatars) on the screen interactively using a remote control or game pad. Viewers use their set-top microphones and TV speakers to talk and listen to other viewers and to hear sounds that are intended to appear to come from specific locations on the screen.

[0005] Conferencing software that supports real-time voice communications over a network is becoming very common in today's society. A distinguishing feature between different conferencing software programs is an ability to support spatialized audio, i.e., the ability to hear sounds relative to the location of the listener—the same way one does in the real world. Many non-spatialized audio conferencing software products, such as NetMeeting, manufactured by Microsoft Corp., Redmond, Wash. and Intel Corp., North Bend, Wash.; CoolTalk, manufactured by Netscape Communications Corp., Mountain View, Calif.; and TeleVox, manufactured by Voxware Inc., Princeton, N.J., are rigid. They do not provide distance-based attenuation (i.e., sounds are not heard relative to the distance between persona locations on the TV screen during the conference). Non-spatialized audio conferencing software does not address certain issues necessary for performing communications in computer scenery. Such issues include: (1) efficient means for joining and leaving a conference; and (2) provision for distance attenuation and other mechanisms to provide the illusion of sounds in real space.

[0006] Spatialized audio conference software does exist. An example is Traveler, manufactured by OnLive! Technologies, Cupertino, Calif., but such software packages exist mainly to navigate 3D space. Although they attempt to spatialize the audio with reference to human representatives in the scene, a sound's real world behavior is not achieved.

[0007] As users navigate through a computer-based scene such as a Virtual Reality Modeling Language (VRML) “world”, they should be able to hear (and to broadcast to other users) audio sounds emanating from sources within the scene. Current systems typically do not do a very good job of realistically modeling sounds. As a result, the sounds not are heard relative to the user's current location as in the real world.

[0008] What is needed is a system and method for providing audio conferencing that provides realistic sounds that appear to emanate from positions in the scene relative to the location of the user's avatar on the TV screen.

SUMMARY OF THE INVENTION

[0009] Briefly stated, the present invention is directed to a system and method for enabling an audio conference server (ACS) to provide an application program with multi-point, weight controllable audio conferencing functions. The present invention achieves realistic sound by providing distance-based attenuation. The present invention associates an energy level with the sound (e.g., weak, medium, strong, etc.) to define the sound within the scene according to the sound's real world behavior.

[0010] The ACS manages a plurality of audio conferences, receives audio data from a plurality of audio clients, mixes the audio data to provide distance-based attenuation and decay characteristics of sounds, and delivers the mixed audio data to a plurality of audio clients. Audio clients include set-top box audio clients and point source audio (PSA) audio clients. Set-top box audio clients can be set-top boxes, computer workstations, and/or personal computers (PCs). PSA audio clients include audio files and audio input lines.

[0011] The ACS mixes the audio data by identifying a decay factor. Pre-defined decay factors include an audio big decay factor, an audio small decay factor, an audio medium decay factor, and a constant decay factor. One can also develop a customized decay factor. A weighted value for a source audio client based on the identified decay factor and the distance between the source audio client and a target audio client is determined. A mix table is generated using the weighted values for each source/target audio client pair. Then, an actual mix value for each target audio client is calculated using the weighted values from the mix table. The present invention also includes means for refining the actual mix value.

[0012] The ACS manages the audio conferences using an ACS shell. The ACS shell is a user interface that provides interactive program access to the ACS using high level methods for creating and managing a proxy audio conference and for creating and managing point source audios. The ACS shell also provides program access to the ACS via low level methods for creating and managing audio conferences.

[0013] The ACS also checks the status of a registered owner of each audio conference using a resource audit service (RAS). The RAS informs the ACS when the registered conference owner stops running. Then, the ACS closes the conference.

[0014] Further features and advantages of the invention, as well as the structure and operation of various embodiments of the invention, are described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the digit(s) to the left of the two rightmost digits in the corresponding reference number.

BRIEF DESCRIPTION OF THE FIGURES

[0015] The present invention will be described with reference to the accompanying drawings, wherein:

[0016]FIG. 1 is a block diagram of an audio conferencing network environment according to a preferred embodiment of the present invention;

[0017]FIG. 2 is a block diagram of a computer system useful for implementing the present invention;

[0018]FIG. 3 is a diagram representing the threads involved in audio conferencing for the present invention;

[0019]FIG. 4 is an exemplar process of an audio conference service's conference owner thread;

[0020]FIGS. 5A and 5B represent a flow diagram of the process of changing a registered owner of an audio conference;

[0021]FIG. 6 represents a diagram representing the play back of a PSA;

[0022]FIG. 7 represents a graph showing pre-defined decay factors for four categories of sounds;

[0023]FIG. 8 is an exemplary mix table with audio mix equations for target audio clients;

[0024]FIG. 9A is a flow diagram representing the functionality of the mixer thread;

[0025]FIG. 9B is a flow diagram representing the audio mixing process;

[0026]FIG. 10 is a diagram representing program access and internal interfaces to the audio conference classes of the ACS shell;

[0027]FIG. 11 is a list of the methods contained in an ACProxy class;

[0028]FIG. 12 is a list of the methods contained in a PointSourceAudio class;

[0029]FIG. 13 is a list of the methods contained in an AudioConferenceService class;

[0030]FIG. 14 is a list of the methods contained in an AudioConference class;

[0031]FIG. 15 is a flow diagram representing the addition of an audio client to a proxy audio conference; and

[0032]FIG. 16 is an exemplary flow diagram representing an audio conference in a service application using the lower level methods of the AudioConference and AudioConferenceService classes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The preferred embodiment of the present invention is discussed in detail below. While specific configurations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope of the invention.

[0034] Overview of the Invention

[0035] The present invention is directed to a system and method for enabling an audio conference server (ACS) to provide multi-point, weight controllable audio conferencing functions to application programs. The present invention allows application programs to incorporate audio conference features without being concerned with the details of audio processing, audio delivery, and audio data mixing. The present invention allows multiple applications to share one conference. Applications can also use a conference created by another application. The present invention allows viewers in an audio conference to hear sounds and talk to other viewers across an interactive TV network.

[0036] In an application service that incorporate the ACS, the ACS enables the application service to have audio clients. Audio clients are displayed as points on a TV screen from which sound appears to emanate. Approaching a source of sound makes the sound grow louder. Moving away from the source of sound makes the sound grow fainter. Audio clients can be point sources of sound, referred to as point source audios (PSAs), from audio files or an audio input line. Viewers having conversations with others on a network using a set-top box (STB) microphone and TV speaker are another type of audio client, often referred to as a set-top box (STB) audio client. The STB audio client includes a set-top application for controlling an audio stream of data emanating from the STB microphone or the TV speaker.

[0037] The set-top application, application service, and the ACS typically reside on separate systems, thus requiring a network communications' set-up among them. The ACS manages the data exchange and audio mixing among clients and servers. The ACS uses a COBRA (Common Object Request Broker Architecture) Interface Definition Language (IDL) interface for setting up an audio conference, and a User Datagram Protocol/Internet Protocol (UDP/IP) interface for transmitting and receiving audio streams of data. Both COBRA IDL and UDP/IP interfaces are well known to persons skilled in the relevant art(s).

[0038] The ACS receives audio streams of data from STB and PSA audio clients, mixes the audio streams of data to enable distance-based attenuation of sounds as well as decay characteristics according to the sound's behavior, and delivers the mixed audio streams to the designated STB audio clients. Audio mixing adjusts the volume of a persona's voice (from a STB) or sound emanating from a PSA relative to the distance between the location of the sound and the audio client's persona on the TV screen: the greater the distance, the fainter the audio. Also, if the sound is a low-energy sound, such as a wind chime, the sound will decay quickly. If the sound is a high energy sound, such as a waterfall, the sound will decay slowly.

[0039] A user can interactively interface with the ACS using an ACS shell. The ACS shell, written in TCL (a tool command language developed by John Ousterhout of the University of California, Berkeley), is an IDL-based client that connects to the ACS. The ACS shell enables developers to prototype applications, and operators to monitor and control audio conferences.

[0040]FIG. 1 is a block diagram of an exemplary audio conferencing network environment <b>100</b> in which the present invention is implemented. The audio conferencing network <b>100</b> includes a headend server <b>102</b> and a plurality of workstations <b>120</b>. The workstations <b>120</b> are connected to the headend server <b>102</b> via a communication bus (not explicitly shown). A typical communication bus architecture includes any one of coaxial, fiber-optic, and 10BaseT cabling, all of which are well known to persons skilled in the relevant art(s).

[0041] The headend server <b>102</b> houses an audio conference server (ACS) <b>104</b>, an application service <b>106</b>, and an ACS shell <b>110</b>. The headend server <b>102</b> may also contain PSAs <b>108</b>. The application service <b>106</b> is an application that incorporates the ACS <b>104</b>. PSAs <b>108</b> can be audio files or analog input lines. The ACS <b>104</b> enables the application service <b>106</b> to incorporate audio conference features without being concerned with the details of audio processing, audio delivery, and audio data mixing. The ACS shell <b>110</b> is an IDL client that connects to the ACS <b>104</b> to provide a user interactive interface to monitor and control the full range of ACS functions.

[0042] Each workstation <b>120</b> contains a set-top box (STB) <b>112</b> and a TV <b>122</b>. The TV <b>122</b> includes a TV screen <b>125</b> and a speaker <b>126</b>. The TV screen <b>125</b> displays the audio clients (PSAs <b>108</b> and/or STBs <b>112</b>) that are included in the audio conference while the sounds emanating from the displayed audio clients (PSAs <b>108</b> and/or STBs <b>112</b>) are heard by a viewer <b>124</b> via the TV speaker <b>126</b>. The STB <b>112</b> contains a set-top application <b>114</b>, an ACS client library <b>116</b>, and a microphone <b>118</b>. The ACS client library <b>116</b> contains application program interfaces (APIs) that are exported to the set-top application <b>114</b>. The APIs enable the set-top application <b>114</b> to control an audio stream of data emanating from the microphone <b>118</b> or the TV speaker <b>126</b>. An ACAudioClient class contains the API interface between the set-top application <b>114</b> and the ACS client library <b>116</b>. The ACAudioClient class contains methods that enable set-top applications <b>114</b> to join or leave an audio conference, start and stop audio conferencing ability for STB <b>112</b> audio clients, and to control and monitor audio talking.

[0043] The ACS <b>104</b> enables the application server <b>106</b> to have audio clients. The functionality of the ACS <b>104</b> is two-fold. First, the ACS <b>104</b> manages the audio stream of data. For example, audio sounds emanating from each STB <b>112</b> are interfaced to the ACS <b>104</b> over the communication bus using a UDP/IP interface <b>128</b>. When a viewer <b>124</b> speaks into the microphone <b>118</b>, the viewer's voice is imported from the microphone <b>118</b> to the set-top box <b>112</b> in a digitized pulse code modulation (PCM) format. PCM is a digital transmission technique by which a continuous signal is periodically sampled, quantized, and transmitted in digital binary code. PCM is well known to persons skilled in the relevant art(s). (Note that when a viewer is not speaking, silence suppression is provided to avoid flooding the network.) The ACS client library <b>116</b> sends the PCM audio data stream as UDP/IP data packets <b>128</b> over the communication bus to the ACS <b>104</b> in the headend server <b>102</b>. UDP/IP protocol is used for real-time audio data delivery. The ACS <b>104</b> mixes the audio data stream accordingly, and sends the mixed audio data stream to each designated viewer's set-top TV speaker <b>126</b> via UDP/IP over the communication bus through the appropriate STBs <b>112</b>. Each designated viewer hears the sound relative to that viewer's position to the sound on the TV screen <b>125</b> according to the real life characteristics of the sound. If the set-top application <b>114</b> has turned on the local echo feature, the viewer's voice is also sent locally to the viewer's set-top speaker <b>126</b>. The echo is in real time; there is no network delay.

[0044] Second, the ACS <b>104</b> manages the audio conference. Conference management is performed using IDL. The application service <b>106</b> communicates with the ACS <b>104</b> over the communication bus using an IDL interface <b>130</b>. The set-top application <b>114</b> communicates with the application service <b>106</b> over the communication bus using an IDL interface <b>131</b>. Communications from the set-top application <b>114</b> that are relevant to audio conference management are translated by the application service <b>106</b> into a conference management command and passed to the ACS <b>104</b> via IDL interface <b>130</b>. The ACS <b>104</b> handles IDL requests from the application service <b>106</b>. The IDL interface <b>130</b> comprises software methods that when executed, perform various ACS functions. These methods, when called by the application service <b>106</b> are executed transparently. The methods perform such functions as creating and deleting a conference, adding and removing participants to and from the conference, respectively, changing the volume mix balance between participants in the conference, etc. Although IDL is relatively slow, IDL was chosen for its reliability.

[0045] Implementation of the Invention

[0046] The headend server <b>102</b> is preferably implemented using a computer system, such as exemplary computer system <b>200</b> shown in FIG. 2. Alternatively, the headend server <b>102</b> comprises a plurality of computer systems, each like computer system <b>200</b>. In an alternate embodiment, the application service <b>106</b> and the PSA <b>108</b> are implemented using a single computer system, and the ACS <b>104</b> is a separate computer system. In another embodiment, the application service <b>106</b> is implemented using a single computer system, and the ACS <b>104</b> and the PSA <b>108</b> are implemented on a separate computer system. Other distributions of the application service <b>106</b>, the ACS <b>104</b>, and the PSA <b>108</b> among computer systems are within the scope and spirit of the present invention.

[0047] The computer system <b>200</b> includes one or more processors, such as processor <b>202</b>. The processor <b>202</b> is connected to a communication bus <b>204</b>. The computer system <b>200</b> also includes a main memory <b>206</b>, preferably random access memory (RAM), and a secondary memory <b>208</b>. The secondary memory <b>208</b> includes, for example, a hard disk drive <b>210</b> and/or a removable storage drive <b>212</b>, representing a floppy disk drive, a magnetic tape drive, a compact disk drive, etc. The removable storage drive <b>212</b> reads from and/or writes to a removable storage unit <b>214</b> in a well known manner.

[0048] Removable storage unit <b>214</b>, also called a program storage device or a computer program product, represents a floppy disk, magnetic tape, compact disk, etc. The removable storage unit <b>214</b> includes a computer usable storage medium having stored therein computer software and/or data, such as an object's methods and data.

[0049] The computer system <b>200</b> can communicate with other computer systems via network interface <b>215</b>. The network interface <b>215</b> is a network interface circuit card that connects computer system <b>200</b> to other computer systems via network <b>216</b>. The other computer systems can be computer systems such as computer system <b>200</b>, set-top box audio clients <b>112</b>, or PCs and/or workstations. The network <b>216</b> can be one of fiber optics, coaxial, or 10BaseT.

[0050] Computer programs (also called computer control logic), including object-oriented computer programs, are stored in main memory <b>206</b> and/or the secondary memory <b>208</b>. Such computer programs, when executed, enable the computer system <b>200</b> to perform the features of the present invention as discussed herein. In particular, the computer programs, when executed, enable the processor <b>202</b> to perform the features of the present invention. Accordingly, such computer programs represent controllers of the computer system <b>200</b>.

[0051] In another embodiment, the invention is directed to a computer program product comprising a computer readable medium having control logic (computer software) stored therein. The control logic, when executed by the processor <b>202</b>, causes the processor <b>202</b> to perform the functions of the invention as described herein.

[0052] In yet another embodiment, the invention is implemented primarily in hardware using, for example, one or more state machines. Implementation of these state machines to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

[0053] ACS Threads

[0054] The ACS <b>104</b> is a multi-threaded process having multiple threads, each thread executing a separate audio conference function. The ACS <b>104</b> contains a thread for managing audio conferences, a thread for monitoring the system, a thread for keeping track of conference ownership, a thread for receiving audio data from audio clients, and a thread for mixing and delivering audio data.

[0055]FIG. 3 is a diagram <b>300</b> representing the multiple threads of the ACS <b>104</b>. Upon initialization, the ACS <b>104</b> creates three threads. The first thread is a conference manager thread <b>302</b> that handles incoming IDL method calls from the application service <b>106</b>. The second thread is a conference recycler thread <b>304</b> that monitors the system and performs appropriate garbage collection. The third thread is a resource audit service (RAS) pinger thread <b>306</b> that checks the status of the registered owner of a conference.

[0056] Users interactively interface with the ACS <b>104</b> via the ACS shell. The ACS shell enables program access to low level IDL methods. The conference manager thread <b>302</b> handles all incoming IDL method calls from the application service <b>106</b>. The low level IDL methods that are handled by the conference manager thread <b>302</b> are discussed below (under ACS Shell).

[0057] The ACS <b>104</b> needs to know if the application that created an audio conference is still running. When the audio conference is generated and used by the same application, the application reports that it is alive by pinging the ACS <b>104</b>. The conference recycler thread <b>304</b> monitors the pinging. When the pinging stops, the conference recycler thread <b>304</b> terminates the audio conference.

[0058] When the audio conference is generated by one application and used by a different application, the above method of pinging the conference recycler thread <b>304</b> is not applicable. For example, a game launcher application generates a conference that is used by a game. To accommodate this situation, the RAS pinger thread <b>306</b> is used. The RAS pinger thread <b>306</b> is responsible for pinging a resource audit service (RAS) to obtain information about the existence of the registered owner of a conference. Conference ownerships are registered with the RAS via a registered owner interface. Using the RAS pinger thread <b>306</b>, the RAS informs the ACS <b>104</b> when the registered conference owner stops running. The incorporation of the RAS pinger thread <b>306</b> enables an application claiming ownership of an audio conference to pass audio conference ownership to another application using the registered owner interface.

[0059] An audio conference ownership transfer will now be discussed with reference to FIGS. 4, 5A and <b>5</b>B. FIG. 4 is an exemplar process <b>400</b> representing a conference ownership transfer using the RAS pinger thread <b>306</b>. Two applications are shown in FIG. 4. The first application is a box office application <b>402</b>. The second application is a theatre application <b>404</b>. The box office application <b>402</b> sells tickets to viewers. Since no one can enter the theatre without purchasing a ticket from the box office, the box office application <b>402</b> creates the audio conference and registers itself as the owner of the conference with the ACS <b>104</b>. Once the tickets have all been sold and the attraction is ready to begin, the viewers enter the theatre room. At this time the box office application <b>402</b> passes audio conference ownership to the theatre application <b>404</b> and registers the theatre application <b>404</b> as the owner of the conference with the ACS <b>104</b>.

[0060] The ACS <b>104</b> keeps a lookup table <b>406</b> in which each audio conference has a matching entry <b>408</b>. The matching entry <b>408</b> is the registered owner of the conference. When the theatre application <b>404</b> terminates, the ACS <b>104</b> detects the termination and closes the conference.

[0061] The RAS <b>410</b> keeps track of a registered application. The RAS <b>410</b> pings the registered application using IDL and also checks for its existence via the operating system. When the registered application ceases to exist, the RAS <b>410</b> notifies the ACS <b>104</b>. If the application was the registered owner of the conference, then ACS <b>104</b> closes the conference and cleans up. In the above example, when the viewers leave the theatre and the theatre application <b>404</b> ceases to exist, the RAS <b>410</b> reports it to the ACS <b>104</b>. The ACS <b>104</b> looks at the table, sees that the registered owner is gone, and closes the conference.

[0062]FIGS. 5A and 5B represent a flow diagram <b>500</b> of the process of changing a registered owner of a conference. With reference to FIG. 5A, in step <b>502</b> a first application starts the conference, and control passes to step <b>504</b>. In step <b>504</b>, the owner of the conference is registered with the ACS <b>104</b> (shown as arrow <b>412</b> in FIG. 4). The ACS <b>104</b> enters the owner of the conference in the lookup table <b>406</b> in step <b>506</b><i>a </i>(shown as arrow <b>414</b> in FIG. 4). The ACS <b>104</b> then informs the RAS <b>410</b> that it is interested in knowing when the first application ceases to exist in step <b>506</b><i>b </i>(shown as arrow <b>416</b> in FIG. 4). Control then passes to decision step <b>507</b>.

[0063] In decision step <b>507</b>, it is determined whether the first application wants to move the conference. If the first application does not want to move the conference, control stays with decision step <b>507</b>. If the first application wants to move the conference, control passes to step <b>508</b>.

[0064] In step <b>508</b>, the first application moves the conference ownership to a second application (shown as arrow <b>418</b> in FIG. 4). Conference ownership in the second application is then passed to the ACS <b>104</b> in step <b>510</b> (shown as arrow <b>420</b> in FIG. 4).

[0065] Referring to FIG. 5B, in step <b>512</b>, the second application is registered as the owner of the conference, and the ACS <b>104</b> rewrites the lookup table <b>406</b> to reflect the new owner of the conference in step <b>514</b><i>a </i>(shown as arrow <b>422</b> in FIG. 4). The ACS <b>104</b> then informs the RAS <b>410</b> that it is interested in knowing when the second application ceases to exist in step <b>514</b><i>b </i>(shown as arrow <b>424</b> in FIG. 4). Control then passes to decision step <b>515</b>.

[0066] In decision step <b>515</b>, it is determined whether the second application has ceased. If the second application is still running, control remains in step <b>515</b>. When the second application ceases, control passes to step <b>516</b>. P In step <b>516</b>, the RAS <b>410</b> reports the loss of the second application to the ACS <b>104</b> (shown as arrow <b>426</b> in FIG. 4). Control then passes to decision step <b>517</b>.

[0067] In decision step <b>517</b>, it is determined whether the second application is registered as the owner of the conference in the lookup table. If the second application is not registered as the owner of the conference in the lookup table, control remains in step <b>517</b>. If the second application is registered as the owner of the conference in the lookup table, control passes to step <b>518</b>. In step <b>518</b>, the ACS <b>104</b> closes the conference.

[0068] Returning to FIG. 3, when a new conference starts, two new threads are generated in the ACS <b>104</b>. The first new thread is a net listener thread <b>308</b>. The net listener thread <b>308</b> receives audio data from audio clients <b>112</b> via the network. The second new thread is a mixer thread <b>310</b>. The mixer thread <b>310</b> performs actual audio mixing and delivers the mixed audio data to the audio clients <b>112</b> through the network. Actual audio mixing will be discussed below.

[0069] The net listener thread <b>308</b> receives upstream audio data packets that are sent from the STB <b>112</b>. The net listener thread <b>308</b> listens to the network. Whenever STBs <b>112</b> send data over the network via UDP/IP <b>128</b>, the net listener thread <b>308</b> gathers the audio data packets and passes them to the mixer thread <b>310</b> via a shared buffer (not shown) located between the net listener thread <b>308</b> and the mixer thread <b>310</b>.

[0070] A PSA can be played from the application service <b>106</b> or from within the ACS <b>104</b>. FIG. 6 is a diagram <b>600</b> representing the play back of a PSA <b>108</b>. When PSAs <b>108</b> are played from the application service <b>106</b>, one thread <b>604</b>, called thAudioSource, per PSA is generated in the application service <b>106</b>. The thAudioSource thread <b>604</b> gets audio from the file (i.e., PSA <b>108</b>) and sends it to the ACS <b>104</b> via UDP/IP. After the application service <b>106</b> sends the PSA <b>108</b> to the ACS <b>104</b>, the process is the same as described for a set-top audio client <b>112</b> (i.e., the data is received by the net listener thread <b>308</b> and passed to the mixer thread <b>310</b> where it is mixed and sent to the designated audio clients <b>112</b> over the network via UDP/IP).

[0071] As previously stated, an application can play a PSA <b>108</b> in the ACS <b>104</b>. When an application plays the PSA <b>108</b> in the ACS <b>104</b>, the net listener thread <b>308</b> directly fetches the audio data from the file (i.e., PSA <b>108</b>). This reduces the network traffic of the audio stream.

[0072] Audio Mixing and Delivery

[0073] As previously stated, the mixer thread <b>310</b> performs actual audio mixing and delivers the mixed audio data to the audio clients <b>112</b> through the network via UDP/IP <b>128</b>. The present invention provides audio mixing with distance-based attenuation. When an audio client (PSAs <b>108</b> or STBs <b>112</b>) moves closer to another audio client (PSAs <b>108</b> or STBs <b>112</b>), the sound gets louder, and when the audio client (PSAs <b>108</b> or STB <b>112</b>) retreats, the sound gets quieter. A PSA <b>108</b> might be, for example, the sound of a snoring dragon. When an audio client represented by a STB <b>112</b> moves closer to the dragon, the snoring sounds louder, and when the STB <b>112</b> retreats, the snoring sound is quieter.

[0074] The ACS <b>104</b> accomplishes this by implementing decay characteristics for categories of sounds. FIG. 7 represents a graph <b>700</b> showing pre-defined decay factors for four categories of sounds. Graph <b>700</b> plots volume vs. distance for each pre-defined decay factor.

[0075] The first pre-defined decay factor represents a sound of constant volume regardless of distance. This plot is identified as audioConstant <b>702</b>. AudioConstant sounds are heard at the same volume from anywhere on the TV screen <b>126</b>. The second predefined decay factor represents a sound of loud volume. This plot is identified as audioBig <b>704</b>. AudioBig sounds can be heard by an audio client (PSA <b>108</b> or STB <b>112</b>) anywhere on the TV screen <b>126</b> (shown on TV screen <b>706</b>), and even several screens away. The third pre-defined decay factor represents a sound of low volume. This plot is identified as audioSmall <b>712</b>. AudioSmall sounds can only be heard by audio clients (PSAs <b>108</b> and STBs <b>112</b>) who are near the sound source on the TV screen <b>126</b> (shown on TV screen <b>714</b>). The small sound (audioSmall <b>712</b>) decays to zero inversely and more quickly than the big sound (audioBig <b>704</b>). The last pre-defined decay factor represents a medium sound, i.e., a sound that falls between audioBig <b>704</b> and audioSmall <b>712</b>. This plot is identified as audioMedium <b>708</b>. AudioMedium sounds can be heard on approximately half of the TV screen <b>126</b>, but not beyond the TV screen <b>126</b> (shown on TV screen <b>710</b>). Medium sounds decay linearly, in between the small and big sounds. Developers can also customize decay factor values. A plot of an exemplary custom decay factor <b>716</b> is also shown in graph <b>700</b>.

[0076] When audio clients (PSAs <b>108</b> and STBs <b>112</b>) are added to the conference, the application specifies the decay factor for that audio client (PSA <b>108</b> or STB <b>112</b>). As previously stated, audio data received from audio clients (STBs <b>112</b>) is received via the net listener thread <b>308</b>. The mixer thread <b>310</b> performs the actual audio mixing and delivers the mixed audio data to the audio clients (STBs <b>112</b>) through the network via UDP/IP <b>128</b>. The actual audio mixing is accomplished by generating an audio mix table. An exemplary audio mix table <b>800</b> is shown in FIG. 8. The mix table <b>800</b> contains weighted values for each source audio client (STB <b>112</b>) in relationship to each target audio client (PSA <b>108</b> or STB <b>112</b>) in the conference. A target audio client (STB <b>112</b>) is the audio client that is receiving the sound. According to graph <b>700</b>, the target audio client (STB <b>112</b>) always resides at location (0,0). A source audio client (PSA <b>108</b> or STB <b>112</b>) is the audio client from which the sound is emanating. Weighted values for each source audio client (STB <b>112</b>) are extracted from graph <b>700</b> according to the distance between the target audio client (STB <b>112</b>) and the source audio client (PSA <b>108</b> or STB <b>112</b>) using the decay factor specified for the source audio client (PSA <b>108</b> or STB <b>112</b>) when that source audio client (PSA <b>108</b> or set-top box <b>112</b>) was added to the conference. The weighted values range from 0.0 to 1.0, with 0.0 indicating no volume and 1.0 indicating maximum volume.

[0077] The mix table <b>800</b> shows that the weight of a target audio client (STB <b>112</b>) to itself is 0.0. Thus the target audio client (STB <b>112</b>) will not be able to hear its own echo.

[0078] The audio mix to be delivered to target audio client audio<b>1</b><b>802</b> is 0.0 for audio client audio<b>1</b><b>802</b>, 1.0 for audio client audio<b>2</b><b>804</b>, and 0.7 for audio client audio<b>3</b><b>806</b>. This indicates that audio client audio<b>1</b><b>802</b> will hear audio client audio<b>2</b><b>804</b> at maximum volume and audio client audio<b>3</b><b>806</b> at 70% of the maximum volume. Equation <b>808</b> represents the audio mix for target audio client audio<b>1</b><b>802</b>. Equations <b>810</b> and <b>812</b> represent the audio mix for target audio clients audio<b>2</b><b>804</b> and audio<b>3</b><b>806</b>, respectively.

[0079] The mixer thread <b>310</b> also refines the mixed audio using the following functions: gain control, fading in/fading out, floating point operation elimination, mixing adaption, mixing cut-off and stream audio. The gain control function controls the gain to avoid transmitting excess energy audio data after calculating the weighted sum. The fading in/fading out function avoids the delivery of audio data in a step-wise manner to the speaker output (which results in discontinuity on the user side) by smoothing the audio data on fading in and fading out. The floating point operation elimination function avoids floating point operation by using pre-calculated weight functions instead of performing actual floating point multiplication. The mixing adaption function is used to adapt an actual mix calculation for a source audio client to the available CPU resources. The mixing cut-off function provides better scalability by allowing a mixing cut-off of three, in which the three nearest talking audio clients are selected for the actual mix. The stream audio function prepares stream audio for two purposes: (1) playing ambient background music, such as radio in cyber-space; and (2) using it as an audio source forwarded from another conference. An example of refining an audio mix using both the mixing adaption and mixing cut-off functions follows.

[0080] After the mixer thread <b>310</b> mixes the audio data, the mixed PCM audio data packet is sent over the network via UDP/IP <b>128</b> to the corresponding audio clients (STBs <b>112</b>). Whether the full active mix for each audio client (STBs <b>112</b>) is sent depends on the availability of CPU resources. If the CPU resources are busy, the active mix for any one audio client (STB <b>112</b>) will be reduced. For example, if an audio client's active mix is equivalent to:

audioX=0.1×audio<b>1</b>+0.3×audio<b>2</b>+0.5×audio<b>3</b>+0.7×audio<b>4</b>+1.0×audio<b>5</b>+0.0×audioX,

[0081] and adequate CPU resources were available, the entire active mix could be delivered to audioX. Alternatively, if CPU resources were busy, then the active mix would be reduced. The reduction might be to delete audio<b>1</b> and audio<b>2</b> since they are only heard by audioX at 10% and 30% of the maximum volume, respectively.

[0082]FIG. 9A is a flow diagram <b>900</b> representing the functionality of the mixer thread <b>310</b>. Flow diagram <b>900</b> begins by receiving audio data from the net listener thread <b>308</b> and from PSAs <b>108</b> generated in the application service <b>106</b> or the ACS <b>104</b> in step <b>902</b>. Control then passes to step <b>904</b>.

[0083] In step <b>904</b>, audio mixing is performed for each source audio client in the conference to provide spatialized audio. Control then passes to step <b>906</b> where delivery of the actual audio mix to the target audio clients (STBs <b>112</b>) is performed.

[0084]FIG. 9B is a flow diagram <b>910</b> representing the audio mixing process <b>904</b>. Flow diagram <b>910</b> begins by identifying the decay factor for each source audio client in step <b>912</b>. Control then passes to step <b>914</b>.

[0085] In step <b>914</b>, the distance between the target audio client and each source audio client is determined. Using the distances determined in step <b>914</b>, a weighted value is extracted using the identified decay factors from step <b>912</b> for each source audio client in step <b>916</b>. Control then passes to step <b>918</b>.

[0086] In step <b>918</b>, the weighted values determined in step <b>916</b> are entered into a mix table for each source/target audio client pair. The actual mix values are calculated in step <b>920</b> for each target audio client. The resultant audio mix values are refined in step <b>922</b>.

[0087] ACS Shell

[0088] The ACS shell <b>110</b>, written in TCL (a tool command language developed by John Ousterhout of the University of California, Berkeley), is an IDL client that connects to the ACS <b>104</b> and provides an interactive interface to monitor and control the full range of ACS functions. The ACS shell <b>110</b> enables developers to quickly prototype an application and operators to monitor and control audio conferences. The ACS shell <b>110</b> provides an ACS shell command prompt for prompting a user to enter ACS shell commands. The ACS shell prompt identifies the shell, time, and history, for example, acsshell &lt;10:09 am&gt;[1]%. ACS shell commands enable the user to examine the execution of and interact with audio conferences and audio clients (PSAs <b>108</b> and STBs <b>112</b>).

[0089] ACS shell commands that provide audio conferencing functionality are divided into four distinct classes. FIG. 10 is a diagram <b>1000</b> representing program access and internal interfaces to the ACS shell audio conferencing classes, as well as the interrelationship among classes. The classes include an ACProxy class <b>1002</b>, a PSA class <b>1004</b>, and AudioConference and AudioConferenceService classes <b>1006</b>.

[0090] The easiest way to use the ACS <b>104</b> is through the ACProxy class <b>1002</b>. The ACProxy class provides a higher level of functionality than the lower level AudioConference and AudioConferenceService classes <b>1006</b>. The ACProxy class <b>1002</b> provides most, but not all, of the functionality implemented by the lower level classes <b>1006</b>. The ACProxy class <b>1002</b> also calculates sound relationships (mixed audio) automatically when audio clients change position.

[0091] Methods <b>1100</b> contained in the ACProxy class <b>1002</b> are shown in FIG. 11. The ACProxy methods <b>1100</b> enable the creation of a proxy audio conference. There are fourteen (<b>14</b>) methods in the ACProxy class <b>1002</b>. The methods include:

[0092] (1) ACProxy() method <b>1102</b>

[0093] (2) ˜ACProxy() method <b>1104</b>;

[0094] (3) AddClient() method <b>1106</b>;

[0095] (4) AddPSA() method <b>1108</b>;

[0096] (5) Audios() method <b>1110</b>;

[0097] (6) DemuteAudio() method <b>1112</b>;

[0098] (7) GetAudioLocation() method <b>1114</b>;

[0099] (8) GetConfInfo() method <b>1116</b>;

[0100] (9) MoveAudio() method <b>1118</b>;

[0101] (10) MuteAudio() method <b>1120</b>;

[0102] (11) RegisterOwner() method <b>1122</b>;

[0103] (12) RegisterOwnerByName() method <b>1124</b>;

[0104] (13) RemoveAudio() method <b>1126</b>; and

[0105] (14) UnregisterOwner() method <b>1128</b>.

[0106] The ACProxy() <b>1102</b> and ˜ACProxy() <b>1104</b> methods allow the opening and closing of a proxy audio conference, respectively. Methods AddClient() <b>1106</b> and AddPSA() <b>1108</b> add STB <b>112</b> and PSA <b>108</b> audio clients to the proxy audio conference, respectively. Audio clients (PSAs <b>108</b> and STBs <b>112</b>) are identified using client IDs. The method Audios() <b>1110</b> lists the audio client ID numbers of all audio clients (PSAs <b>108</b> and STBs <b>112</b>) in the proxy audio conference. The audio of a PSA <b>108</b> is enabled and disabled using methods DemuteAudio() <b>1112</b> and MuteAudio() <b>1120</b>, respectively.

[0107] A proxy audio conference registers ownership of the application to the ACS <b>104</b>. The ACS <b>104</b> then pings the RAS <b>410</b> to see if the application continues to exist. To transfer conference ownership to another application, methods RegisterOwner() <b>1122</b> and RegisterOwnerByName() <b>1124</b> are invoked. The RegisterOwner() method <b>1122</b> transfers ownership of the proxy audio conference using an object reference. The RegisterOwnerByName() method <b>1124</b> transfers ownership of the audio conference using the name of the application. The UnregisterOwner() method <b>1128</b> removes the previous ownership of the audio conference.

[0108] The ACProxy class <b>1002</b> allows one to specify a TV screen location for an audio client (PSA <b>108</b> or set-top box <b>112</b>) and calculates the changes in sound automatically when audio clients (PSAs <b>108</b> and STBs <b>112</b>) move. This is accomplished by invoking the MoveAudio() method <b>1118</b>. The X, Y coordinate location of an audio client (PSA <b>108</b> or STB <b>112</b>) is displayed when the GetAudioLocation() method <b>1114</b> is invoked.

[0109] To remove audio clients (PSAs <b>108</b> or STBs <b>112</b>), the RemoveAudio() method <b>1126</b> is invoked. Audio conference information is displayed by invoking the GetConfInfo() method <b>1116</b>.

[0110] An example of how to add audio clients to a proxy conference by invoking methods from the ACProxy class <b>1002</b> is shown in FIG. 15. FIG. 15 is a flow diagram <b>1500</b> representing the addition of an audio client to a proxy audio conference. Flow diagram <b>1500</b> begins by adding an audio client to the proxy audio conference in step <b>1502</b>. The audio client can be a STB <b>112</b> or a PSA <b>108</b>. If the audio client is a STB <b>112</b>, the AddClient() method <b>1106</b> is invoked. If the audio client is a PSA <b>108</b>, the AddPSA() method <b>1108</b> is invoked. The audio client ID and the decay factor (<b>702</b>, <b>704</b>, <b>708</b>, <b>712</b>, or <b>716</b>) must be specified in the parenthetical of the AddClient() method <b>1106</b> or the AddPSA() method <b>1108</b>. Control then passes to step <b>1504</b>.

[0111] In step <b>1504</b>, the audio client is located onscreen by invoking the MoveAudio() method <b>1118</b>. If the audio client is a STB <b>112</b>, the character representing the audio client is located onscreen. If the audio client is a PSA <b>108</b>, the sound source representation of the audio client is located onscreen. When locating an audio client onscreen, the audio client ID and the X, Y coordinates of the audio client must be specified in the parenthetical. The origin, (0,0), is the lower left corner of the TV screen.

[0112] Referring back to FIG. 10, the most complete method of using the ACS shell <b>110</b> is by accessing the PSA class <b>1004</b>, and the AudioConference and AudioConferenceService classes <b>1006</b> directly. PSAs <b>108</b> are initiated by accessing the PSA class <b>1004</b>. As previously stated, PSAs <b>108</b> can be files or audio lines.

[0113] The PointSourceAudio class methods <b>1200</b> are shown in FIG. 12. These methods allow the user to instance or delete a point source as well as play, pause, stop, and resume play of a point source. The PointSourceAudio class <b>1004</b> contains six (6) methods (<b>1202</b>-<b>1212</b>). The six methods include:

[0114] (1) PointSourceAudio() method <b>1202</b>;

[0115] (2) ˜PointSourceAudio() method <b>1204</b>;

[0116] (3) Play() method <b>1206</b>;

[0117] (4) Stop() method <b>1208</b>;

[0118] (5) Pause() method <b>1210</b>; and

[0119] (6) Resume() method <b>1212</b>.

[0120] The PointSourceAudio() method <b>1202</b> instantiates a PSA object. To play the audio source, the Play() method <b>1206</b> is invoked. To stop playing the audio source, the Stop() method <b>1208</b> is invoked. To resume playing the audio source, the Resume() method <b>1212</b> is invoked. The Pause() method <b>1210</b> pauses the playing of the audio source. To clean up resources and close the device used by the PSA <b>108</b>, the ˜PointSourceAudio() method <b>1204</b> is invoked.

[0121] Referring back to FIG. 10, the audio conference methods provided by the AudioConferenceService and AudioConference classes <b>1006</b> are direct IDL calls to the ACS <b>104</b>. Thus, the conference manager thread <b>302</b> handles these incoming method calls. The ACProxy class <b>1002</b> and the PointSourceAudio class <b>1004</b> are wrappers to these IDL interfaces.

[0122] The AudioConferenceService methods <b>1300</b> are shown in FIG. 13. The AudioConferenceService class <b>1006</b> contains five (5) methods (<b>1302</b>-<b>1310</b>). The five (5) methods include:

[0123] (1) OpenConference() <b>1302</b>;

[0124] (2) CloseConference() <b>1304</b>;

[0125] (3) GetConferenceByTicket() <b>1306</b>;

[0126] (4) ListConference() <b>1308</b>; and

[0127] (5) HctAudioStat() <b>1310</b>.

[0128] The OpenConference() <b>1302</b> and CloseConference() <b>1304</b> methods create and close an audio conference. An audio conference is identified by a ticket number. To locate a conference name by ticket number, the GetConferenceByTicket() method <b>1306</b> is invoked. One can also obtain a listing of the online conferences by invoking the ListConference() method <b>1308</b>. Information about STB <b>112</b> audio clients can be obtained by invoking the HctAudioStat() method <b>1310</b>. The data provided for STB <b>112</b> audio clients includes such information as:

[0129] (1) the host IP address of the ACS <b>104</b>;

[0130] (2) the UDP port on the ACS server that handles the UDP/IP audio data packets;

[0131] (3) the port on the set-top side that handles the UDP/IP audio data packets;

[0132] (4) the ID of the set-top where this audioID is mapped to;

[0133] (5) the number that identifies each audio conference;

[0134] (6) the date and time that the audio client (<b>112</b>) was created;

[0135] (7) the most recent access of the client or ping;

[0136] (8) the most recent time that the ACS <b>104</b> received audio data from this audio client (<b>112</b>); and

[0137] (9) the most recent time that the ACS <b>104</b> sent audio data to this audio client (<b>112</b>).

[0138] The methods for the AudioConference class <b>1400</b> are shown in FIG. 14. There are sixteen methods in the AudioConference class <b>1006</b>. They include:

[0139] (1) NewAudio() <b>1402</b>;

[0140] (2) DeleteAudio() <b>1404</b>;

[0141] (3) RegisterOwner() <b>1406</b>;

[0142] (4) RegisterOwnerByName() <b>1408</b>;

[0143] (5) UnregisterOwner() <b>1410</b>;

[0144] (6) SetMixOne() <b>1412</b>;

[0145] (7) SetMix() <b>1414</b>;

[0146] (8) GetMxOne() <b>1416</b>;

[0147] (9) GetMix() <b>1418</b>;

[0148] (10) SetTimeOut() <b>1420</b>;

[0149] (11) GetTimeOut() <b>1422</b>;

[0150] (12) AudioIds() <b>1424</b>;

[0151] (13) AudioStat() <b>1426</b>;

[0152] (14) ConfStat() <b>1428</b>;

[0153] (15) HctAudioStat() <b>1430</b>; and

[0154] (16) Ping() <b>1432</b>.

[0155] The methods in the AudioConferenceService and AudioConference class <b>1300</b> and <b>1400</b>, respectively, provide greater control over the audio conference than the ACProxy class methods <b>1100</b>. As one can see from the lists of methods, one can open and close audio conferences, create and delete audio clients, set an automatic timeout, and set the volume levels between clients.

[0156] The RegisterOwner() method <b>1406</b>, the RegisterOwnerByName() method <b>1408</b>, and the UnregisterOwner() method <b>1410</b> are similar to the ACProxy methods <b>1122</b>, <b>1124</b>, and <b>1128</b>. For descriptions of these methods (<b>1406</b>, <b>1408</b>, and <b>1410</b>) refer to the discussion of the ACProxy methods (<b>1122</b>, <b>1124</b>, and <b>1128</b>) given above.

[0157] Methods NewAudio() <b>1402</b> and DeleteAudio() <b>1404</b> create an audio client for a given STB <b>112</b> and delete an audio client, respectively. The methods SetMixOne() <b>1412</b> and GetMixOne() <b>1416</b> set the volume and return the volume setting of the conversations between two audio clients (PSAs <b>108</b> and STBs <b>112</b>), respectively. The SetMix() method <b>1414</b> and the GetMix() method <b>1418</b> set the volume and return the volume setting between many audio client pairs. The SetTimeOut() method <b>1420</b> sets a number of seconds of inactivity, after which the conference is closed automatically. A conference can be set to never close, if desired. The GetTimeOut() method <b>1422</b> finds the length in seconds of the timeout. The Ping() method <b>1432</b> pings a given audio conference and returns an exception if the conference does not exist.

[0158] The remaining methods (<b>1424</b>-<b>1430</b>) all deal with providing information about the conference. The AudioIds() method <b>1424</b> returns a list of audio IDs in a conference. The AudioStat() method <b>1426</b> returns information about an audio client, given the audio ID. The ConfStat() method <b>1428</b> returns information about an audio conference, and the HctAudioStat() method <b>1430</b> returns information about an audio client, given the STB ID.

[0159]FIG. 16 is an exemplary flow diagram <b>1600</b> representing an audio conference in a service application using the lower level methods of the AudioConferenceService and AudioConference class <b>1300</b> and <b>1400</b>. The procedural steps in flow diagram <b>1600</b> are presented as a guide. Other procedural step distributions for a service application using these lower level methods are within the scope and spirit of the present invention.

[0160] Flow diagram <b>1600</b> begins in step <b>1602</b>. In step <b>1602</b> an audio conference in an application is opened by invoking the OpenConference() method <b>1302</b>. The ticket number identifying the audio conference or audio conference ID is returned. Control then passes to step <b>1604</b>.

[0161] In step <b>1604</b>, a set-top box audio client <b>112</b> is added to the conference by invoking the NewAudio() method <b>1402</b>. The NewAudio() method <b>1402</b> returns an audio ID representative of the set-top box <b>112</b>. Additional audio clients can be created by invoking the NewAudio() method <b>1402</b> or the PointSourceAudio() method <b>1202</b>. Control then passes to step <b>1606</b>.

[0162] In step <b>1606</b>, the volume between audio clients is set by invoking the SetMixOne() method <b>1412</b>. The audio ID of both the receiver and the sender, as well as the mix (i.e., the weighted factor) must be included in the parenthetical of the SetMixOne() method <b>1412</b>. If one needed to set up the relative volumes between many viewers, the SetMix() method <b>1414</b> would be invoked. The SetMix() method <b>1414</b> combines multiple IDL calls into one call. Control then passes to step <b>1608</b>.

[0163] In step <b>1608</b>, the audio client is removed by invoking the DeleteAudio() method <b>1404</b>. To delete an audio client, the audio ID of the audio client to be deleted must be specified in the parenthetical of the DeleteAudio() method <b>1404</b>. Control then passes to step <b>1610</b>, where the audio conference in the application is closed by invoking the CloseConference() method <b>1304</b>.

[0164] Conclusion

[0165] While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.