Method and apparatus for video conferencing having dynamic picture layout

RELATED APPLICATION

0001This application is a continuation of U.S. application Ser. No. 10/601,095, filed Jun. 20, 2003. The entire teachings of the above application are incorporated herein by reference.

BACKGROUND

0002Video conferencing systems allow for simultaneous exchange of audio, video and data information among multiple conferencing sites. Systems known as multipoint control units (MCUs) perform switching functions to allow multiple sites to intercommunicate in a conference. The MCU links the sites together by receiving frames of conference signals from the sites, processing the received signals, and retransmitting the processed signals to appropriate sites. The conference signals include audio, video, data and control information. In a switched conference, the video signal from one of the conference sites, typically that of the loudest speaker, is broadcast to each of the participants. In a continuous presence conference, video signals from two or more sites are spatially mixed to form a composite video signal for viewing by conference participants. The continuous presence or composite image is a combined picture that may include live video streams, still images, menus or other visual images from participants in the conference.

0003In a typical continuous presence conference, the video display is divided into a composite layout having areas or regions (e.g., quadrants). Sites are selected at conference setup from the sites connected in the conference for display in the regions. Common composite layouts include four, nine or sixteen regions. The layout is selected and then fixed for the duration of the conference.

0004Some conference arrangements provide different composite signals or video mix such that each site may view a different mix of sites. Another arrangement uses voice activated quadrant selection to associate sites with particular quadrants. That arrangement enables conference participants to view not only fixed video mix sites, but also a site selected on the basis of voice activity. However, the layout in terms of number of regions or quadrants is fixed for the conference.

SUMMARY

0005There is a need for a capability for conferencing that enables conference sites to receive composite video signals that present an appropriate or best impression. There is a further need to have the best impression be provided dynamically as the number of conference participants or the number of video sources changes during a conference.

0006Accordingly, the present approach provides a method and apparatus for conferencing that dynamically determines conference parameters such as image layout, image resolution and bandwidth. According to an aspect of the present approach, the conference parameters are dynamically determined based on the number of video sources or participants currently connected to the conference.

0007According to a method of the present approach, at least two sites are connected to a conference. At least two video signals are received from the connected sites. One or more conference parameters are defined based on the number of sources or sites connected in the conference. The received video signals are processed to provide a composite video signal according to the defined conference parameters and the composite video signal is transmitted to at least one of the connected sites. Upon one or more other sites or other video sources connecting to the conference or one or more of the connected sites or video sources disconnecting from the conference or both, such that the number of sites or video sources currently connected in the conference changes, one or more conference parameters is redefined. The received video signals are processed according to the redefined conference parameters to provide a second composite video signal that is transmitted to at least one of the connected sites.

0008The conference parameters include composite image layout, image resolution, and conference bandwidth rate. The composite image layout includes areas or regions, the number of which may depend on the number of currently connected sites. The image resolution may include image formats selected based on image clarity and image motion. The conference bandwidth rate includes defining a standard rate and a low rate depending on the capabilities of the conference sites.

0009According to an apparatus of the present approach, a multipoint control unit for conferencing includes an interface unit for receiving video signals from at least two sites connected in a conference; a control processor for dynamically defining one or more conference parameters based on the number of sites or video sources connected in the conference; and a video processor for processing the received video signals to provide a composite video signal according to the defined conference parameters.

BRIEF DESCRIPTION OF THE DRAWINGS

0010The foregoing and other objects, features and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.

0011<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of an MCU configuration.

0012<figref idref="DRAWINGS">FIG. 2</figref> is a schematic block diagram of an embodiment of a VPU.

0013<figref idref="DRAWINGS">FIG. 3</figref> is a block diagram of an MCU configuration illustrating data flow for continuous presence conferencing.

0014<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram illustrating image tiling in a continuous presence conference.

0015<figref idref="DRAWINGS">FIG. 5A</figref> is a diagram illustrating a conference arrangement with two connected sites.

0016<figref idref="DRAWINGS">FIG. 5B</figref> illustrates images transmitted to the sites in <figref idref="DRAWINGS">FIG. 5A</figref>.

0017<figref idref="DRAWINGS">FIG. 5C</figref> illustrates another image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 5A</figref>.

0018<figref idref="DRAWINGS">FIG. 6A</figref> is a diagram illustrating a conference arrangement with three connected sites.

0019<figref idref="DRAWINGS">FIG. 6B</figref> illustrates an image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 6A</figref>.

0020<figref idref="DRAWINGS">FIG. 6C</figref> illustrates another image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 6A</figref>.

0021<figref idref="DRAWINGS">FIG. 7A</figref> is a diagram illustrating a conference arrangement with five connected sites.

0022<figref idref="DRAWINGS">FIG. 7B</figref> illustrates an image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 7A</figref>.

0023<figref idref="DRAWINGS">FIG. 7C</figref> illustrates another image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 7A</figref>.

0024<figref idref="DRAWINGS">FIG. 8A</figref> is a diagram illustrating a conference arrangement with eight connected sites.

0025<figref idref="DRAWINGS">FIG. 8B</figref> illustrates an image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 8A</figref>.

0026<figref idref="DRAWINGS">FIG. 9A</figref> is a diagram illustrating a conference arrangement with ten connected sites.

0027<figref idref="DRAWINGS">FIG. 9B</figref> illustrates an image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 9A</figref>.

0028<figref idref="DRAWINGS">FIG. 10</figref> illustrates another image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 7A</figref>.

0029<figref idref="DRAWINGS">FIG. 11A</figref> illustrates a conference arrangement with five connected sites, one of which provides two video signals.

0030<figref idref="DRAWINGS">FIG. 11B</figref> illustrates an image layout for the conference arrangement of <figref idref="DRAWINGS">FIG. 11A</figref>.

0031<figref idref="DRAWINGS">FIG. 12</figref> illustrates a flow diagram associated with control of video processing in accordance with the present approach.

DETAILED DESCRIPTION

0032A video teleconferencing system capable of continuous presence conferencing is described in U.S. Pat. No. 5,600,646, the entire teachings of which are incorporated herein by reference. That system, with certain modifications, is now described briefly herein followed by a description of the continuous presence configuration of the present invention.

0033Throughout the specification, the term “site” is used to refer collectively to a location having an audiovisual endpoint terminal and a conference participant or user.

0034Referring now to <figref idref="DRAWINGS">FIG. 1</figref>, there is shown an embodiment of an MCU <b>10</b> of the type disclosed in the above-referenced U.S. Pat. No. 5,600,646. The MCU <b>10</b> also includes H.<b>323</b> functionality as disclosed in U.S. Pat. No. 6,404,745 (incorporated herein by reference in its entirety). In addition, video processing in the MCU has been enhanced, as will be described further herein. The features described herein for MCU <b>10</b> can be embodied in a Tandberg MCU.

0035The MCU <b>10</b> includes at least one Network Interface Unit (NIU) <b>120</b>, at least one Bridge Processing Unit (BPU) <b>122</b>, a Video Processing Unit (VPU) <b>124</b>, a Data Processing Unit (DPU) <b>126</b>, and a Host Processing Unit (HPU) <b>130</b>. In addition to a host Industry Standard Architecture (ISA) control bus <b>132</b>, the MCU <b>10</b> includes a network bus <b>134</b>, a BPU bus <b>136</b> and an X-bus <b>138</b>. The network bus <b>134</b> complies with the Multi-Vendor Integration Protocol (MVIP) while the BPU bus <b>136</b> and the X-bus are derivatives of the MVIP specification. The HPU <b>130</b> provides a management interface for MCU operations. Each of the foregoing MCU elements is further described in the above-referenced U.S. Pat. Nos. 5,600,646 and 6,404,745.

0036The H.<b>323</b> functionality is provided by the addition of a Gateway Processing Unit (GPU) <b>128</b> and a modified BPU referred to as a BPU-G <b>122</b>A. The GPU <b>128</b> runs H.<b>323</b> protocols for call signaling and the creation and control of audio, video and data streams through an Ethernet or other LAN interface <b>140</b> to endpoint terminals. The BPU-G <b>122</b>A is a BPU <b>122</b> that is programmed to process audio, video and data packets received from the GPU <b>128</b>.

0037The MCU operation is now described at a high-level, initially for circuit-based conferencing and then for packet-based H.<b>323</b> conferencing. In circuit-based conferencing, digital data frames from H.<b>320</b> circuit-based endpoint terminals are made available on the network bus <b>134</b> through a network interface <b>142</b> to an NIU <b>120</b>. The BPUs <b>122</b> process the data frames from the network bus <b>134</b> to produce data frames which are made available to other BPUs <b>122</b> on the BPU bus <b>136</b>. The BPUs <b>122</b> also extract audio information from the data frames. The audio information is decoded to PCM data and made available on the BPU bus <b>136</b> for mixing with audio from other H.<b>320</b> terminals by respective BPUs <b>122</b> in a conference. The BPUs <b>122</b> combine compressed video information and mixed encoded audio information into frames that are placed on the network bus <b>134</b> for transmission to respective H.<b>320</b> terminals.

0038In a standard conference, the BPUs <b>122</b> perform video switching within a conference by selecting video data frames from timeslots on the BPU bus <b>136</b> and routing the frames to respective terminals in the conference. A particular BPU <b>122</b> selects the appropriate video data frames based upon a conference selection process. Typically, the selection process is based upon a comparison of the voice levels of the conference sites. The loudest conference site is designated the current broadcaster to be viewed by all other conference locations while the current broadcaster typically views another site. In other conference selection processes, an MCU operator or a particular audiovisual terminal operating in a chair control mode selects a site as the current broadcaster.

0039In cases where the audiovisual terminals operate at different transmission rates or with different compression algorithms or are to be mixed into a composite image, multiple video inputs are sent to the VPU <b>124</b> where the video inputs are decompressed, mixed and recompressed into a single video stream. This single video stream is then passed back through the BPU <b>122</b> which switches the video stream to the appropriate endpoint terminals.

0040For packet-based H.<b>323</b> conferencing, the GPU <b>128</b> makes audio, video and data packets available on the network bus <b>134</b>. The data packets are processed through the DPU <b>126</b>. The BPU-G <b>122</b>A processes audio and video packets from the network bus <b>134</b> to produce audio and video broadcast mixes which are placed on the network bus <b>134</b> for transmission to respective endpoint terminals through the GPU <b>128</b>. In addition, the BPU-G <b>122</b>A processes audio and video packets to produce data frames which are made available to the BPUs <b>122</b> on the BPU bus <b>136</b>. In this manner, the MCU <b>14</b> serves a gateway function whereby regular BPUs <b>122</b> and the BPU-G <b>122</b>A can exchange audio and video between H.<b>320</b> and H.<b>323</b> terminals transparently.

0041Having described the components of the MCU <b>10</b> that enable the basic conference bridging functions, a high level description of the flexibility provided by the VPU <b>124</b> is now described with reference to the functional block diagram of <figref idref="DRAWINGS">FIG. 2</figref>. In the MCU <b>10</b>, compressed video information from up to five audiovisual terminals that are in the same conference are routed to a particular VPU <b>124</b> over the BPU bus <b>136</b>. The VPU <b>124</b> comprises five video compression processors (VCP<b>0</b>-VCP<b>4</b>), each having a video decoder/encoder pair <b>102</b>-<i>i</i>, <b>106</b>-<i>i</i>, and pixel scaling blocks <b>104</b>-<i>i</i>, <b>108</b>-<i>i. </i>

0042A video decoder/encoder pair <b>102</b>-<i>i</i>, <b>106</b>-<i>i </i>is assigned to the compressed video information stream associated with each particular site in the conference. Each video decoder <b>102</b>-<i>i </i>decodes the compressed video information using the algorithm that matches the encoding algorithm of its associated site. Included as part of the video decoder <b>102</b>-<i>i </i>may be the processing to determine the framing, packets, and checksums that may be part of the transmission protocol. It should be noted that a processor encoded video stream can be assigned to multiple sites (e.g., a continuous presence application having more than five sites in the conference). In addition, a decoder/encoder pair <b>102</b>-<i>i</i>, <b>106</b>-<i>i </i>can switch among the sites within a conference.

0043The decoded video information (e.g., pixels) is scaled up or down, if necessary, by a pixel scaling block <b>104</b>-<i>i </i>to match the pixel resolution requirements of other sites in the conference that will be encoding the scaled pixels. For example, a desktop system may encode at a resolution of 256×240 pixels while an H.<b>320</b> terminal may require a pixel resolution of 352×288 pixels for a Common Intermediate Format (CIF) image. Other common formats include Quarter Common Intermediate Format (QCIF) (176×144 pixels), 4CIF (704×576), SIF (352×240), 4SIF (704×480), VGA (640×480), SVGA (800×600) and XGA (1024×768).

0044The VPU <b>124</b> includes a pixel bus <b>182</b> and memory <b>123</b>. The system disclosed in U.S. Pat. No. 5,600,646 uses a time division multiplex bus. In the embodiment disclosed herein, the pixel data is not identified and separated by time slots on pixel bus <b>182</b>. Rather, the decoded pixels from each video source are temporarily stored in memory <b>123</b> (addressable RAM). Thus, instead of fetching pixel data by reading from the bus at certain time slots reserved for the respective inputs as disclosed in the '646 patent, the RAM is addressed to read out the pixel data of interest onto the bus at the time the data is needed.

0045In particular, each decoder <b>102</b>-<i>j </i>outputs pixels onto pixel bus <b>182</b> to memory <b>123</b>. Each encoder <b>106</b>-<i>j </i>may retrieve any of the images from the memory <b>123</b> on the pixel bus for re-encoding and/or spatial mixing or compositing. Another pixel scaling block <b>108</b>-<i>j </i>is coupled between the pixel bus <b>182</b> and the encoder <b>106</b>-<i>j </i>for adjusting the pixel resolution of the sampled image as needed.

0046A continuous presence application is now described with reference to <figref idref="DRAWINGS">FIGS. 3 and 4</figref>. For simplicity the endpoint terminals as shown are H.<b>320</b> terminals. In <figref idref="DRAWINGS">FIG. 3</figref>, data from sites <b>38</b> arrive over a communications network to respective NIUs <b>120</b>. Five sites <b>38</b> (A, B, C, D, E) are connected in the conference. Sites A and B are shown connected to a particular NIU <b>120</b> which supports multiple codec connections (e.g., a T<b>1</b> interface). The other sites C, D, and E connect to NIUs <b>120</b> supporting only a single codec connection (e.g., an ISDN interface). Each site <b>38</b> places one or more octets of digital data onto the network bus <b>134</b> as unsynchronized H.<b>221</b> framed data. The BPUs <b>122</b> then determine the H.<b>221</b> framing and octet alignment. This aligned data is made available to all other units on the BPU bus <b>136</b>. The BPUs <b>122</b> also extract audio information from the H.<b>221</b> frames and decode the audio into 16 bit PCM data. The decoded audio data is made available on the BPU bus <b>136</b> for mixing with audio data from other conference sites.

0047Aligned H.<b>221</b> frames are received by the VPU <b>124</b> for processing by encoder/decoder elements called video compression processors (VCPs). The VPU <b>124</b> has five VCPs (<figref idref="DRAWINGS">FIG. 2</figref>) which in this example are respectively assigned to sites A, B, C, D, E. A VCP on the VPU <b>124</b> which is assigned to site E is functionally illustrated in <figref idref="DRAWINGS">FIG. 4</figref>. Compressed video information (H.<b>261</b>) is extracted from the H.<b>221</b> frames and decoded by the VCP as image X. The decoder video image X is placed on the pixel bus <b>182</b> through a scaling block. <figref idref="DRAWINGS">FIG. 4</figref> shows the pixel bus <b>182</b> with decoded video frames from each site A, B, C, D, E successively retrieved from memory <b>123</b> identified by their respective RAM addresses. The VCP assigned to site E receives the decoded video frames from sites A, B, C and D which are then tiled (spatially mixed) into a single composite image I. The tiled image I is then encoded as H.<b>261</b> video within H.<b>221</b> framing and placed on the BPU bus <b>136</b> (<figref idref="DRAWINGS">FIG. 3</figref>) for BPU processing as described above.

0048A similar process can be used to provide transcoding for sites that use different bandwidths. In particular, U.S. Pat. No. 5,600,646 discloses bandwidth transcoding, also referred to as rate matching. Taking into account the embodiment described above for an MCU having a VPU configured with memory accessible across a pixel bus, the processing is similar as described therein.

0049Having described an embodiment of a video conferencing system capable of continuous presence conferencing, an improved continuous presence capability of the present invention is now described which provides dynamic conference parameters, including picture layout, transcoding and bandwidth, based on the number of sites connected to a conference.

0050As noted, a limitation associated with conventional continuous presence applications is that the picture layout is fixed during the conference even if participants enter or leave. Typically, the screen split (e.g., 4-way, 9-way, 16-way) and the positions of the views of the respective participants are set up manually or randomly at initiation of the conference. This can result in an inconvenient layout, format and/or resolution. The present approach allows an MCU operating in continuous presence mode to automatically select the “best impression” for viewing at the endpoint sites without requiring any user intervention. The best impression includes appropriate or ideal picture layout, resolution and bandwidth. The selection is made according to procedures that take into account the number of participant in the conference and the capabilities of the endpoint terminals.

0051Three aspects of the present approach include dynamic layout switching, dynamic resolution switching, and dynamic video transcoding. Each of these aspects is now described in further detail.

0052Regarding dynamic layout switching, the MCU adjusts the picture layout according to the following procedures. With one or two participants, the MCU processes the video signals from the participants to display a picture layout with full screen showing each participant. In another embodiment, the MCU can use a picture layout having a 2/N split where for N=4, the screen contains two pictures at the same size as if the whole screen was filled with four equally sized pictures (4/4 split). Note that the parts of the screen not covered with a picture may be filled with a background color, logo or other image.

0053<figref idref="DRAWINGS">FIG. 5A</figref> shows an example of two participants (sites A and B) connected to an MCU. <figref idref="DRAWINGS">FIG. 5B</figref> shows the images <b>200</b>, <b>202</b> that are displayed with full screen picture layout at the respective sites B and A. <figref idref="DRAWINGS">FIG. 5C</figref> shows an example of a 2/4 split layout with a common image <b>203</b> displayed at both sites. In this case, the image from site A is located in region <b>205</b> and the image from site B is located in region <b>207</b>.

0054When more participants join the conference, the MCU may automatically change the picture layout to display more participants, depending on the number of participants now connected in the conference. <figref idref="DRAWINGS">FIG. 6A</figref> shows an example in which three participants (sites A, B, C) are joined in the conference. In this case, a four region or quadrant image layout, referred to as CP<b>4</b>, is selected to form the image <b>204</b> shown in <figref idref="DRAWINGS">FIG. 6B</figref>. In the particular 3/4 split layout shown, the image from site A is located in quadrant <b>206</b>, the image from site B is located in quadrant <b>208</b> and the image from site C is located in quadrant <b>210</b>. Quadrant <b>212</b> contains a logo, background color or other image. If a fourth site next joins the conference, the layout remains the same except that the image from the fourth site is placed in quadrant <b>212</b>.

0055Note that other layouts can be used to handle the three participants, such as a 3/3 split image layout forming the image <b>213</b> shown in <figref idref="DRAWINGS">FIG. 6C</figref>. For the image <b>213</b>, the image from site A is located in region <b>215</b>, the image from site B is located in region <b>217</b> and the image from site C is located in region <b>219</b>.

0056In the case in which five participants are now included in the conference, the MCU again dynamically changes the picture layout to select the best impression. <figref idref="DRAWINGS">FIG. 7A</figref> shows five participants (sites A, B, C, D, E) connected in a conference. For this case a nine region image layout, referred to as CP<b>9</b>, is selected to form image <b>220</b> shown in <figref idref="DRAWINGS">FIG. 7B</figref>. In the CP<b>9</b> layout, the images from sites A, B, C, D and E are located in respective regions <b>222</b>, <b>224</b>, <b>226</b>, <b>228</b> and <b>230</b>. The remaining regions <b>232</b>, <b>234</b>, <b>236</b> and <b>238</b> contain a logo, background color or other image.

0057Another layout for handling the five participants is the 5/6 split image layout that forms image <b>223</b> shown in <figref idref="DRAWINGS">FIG. 7C</figref>. In this case, images from sites A to E are located in respective regions <b>225</b>, <b>227</b>, <b>229</b>, <b>231</b> and <b>233</b>. Region <b>235</b> contains a logo, background color or other image.

0058Upon three additional participants (sites F, G, H) being added to the conference, the conference now includes eight participants (sites A, B, C, D, E, F, G, H) as shown in <figref idref="DRAWINGS">FIG. 8A</figref>. In this case, the MCU keeps the CP<b>9</b> picture layout to provide the best impression, forming image <b>221</b> shown in <figref idref="DRAWINGS">FIG. 8B</figref>. In the CP<b>9</b> layout, the images from sites A, B, C, D and E are again located in respective regions <b>222</b>, <b>224</b>, <b>226</b>, <b>228</b> and <b>230</b>. Sites F, G and H are located in respective regions <b>232</b>, <b>234</b> and <b>236</b>. The remaining region <b>238</b> contains a logo, background color or other image.

0059With the addition of two more participants (sites I and J), the conference now includes ten participants (sites A, B, C, D, E, F, G, H, I, J) as shown in <figref idref="DRAWINGS">FIG. 9A</figref>. The MCU again dynamically changes the picture layout to select the best impression. For this case a sixteen region image layout, referred to as CP<b>16</b>, is selected to form image <b>240</b> shown in <figref idref="DRAWINGS">FIG. 9B</figref>. In the CP<b>16</b> layout, the images from sites A, B, C, D, E, F, G, H, I, J are located in respective regions <b>242</b>, <b>244</b>, <b>246</b>, <b>248</b>, <b>250</b>, <b>252</b>, <b>254</b>, <b>256</b>, <b>258</b> and <b>260</b>. The remaining regions <b>262</b>, <b>264</b>, <b>266</b>, <b>268</b>, <b>270</b> and <b>272</b> contain a logo, background color or other image. These remaining regions can be used to accommodate the images of up to six additional participants. For conferences having more than 16 participants using a CP<b>16</b> layout, the MCU selects which sites to display from among the participants using random selection, voice loudness, or other criteria.

0060As described, the MCU can be configured to support any number of picture layouts, with the MCU automatically changing between these layouts as the number of participants increases or decreases as sites are added or disconnected from an active conference. Note that while the foregoing has shown image layouts in which the regions are contiguous, other layouts can be used in which the regions are non-contiguous. In a particular embodiment, the MCU is configured to support the following picture layouts: full screen showing one participant only, CP<b>4</b> showing four participants, CP<b>9</b> showing nine participants, and CP<b>16</b> showing sixteen participants. The table below shows a possible mapping between the number of participating sites and/or video sources and a selected picture layout. <tables id="TABLE-US-00001" num="1"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="OFFSET" colwidth="21PT" align="left" /><colspec colname="1" colwidth="126PT" align="left" /><colspec colname="2" colwidth="70PT" align="left" /><thead><row><entry /><entry /></row><row><entry /><entry namest="OFFSET" nameend="2" align="center" rowsep="1" /></row><row><entry /><entry>Number of sites or video sources</entry><entry /></row><row><entry /><entry>in conference</entry><entry>Picture layout</entry></row><row><entry /><entry namest="OFFSET" nameend="2" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /><entry>1 or 2</entry><entry>Full screen</entry></row><row><entry /><entry>3 or 4</entry><entry>CP4</entry></row><row><entry /><entry>5 to 9</entry><entry>CP9</entry></row><row><entry /><entry>Greater than 9</entry><entry>CP16</entry></row><row><entry /><entry namest="OFFSET" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

0061Note that while the foregoing has shown image layouts in which the regions are of the same size, other layouts can be used in which the size of the regions varies. As an example, one layout <b>278</b> could be similar to the CP<b>9</b> layout, except that one main picture, e.g., showing the present speaker, occupies four regions <b>280</b> of the nine regions. This picture layout is shown in <figref idref="DRAWINGS">FIG. 10</figref> for a conference of five sites corresponding to the conference shown in <figref idref="DRAWINGS">FIG. 7A</figref>.

0062Further, the participants or sites are not limited to transmit only one single video signal. One site may e.g., transmit one video signal from a camera representing a picture of a presenter, and another video signal from a PC representing data from a presentation application (e.g., PowerPoint). Thus, the picture layout can depend on the number of video signals or sources rather than the number of participants. <figref idref="DRAWINGS">FIGS. 11A and 11B</figref> provide a configuration and picture layout that illustrate this feature for a conference of five sites. Note that site A provides two video signals, one signal <b>281</b> that corresponds to the participant at site A and another signal <b>283</b> that corresponds to a presentation. The picture layout in <figref idref="DRAWINGS">FIG. 11B</figref> is arranged to include the video signals from sites A, B, C, D and E plus the video signal for the presentation from site A at <b>285</b>. Note that other layouts can be used and that the dynamic aspect described above with respect to additions or removals of sites from the conference applies as well to addition or removal of other video signals such as the presentation signal from site A.

0063To form the images shown in any of the continuous presence layouts described above, the composite image can be processed from the video signals received from the sites as described above with respect to FIGS. <b>1</b> to <b>4</b> of the MCU embodiment. A process for controlling the dynamic picture layout selection is shown in the example control flow diagram of <figref idref="DRAWINGS">FIG. 12</figref>. In this example, a conference has between one and five participants at different points during the operation of the conference. Steps <b>300</b> through <b>324</b> relate to changes that occur in the conference due to sites being added, while steps <b>326</b> through <b>340</b> relate to changes that occur due to sites disconnecting from the conference.

0064At step <b>300</b>, the conference starts from a conference state of zero sites connected. At <b>302</b>, a first site (site <b>1</b>) connects to the conference and at <b>304</b> is provided with a video signal of its own video image (denoted “site<b>1</b>Tosite<b>1</b>”) using a video mode referred to as “selfview”. At <b>306</b> the conference state is one site connected. A second site (site <b>2</b>) is connected to the conference at <b>308</b>. At <b>310</b>, using a video mode referred to as “voice switched”, each of sites <b>1</b> and <b>2</b> receives a video signal of the other site (denoted “siteXTositeY”). This video image corresponds to the images <b>200</b>, <b>202</b> shown in <figref idref="DRAWINGS">FIG. 5B</figref>. Referring again to <figref idref="DRAWINGS">FIG. 12</figref>, at <b>312</b> the conference state is two sites

0065A third site (site <b>3</b>) joins the conference at <b>314</b>. With three sites in the conference, a CP<b>4</b> image layout is selected for forming a composite image that includes image data from each of the three sites. The composite image (denoted herein as “CpPictureTo_AllSites”) is sent to all three sites at <b>316</b> and corresponds to the image <b>204</b> shown in <figref idref="DRAWINGS">FIG. 6B</figref>. Referring again to <figref idref="DRAWINGS">FIG. 12</figref>, the conference state at <b>318</b> is then three sites connected.

0066At <b>320</b>, two additional sites (sites <b>4</b> and <b>5</b>) are connected to the conference. A CP<b>9</b> image layout is selected to form a composite image that includes image data from each of the five sites. The composite image is sent to all five sites at <b>322</b>, and the conference state at <b>324</b> is now five sites connected. The composite CP<b>9</b> image corresponds to the image <b>220</b> shown in <figref idref="DRAWINGS">FIG. 7B</figref>.

0067At step <b>326</b>, site <b>2</b> disconnects from the conference. Since there are now only four sites (sites <b>1</b>, <b>2</b>, <b>3</b> and <b>5</b>) remaining in the conference, at <b>328</b> a CP<b>4</b> image layout is selected and a composite image is sent to the four remaining sites. At <b>330</b>, the conference state is four sites connected.

0068At <b>332</b>, site <b>1</b> disconnects from the conference. Since there are now only three sites (sites <b>2</b>, <b>3</b> and <b>5</b>) remaining in the conference, the CP<b>4</b> image layout is again used and a composite image is sent to the three remaining sites at <b>334</b>. At <b>336</b>, the conference state is three sites connected.

0069At <b>338</b>, site <b>3</b> disconnects from the conference. With only two sites (site <b>2</b> and site <b>5</b>) remaining, the conference mode switches back to voice switched, wherein each of sites <b>2</b> and <b>5</b> receive a video signal of the other site at <b>340</b>. At <b>342</b> the conference state is two sites connected in the conference.

0070Regarding automatic resolution switching, the present approach provides automatic determination of the best possible video format transmitted from the MCU to the respective sites dependent on the number of participants included in the CP picture and the capabilities of the respective participants. As an example, in a CP<b>16</b> and CP<b>9</b> picture, clarity may be prioritized over correct representation of movements. Thus, a 4CIF resolution is selected as the format transmitted to the sites if they are capable of receiving it, because 4CIF is more suitable for a clear representation than e.g., CIF. However, in the case of a CP<b>4</b> picture, a CIF resolution format is selected for a better representation of movements at the sacrifice of clarity. Note that if an endpoint site is not capable of receiving the preferred format, the transmitted format can fall back to a reserve format for that particular preferred format.

0071Regarding video transcoding, the present approach provides support for video transcoding of a limited number of bandwidths. Because of processing limitations, many MCUs support e.g., only two contemporary bandwidths in the same conference call. However, the two bandwidths do not have to be fixed. Thus, the MCU of the present approach defines the two bandwidths according to a highest “standard” and a lowest “low” bandwidth when the conference is initiated. According to the present approach, the two bandwidths can vary according to an optimal choice dependent on the situation. For example, if all participants are connected at 384 kbps and another participant joins the conference at 128 kbps, the standard bandwidth is set to 384 kbps, and the low bandwidth is set to 128 kbps. The video rate of the 384 kbps participants is still set at 384 kbps. However, the new participant receives video at 128 kbps by means of rate transcoding. Subsequent endpoints connected on lower rates than the standard rate will receive video encoded at the low rate of 128 kbps. But, if an endpoint is connected on less than 64 kbps (or another constant) below the standard rate, the standard video rate will be decreased to that rate. In this way, instead of substantially decreasing the bit rate of a new site if it has a bit rate just below the standard bit rate, the standard bit rate is adjusted to the bit rate of the new site.

0072Thus, while the bandwidths are set according to the bandwidth capabilities of the participants as a whole, the currently available bandwidths for the conference may change when a participant enters or leaves the conference. In other words, even if the allowed number of bandwidths is fixed (e.g., due to limited processing resources in the MCU), the actual bandwidths being used may vary to optimize the utilization of the endpoint's capabilities.

0073Those skilled in the art should readily appreciate that the present conferencing approach described herein is deliverable to a computer in many forms, including but not limited to a) information permanently stored on non-writeable storage media such as ROM devices, b) information alterably stored on writeable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media, or c) information conveyed to a computer through communication media, for example using baseband signaling or broadband signaling techniques, as in an electronic network such as the Internet or telephone modem lines. The operations and methods may be implemented in a software entity executable by a processor or as a set of instructions embedded in a carrier wave. Alternatively, the operations and methods may be embodied in whole or in part using hardware components, such as Application Specific Integrated Circuits (ASICs), state machines, controllers or other hardware components or devices, or a combination of hardware, software, and firmware components.

0074While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.