Remote server environment

CROSS REFERENCE TO RELATED APPLICATIONS

p-0002This application claims priority to Provisional Patent Application Ser. No. 61/357,274, filed Jun. 22, 2010, entitled “Remote Server Environment” which is incorporated herein in its entirety.

BACKGROUND OF THE INVENTION

p-00031. Technical Field

p-0004The invention relates to remote control of a computer system.

p-0005More specifically, the invention relates to frame-by-frame encoding of a desktop computer display, transmitting the encoded view to a thin client device, decoding the stream on the thin client device, and translating inputs made on the thin client device for controlling the remote desktop.

p-00062. Description of the Prior Art

p-0007In computing, there are techniques for sharing a view of a desktop computer. Virtual Network Computing (“VNC”) is a graphical desktop sharing system that uses a remote frame buffer (“RFB”) protocol over wired or wireless network connections to share a view of the desktop to other devices. The client devices decode the video data and display the view of the desktop.

p-0008VNC systems can sometimes be adequate for resource light applications such as remote troubleshooting and collaborative word-processing; however, known VNC solutions suffer a number of major drawbacks.

p-0009First, many modern applications demand low latency times between user input and the corresponding output, as well as with video rendering, in general. For example, participants in a videoconferencing session are often frustrated by delays between speech and output between the participants of a conversation. This can lead to miscommunication. Also, in the world of online gaming, latency leads to unacceptable delays between input and action that disadvantage players having high latency.

p-0010Next, traditional VNC system approaches assumed that the desktop computer and the client system shared a common system for providing input. For example, in a common remote computing scenario, a traveling business person logs into his work computer from a remote location, i.e. a hotel. Traditional VNC systems assumed that the work computer and the hotel computer were each equipped with a keyboard and a mouse or other pointing device. This paradigm has worked in the past, but an expansion of alternative inputting methods render traditional VNC systems obsolete.

SUMMARY OF THE INVENTION

p-0011The invention provides a system in which computer content is encoded and distributed to one or more client devices and in which user gestures on the client device are translated into a digital representation of the computer's native input format, thereby allowing the client device to control the computer.

p-0012Some embodiments of the invention involve a distributed computer system comprising a desktop server networked with one or more client devices, wherein the desktop system is configured to encode content, and wherein the one or more client devices are configured to decode content. The one or more client devices are also configured for transmitting users inputs back to the desktop server, and the desktop device is configured for translating the inputs and controlling an application running on the desktop server.

p-0013Some embodiments of the invention involve a method of capturing screen data on a desktop system, packaging the captured content, streaming the content to one or more client device, decoding the content on the client-side, displaying the decoded content on the client device, accepting user input, transmitting the user input to the desktop system, translating the input into a native format, and controlling the desktop system using the translated user input.

p-0014Some embodiments of the invention involve unique solutions for encoding desktop content efficiently to reduce latency.

p-0015Some embodiments of invention involve specific gesture translation techniques for translating gestures native to a client device into inputs recognized by the desktop server. Some embodiments of the invention involve rendering virtual controllers on a client device.

p-0016Some embodiments of the invention involve peer-to-peer infrastructure for sharing a desktop environment. Some embodiments of the invention involve transmission of a plurality of desktop environments to create a social gaming network of friends.

p-0017Some embodiments of the invention involve a multi-modal system of viewing and controlling a desktop server on a remote device.

BRIEF DESCRIPTION OF THE DRAWINGS

p-0018<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates a computer environment for streaming encoded content from a desktop to one or more client devices, decoding on the client-side, and for delivering control information back to the desktop system for processing according to some embodiments of the invention;

p-0019<figref idrefs="DRAWINGS">FIG. 2A</figref> illustrates an exemplary workflow including the processing steps taken at the desktop-side and at the client-side in a streaming desktop environment according to some embodiments of the invention;

p-0020<figref idrefs="DRAWINGS">FIG. 2B</figref> illustrates a workflow including the processing steps of a desktop server for streaming the desktop environment to a client device using only an onboard CPU according to some specific embodiments of the invention;

p-0021<figref idrefs="DRAWINGS">FIG. 2C</figref> illustrates a workflow including the processing steps of a desktop server for streaming the desktop environment to a client device using an onboard CPU and a graphics accelerator according to some specific embodiments of the invention;

p-0022<figref idrefs="DRAWINGS">FIG. 2D</figref> illustrates a workflow including the processing steps of a desktop server for streaming the desktop environment to a client device using an onboard CPU, a GPU, and a standalone hardware encoder according to some specific embodiments of the invention;

p-0023<figref idrefs="DRAWINGS">FIG. 2E</figref> illustrates a method of multi-processing screen image data to increase encoding speed according to some embodiments of the invention;

p-0024<figref idrefs="DRAWINGS">FIG. 3</figref> illustrates a graphical representation of a touch screen gesture translation table according to some embodiments of the invention;

p-0025<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates a graphical representation of a gyroscope translation according to some embodiments of the invention;

p-0026<figref idrefs="DRAWINGS">FIG. 5A</figref> illustrates a tablet computer with a touch screen interface and virtual controls according to some embodiments of the invention;

p-0027<figref idrefs="DRAWINGS">FIG. 5B</figref> illustrates a tablet computer with a touch screen interface and virtual controls according to some embodiments of the invention;

p-0028<figref idrefs="DRAWINGS">FIG. 5C</figref> illustrates a tablet computer with a touch screen interface and virtual controls according to some embodiments of the invention;

p-0029<figref idrefs="DRAWINGS">FIG. 5D</figref> illustrates a tablet computer with a touch screen interface and virtual controls according to some embodiments of the invention;

p-0030<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates an exemplary system for viewing social gaming network on a client device according to some embodiments of the invention;

p-0031<figref idrefs="DRAWINGS">FIG. 7</figref> illustrates a method of dual-mode video stream encoding, streaming, and playback according to some embodiments of the invention; and

p-0032<figref idrefs="DRAWINGS">FIG. 8</figref> is a block schematic diagram of a machine in the exemplary form of a computer system within which a set of instructions may be programmed to cause the machine to execute the logic steps of the invention.

DETAILED DESCRIPTION OF THE INVENTION

p-0033The invention provides a system in which computer content is encoded at low latency and distributed to one or more client devices and in which user gestures on the client device are translated into a digital representation of the computer's native input format, thereby allowing the client device to control the computer.

p-0034The invention is particularly useful in the case where the client device comprises a thin client without processing capabilities that are sufficient or optimal for running rich media applications. For example, an attractive feature of tablet computers is their slight profile and lightweight construction; however, this form factor inherently limits the space for processing hardware, memory, and cooling systems. Accordingly, tablet computers do not have the same potential for high performance as a larger computer simply because the large computer can physically accommodate this hardware.

p-0035Therefore, the preferred embodiments of the invention involve configuring a desktop computer to run processing-heavy application and streaming the video output to one or more thin client devices including, but not limited to tablet computers, smartphones, personal digital assistants, laptops, portable video players, portable gaming systems, and on-board vehicle computers. Although the term “desktop” is used throughout the disclosure, the term shall include any processing machine that has the computing resources to carry out the invention.

h-0006System Overview

p-0036<figref idrefs="DRAWINGS">FIG. 1</figref> illustrates a computer environment <b>100</b> for streaming content from a desktop system <b>105</b> to one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b>, for decoding the content on the client-side, and for delivering control information back to the desktop system <b>105</b> from the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b>.

p-0037In the presently preferred embodiments, the desktop system <b>105</b> comprises at least one processor <b>106</b> operatively coupled with memory <b>107</b>, a network interface <b>108</b>, and one or more input/output devices <b>111</b>.

p-0038The desktop system <b>105</b> is operatively coupled with a network <b>120</b> via the network interface <b>108</b>. The network <b>120</b> can comprise one or more of any type of computer network including a local area network (LAN), personal area network (PAN), wide area network (WAN), metropolitan area network (MAN), wireless local area network (WLAN), wireless wide area network (WWAN), peer-to-peer network, or other computer network, now know or later developed.

p-0039In some embodiments of the invention, the processor <b>106</b> comprises an on-board graphics processing unit (not shown). In some other embodiments of the invention, the processor <b>106</b> is operatively coupled to video processing expansion card <b>109</b>. In some embodiments of the invention, the processor is operatively coupled with a proprietary expansion card <b>110</b> especially configured for performing the encoding operations disclosed herein.

p-0040In the presently preferred embodiments of the invention, the desktop system <b>105</b> streams encoded content on a per frame basis from the desktop <b>105</b> to the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b>. The one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, and <b>104</b> include one or more local applications, embodied in either hardware or software, which decodes the streamed content and displays the decoded content. The one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, and <b>104</b> also include an input device and a transmission module for accepting user inputs and delivering user input back to the desktop system <b>105</b>.

p-0041The desktop system <b>105</b> further comprises an input translation module <b>112</b> coupled with the processor <b>106</b>. The input translation module <b>112</b> is configured for translating input formats native to the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b> into a corresponding input known to the desktop system <b>105</b>.

p-0042In the presently preferred embodiments of the invention, the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, and <b>104</b> include one or more local applications, embodied in either hardware or software, that translate controls native to the device into controls that are compatible with the desktop system <b>105</b>.

p-0043According to some other embodiments of the invention, the desktop system <b>105</b> receives user inputs from the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b> in first format via said network interface <b>108</b>. If the desktop system <b>105</b> does not recognize the first format, the processor <b>106</b> passes those user inputs to the translation module <b>112</b> and the translation module <b>112</b> translates the first format into a second format that is recognizable by the desktop system <b>105</b>.

p-0044In the presently preferred embodiments of the invention, the desktop system <b>105</b> contains all the hardware and software to capture screen data, scale, perform color conversion, encode, multiplex, and transmit encoded video data. According to these embodiments, there is not a need for an online video encoding engine or intermediary server hub—eliminating time of flight delays that unacceptably contribute to latency.

p-0045In some embodiments of the invention, the processor <b>106</b> includes one or more processing modules for performing some of the computing functions. Likewise, in some embodiments of the invention, the one or more client devices <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b> include one or more processing modules for performing some of the computing functions.

p-0046As used herein, the term “module” refers to any software or hardware processing component or portion therefor that may be used to implement one or more of the processing functions.

p-0047In some embodiments of the invention, a client device is preloaded with hardware or software for receiving image file data, decoding the data, or translating device gestures into desktop control instructions. In some other embodiments, an application for receiving image file data, decoding the data, or translating device gestures is available to client device operators to download and install on their device. In yet other embodiments, a software application for configuring a client device is available through a third-party service, such as an app-store.

p-0048In some embodiments of the invention, a user performs an initial configuration of the system to optimize the streaming of desktop content to his client device. According to these embodiments, one or more databases of settings are made available to the user for storing settings, screen parameters, screen resolutions, etc. According to these embodiments, the computer system <b>105</b> is able to encode and transmit screen data in the appropriate formats in a more streamlined workflow. In some other embodiments, the computer system <b>105</b> or the one or more client device <b>101</b>, <b>102</b>, <b>103</b>, <b>104</b> discover settings, parameters, resolutions, etc. automatically.

h-0007Workflow

p-0049<figref idrefs="DRAWINGS">FIG. 2A</figref> illustrates an exemplary workflow <b>200</b> including the processing steps taken at the remote desktop-side and at the client-side in a streaming desktop environment.

p-0050The workflow <b>200</b> begins with capturing screen data <b>201</b> from the desktop computer at a particular frame rate. In some embodiments of the invention, the screen data is captured as a raster graphic image. In some other embodiments, the screen data is captured as a vector graphics image. In some embodiments of the invention, frames are processed serially, while in other embodiments perform parallel processing, redundant processing, single instruction, multiple data (SIMD) processing, multiple instruction, multiple data (MIMD) processing, or any other processing method now known or later developed.

p-0051Initially, the computer display resolution is modified to best fit the client resolution or display exactly as is on the computer display onto the client display. This provides the best performance and display quality onto the client display. After the frame is captured, the captured image file may be scaled to fit the dimensions of the destination device <b>202</b>. The additional scaling may be done to reduce the file size and therefore bandwidth used on the network. In the presently preferred embodiments of the invention, the dimensions of the destination device are predetermined through automatic discovery or by user specification via a GUI.

p-0052Next, color conversion is performed <b>203</b> on the scaled image data image data to comply with the native color values of the destination device. In the presently preferred embodiments of the invention, the native color values of the destination device are predetermined through automatic discovery or by user specification via a GUI.

p-0053The image file data is then encoded for video playback. In the presently preferred embodiments of the invention, the image file data is encoded at very low latency. In a specific example, a H.264/MPEG-4 codec at low latency is used to encode image data.

p-0054The encoded image file data is then matched with time-stamped audio <b>205</b> and multiplexed <b>206</b>. Next, the multiplexed signal is transmitted <b>207</b> to one or more client devices via a network comprising one or more of a local area network (LAN), personal area network (PAN), wide area network (WAN), metropolitan area network (MAN), wireless local area network (WLAN), wireless wide area network (WWAN), peer-to-peer network, or other computer network, now know or later developed.

p-0055As explained above, the one or more client devices are configured with hardware or software that is configured to receive and decode encoded video from the desktop.

p-0056Accordingly, the workflow <b>200</b> continues with the client device receiving an encoded video stream <b>208</b> and decoding the stream <b>209</b>. The encoded video has already been scaled and converted into device-native colors and resolutions, so the device simply displays the decoded video frames at the frame rate that they are received.

p-0057The workflow <b>200</b> also involves accepting input from the user of a client device <b>211</b> and transmitting the input information back to the desktop <b>212</b>.

p-0058User input takes a wide variety of forms. Most simply, user input comprises a pointer tracking input, pointer clicking input(s), and keyboard input. A pointer tracking input most commonly takes the form of a computer mouse device, but can take other forms such as trackballs, joysticks, pressure sensitive pointing stick, graphics tablet, touchpad and stylus, and touch screen operated with human fingers.

p-0059Many of inputs from these types of pointing tracker devices are relatively straightforward to port from the client device to the desktop since they all involve two-dimensional coordinate values. However, a number of client devices utilize input types not intuitively translatable to the mouse/keyboard paradigm. For example, there is not a simple way to distinguish a screen tap on a touch screen device as a right click of a mouse or a left click of a mouse. Likewise, there is not an intuitive way to “mouse-over” an item using a touchscreen device. Similarly, many mobile devices include a built in gyroscope that uses the tilt of the device as a control.

p-0060Accordingly, the invention involves translating native client device control inputs into control inputs recognizable by the desktop.

p-0061In the presently preferred embodiments of the invention, the client device includes one or more local applications, embodied in either hardware or software, that translate controls native to the device into controls that are compatible with the desktop system.

p-0062In some embodiments of the invention, the client device simply transmits its native input signals and the desktop performs the translations. In some other embodiments of the invention, the client device includes hardware or software for translating inputs.

p-0063The workflow <b>200</b> of <figref idrefs="DRAWINGS">FIG. 2A</figref> continues as the desktop receives the control input <b>213</b> transmitted by the client device and the desktop translates the client device control input into a desktop native control signal <b>214</b>. Translating control signals is discussed in greater detail below. Finally, the workflow <b>200</b> involves delivering the control instructions <b>215</b> to one or more application running on the desktop.

p-0064A key to the invention is that the steps of capturing screen data, encoding the frames, packaging the frames with audio, and transmitting the frames as fast as possible. Therefore, the invention includes various techniques to speed this process up including using graphics accelerators, using dedicated expansion encoder cards, and multi-threading.

p-0065As explained above, the desktop device optionally includes one or more graphics accelerator or peripheral expansion card designed to perform the processing steps of the invention at a greater speed. <figref idrefs="DRAWINGS">FIGS. 2B-2D</figref> illustrate various exemplary workflow solutions according to some embodiments of the invention.

p-0066<figref idrefs="DRAWINGS">FIG. 2B</figref> illustrates a workflow <b>220</b> including the processing steps of a desktop server for streaming the desktop environment to a client device using only an onboard CPU according to some specific embodiments of the invention.

p-0067The workflow <b>220</b> of <figref idrefs="DRAWINGS">FIG. 2B</figref> begins with capturing screen data on a desktop server machine <b>221</b>. According to <figref idrefs="DRAWINGS">FIG. 2B</figref>, the screen capture is performed using a Windows® API and one or more of a device context retrieval function (“GetDC”) function and a Bit-block transfer function (“BitBlt”).

p-0068Next, the captured screen data is scaled <b>222</b> and converted to the appropriate color-space <b>223</b> and encoded <b>224</b>. The example of <figref idrefs="DRAWINGS">FIG. 2B</figref> simply employs software for performing the steps of scaling <b>222</b>, color conversion <b>223</b> and encoding <b>224</b>.

p-0069The encoded image file data is then matched with a time-stamped audio signal and any control input information and multiplexed <b>225</b>. According to this example, the workflow <b>220</b> utilizes one or more of a Real Time Streaming Protocol (RTSP) and a HTTP Live Streaming Protocol.

p-0070Finally, the multiplexed signal is transmitted <b>226</b> to one or more client devices via a PAN, LAN, WAN, etc.

p-0071Some other embodiments of the invention involve using both an on-board CPU processor and a graphics accelerator expansion card for performing the various processing steps. <figref idrefs="DRAWINGS">FIG. 2C</figref> illustrates a workflow <b>230</b> including the processing steps of a desktop server for streaming the desktop environment to a client device using an onboard CPU and a graphics accelerator according to some specific embodiments of the invention.

p-0072The workflow <b>230</b> of <figref idrefs="DRAWINGS">FIG. 2C</figref> begins with capturing screen data on a desktop server machine, scaling the data, and converting to the appropriate color-space <b>231</b> all using a graphics processing unit and a screen capture API.

p-0073Next, the captured screen data is encoded <b>232</b> using the CPU and the GPU. The example of <figref idrefs="DRAWINGS">FIG. 2C</figref> involves GPU accelerated processing using an OpenCL API and a General-Purpose computing on graphics processing units (GPGPU) technique, such as CUDA developed by Nvidia®. These techniques involve using motion estimation, discreet cosine transforms (DCT) and lossless compression techniques, such as Context-adaptive binary arithmetic coding (CABAC).

p-0074The encoded image file data is then matched with a time-stamped audio signal and any control input information and multiplexed <b>233</b>. According to this example, the workflow <b>220</b> utilizes one or more of a Real Time Streaming Protocol (RTSP) and a HTTP Live Streaming Protocol.

p-0075Finally, the multiplexed signal is transmitted <b>234</b> to one or more client devices via a PAN, LAN, WAN, etc.

p-0076As explained above, the desktop device optionally includes one or more graphics accelerator and a standalone hardware encoder.

p-0077<figref idrefs="DRAWINGS">FIG. 2D</figref> illustrates a workflow <b>240</b> including the processing steps of a desktop server for streaming the desktop environment to a client device using an onboard CPU, a GPU, and a standalone hardware encoder according to some specific embodiments of the invention.

p-0078The workflow <b>240</b> of <figref idrefs="DRAWINGS">FIG. 2D</figref> begins with capturing screen data on a desktop server machine <b>241</b> using a GPU and API. According to this example, the workflow <b>240</b> utilizes Microsoft® DirectShow for screen capture.

p-0079After the screen is captured, the captured image file is scaled to fit the dimensions of the destination device and color conversion is performed <b>242</b> on the scaled image data image data to comply with the native color values of the destination device. According to this example, the workflow <b>240</b> utilizes a Microsoft®, DirectX API and a Nvidia® Cg shader.

p-0080Next, encoding is performed using a standalone hardware encoder dedicated to low latency video encoding <b>243</b>.

p-0081The encoded image file data is then matched with a time-stamped audio signal and any control input information and multiplexed <b>244</b>. According to this example, the workflow <b>240</b> utilizes one or more of a Real Time Streaming Protocol (RTSP) and a HTTP Live Streaming Protocol.

p-0082Finally, the multiplexed signal is transmitted <b>245</b> to one or more client devices via a PAN, LAN, WAN, etc.

p-0083Although GPU-assisted encoding and dedicated hardware solutions are an effective way to speed up the encoding and transmission of screen data, multi-processing is also an effective way to encode screen frame data at low latency.

p-0084<figref idrefs="DRAWINGS">FIG. 2E</figref> illustrates a method <b>250</b> of multi-processing screen image data to increase encoding speed according to some embodiments of the invention. The method <b>250</b> begins with virtually dividing the desktop server into a plurality of regions <b>251</b>. The screen regions are then individually captured <b>252</b>. In the presently preferred embodiments of the invention, the screen regions are individually captured using a GPU Screen Capture API, as described above.

p-0085Next, each individual regional frame is encoded using a separate processing core <b>253</b> of the CPU or dedicated encoder peripheral. The encoded regional frame data is then streamed to the client device <b>254</b> via one or more network or peer-to-peer infrastructure.

p-0086The client device receives encoded regional frame data <b>255</b> and decodes each encoded region <b>256</b>. In some embodiments of the invention, the client device comprises a multi-core processor and each core processes a single regional frame.

p-0087Finally, the regional frame data is combined and displayed <b>257</b> on the client device.

p-0088Although specific workflows using specific hardware and software modules are disclosed herein, those with ordinary skill in the art having the benefit of this disclosure will appreciate that a wide variety of hardware and software modules, now known or later developed, in many configurations are equally applicable for carrying out the invention.

h-0008Gesture Translation and Native Input Device Rendering

p-0089As explained above, some embodiments of the invention involve translating control inputs received from a user of a client device into control instructions understandable by a remote desktop server. Table 1 is an exemplary translation table equating touch screen controls with controls common to a two-button mouse.

p-0090<tables id="TABLE-US-00001" num="00001"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="3"><colspec colname="1" colwidth="14pt" align="left" /><colspec colname="2" colwidth="105pt" align="left" /><colspec colname="3" colwidth="98pt" align="left" /><thead><row><entry namest="1" nameend="3" rowsep="1">TABLE 1</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row><row><entry /><entry>Touch Screen Control Input</entry><entry>Two Button Mouse Input</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry /><entry>One Finger Tap</entry><entry>Left Click</entry></row><row><entry /><entry>Two Finger Drag</entry><entry>Window Scroll</entry></row><row><entry /><entry>One Finger Tap and Hold</entry><entry>Right Click</entry></row><row><entry /><entry>Two Finger Tap</entry><entry>Mouse Over</entry></row><row><entry /><entry>Three Finger Drag</entry><entry>Scroll Screen</entry></row><row><entry /><entry>Three Finger Tap</entry><entry>Toggle Keyboard/</entry></row><row><entry /><entry /><entry>Toggle Game Controls</entry></row><row><entry namest="1" nameend="3" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

p-0091Likewise, <figref idrefs="DRAWINGS">FIG. 3</figref> illustrates a graphical representation of a touch screen gesture translation table according to some embodiments of the invention.

p-0092Some other embodiments of the invention involve translating gyroscope control inputs received from a user of a client device into control instructions understandable by a remote desktop server. Many mobile client devices include an integrated gyroscope that roll, pitch, and yaw movements of the device into control instructions. Table 2 is an exemplary translation table equating gyroscope controls with movement controls common to a two-button mouse.

p-0093<tables id="TABLE-US-00002" num="00002"><table frame="none" colsep="0" rowsep="0"><tgroup align="left" colsep="0" rowsep="0" cols="2"><colspec colname="1" colwidth="140pt" align="left" /><colspec colname="2" colwidth="77pt" align="left" /><thead><row><entry namest="1" nameend="2" rowsep="1">TABLE 2</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row><row><entry>Gyroscope Control Input</entry><entry>Mouse Movement Input</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></thead><tbody valign="top"><row><entry>Clockwise Rotation about Yaw Axis</entry><entry>Right</entry></row><row><entry>Counter Clockwise Rotation about Yaw Axis</entry><entry>Left</entry></row><row><entry>Clockwise Rotation about Pitch Axis</entry><entry>Down</entry></row><row><entry>Counter Clockwise Rotation about Pitch Axis</entry><entry>Up</entry></row><row><entry>Clockwise Rotation about Roll Axis</entry><entry>Auxiliary Command 1</entry></row><row><entry>Counter Clockwise Rotation about Roll Axis</entry><entry>Auxiliary Command 2</entry></row><row><entry namest="1" nameend="2" align="center" rowsep="1" /></row></tbody></tgroup></table></tables>

p-0094As shown in Table 2, a clockwise rotation of the device about the yaw axis moves the mouse pointer right, a counterclockwise rotation of the device about the yaw axis moves the mouse pointer left, a clockwise rotation of the device about the pitch axis moves the mouse pointer down, and a counterclockwise rotation of the device about the pitch axis moves the mouse pointer up. Additionally, simultaneous movement of the device about two axis results in a vector translation to move the mouse pointer in a diagonal direction. In some embodiments of the invention, the pitch, roll, and yaw rotations must reach a threshold angle before registering as a movement.

p-0095In some embodiments of the invention, as also shown in Table 2, a clockwise rotation of the device about the roll axis initiates a first auxiliary command and a counterclockwise rotation of the device about the roll axis initiates a second auxiliary command. In some embodiments of the invention, the first auxiliary command and the second auxiliary command comprise right and left mouse clicks.

p-0096In some other embodiments of the invention, the first auxiliary command and the second auxiliary command are application specific. For example, when using the mobile device to show a slide show presentation, rotation of the device about the roll axis moves between previous slides and the next slides. In another example, when using the device in a video gaming application, rotation in the roll axis toggles through an inventory of virtual items or through a hierarchy of menu options.

p-0097<figref idrefs="DRAWINGS">FIG. 4</figref> illustrates a graphical representation of a gyroscope translation according to some embodiments of the invention.

p-0098Some other embodiments of the invention involve translating control inputs received from a user via one or more of an accelerometer, a camera, a microphone, and other input devices now known or later developed. Although specific examples of translations are explicitly disclosed herein, it will be readily apparent to a person having ordinary skill in the art and having the benefit of this disclosure that the invention can perform any type of gesture translation that is required to control the desktop system with a client device.

p-0099In some other embodiments of the invention, the desktop system renders a depiction of a mouse or a game controller, along with a frame-by-frame video rendering. Additionally, the desktop system translates touch screen input into mouse or game controller input based on what portions of the virtual mouse or game controller are touched, and for how long.

p-0100<figref idrefs="DRAWINGS">FIGS. 5A-5D</figref> illustrate examples of client rendering of input controls on a client device. <figref idrefs="DRAWINGS">FIG. 5A</figref> illustrates a tablet computer <b>501</b> with a touch screen interface and virtual controls <b>502</b>, <b>503</b> according to some embodiments of the invention. According to <figref idrefs="DRAWINGS">FIG. 5A</figref>, the virtual controls <b>502</b>, <b>503</b> comprise a virtual joystick and a set of virtual buttons. In these embodiments, a user taps the virtual buttons and moves the virtual joystick to control gameplay.

p-0101<figref idrefs="DRAWINGS">FIGS. 5B-5D</figref> illustrate other examples of a tablet computer <b>501</b> with a touch screen interface and virtual controls <b>502</b>, <b>503</b> according to some embodiments of the invention. In the presently preferred embodiments of the invention, the virtual controls <b>502</b>, <b>503</b> provide the user with the ability to toggle between control types.

h-0009Peer-to-Peer Systems and Social Gaming Networks

p-0102In addition to rendered images of a mouse or a game controller, some embodiments of the invention involve receiving information about other people's desktops and rendering an image of another user's screen.

p-0103In some embodiments of the invention, the application on the client device includes a module for receiving encoded video streams from more than one desktop simultaneously. In some embodiments, the application includes a module for displaying a selection screen in which a user can chose one desktop from the many remote desktops to view.

p-0104Some embodiments of the invention involve a peer-to-peer architecture in which applications running on a first user's desktop are encoded and streamed to a second user's device via a peer-to-peer infrastructure.

p-0105Some embodiments involve a module configured for listing a social network of buddies and for rendering one or more buddies' desktop upon selection of a thumbnail image. Likewise, some embodiments of the invention involve a gaming social network feature.

p-0106A large part of the enjoyment in playing video games is the social interaction between online friends and integrating game experiences of your friends with your experiences. Accordingly, some embodiments of the invention involve a client device receiving an encoded video stream of one or more of a user's friends' gaming experiences and viewing a decoded video of those experiences on the user's device.

p-0107<figref idrefs="DRAWINGS">FIG. 6</figref> illustrates an exemplary system <b>600</b> for viewing social gaming network on a client device <b>601</b> according to some embodiments of the invention. The system <b>600</b> comprises one or more game servers <b>620</b> coupled with a user's computer <b>602</b> and with a plurality of client computers c₁, c₂, . . . , c_nvia one or more network <b>610</b> including a local area network (LAN), personal area network (PAN), wide area network (WAN), metropolitan area network (MAN), wireless local area network (WLAN), wireless wide area network (WWAN), peer-to-peer network, or other computer network, now know or later developed.

p-0108In some embodiments of the invention, the client device <b>601</b> includes a hardware or software module configured for displaying a user's gameplay experience in a main frame <b>604</b> and for displaying one or more other players' gameplay experiences in a buddy frame <b>605</b>.

p-0109The one or more game servers <b>620</b> render a unique gaming environment individually for the user computer <b>602</b> and for each of the client computers c₁, c₂, . . . , c_n. The user computer <b>602</b>, as well as the client computers c₁, c₂, . . . , c_n, encode the gameplay that is displayed on their desktops and stream the encoded data to the client device <b>601</b> over the one or more network <b>610</b>.

p-0110The client device <b>601</b> is configured with a hardware or software module configured for decoding the video data and rendering a user's gameplay in a main window <b>604</b> as well as rendering the gameplay of the users of the client computers c₁, c₂, . . . , c_nin the main frame <b>604</b> and the buddy frame <b>605</b>, respectively.

h-0010Automatic Video Decoder Adjustment for Passive Viewing/Multi-Modal

p-0111As explained above, it is often the case that a user of a client device operates the device with the expectation that inputs on the device will seamlessly result in the intended result, thereby necessitating low latency encoding and avoiding buffering. However, buffering a video stream before encoding or decoding increases the quality of the video playback, i.e. smoothness of the video data.

p-0112Despite the general scenario that low latency is the paramount attribute, it is sometimes the case that a user will consume video content passively or semi-passively without providing regular inputs, i.e. watching a movie or other non-interactive video. Therefore, some embodiments of the invention involve systems and methods for automatically determining when a user is consuming content passively and applying a buffer to increase video quality.

p-0113<figref idrefs="DRAWINGS">FIG. 7</figref> illustrates a method <b>700</b> of multi-modal video stream encoding, streaming, decoding, and playback according to some embodiments of the invention. The method <b>700</b> begins on the server-side with capturing the display on a desktop server <b>701</b>. The captured desktop display is encoded at a first frame rate and packaged for one or more client devices <b>702</b>. The encoded video is then streamed to one or more client device <b>703</b>. Once streamed video is received by an application on the client-side from the server-side, the video is decoded and played back on the client device at the first frame rate <b>704</b>.

p-0114The client-side application is configured for accepting user inputs and transmitting them back to the desktop server for translation and control. Accordingly, the method waits for a user input event <b>705</b>. If the client device receives a user input, the client device will continue to operate at a first frame rate <b>704</b> and will transmit the input to the desktop server <b>709</b> for translation. If a user input is not received at <b>705</b>, the method <b>700</b> determines if a threshold time has been reached <b>706</b>. If a threshold time has not been reached, the method <b>700</b> simply waits longer for a user input.

p-0115In the event of a threshold time being reached without the receipt of a user input, the method will begin to operate in a second mode of buffering and display <b>707</b>. The second mode comprises decoding the stream video and playing it back at a lesser frame rate than the encoded video was streamed, thereby building a buffer and resulting in a smoother video playback.

p-0116The method <b>700</b> again waits for a user input event <b>708</b>. Until an event occurs, the method <b>700</b> continues to operate in a second mode <b>707</b>. However, in the event of a user input, the method <b>700</b> catches up frames to match the display of the desktop server <b>710</b> and reverts to displaying at the first frame rate <b>704</b> and transmits the input to the desktop server <b>709</b>.

p-0117The method continues by the desktop server receiving the transmitted user inputs <b>711</b>, translating the inputs into control instructions <b>712</b>, and sending control instructions to the relevant application <b>713</b>.

p-0118In some embodiments of the invention, the step of catching up frames <b>710</b> simply involves trashing all of the buffered frames and beginning display of the current desktop frame. In some other embodiments, the step of catching up frames <b>710</b> involves speeding up frame rate faster than the first frame rate until the video catches up with the current desktop frame.

p-0119In one particular example, the dual mode aspect occurs as follows. In a desktop server-client environment, a desktop sever displays a movie, encodes the video, multiplexes the encoded video with audio, and streams the multiplexed signal to a client device. Suppose the video encoding is performed at 30 frames per second (fps). Initially, the video decoder in the client side decodes the stream at thirty fps to ensure that latency is not noticed by the user of the client device. However, after the threshold time period, the decoder begins to decode the video by a lesser amount that is unperceivable by the user. For example, the client side application might begin playing back the video at 29.5 fps. This difference is likely not perceivable by a human user, but the net result is that a buffer is built up. Thereafter, if the network conditions are slowed for whatever reason, the buffer allows the playback of a slightly delayed video without interruption.

p-0120Next, in the event of a user input, the client side application catches up to the current desktop frame and reverts to the first, non-buffering mode.

p-0121<figref idrefs="DRAWINGS">FIG. 8</figref> is a block schematic diagram of a machine in the exemplary form of a computer system within which a set of instructions may be programmed to cause the machine to execute the logic steps of the invention.

p-0122In alternative embodiments, the machine may comprise a network router, a network switch, a network bridge, personal digital assistant (PDA), a cellular telephone, a Web appliance or any machine capable of executing a sequence of instructions that specify actions to be taken by that machine.

p-0123The computer system <b>800</b> includes a processor <b>802</b>, a main memory <b>804</b> and a static memory <b>806</b>, which communicate with each other via a bus <b>808</b>. The computer system <b>500</b> may further include a display unit <b>810</b>, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT). The computer system <b>800</b> also includes an alphanumeric input device <b>812</b>, for example, a keyboard; a cursor control device <b>814</b>, for example, a mouse; a disk drive unit <b>816</b>, a signal generation device <b>818</b>, for example, a speaker, and a network interface device <b>820</b>.

p-0124The disk drive unit <b>816</b> includes a machine-readable medium <b>824</b> on which is stored a set of executable instructions, i.e. software, <b>826</b> embodying any one, or all, of the methodologies described herein below. The software <b>826</b> is also shown to reside, completely or at least partially, within the main memory <b>804</b> and/or within the processor <b>802</b>. The software <b>826</b> may further be transmitted or received over a network <b>828</b>, <b>830</b> by means of a network interface device <b>820</b>.

p-0125In contrast to the system <b>800</b> discussed above, a different embodiment uses logic circuitry instead of computer-executed instructions to implement processing entities. Depending upon the particular requirements of the application in the areas of speed, expense, tooling costs, and the like, this logic may be implemented by constructing an application-specific integrated circuit (ASIC) having thousands of tiny integrated transistors. Such an ASIC may be implemented with CMOS (complimentary metal oxide semiconductor), TTL (transistor-transistor logic), VLSI (very large systems integration), or another suitable construction. Other alternatives include a digital signal processing chip (DSP), discrete circuitry (such as resistors, capacitors, diodes, inductors, and transistors), field programmable gate array (FPGA), programmable logic array (PLA), programmable logic device (PLD), and the like.

p-0126It is to be understood that embodiments may be used as or to support software programs or software modules executed upon some form of processing core (such as the CPU of a computer) or otherwise implemented or realized upon or within a machine or computer readable medium. A machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine, e.g. a computer. For example, a machine readable medium includes read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other form of propagated signals, for example, carrier waves, infrared signals, digital signals, etc.; or any other type of media suitable for storing or transmitting information.

p-0127Although the invention described herein with reference to the preferred embodiments, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the invention. Accordingly, the invention should only be limited by the Claims included below.