Automatic discovery and mirroring of server-client remote user interface (RUI) session on a companion device and synchronously controlling both sessions using RUI on companion device

FIELD OF THE INVENTION

0001The present application relates generally to constraining the availability of real time and non-real time (NRT) content to the geographic locality of the broadcast TV signals that are associated with the RT content.

BACKGROUND OF THE INVENTION

0002Remote User Interfaces (RUI) are essentially user interfaces sent from a device to be controlled to a device intended to do the controlling for presentation on the intended controlling device as a graphical user interface. RUIs advantageously facilitate the implementation of user interfaces dynamically without having to pre-program the device intended to do the controlling with application graphics and input responses.

0003As understood herein, a larger rendering device such as a TV may be provided with an RUI from a server to allow a user of the TV to control the server using the remote commander of the TV to navigate and operate the RUI during a RUI “session”. The server may implement functions of a set top box, as but one example, and the RUI may accordingly be an electronic programming guide (EPG) that permits establishing, during the session, the channel being shown on the TV.

0004As further understood herein, a user might happen to possess a companion device such as a tablet computer, smart phone, and the like while watching TV during the server-TV TV RUI session, and that controlling the server using that companion device, which typically has more functionality than a TV RC, may be desirable while the RUI session is in force between the other two components, namely, TV and server. However, as further understood herein, simply presenting multiple RUIs on multiple devices without linking them to keep them reconciled with each other is less than desirable.

SUMMARY OF THE INVENTION

0005A server provides a remote user interface (RUI) and video to a main screen during an RUI session, which also includes the status of the RUI. The main screen and the session to the main screen are discoverable by a companion device, which pairs with the main screen and/or server. Once the screens are paired at the server, the server mirrors the graphical part of the RUI of the session on both the main screen and on a new session the server creates on the companion device, which may only have a standard HTML5 browser. In this case, graphics presentation commands for the RUI sent from the server to the main screen are also translated into a protocol compatible with an application running on an HTML5 browser. This application is capable of producing the same graphical output on the companion device and can be downloaded from the server prior to starting the mirror session on the companion device or can be present on the companion device locally before starting the mirror session. The companion screen may have a touch screen. When the companion screen has a touch screen it can replace the need for a remote control on the main screen by providing touch commands directly on the RUI graphics. The companion device sends back the data for the touch or multi-touch to the server, which knows what RUI elements are positioned where on the companion device, processing the commands accordingly as if they came from the main screen.

0006The RUI can be, without limitation, based on the RVU protocol or X11 protocol or any custom protocol that supports server side scene graphics such that the server maintains the rendering control over the assembly of the graphics.

0007Accordingly, a companion device to an audio video display device (AVDD) includes a processor, a display presenting demanded images under control of the processor, and a computer readable storage medium bearing instructions executable by the processor to receive from a server information pertaining to rendering a remote user interface (RUI). A version of the RUI is presented on the AVDD. Display locations on the RUI are selectable by a user to send information correlatable to control commands back to the server. The companion device processor receives from the server updates to the RUI such that the RUI presented on the companion device is synchronized with the version presented on the AVDD.

0008In example embodiments the information pertaining to rendering the RUI is formatted for rendering the RUI in HTML5. In general, HTML5 is an example hypertext markup language that supports graphics rendering commands using vector graphics, in addition to bitmap graphics commands. The version of the RUI presented on the AVDD, however, need not be through an HTML5 client. Also, the RUI presented on the display of the companion device can be modified from the version presented on the AVDD as appropriate for the display of the companion device.

0009The instructions executable by the companion device processor can, if desired, cause the processor to establish communications with the server. The companion device may establish communications only with the AVDD and not the server, with the server information pertaining to rendering the RUI being received through the AVDD by the companion device.

0010In an example embodiment, the instructions executable by the processor cause the processor to execute, on an HTML5 base, a protocol application which transcodes the information pertaining to rendering the RUI to HTML5 graphics. The information pertaining to rendering the RUI received by the companion device can include extensible markup language (XML) commands and/or JavaScript Object Notation (JSON) formatted commands.

0011In another aspect, a server includes a processor and a computer readable storage medium accessible to the processor and bearing instructions which when executed by the processor cause the processor to send to an audio video display device (AVDD) a graphics rendering command to render, on a display of the AVDD, a remote user interface (RUI) containing one or more RUI elements. In synchronization with sending the graphics rendering command to the AVDD, the server sends to a companion device a corresponding graphics rendering command that causes the companion device to render the RUI or a substantially similar version thereof on a display of the companion device. By “substantially similar version” is meant a version of the RUI presented on the AVDD modified for the presentation capabilities of the companion device. The server processor receives from the companion device a user selection signal pertaining to the RUI, correlates the user selection signal to a control command, and executes the control command.

0012In another aspect, a method includes receiving, at an audio video display device (AVDD) and a companion device paired with the AVDD, graphics rendering commands to render on each device a respective remote user interface (RUI). The method also includes receiving, at both the AVDD and companion device, respective RUI modification commands to change the respective RUIs in synchronization with each other.

0013The details of the present invention, both as to its structure and operation, can be best understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which:

BRIEF DESCRIPTION OF THE DRAWINGS

0014<figref idref="DRAWINGS">FIG. 1</figref> is a block diagram of a non-limiting example system in accordance with present principles;

0015<figref idref="DRAWINGS">FIG. 2</figref> is a flow chart of example overall logic;

0016<figref idref="DRAWINGS">FIG. 3</figref> is a flow chart of example alternate overall logic;

0017<figref idref="DRAWINGS">FIG. 4</figref> is a block diagram of a first example implementation;

0018<figref idref="DRAWINGS">FIG. 5</figref> is a block diagram of a second example implementation; and

0019<figref idref="DRAWINGS">FIGS. 6-9</figref> are screen shots illustrating the results of present principles.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

0020Referring initially to the non-limiting example embodiment shown in <figref idref="DRAWINGS">FIG. 1</figref>, a system <b>10</b> includes a client such as an audio video display device (AVDD) <b>12</b>. The AVDD may be implemented by a TV including a TV tuner, 16 communicating with a processor <b>18</b> accessing a tangible computer readable storage medium <b>20</b> such as disk-based or solid state storage. The AVDD may also be implemented, by way of non-limiting example, a large tablet computer, a personal computer, an Internet Protocol (IP) TV such as the one shown in <figref idref="DRAWINGS">FIG. 1</figref>, etc.

0021The AVDD <b>12</b> can output audio on one or more speakers <b>22</b>. In examples, the AVDD <b>12</b> can receive streaming video from the Internet using a built-in wired or wireless modem <b>24</b> communicating with the processor <b>12</b> which may execute a software-implemented implemented browser <b>26</b>. Video is presented under control of the processor <b>18</b> on a display <b>28</b> such as but not limited to a high definition TV (HDTV) flat panel display, and may be a touch screen display. The display <b>28</b> may be a 40″ or larger display. User commands to the processor <b>18</b> may be wirelessly received from a remote control (RC) <b>30</b> using, e.g., rf or infrared. Audio-video display devices other than a TV may be used, e.g., smart phones, game consoles, personal digital organizers, notebook computers and other types of computers, a networked Blu-ray or DVD player with HDMI output connected to a TV or monitor, a game console with HDMI output connected to a TV or monitor, an HDMI Stick with networking and processing capabilities connected to a TV or Monitor, a IP client STB connected by HDMI to TV or Monitor, etc.

0022TV programming from one or more terrestrial TV broadcast sources <b>32</b> as received by a terrestrial broadcast antenna <b>34</b> which communicates with the AVDD <b>12</b> may be presented on the display <b>28</b> and speakers <b>22</b>. The terrestrial broadcast programming may conform to digital ATSC standards and may carry within it a terrestrial broadcast EPG, although the terrestrial broadcast EPG may be received from alternate sources, e.g., the Internet via Ethernet, or cable communication link, or satellite communication link.

0023TV programming from a cable TV head end <b>36</b> may also be received at the TV for presentation of TV signals on the display <b>28</b> and speakers <b>22</b>. When basic cable only is desired, the cable from the wall typically carries TV signals in QAM or NTSC format and is plugged directly into the “F-type connector” <b>38</b> on the TV chassis in the U.S., although the connector used for this purpose in other countries may vary. In contrast, when the user has an extended cable subscription for instance, the signals from the head end <b>36</b> are typically sent through a STB <b>40</b> which may be separate from or integrated within the TV chassis but in any case which sends HDMI baseband signals to the TV. Other types of connections may be used, e.g., MOCA, USB, 1394 protocols, DLNA.

0024In an example implementation, the STB <b>40</b> may perform the role of the server described below. Other example components which may perform the role of server include a gateway device, a media server, a cloud-based server <b>41</b> communicating with the AVDD <b>12</b> over the Internet as shown and performing the function of, e.g., the STB <b>40</b>, and a PC.

0025Similarly, HDMI baseband signals transmitted from a satellite source <b>42</b> of TV broadcast signals received by an integrated receiver/decoder (IRD) <b>44</b> associated with a home satellite dish may be input to the AVDD <b>12</b> for presentation on the display <b>28</b> and speakers <b>22</b>. Also, streaming video may be received from the Internet <b>46</b> for presentation on the display <b>28</b> and speakers <b>22</b>. The streaming video may be received at the computer modem <b>24</b> or it may be received at an in-home modem <b>48</b> that is external to the AVDD <b>12</b> and conveyed to the AVDD <b>12</b> over a wired or wireless Ethernet link and received at an RJ45 or 802.11x antenna on the TV chassis. Note that both broadcast TV programs on a schedule can be sent through any of the channels above along with augmented content, either real time or non-real time (NRT) content, as described further below.

0026Also, in some embodiments a video camera <b>50</b>, which may be integrated in the chassis if desired or mounted separately and electrically connected thereto, may be connected to the processor <b>16</b> to provide to the processor <b>16</b> video images of viewers looking at the display <b>28</b>. The video camera <b>50</b> may be provided with a wide angle lens. The video camera <b>50</b> may have its own camera processor communicating with the TV processor <b>18</b>, or it may be a simple imaging devices such as a CCD or other imager that is controlled by the processor <b>18</b>. Furthermore, a microphone <b>52</b> may be provided on the chassis or separate therefrom and can be electrically connected to the processor <b>16</b> to provide viewer-generated voice commands to the processor <b>16</b>.

0027A companion device <b>54</b> may also be provided. Example embodiments of companion devices include but are not limited to tablet computers, or smart phones, PCs, or other mobile wireless computing device. The companion device may include a visual display <b>56</b> such as a touch screen display controlled by a companion processor <b>58</b> accessing one or more computer readable storage media <b>60</b> to execute present logic. Additional input devices <b>62</b> may be includes, such as but not limited to voice recognition devices, keypads, point and click devices, etc. The companion device <b>54</b> may communicate with a wireless network interface <b>64</b> of the AVDD <b>12</b> and with a wireless network interface <b>65</b> of the server, e.g., STB <b>40</b>, using a companion device network interface <b>66</b>. The interfaces <b>64</b>, <b>65</b>, <b>66</b> may be, without limitation, WiFi interfaces, Bluetooth interfaces, or other appropriate wireless communication interfaces including wireless telephony interfaces.

0028Now referring to <figref idref="DRAWINGS">FIG. 2</figref>, the companion device <b>54</b> connects (“pairs”) with the AVDD <b>12</b> (referred to in the flow chart as “TV” for illustration only) and/or to the server, e.g., the cloud server <b>41</b>, the STB <b>40</b>, or other apparatus performing the server functions described herein. In a preferred embodiment the companion device <b>54</b> connects to the server. Although as described below in reference to <figref idref="DRAWINGS">FIG. 3</figref> the companion device may connect only to the TV.

0029The connection of the companion device <b>54</b> to the server, which may be a wireless communication connection, preferably is automatic and is done by the companion device. The companion device may discover either the RUI being sent from the server to the AVDD <b>12</b>, and/or the RUI session between the server and AVDD, or may itself initiate a first RUI session between the AVDD and server and a second RUI session between the companion device and server which mirrors the first RUI session between the AVDD and server.

0030Among the non-limiting ways this can be accomplished, the companion device can initiate communication using the Discovery and Launch (DIAL) protocol developed under the auspices of Netflix. For example, the companion device <b>54</b> can use the DIAL protocol to launch an RUI client application on the AVDD, and the AVDD in response establishes a communication connection to the server. The companion device <b>54</b> sends the RUI client on the AVDD an extra identifying parameter using DIAL, which in turn sends the identifying parameter to the server. The extra identifying parameter is interpreted by the server to mean that a client with the identity defined by the identifying parameter is connected to the server. The extra identifying parameter can indicate the model of the companion device, which the server can correlate to capabilities so as to appropriately modify the RUI for the display of the companion device, or the extra identifying parameter can indicate the capabilities directly.

0031Then the companion device <b>54</b> establishes communication with the server using the same identifying parameter, which is recognized by the server as indicating that a companion device is connected to the server, paired with the AVDD <b>12</b>, and requiring a separate RUI that mirrors the RUI provided by the server to the AVDD <b>12</b>.

0032Another approach is the server advertising its RUI on the network as well as any active sessions using Universal Plug-n-Play (UPnP), the “Bonjour” discovery feature provided by Apple, Inc., the “ZeroConf” feature provided by Microsoft Corp. The companion device <b>54</b> then connects to the instantiation of the active RUI session that is on the AVDD, therefore mirroring it.

0033Yet another approach is to provide a separate ‘pairing registration screen’ as part of the RUI. Once the user of the companion device <b>54</b> selects a device to pair to, the companion device so informs the server which in response sends to the companion device an RUI mirroring the RUI being sent from the server to the AVDD.

0034Moving to block <b>72</b>, the server sends its RUI to the AVDD. In one non-limiting example, the RUI sent to the AVDD may be sent using a low level graphics card protocol via a transmission control protocol/Internet Protocol (TCP/IP) socket. This is because the RUI information sent to the AVDD (as well as the mirrored RUI information sent to the companion device <b>54</b>) essentially simply defines how to render RUI elements on the rendering device, be it the AVDD <b>12</b> or companion device <b>54</b>. The underlying functionality of the RUI elements, when selected, need not be known to the RUI client on the AVDD <b>12</b> or the RUI client on the companion device <b>54</b>, but only to the server. Selection of an RUI element by a user results in the rendering device simply sending a signal back to the server indicating the screen location at which the selection occurred, which the server then correlates to the appropriate function underlying the selected RUI element. Another method is that some functionality to recognize the selection of a RUI element is available on the RUI client on the AVDD <b>12</b> or the RUI client on the companion device <b>54</b> and some processing is done on the RUI client to indicate the selection as well as sending a signal back to the server indicating the screen location at which the selection occurred, which the server then correlates to the appropriate function underlying the selected RUI element.

0035Proceeding to block <b>74</b>, the server sends to the companion device <b>54</b> a second RUI which mirrors the first RUI sent to the AVDD <b>12</b>, modified if desired as appropriate for the display of the companion device <b>54</b>. In other words, instead of having exactly the same graphics on the companion device <b>54</b> as on the AVDD <b>12</b>, the server can send a companion-optimized version to the companion device <b>54</b> that, for instance, omits a video picture window otherwise presented on the AVDD <b>12</b>, or that has less text information, or that has extra text information, e.g., uniform resource locator (URL) hyperlinks to relevant information about a program, channel, or actress. The user experience on the companion device <b>54</b> still feels like a synchronized experience, however, since the general appearance and functionality of the RUI on the companion device <b>54</b> mirrors that of the RUI on the AVDD <b>12</b> and is kept synchronized therewith by the server.

0036Thus, by “mirror” is meant that the RUI that the server sends to the companion device <b>54</b> changes responsive to user input correspondingly to how the RUI sent by the server to the AVDD <b>12</b> changes, responsive to the same input, and vice-versa. User input on the companion device RUI thus is propagated by the server correspondingly to the RUI on the AVDD. If desired, “mirroring” may also mean that the two RUIs have the same functions when operated by a user.

0037In an example embodiment, the companion device <b>54</b> may use only a relatively simple hypertext markup language (HTML)-5 browser. The server bears the burden of ensuring the RUI graphics sent to the AVDD <b>12</b> and the RUI graphics sent to the companion device <b>54</b> executing the HTML5 browser are mirrored synchronously with each other. Note that if desired, the companion device <b>54</b> may be computationally lightweight and inexpensive and need not support video playback, i.e., the companion device <b>54</b> need not execute video codecs or encryption capability.

0038With this in mind, in example implementations the server generates HTML5-compatible graphics rendering commands to the companion device <b>54</b> that mirror the graphics rendering commands being sent to the AVDD <b>12</b>, which may or may not be HTML5. In embodiments in which the RUI sent from the server to the AVDD <b>12</b> are in a low level graphics protocol such as a TCP/IP socket, HTML5 features such as Canvas and WebSocket provide for transcoding the TCP/IP low level protocol into HTML5, noting that Canvas offers full pixel level manipulation low level graphics and WebSocket replicates the behavior of TCP/IP. Note further that Websocket provides for full duplex communication over an HTTP connection, whereas Canvas provides support for bitmap graphics and vector graphics rendering.

0039Note further that HTML5 is a framework on top of which an application executes. On the companion device <b>54</b>, the application may be referred to as a protocol application which is configured to set up a WebSocket link and receive compressed extensible markup language (XML) or JavaScript Object Notation (JSON) formatted commands, which the protocol application extracts to render HTML5 graphics. For example, the extracted commands may be a graphics vector command such as “draw a box at screen location x,y of size h,w with color c,alpha.” In other words, the protocol application executing on the companion device <b>54</b>, which in example implementations may be written in JavaScript, HTML, and Cascading Style Sheets (CSS), determines how to turn the RUI commands from the server into suitable HTML5 graphics.

0040The protocol application may be downloaded from the server to the companion device <b>54</b>, bearing in mind that the server owns the other half of the protocol (i.e., serving out the compressed XML or JSON formatted commands.) However, the protocol application can be pre-loaded into the HTML5 browser of the companion device <b>54</b> at time of sale or any time prior to using the companion device <b>54</b> to control the server.

0041Block <b>76</b> indicates that the AVDD <b>12</b> and companion device <b>54</b> render their respective RUIs in synchronization with each other according to the respective RUI graphics commands sent to each component by the server. Thus, when a user inputs, by means of, e.g., a touch screen on the companion device <b>54</b>, a selection of an element on the RUI at block <b>78</b>, the companion device sends, at block <b>80</b>, the selection (typically by sending the screen location of the user's touch) to the server. At block <b>82</b> the server correlates the screen location to a UI element and its function to execute the function, e.g., change channel, change volume, perform a trick play function, record, etc. The server also updates both RUIs if appropriate in synchronization with each other to keep the two RUIs mirrored with each other. Updated RUI rendering commands are sent from the server to both the AVDD <b>12</b> and companion device <b>54</b> accordingly.

0042Note that updating the AVDD RUI need not occur for every RUI input by a user on the companion device. For example, suppose the RUI is an electronic program guide (EPG) with many rows of channel selector elements, and six rows at a time are presented on the AVDD while only three rows at a time are presented on the companion device. If a user scrolls up one row on the companion device, causing a formerly top row to disappear, but the new bottom row and former top row of the RUI on the companion device appear within the six rows of the RUI being presented on the AVDD, the server need not command the AVDD to scroll the EPG on the AVDD for this specific case. The server maintains the session with both devices, however, updating both sessions, meaning updating the states of both RUIs, so that further scrolling by a user on the RUI of the companion device which results in new rows of the hypothetical EPG to be shown on the companion device that are not currently shown on the AVDD will cause the server to update the EPG on the AVDD to scroll accordingly, corresponding to the scroll location on the companion device. This works in the opposite direction as well, in which user input to the RUI on the AVDD is sent to the server which correspondingly changes the RUI on the companion device to remain synchronized with the RUI on the AVDD. In general, every RUI element presented on the companion device, which typically has a smaller display than the AVDD, is presented in the RUI shown on the AVDD, and if a user places a cursor focus in a RUI of one of the devices (AVDD or companion), the RUI of the other device is immediately updated by the server to present the same focus.

0043It may now be appreciated that the two RUIs (on the AVDD <b>12</b> and companion device <b>54</b>) are linked synchronously to each other such that what the user sees on the companion device display <b>56</b> mirrors what is seen on the AVDD display <b>28</b>. Note that the companion device <b>54</b> is relieved of the burden of generating the RUI and sending the RUI to the AVDD <b>12</b> since the server bears the burden of ensuring the RUI graphics sent to the AVDD <b>12</b> and the RUI graphics sent to the browser of the companion device <b>54</b> are mirrored synchronously with each other.

0044Note further that a user can operate the companion device <b>54</b> as a touch screen control of what he is seeing on the AVDD, which substantially improves the navigation (touch rather than pointer) and yet still allows everyone in the room to share with the selection process, just as they would with a conventional remote control. This approach is very applicable when the server is in charge of rendering each part of the graphics on the two clients at all times, without any client side composition taking place. Therefore the server can keep the rendering on multiple clients synchronized quite easily.

0045<figref idref="DRAWINGS">FIG. 3</figref> illustrates an alternate embodiment in which the server does not transcode the RUI graphics to HTML5 for transmission to the companion device <b>54</b>, but instead the transcoding task is performed by the AVDD <b>12</b>. In this implementation the server would not need to know that there is a companion device at all, so no server modification is required.

0046Accordingly, at block <b>84</b> the companion device <b>54</b> establishes communication with the AVDD <b>12</b>. At block <b>86</b> the server sends the RUI to the AVDD <b>12</b>, which in turn, at block <b>88</b>, transcodes the RUI for HTML5 and sends the instructions for rendering the RUI to the companion device <b>54</b>. The AVDD <b>12</b> also presents the RUI on its own display, but does not yet send back to the server a message indicating that the RUI or RUI element has in fact been displayed. Thus, both devices render the RUI or RUI element at block <b>90</b>, the companion device marginally later than the AVDD. When the companion device has completed rendering the RUI or RUI element, it reports this completion to the AVDD, which then reports back to the server that the RUI rendering on the AVDD is complete. Accordingly, the AVDD does not report to the server that the RUI or RUI element has been rendered immediately upon completion of rendering of the RUI or RUI element on the AVDD, but instead waits until the companion device has reported to the AVDD that the RUI or RUI element has been rendered on the companion device, at which time the AVDD reports RUI rendition to the server.

0047At block <b>92</b> the companion device receives a user selection. At block <b>94</b> the companion device sends the screen location of the selection to the AVDD, which relays the selection to the server. The server updates the AVDD RUI at block <b>96</b>, sending it to the AVDD, which transcodes the updated RUI and sends it to the companion device <b>54</b>.

0048<figref idref="DRAWINGS">FIGS. 4 and 5</figref> show example block diagrams of a server, a companion device, and an audio video display device (AVDD, labeled “TV” in <figref idref="DRAWINGS">FIGS. 4 and 5</figref> for example description). The server, AVDD, and companion device shown in <figref idref="DRAWINGS">FIGS. 4 and 5</figref> may be implemented by any of the above-described corresponding appliances.

0049In <figref idref="DRAWINGS">FIG. 4</figref>, a server <b>100</b> sends RUI graphic rendering commands over a TCP/IP connection <b>102</b> to an AVDD <b>104</b>, along with video to be rendered on the AVDD <b>104</b>. In parallel, the server <b>100</b> sends to a HTML5 capable companion device <b>106</b> only RUI graphics rendering commands over a Websockets communication link <b>108</b>, without sending video. As described above, the graphics rendering commands, which may be compressed XML or BON formatted commands, are transcoded to HTML5 by the protocol application being executed by the companion device <b>106</b> to render HTML5 graphics that mirror the RUI presented on the AVDD <b>104</b>. UI commands are received by server <b>100</b> back from the companion device <b>106</b> over a link <b>110</b>, with the server then propagating new RUI rendering commands to both the AVDD <b>104</b> and companion device <b>106</b> to reflect the UI commands in synchronization, to maintain the RUIs linked in a mirrored fashion.

0050<figref idref="DRAWINGS">FIG. 5</figref> shows a system that is substantially identical to that shown in <figref idref="DRAWINGS">FIG. 4</figref>, except that in addition to sending RUI graphics rendering commands to the companion device <b>106</b>, the server <b>100</b> also sends video to the companion device <b>106</b> for rendering of the video on the display of the companion device.

0051By way of example of the above principles, assume the logic of <figref idref="DRAWINGS">FIG. 2</figref> (server synchronizes both RUIs) is invoked, and refer now to <figref idref="DRAWINGS">FIGS. 6-9</figref>. In the example shown in <figref idref="DRAWINGS">FIGS. 6 and 7</figref>, both the AVDD display <b>28</b> and the companion device display <b>56</b> present the same RUI, in this case, an EPG. Note that in the hypothetical shown, owing to the smaller companion device display <b>56</b>, it presents only two rows of the EPG whereas the larger AVDD display <b>28</b> presents three.

0052Now assume that the user has touched the region of the companion device display <b>56</b> corresponding to show E, as indicating by border highlight <b>200</b> in <figref idref="DRAWINGS">FIG. 7</figref>. According to above principles, the companion device <b>54</b> sends the screen location of the touch to the server, which correlates the touch to a user command to present information on show E. The server executes this command by maintaining the RUIs on the both devices synchronized, in this case, by presenting show E information on both the AVDD display <b>28</b> and the companion device display <b>56</b>, as shown in <figref idref="DRAWINGS">FIGS. 8 and 9</figref>. Because the AVDD display <b>28</b> is larger than the companion device display <b>56</b>, it presents the same information as the companion device <b>56</b> and more, in this case, the name of the star of show E. Other than that, the RUIs are maintained to present identical appearances. Both RUIs include a “select now” element which if selected from the AVDD RUI using, e.g., the RC <b>30</b> or from the companion device using a touch on the display <b>56</b> cause the server to present show E on the AVDD and, when video capable and enabled for video, on the companion device as well.

0053Note that principles above may be applied to synchronized web pages, in which first and second devices communicate with a server which synchronizes presentation of the same web page on both devices. User activity on the web page of a first one of the devices is sent to the server, which automatically changes the web page on the other (second) device to mirror the web page being operated on by the user of the first device.

0054In the case when the companion device is not capable of receiving video it may receive alternatives in the blank video space. This could be occasional iframes from the video, presented just as a bitmap image. It could also be ads relevant to the user and content being watched. It could also be interactive content like voting, real time social networking, closed captions or other accessibility features that an accessibility user would need that the others in the room would not.

0055While the particular AUTOMATIC DISCOVERY AND MIRRORING OF SERVER-CLIENT REMOTE USER INTERFACE (RUI) SESSION ON COMPANION DEVICE AND SYNCHRONOUSLY CONTROLLING BOTH SESSIONS USING RUI ON COMPANION DEVICE as herein shown and described in detail, it is to be understood that the subject matter which is encompassed by the present invention is limited only by the claims.