User input back channel for wireless displays

[0001] This application claims priority to:

Provisional Patent Application (US) number 61/435194, filed January 21, 2011;

Provisional Patent Application (US) 61/447592, filed February 28, 2011;

Provisional Patent Application (US) 61/448312, filed Mar. 2, 2011;

Provisional Patent Application (US) 61/450101, filed March 7, 2011;

Provisional Patent Application (US) 61/467535, filed March 25, 2011;

Provisional Patent Application (US) 61/467543, filed March 25, 2011;

Provisional Patent Application (US) 61/514863, filed Aug. 3, 2011; and

Provisional Patent Application (US) 61/544445, filed October 7, 2011;

the contents of each of which are fully incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present disclosure relates to techniques for transmitting data between a wireless source device and the wireless device receiver.

BACKGROUND

[0003] Wireless Systems display (WD) or Wi-Fi-display (WFD) includes a wireless source device and one or more wireless receivers. The source device and each of the receivers may be mobile devices or wired devices with wireless links. One or more of a source device and a destination device may include, for example, mobile phones, portable computers with wireless cards, personal digital assistants (PDA), portable media players, or other such devices with wireless capability, comprising the so-called smart phones and smart touch panel or tablet computers, or any type of wireless displays, video game devices or other types of wireless devices. One or more source devices and sink devices may also include wired devices such as televisions, desktop computers, monitors, projectors, etc., which include support for communications.

[0004] The source device sends the multimedia data, for example, audio-video (AV) -data, in one or more sink devices participating in a particular session sharing multimedia. Multimedia data can be played on the local device display source, and on each of the displays devices receivers. More specifically, each of the participating receivers devices renders received multimedia data on its screen, and audio equipment.

SUMMARY OF THE INVENTION

[0005] This disclosure, in general, describes a system in which a wireless device receiver may communicate with a wireless device receiver. As part of the wireless communication session source device may transmit audio and video data to the wireless device receiver, and the wireless device receiver may transmit the user input received at the wireless receiver device back to the wireless source device. Thus, the user of the wireless device receiver can manage wireless source device and manage the content that is transmitted from the wireless device to the wireless source device receiver.

[0006] In one example, the method of matching characteristics between a wireless receiver and a wireless source device comprises transmitting a message to the wireless source device, wherein the message identifies a list of supported categories of input and a plurality of lists of supported types, each of the supported categories Input from the list of supported categories of input has an associated list of supported types.

[0007] In another example, the wireless device receiver configured to coordinate with characteristics of the wireless source device. The wireless device receiver includes a memory retains instructions and one or more processors configured to execute the instructions. In carrying out the instructions one or more processors instruct transmission of a message to the wireless source device, and the message identifies a list of supported categories of input and a lot of lists of supported types, with each of the supported input categories from the list of supported categories of input has an associated list of supported types.

[0008] In another example, the computer readable storage medium stores instructions that, when executed by one or more processors instruct one or more processors perform a method of matching characteristics between the wireless device and the wireless receiver source unit. The method includes transmitting a message to the wireless source device, and the message identifies a list of supported categories of input and a lot of lists of supported types, each of the supported input categories from the list of supported categories of input has an associated list of supported types.

[0009] In another example, the wireless device receiver configured to coordinate with characteristics of the wireless source device. The wireless device receiver includes means for transmitting a message to the wireless source device, and the message identifies a list of supported categories of input and a lot of lists of supported types, each of the supported input categories from the list of supported categories of input has an associated list of supported types.

[0010] In another example, the method of matching characteristics between a wireless receiver and a wireless source device includes receiving a message from a wireless device receiver, wherein the message identifies a list of supported categories of input and a plurality of lists of supported types, each of the supported categories Input from the list of supported categories of input has an associated list of supported types.

[0011] In another example, the wireless source device is adapted to align with the characteristics of the wireless device receiver. Wireless source device includes a memory retains instructions and one or more processors configured to execute the instructions, wherein when the document one or more processors are instructed receiving a message from a wireless device receiver, wherein the message identifies a list of supported categories I and many lists of supported types, with each of the supported input categories from the list of supported categories of input has an associated list of supported types.

[0012] In another example, the computer readable storage medium stores instructions that, when executed by one or more processors instruct one or more processors perform a method of matching characteristics between the wireless device and the wireless receiver source unit. The method includes receiving a message from a wireless device receiver, and the message identifies a list of supported categories of input and a plurality of lists of supported types, wherein each of the supported input category from supported input category has an associated list of supported types.

[0013] In another example, the wireless source device is adapted to align with the characteristics of the wireless device receiver. The wireless source device includes means for receiving a message from the wireless device receiver, and the message identifies a list of supported categories of input and a lot of lists of supported types, each of the supported input categories from the list of supported categories of input has an associated list of supported types.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1A is a block diagram illustrating an example of a source / receiver that may implement techniques of this disclosure.

[0015] FIG. 1B is a block diagram illustrating an example of a source / receiver device with two receivers.

[0016] FIG. 2 shows an example of a source device that may implement techniques of this disclosure.

[0017] FIG. 3 shows an example of a receiver device that may implement techniques of this disclosure.

[0018] FIG. 4 shows a block diagram of a transmitter and a receiver system that may implement techniques of this disclosure.

[0019] FIG. 5A and 5B show exemplary message sequence for performing characteristics of the consents of the technology according to the disclosure.

[0020] FIG. 6 shows an exemplary data packet which can be used to deliver the user input data received by the device receiver, a source device.

[0021] FIG. 7A and 7B are flowcharts of a method of illustrating the technology of this disclosure that can be used for matching characteristics between the source device and destination device.

[0022] FIG. 8A and 8B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with the user input data.

[0023] FIG. 9A and 9B are flowcharts of a method of illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with the user input data.

[0024] FIG. 10A and 10B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with timestamp information and the user input data.

[0025] FIG. 11A and 11B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with timestamp information and the user input data.

[0026] FIG. 12A and 12B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets, which include voice commands.

[0027] FIG. 13A and 13B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with commands multisensory user input.

[0028] FIG. 14A and 14B are flowcharts of a method illustrating the technology of this disclosure that can be used for transmitting and receiving data packets with the user input data are redirected from the external device.

[0029] FIG. 15A and 15B are flowcharts of a method illustrating the technology of this disclosure that can be used to transmit and receive data packets.

DETAILED DESCRIPTION OF THE INVENTION

[0030] This disclosure, in general, describes a system in which a wireless device receiver may communicate with a wireless device receiver. As part of the wireless communication session source device may transmit audio and video data to the wireless device receiver, and the wireless device receiver may transmit the user input received at the wireless receiver device back to the wireless source device. Thus, the user of the wireless device receiver can manage wireless source device and manage the content that is transmitted from the wireless device to the wireless source device receiver.

[0031] FIG. 1A is a block diagram illustrating an exemplary system 100 sources / receivers, which may implement one or more of the techniques of this disclosure. As shown in FIG. 1A, system 100 includes a source device 120, which communicates with a receiver 160 via communications channel 150. The source device 120 may include a memory that stores audio-video (A / V-) data 121, display 122, speaker 123, audio encoder 124 (also referred to as encoder 124), the module 125 controls the audio and video block 126 of transceiver (TX / RX). The device receiver 160 can include a display 162, a speaker 163, an audio decoder 164 (also referred to as decoder 164), the control unit 166 of the transceiver device 167 a user input (UI) module 168 and a user input (UIPM). The illustrated components comprise only one exemplary configuration of a system 100 for sources / receivers. Other configurations may include fewer components than the illustrated components or may include additional components as compared to the illustrated components.

[0032] In the example of FIG. 1A, the source device 120 can display the video portion of the audio-video data 121 on the display 122 and can output the audio portion of the audio-video data 121 to the speaker 123. The audio-video data 121 can be stored locally on the source device 120, available from an external storage medium For example, file server, hard disk, external storage device, Blu-Ray-ROM, DVD or other physical storage medium, or may be streamed to the source device 120 via a network connection, for example, the Internet. In some cases, audio-video data 121 may be captured in real time via the camera and microphone source unit 120. Audio-video data 121 may comprise multimedia content such as movies, television shows or music, but also may include content in real time generated by the source device 120. Such content in real time, for example, may be generated by applications running on the source device 120 or the captured image data, for example, as part of a video telephony session. As described in more detail, a real-time content may in some cases include the video frame with the user input items available for selection by the user. In some cases, audio-video data 121 may include video frames, which are a combination of different types of content such as a movie or a video frame TV program that has a user input points superimposed on the video frame.

[0033] In addition to rendering the audio-video data 121 locally via the display 122 and speaker 123, audio encoder 124 of source device 120 may encode the audio-video data 121, and block 126 transceiver may transmit the encoded data over the channel 150 communication destination device 160. Block 166 transceiver device receiver 160 receives the encoded data and the audio decoder 164 decodes the encoded data and outputs the decoded data through the display 162 and speaker 163. Thus, audio and video data, which is performed by the rendering display 122 and speaker 123 may be simultaneously produced by rendering by the display 162 and speaker 163. The audio data and video data can be placed in frames and audio frames can be synchronized in time with the video frames when performing rendering.

[0034] Audio-video encoder 124 and audio decoder 164 may implement any number of standards for audio and video compression, for example, the standard ITU-T H.264, alternatively referred to as MPEG-4 Part 10 advanced video coding (AVC), or new developed highly efficient video coding standard (HEVC), sometimes referred to as standard H.265. It may also be used a plurality of types of their own or other standardized compression techniques. Generally speaking, the audio decoder 164 is configured to perform operations mutually inverse coding audio encoder 124. Although not shown in FIG. 1A, in some aspects, A / V-encoder 124 and decoder 164 A / V can be integrated with an audio encoder and decoder, and may include appropriate units multiplexer-demultiplexer or other hardware and software to handle encoding both audio and video in a common data stream or separate data streams.

[0035] As described in greater detail below, A / V-encoder 124 may also perform other coding functions, in addition to implementing the video compression standard, as described above. For example, A / V-coder 124 can add various types of metadata to the A / V-data 121 before transmitting A / V-data 121 to destination device 160. In some cases, A / V-data 121 may be stored or received in the device -source 120 in coded form and therefore do not require additional compression by the A / V-124 encoder.

[0036] Although FIG. 1A shows a communication channel 150 that carries the working and operating video audio data separately, it should be understood that in some cases, workers working audio and video data may be a part of the data stream. If applicable, the multiplexer-demultiplexer blocks may correspond to ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP). Audio encoder 124 and audio decoder 164 may be implemented as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete logic, software, hardware, firmware or any combinations thereof. Each of the audio encoder 124 and an audio video decoder 164 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder / decoder (codec). Thus, each source device 120 and destination device 160 may comprise a specialized machines adapted to perform one or more of the techniques of this disclosure.

[0037] The display 122 and the display 162 may comprise any number of devices video output such as a display on a cathode ray tube (CRT), liquid crystal display (LCD), a plasma display, light-emitting diode (LED) display, an organic LED (OLED ) or other type of display device. In these and other examples, the displays 122 and 162 may be emissive or transmissive display screens. The display 122 and the display 162 may also be a touch screen, so that they simultaneously represent both an input device and a display device. These touchscreens can be capacitive, resistive or other type of touch panel that allows a user to provide user input to the appropriate device.

[0038] The speaker 123 may comprise any set of audio output devices, such as headphones, a speaker system, a system with multiple speakers and surround sound system. Further, although the display 122 and speaker 123 are shown as part of the source device 120 and the display 162 and speaker 163 are shown as part of the receiver device 160, source device 120 and destination device 160 may actually be the system devices. As one example, the display 162 may be a television receiver, the speaker 163 may be a surround sound system, and decoder 164 may be part of the external consoles connected, wired or wirelessly, to the display 162 and the dynamics 163. In other cases, the destination device 160 may be a single device, for example, a tablet computer or smartphone. In still other cases, the source device 120 and destination device 160 are similar devices, for example, both are smartphones, tablet computers, etc. In this case, one device can act as a source and the other can operate as a receiver. These roles may even be reversed in subsequent communications. In still other cases the source device may comprise a mobile device, for example, a smart phone, laptop or tablet computer, and a destination device may comprise a stationary device (e.g., wire AC power), and in this case, the source device may delivering audio and video data for presentation through the mass destination device.

[0039] Block 126 of the transceiver and the block 166 of transceiver devices may include various mixers, filters, amplifiers or other components designed for signal modulation, and one or more antennas, and other components designed to transmit and receive data. Communication channel 150 generally represents any suitable communication medium, or collection of different communication media for transmitting video data from source device 120 to destination device 160. The communication channel 150 is typically a relatively short-range communication channel, like Wi-Fi, Bluetooth technology and etc. However, the communication channel 150 is not necessarily limited in this respect, and may comprise any wireless or wired communication medium, for example, a radio frequency (RF) spectrum or one or more physical transmission lines, or any combination of wireless and wired media. In other embodiments, communications channel 150 can even be part of a packet switched network such as a wired or wireless local area network, a wide area network or a global network, such as Internet. Additionally, the communication channel 150 may be used by the source device 120 and destination device 160 to establish a communication link between the peers. The source device 120 and destination device 160 may transmit communications over the channel 150 using a communication protocol such as the standard of the IEEE 802.11 family of standards. The source device 120 and destination device 160, for example, can communicate according to the standard Wi-Fi Direct, so that the source device 120 and destination device 160 communicate with each other directly without the use of mediators such as wireless access points or so called a public access point. The source device 120 and destination device 160 can also carry out the establishment of a direct tunnel link (TLDS), to prevent or reduce network congestion. Techniques of this disclosure may sometimes be described with respect to Wi-Fi, but it is assumed that aspects of these technologies may also be compliant with other communications protocols. By way of example, and not limitation, wireless communication between the source device 120 and destination device can use multiplexing techniques with orthogonal frequency division multiplexing (OFDM). It may also be used a plurality of other wireless communication technologies, including, but not limited to Multiple Access with Time Division Multiplexing (TDMA) multiple access frequency division multiplexing (FDMA), code division multiple access (CDMA) systems, or any combination OFDM, FDMA, TDMA and / or CDMA. WiFi Direct and TDLS intended to establish sessions with respect to short-range communication. The relatively short distance in this context may mean, for example, less than 70 meters, although in a noisy or cluttered environment distance between devices may be even smaller, for example less than 35 meters.

[0040] In addition to decoding and rendering of the data received from the source device 120, destination device 160 may also receive user inputs from user input device 167. User input device 167, for example, be a keyboard, a mouse, trackball or touch pad, touch screen, voice recognition module, or any other such user input device. UIPM 168 formats the command user input received through the user input device 167, the structure of the data packet, the interpretation of which the source device 120 permits. Such data packets are transmitted by transceiver 166 to the source device 120 via connection 150. Block 126 transceiver receives data packets, and a module 125 A / V-control parses the data packets in order to interpret the user input command, which is done through the user input device 167. Based on the command received in the packet data unit 125 A / V-control can change and transmit the encoded content. Thus, the user of the device receiver 160 can operate the working and operating video data audio data transmitted by the source device 120, remotely and without direct interaction with the source device 120. Examples of the types of commands that the user-receiver unit 160 may transmit to the source device 120 include commands to rewind, fast forward, pause and playback audio and video, as well as commands for zoom, rotate, scroll, etc. Users can also choose from, for example, to items from the menu selection and transfer back to the source device 120.

[0041] Additionally, users receiver device 160 may be able to run and manage applications on the source device 120. For example, the user device receiver 160 may be able to run the application for editing photos stored on the source device 120, and use the application To edit a photo that is stored locally on the device, the source device 120. Receiver 160 can provide the user with such possibilities, it seems that the picture is edited locally on the device receiver 160, whereas in fact the photo is edited in the source device 120. Using this configuration, the user device may be able to use the characteristics of the device for use with multiple devices. For example, source device 120 may be a smartphone with large amounts of memory and high-processing performance. The user of the source device 120 can use a smartphone in all environments and situations, which are typically used in smart phones. However, when watching a movie user may want to watch a movie on a device with a large screen display, and in this case, the receiver 160 may be a tablet computer, or even a large display device or television receiver. Optionally send or respond to a mail message, the user may wish to use the device with the keyboard, and in this case, the receiver 160 may be a portable computer. In both cases, the amount of processing in this case may be performed by the source device 120 (smart phone in this example), even when the user interacts with the device receiver. In this particular context, the work volume due to processing performed by the source device 120, destination device 160 may be a less expensive device with fewer resources than if the device receiver 160 is required to perform the processing performed by the source device 120. As device -source and destination device may allow reception of user input (e.g., commands, touch screen), in some examples, and techniques of this disclosure can facilitate two-way interaction by matching and / or identification of the device characteristics in any given session.

[0042] In some configurations, the module 125 A / V-control may be a process operating system, the operating system is performed by the source device 125. However, in other configurations of modules 125 A / V-control process may be a software application running source device 120. In this configuration, the user input command can be interpreted by a software process, so the user device receiver 160 communicates directly with the application running on the source device 120 and not the operating system running on the source device 120. Through interaction with the application itself, rather than with an operating system, a user device receiver 160 may have access to a library of commands that are not proper for the operating system of the source device 120. Further, the interaction with the application itself may provide a simpler gear and command processing via devices running on other platforms.

[0043] The source device 120 may respond to a user input device used in the wireless receiver 160. In this environment, the interactive application, user input device used in the wireless receiver 160 may be sent back to the wireless display source 150 via communications. In one example, the architecture of the reverse channel, also called "reverse channel user interface (UIBC)", can be implemented to provide the ability to sink devices 160 to transmit a user input applied to the device receiver 160, a source device 120. Architecture reverse channel may include upper message to transport the user input and the lower layer frames to negotiate the characteristics of the user interface on the device receiver 160 and source device 120. UIBC may reside on top of the transport layer based on Internet protocol (IP) between destination device 160 and source device 120. Thus, UIBC may be above the transport layer communication model in accordance with the Open Systems Interconnection (OSI). In one example, OSI-link includes seven levels (1 - physical, 2 - fold, 3 - net 4 - transport, 5 - session, 6 - 7 and representation - applied). In this example, finding the transport means above the levels 5, 6 and 7. To facilitate the reliable transmission and delivery sequence of data packets containing data of the user input, UIBC can be configured to run on top of other protocols, packet communication, such as management protocol transmission / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP). UDP and TCP can operate in parallel architecture in OSI-layers. TCP / IP can provide the ability to sink devices 160 and source device 120 to implement the technology in the case of the retransmission packet loss.

[0044] In some cases, there may be a mismatch between a user input interface disposed in a source device 120 and destination device 160. In order to solve the potential problems created by such a mismatch, and improve possibilities of users in this case, alignment performance interface user input can be performed between the source device 120 and destination device 160 to establish a communication session or more than once during the communication session. As part of this approval process, the source device 120 and destination device 160 may agree about an agreed resolution of the screen. When the destination device 160 transmits the coordinate data associated with the user input, the device receiver 160 can be scaled coordinate data received from the display 162 so that they coincide with the agreed-resolution screen. In one example, if the destination device 160 has a resolution of 1280x720, a source device 120 has a resolution of 1600x900, the device may for example be used as a coordinated 1280x720 resolution. Coordinated resolution can be selected on the basis of an authorization device receiver 160, although may also be used in the resolution of the source device 120 or some other solution. In the example in which the device is used with 1280x720 receiver, receiver device 160 may scale the received coordinates by a factor of X 1600/1280 to transmit the coordinates in the source device 120, and similarly, the destination device 160 may scale the received Y coordinate 900 / 720 to transmit the coordinates to the source device 120. In other configurations, the source device 120 can scale the resulting coordinates to an agreed solution. Scaling can increase or decrease the range of coordinates on the basis of using the device receiver 160 display a higher resolution than the source device 120, or vice versa.

[0045] Additionally, in some cases, the resolution of the device receiver 160 can be varied during the session, potentially creating a mismatch between the display 122 and display 162. To enhance the capabilities of the user experience and to ensure proper functionality, the system 100 source / receiver may implement techniques to reduce or prevent mismatches of user interaction through the implementation of technologies for the normalization of the screen. The display 122 of the source device 120 and the display device 162, the receiver 160 may have a different resolution and / or high aspect ratio. Additionally, in some environments device user receiver 160 may be able to resize the display window for the video data received from the source device 120, so that the rendering of video data received from the source device 120, in the window that does not completely cover the display 162 device -receiver 160. In another exemplary environment, user device receiver 160 may have the option of viewing the content in landscape mode or in portrait mode, each of which has a unique coordinate and high aspect ratio. In such cases, the coordinates associated with the user input received in the receiver device 160, such as the coordinate point at which the event occurs mouse click or touch input, may not be processed by the source device 120 without modification of coordinates. Accordingly, techniques of this disclosure may include converting coordinate user input received at the receiver device 160, to the coordinates associated with the source device 120. This conversion is also referred to as normalization herein, and as explained in more detail below, This conversion can be performed based on the receiver or on the basis of the source.

[0046] User input is received through the receiver unit 160 may be made through the UI-module 167, e.g., on the driver level and transferred to the operating system of the device receiver 160. The operating system on the device receiver 160 can receive the coordinates (xSINK, ySINK), associated with a place on the surface of the display, where the user input occurred. In this example, (xSINK, ySINK) can represent the coordinates of the display 162 on which the event occurred mouse click or touch. A display window which rendering is performed on the display 162 may have a length in the coordinate X (LDW) and width (WDW) Y coordinate, which describe the size of the display window. The window display may also have coordinates (aDW, bDW) the upper left corner, which describes the location of the display window. On the basis of LDW, WDW and upper left coordinates (aDW, bDW) can be determined part of the display 162, met through the display window. For example, the upper-right corner of the display may be in the coordinates (aDW + LDW, bDW), the lower left corner of the display window can be located by coordinates (aDW, bDW + WDW), and the lower right corner of the display may be in the coordinates (aDW + LDW , bDW + WDW). The device receiver 160 can handle input as UIBC-input if the input is taken to the coordinates within the display window. In other words, the input from the associated coordinates (xSINK, ySINK) can be processed as UIBC-entry if the following conditions are satisfied:

aDW≤xSINK≤aDW + LDW (1)

bDW≤ySINK≤bDW + WDW (2)

[0047] After determining that the user input is UIBC-entering coordinates associated with the input can be normalized by UIPM 168 before being sent to the source device 120. The inputs which are determined as being outside the display window, can be handled locally by the device receiver 160 as a non-UIBC-inputs.

[0048] As mentioned above, the normalization of the input coordinates can be performed either on the basis of the source or through the receiver. When implementing normalization based receiver, the source device 120 can send the supported resolution (LSRC, WSRC) for displaying the display 122, video data or independent of the video data to destination device 160. Supported resolution display, for example, it may be transferred as part of the matching session characteristics or can be transferred to another time during a communication session. The device receiver 160 can determine the resolution (LSINK, WSINK) display for the display 162, the resolution (LDW, WDW) of the window for displaying the content window received from the source device 120 and the coordinate (aDW, bDW) of the upper left corner of the display window . As described above, when the coordinate (xSINK, ySINK), corresponding to the user input is determined to be within the display window, the operating system device receiver 160 may convert a coordinate (xSINK, ySINK) into coordinates (xSRC, ySRC) Source using conversion functions . Approximate function for converting (xSINK, ySINK) in (xSRC, ySRC) may be as follows:

xSRC = (xSINK-aDW) * (LSRC / LDW) (3)

ySRC = (ySINK-bDW) * (WSRC / WDW) (4)

[0049] Thus, when transmitting coordinates corresponding to the received user input, the device receiver 160 can transmit coordinate (xSRC, ySRC) for user input received at (xSINK, ySINK). As described in more detail below, the coordinate (xSRC, ySRC), for example, may be transmitted as part of a data packet used to transmit the user input received at the receiver device 160 to the source device 120 through UIBC. In all other parts of this disclosure, in which the input coordinates are described as being included in the packet data, these coordinates can be converted into coordinates of the source, as described above, in cases where the system 100 source / receiver implements normalization based receiver.

[0050] When the system 100 sources / receivers implements normalization based on user inputs for the source identified by the UIBC-input instead of local inputs (ie, within the display window, not outside the display window), the above calculation can be performed in the source device 120 instead of a destination device 160. To simplify these calculations, receiver device 160 may transmit to the source device 120 the values ​​for LDW, WDW and location information for a window display (e.g., aDW, bDW), and the coordinates for (xSINK, ySINK). Using these values ​​transmitted by the source device 120 can determine the values ​​for (xSRC, ySRC) according to the above equations 3 and 4.

[0051] In other implementations, the normalization on the basis of the receiver, destination device 160 may transmit the coordinates (xDW, yDW) for user input, which describe in what location within the display window occurs a user input event, unlike, where on the display 162 a user input event occurs. In this embodiment, the coordinates (xDW, yDW) can be transmitted to the source device 120, together with values ​​for (LDW, WDW). On the basis of these values ​​received, the source device 120 can determine (xSRC, ySRC) according to the following conversion functions:

xSRC = xDW * (LSRC / LDW) (5)

ySRC = yDW * (WSRC / WDW) (6)

Receiver device 160 may determine xDW yDW and based on the following features:

xDW = xSINK-aDW (7)

yDW = ySINK-bDW (8)

[0052] When this disclosure describes coordinates of transmission associated with the user input in the data packet, such as transfer of these coordinates may include a normalization based on the source or on the basis of the receiver, as described above, and / or may include any additional the information necessary to perform normalization on the basis of the source or through the receiver.

[0053] UIBC can be designed with the ability to transport various types of user input data comprising middleware user input data. For example, the source device 120 can be operated under the operating system iOS®, while the destination device 160 is running another operating system, such as the Android® and Windows®. Regardless of the platform UIPM 168 can encapsulate the received user input in a form understandable to the module 125 A / V-control. A number of different types of user input formats can be supported by UIBC, so as to allow a plurality of different types of devices, sources and receivers use the protocol whether or not to operate the source devices and receivers on different platforms. Formats universal input may be specified, and the platform-specific input formats can be supported, thus providing flexibility due to the fact that the user input can be transmitted between the source device 120 and destination device 160 via UIBC.

[0054] In the example of FIG. 1A, the source device 120 may include a smart phone, a tablet computer, laptop computer, desktop computer, television receiver with support for Wi-Fi or any other device capable of transmitting audio and video data. The device is similar to the receiver 160 may include a smart phone, a tablet computer, laptop computer, desktop computer, television receiver with support for Wi-Fi or any other device capable of receiving audio and video data, and receive user input. In some cases, the destination device 160 may include a system of devices, so that the display 162, speaker 163, UI unit 167, and A / V-encoder 164 are parts separate but interacting devices. The source device 120 may similarly be a system of devices instead of a single device.

[0055] In this disclosure, the term "source device" is generally used to mean a device that transmits audio-video data, and the term "destination device" is generally used to mean a device that accepts audio video from a source device. In many cases, source device 120 and destination device 160 may be similar or identical devices, in that one device operates as the source, while the other operates as a receiver. Furthermore, these roles may be reversed in different communication sessions. Thus, the destination device in a communication session may become a source device in a subsequent communication session, or vice versa.

[0056] FIG. 1B is a block diagram illustrating an exemplary system 101 sources / receivers that may implement techniques of this disclosure. System 101 source / receiver includes a source device 120 and destination device 160, each of which can function and operate in the manner described above for FIG. 1A. System 101 source / receiver further comprises a receiver unit 180. Similarly ustro ystvu receiver 160 described above, the receiver 180 may receive audio and video from a source device 120 and transmit the user command to the source device 120 through the established UIBC. In some configurations, destination device 160 and destination device 180 can operate independently, and output of audio and video source device 120 may be displayed simultaneously in the device receiver 160 and the receiver device 180. In alternative configurations, the device -receiver 160 may be the primary receiver device and receiver device 180 may be a secondary device receiver. In this exemplary configuration, the device receiver 160 and the receiver device 180 can connect and the device receiver 160 can display the video data while a destination device 180 outputs corresponding audio data. Additionally, in some configurations, the device receiver 160 can display only transmitted video data, while the receiver apparatus 180 outputs only the audio data transmitted.

[0057] FIG. 2 is a block diagram showing one example of a source device 220. The source device 220 may be a device similar to the source device 120 in FIG. 1A, and can operate identically to source device 120. The source device 220 includes a local display 222, a local speaker 223, processor 231, memory 232, a transport unit 233 and the wireless modem 234. As shown in FIG. 2, source device 220 may include one or more processors (i.e., CPU 231), which encode and / or decode A / V-data transport, storage and display. A / V-data, for example, may be stored in a memory 232. The memory 232 may store the entire A / V-file can contain a smaller or a buffer that maintains a portion of A / V-file, such as streamed from another device or source. The transport unit 233 may process the encoded A / V-data transport network. For example, the encoded A / V-data may be processed by processor 231 and encapsulated by the transport unit 233 in units of the network access layer (NAL) for network communication. NAL-unit may be sent via the wireless modem 234 to a wireless destination device over a network connection. Wireless modem 234, for example, may be Wi-Fi-modem arranged to implement one of a family of standards IEEE 802.11.

[0058] The source device 220 may also locally process and display the A / V-data. In particular, the display processor 235 may process the video data to be displayed on the local display 222, an audio processor 236 may process audio data to output to a speaker 223.

[0059] As described above with respect to the source device 120 of FIG. 1A, the source device 220 may also receive commands from the user input device receiver. Thus, the wireless modem 234 of the source device 220 receives the encapsulated data packets, for example, NAL-units and sends the encapsulated data unit into a transport block 233 for decapsulation. For example, the transport unit 233 can extract data packets from NAL-units, and the processor 231 may parse the data packets to retrieve the user input commands. Based on the user input commands, the processor 231 may regulate the encoded A / V-data transmitted by the source device 220 to destination device. Thus, the functionality described above in relation to module 125 A / V-control of FIG. 1A, it may be implemented, or fully or partially, by the processor 231.

[0060] The processor 231 of FIG. 2 generally represents any of the plurality of processors, including, but only one or more digital signal processors (DSP), general purpose microprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other equivalent integrated or discrete logic circuitry, or some combination thereof. The memory device 232 of FIG. 2 may comprise any of a variety of volatile or non-volatile memory devices including, but not limited to, random access memory (RAM), for example, synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, etc. The memory 232 may comprise a computer-readable storage medium for storing audio and video data as well as other types of data. The memory 232 may further store instructions and code that are executed by processor 231 as part of the performance of various techniques described in this disclosure.

[0061] FIG. 3 shows an example of a receiver apparatus 360. The device receiver 360 can be a device similar to the device receiver 160 of FIG. 1A, and can operate identically receiver apparatus 160. The device receiver 360 includes one or more processors (i.e., CPU 331), a memory 332, a transport unit 333, a wireless modem 334, display processor 335, the local display 362 , an audio processor 336, a speaker 363 and a user input interface 376. The device receiver 360 receives the wireless modem 334 encapsulated data units sent from the source device. Wireless modem 334, for example, may be Wi-Fi-modem arranged to implement one or more standards of the IEEE 802.11 family of standards. The transport unit 333 may decapsulate encapsulated data units. For example, the transport unit 333 can extract the coded video data units from the encapsulated data and send the encoded A / V-data to the processor 331 for decoding and performing rendering for output. Display processor 335 may process the decoded video data to be displayed on the local display 362 and audio processor 336 can process the decoded audio data for output to a speaker 363.

[0062] In addition to rendering audio and video data, the wireless device receiver 360 can also receive data through a user input interface 376 for user input. The interface 376 for user input can be any number of user input device, including, but not limited to, the interface touchscreen display, keyboard, mouse, the module processing voice commands capture unit gestures (e.g., the characteristics of the capture input through the camera) or any other a number of user input devices. User input received via the user input interface 376 may be processed by processor 331. This processing may include generating data packets that include the user input command received in accordance with the techniques described in this disclosure. After forming, the transport unit 333 can process data packets for transport network to the wireless device, the source of UIBC.

[0063] The processor 331 of FIG. 3 may comprise one or more of a wide range of processors, for example, one or more digital signal processors (DSP), general purpose microprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other equivalent integrated or discrete logic circuitry or some combination thereof. The memory device 332 of FIG. 3 may comprise any of a variety of volatile or non-volatile memory devices including, but not limited to, random access memory (RAM), for example, synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash memory, etc. The memory 232 may comprise a computer-readable storage medium for storing audio and video data as well as other types of data. The memory 332 may further store instructions and code that are executed by processor 331 as part of the performance of various techniques described in this disclosure.

[0064] FIG. 4 shows a block diagram of an exemplary system 410 transmitter and receiver system 450 that may be used by transceiver 126 and transceiver 166 of FIG. 1A to exchange data over the channel 150 connection. In system 410, traffic data transmission device for data streams is provided from a room a data source 412 to a TX data processor 414 (TX). Each data stream may be transmitted over a respective transmit antenna. Processor 414-TX data formats, codes, and interleaves the traffic data for each data stream based on a particular coding scheme selected for that data stream.

[0065] The coded data for each data stream may be multiplexed with pilot data using Orthogonal Frequency Division Multiplexing (OFDM). It may also be used a plurality of other wireless communication technologies, including, but not limited to Multiple Access with Time Division Multiplexing (TDMA) multiple access frequency division multiplexing (FDMA), code division multiple access (CDMA) systems, or any combination OFDM, FDMA, TDMA and / or CDMA.

[0066] Referring to FIG. 4, the pilot data is typically a known data pattern that is processed in a known manner and may be used at the receiver system to estimate the channel response. The multiplexed pilot and coded data for each data stream is then modulated (e.g., symbol mapped) based on a particular modulation scheme (e.g., binary phase shift keying (BPSK), quadrature phase shift keying (QPSK), M-PSK or M-QAM (Quadrature Amplitude modulation) wherein M can be a power of two) selected for that data stream to provide modulation symbols. The data rate, coding, and modulation for each data stream may be determined by instructions performed by a processor 430 that can be coupled to memory 432.

[0067] The modulation symbols for all data streams are then provided to a TX MIMO processor 420-which may further process the modulation symbols (e.g., for OFDM). TX MIMO-processor 420 can then provide NT modulation symbol streams to NT transmitters (TMTR) 422a-422t. In certain aspects, TX MIMO processor 420-applies beamforming weights to the symbols of the data streams and to the antenna from which the symbol is transmitted.

[0068] Each transmitter 422 can receive and process a respective symbol stream to provide one or more analog signals, and further leads to conditions (e.g., amplifies, filters, and upconverts) the analog signals to provide a modulated signal, suitable for transmission over the MIMO-channel. NT modulated signals from transmitters 422a-422t are then transmitted from NT antennas 424a-424t, respectively.

[0069] The system receiver 450, the transmitted modulated signals are received by NR antennas 452a-452r, and the received signal from each antenna 452 is provided to a respective receiver (RCVR) 454a-454r. The receiving device 454 leads to conditions (e.g., filters, amplifies, and downconverts) a respective received signal, digitizes the conditioned signal to provide samples, and further processes the samples to provide a corresponding "received" symbol stream.

[0070] An RX data processor 460 (RX) then receives and processes the NR received symbol streams from NR receivers 454 based on a particular receiver processing technique to provide NT "detected" symbol streams. Processor 460 RX-data then demodulates, deinterleaves, and decodes each detected symbol stream to recover the traffic data for the data stream. The processing by RX processor 460 is complementary to the data-processing performed by TX MIMO-processor 420 and TX processor 414 in the system-data transmission device 410.

[0071] Processor 470, which can be coupled to memory 472 periodically determines which precoding matrix to use. The reverse link message may comprise various types of information regarding the communication link and / or the received data stream. The reverse link message is then processed by processor 438 TX-data, which also receives traffic data for a number of data streams from a source 436, modulated by a modulator 480, is conditioned by transmitters 454a-454r, and transmitted back to the system 410 transmits device.

[0072] The system 410 transmits the modulated signals from receiver system 450 of the receiving device are received by antennas 424 provides conditioned by receivers 422, demodulated by a demodulator 440, and processed by processor 442 RX-data to extract the reverse link message transmitted System 450 via the receiver. Processor 430 then determines which precoding matrix to use for determining the beamforming weights orientation, and then processes the extracted message.

[0073] FIG. 5A is a flowchart illustrating an exemplary sequence of communications between the source device 520 and destination device 560 as part of the coordination session characteristics. Matching characteristics can be implemented as part of a larger process of establishing a session between the source device 520 and destination device 560. The session is, for example, may be established via Wi-Fi Direct or TDLS as a reference standard connection. After the establishment of Wi-Fi Direct- or TDLS-session destination device 560 may initiate a TCP-connection to the source device 520. As part of the establishment of TCP-connection port control protocol streaming real-time (RTSP) can be set to operate communication session between the source device 520 and destination device 560.

[0074] The source device 520 may generally operate in the manner described above for source device 120 of FIG. 1A, and the destination device 560 may generally operate in the manner described above for receiver unit 160 of FIG. 1A. Once the source device 520 and destination device 560 to establish a connection, the source device 520 and destination device 560 may determine a set of parameters to be used for subsequent communication session as part of the data exchange in agreement characteristics.

[0075] The source device 520 and destination device 560 can coordinate characteristics through a series of messages. Messages, for instance, may be messaging protocol streaming in real time (RTSP). At any stage in coordination with the recipient of the message can RTSP request-reply-RTSP response that includes a status code RTSP, different RTSP OK, in which case the exchange of messages can be repeated with a different set of parameters or characteristics of the session negotiation may fail.

[0076] The source device 520 may send a first message (message requesting RTSP OPTIONS) into the receiver 560 to determine a set of RTSP-ways that support the destination device 560. When receiving the first message from the source device 520, the device receiver 560 can meet the second message (response message to RTSP OPTIONS), which lists the RTSP-processes supported by the receiver 560. The second message may also include the RTSP OK state code.

[0077] After sending a second message to the source device 520, a destination device 560 may send a third message (message requesting RTSP OPTIONS), to define a set of RTSP-ways, which supports the source device 520. Upon receipt of the third message from the device receiver 560 the source device 520 can respond to the fourth message (RTSP OPTIONS response), which enumerates-RTSP methods supported by the source device 520. The fourth message may also include a state code RTSP OK.

[0078] After sending the fourth message source device 520 may send a fifth message (request message RTSP GET_PARAMETER), to specify a list of features that are of interest to the source device 520. The destination device 560 may respond the sixth message (to answer RTSP GET_PARAMETER). The sixth message can contain a status code of RTSP. If the status code RTSP - OK, the sixth message can also include parameters for the response parameter indicating a fifth message supported by the device receiver 560. The device receiver 560 can ignore the parameters in the fifth message which destination device 560 do not support.

[0079] Based on the sixth message, the source 520 can determine the optimal set of parameters to be used for the session, and may send a seventh message (request message RTSP SET_PARAMETER) to destination device 560. The seventh message may contain a set of parameters that must be used during a session between the source device 520 and destination device 560. The seventh message may include wfd-url-presentation, which describes uniform resource identifier (URI), to be used in the request for the establishment of RTSP, to install communications session. Wfd-presentation-url specifies the URI, which destination device 560 may be used for subsequent messages in the messaging session establishment. Values ​​wfd-url0 and wfd-url1, specified in this parameter can correspond to the values ​​rtp-port0 and rtp-port1 in wfd-client-rtp-ports in the seventh report. RTP in this case generally means a real-time protocol, which can run over UDP.

[0080] When receiving the seventh message destination device 560 to eight messages can respond with status code RTSP, indicating whether or not the ending centrally configure, as defined in the seventh message. As mentioned above, as the source device and destination device are reversed or changed in different sessions. The order of messages that establish a session connection, in some cases, may define a device which operates as the source, and set a device that works as a receiver.

[0081] FIG. 5B is a block diagram illustrating another exemplary sequence of messaging between the source device 560 and destination device 520 as part of the coordination session characteristics. The sequence of transmitting messages according to FIG. 5B is designed to provide a more detailed view of the transmission sequence described above for FIG. 5A. FIG. 5B, a message "1b. GET_PARAMETER RESPONSE" shows an example of a message that identifies the list of supported input categories (for example, universal and HIDC) and a plurality of lists of supported input types. Each of the categories supported by input from the list of supported categories of input has an associated list of supported types (for example, generic_cap_list and hidc_cap_list). FIG. 5B, a message "2a. SET_PARAMETER REQUEST" is an example of the second message, which identifies a second list of supported input categories (for example, universal and HIDC) and a plurality of second lists the supported types. Each of the categories supported by the input of a second list of supported categories of input has an associated second list of supported types (for example, generic_cap_list and hidc_cap_list). Post "1b. GET_PARAMETER RESPONSE" identifies the categories of input and input types supported by the device 560. The receiver-message "2a. SET_PARAMETER REQUEST" identifies the categories of input and input types supported by the source device 520, but it may not be a comprehensive list of all categories of input and input types supported by the source device 520. Instead, the message "2a. SET_PARAMETER REQUEST" can identify only the categories of input and input types identified in the message "1b. GET_PARAMETER RESPONSE" as supported by the device receiver 560. Thus , the categories of input and input types identified in the message "2a. SET_PARAMETER REQUEST", can be a subset of the categories of input and input types identified in the message "1b. GET_PARAMETER RESPONSE".

[0082] FIG. 6 is a conceptual diagram illustrating one example of a packet of data that can be generated by the device receiver, and transmitted to the source device. Aspects of the packet data 600 will be explained with reference to FIG. 1A, but for explanation technology may be applicable to additional types of systems source / receivers. Packet data 600 may include a data packet header 610, followed by the payload data 650. Operating data 650 may further include one or more headers of operating data (e.g., header 630 of operating data). Packet data 600, for example, may be transmitted from the device receiver 160 of FIG. 1A to source device 120 so that the user-receiver unit 160 may control the audio-video data transmitted by the source device 120. In this case, the job data 650 may include user input data received in the receiver device 160. Blue data 650, for example, can identify one or more user commands. The device receiver 160 can receive one or more user commands, and based on the received commands may generate a data packet header 610 and payload data 650. Based on the content header 610 of the data packet for the packet data 600, source device 120 may parse the payload data 650, data to identify a user input device adopted in the receiver 160. Based on user input data contained in the operation data 650, source device 120 may in some way modify the audio and video data transmitted from the source device 120 to destination device 160.

[0083] As used in this disclosure, the terms "parse" and "parses" generally means a process of analyzing a bitstream to extract data from the bitstream. After extracting the data may be processed, for example, by source device 120. Data extraction may for example include an identification of how information is formatted in the bit stream. As further described below, the data packet header 610 may define a standardized format, which is known as to the source device 120 and receiver device 160. However, the operational data 650 may be formatted one of many possible ways. By parsing the header 610 of the data packet source device 120 may determine how the payload data 650 is formatted, and thus, the source device 120 may parse the payload data 650 to retrieve the operational data 650 from one or more user input commands. This enables flexibility in any type of operational data that can be supported in communication source / receivers. As further described below, the job data 650 may also include one or more headers of operating data, for example, the header 630 of operational data. In such cases, the source device 120 can parse the header of the data packet 610 to determine the format for the header 630 operational data and then parse the header 630 operational data to determine the format for the remaining 650 operating data.

[0084] The circuit 620 is a conceptual illustration of how the header 610 may be formatted data packet. Numbers 0-15 in row 615 are intended to identify the locations of the bits in the data packet header 610, and are not intended to represent the actual information contained in the header 610 of the data packet. Data packet header 610 includes a version field 621, a flag 622 timestamp, a reserved field 623, a category field 624 input field 625 length field 626 and an optional time stamp.

[0085] In the example of FIG. 6, the version field 621 is a 3-bit field that may indicate the version of a particular communication protocol, implemented by a receiver device 160. The value of the version field 621 may notify the source device 120 how to parse the remainder of the data packet header 610, as well as how to parse the operational data 650. In the example of FIG. 6, field 621 version is a three-bit field, which should provide a unique identifier for eight different versions. In other examples, more or fewer bits can be allocated to the field 621 version.

[0086] In the example of FIG. 6, the flag 622 a timestamp (T) is a 1-bit field which indicates whether or not a timestamp field 626 in the header 610 of the data packet. Timestamp field 626 is 16-bit field containing a timestamp based on the multimedia data which are formed by the source device 120 and transmitted to the destination device 160. The timestamps may be for example a sequence of values ​​assigned to frames by the video source device 120 to a frame transmission device 160. The receiver 622 flag the timestamp, for example, may include a "1" to indicate that the time stamp field 626 is present and may include a "0" to indicate that the Timestamp field 626 is not present. When parsing the header 610 of the data packet and determining that the timestamp field 626 is present, the source device 120 may handle a time stamp field 626 is included in the timestamp. When parsing the header 610 of the data packet and determining that the timestamp field 626 is not present, the source device 120 may begin parsing the job data 650 after parsing the length field 625, as a timestamp is not present in the header 610 of the data packet.

[0087] If present, the timestamp field 626 may include a time stamp to identify a frame of video data that are displayed on the wireless device receiver 160, data obtained when the user input of operating data 650. The timestamp may, for example, is added to the frame by video source device 120 to the frame transmission source device 120 in the video receiver unit 160. Accordingly, the source device 120 can generate a frame of video and embedded into the video data frame as a metadata, for example, a timestamp. The source device 120 can transmit a video frame with a time stamp in the destination device 160 and destination device 160 may display a frame of video. Although the video frame is displayed by the device receiver 160, a device receiver 160 may receive a user command from the user. When the destination device 160 generates a data packet to transmit the user command to the source device 120, destination device 160 may include a timestamp field 626 the timestamp of the frame that is displayed by the device receiver 160 when receiving a user command.

[0088] When receiving the packet 600 data field 626 timestamp present title, the wireless source device 120 may identify the frame of the video displayed on the device receiver 160 when the data user input of operating data 650 received and process data the user input based on the content of the frame identified by the timestamp. For example, if the user input data is the touch command is applied to the touch screen, or by clicking the mouse, the source device 120 can determine the content of the frame to be displayed when the user uses the touch command to display or clicks. In some cases, the content of the frame may be required to properly process the operational data. For example, a user input based on user touch or mouse clicks may depend on what is shown on the screen while touch or click. Touching or clicking may correspond to, for example, the icon or menu item. In cases where the display content is changed, timestamp field 626 present in the timestamp, can be used by the source device 120 to associate the touch or click with the correct icon or menu item.

[0089] The source device 120 additionally or alternatively may compare the timestamp in the timestamp 626 with a time stamp applied to the current frame is prepared by means of rendering video. By comparing the time stamp field from time stamp 626 with current time stamp of the source device 120 can determine the time of transmission and acknowledgment. The transmission time and an acknowledgment, in general, corresponds to the amount of time that elapses from the time when a frame transmitted by the source device 120, until the user input is based on this frame is taken back to the source device 120 from the device receiver 160. The transmission time and an acknowledgment may provide the source device 120 indicator system delay time, and if the time of transmission and acknowledgment exceeds the threshold, then the source device 120 may ignore the data of the user input contained in the job data 650, in accordance with this assumption, that the input command is applied to the irrelevance of the frame to display. When the time of transmission and an acknowledgment is below the threshold, the source device 120 can process user input and adjust the audio video content transmitted in response to user input data. The threshold values ​​may be programmable, and various types of devices (or different combinations of source / receiver) may be configured to define different thresholds for the transmission time and an acknowledgment that are acceptable.

[0090] In the example of FIG. 6, a reserved field 623 is an 8-bit field which does not include the information used by the source 120 while parsing the header of the data packet 610 and job data 650. However, future versions of a particular protocol (identified in field 621 version) can use the reserved field 623, and in this case, source device 120 may use the information in a reserved field 623 for parsing the header 610 of the data packet and / or parsing job data 650. The reserved field 623 in combination with the version field 621 provides options for potentially expanding and adding features to the format of data packets without a fundamental change has already used the format and features.

[0091] In the example of FIG. 6, the input category field 624 is a 4-bit field to identify a category for entering user input data contained in the operation data receiver 650. The apparatus 160 may classify the user input data to determine the type of input. Categorizing the user input data, for example, may be based on the device from which the command is received, either on the basis of the properties of the pitch. Field value input category 624, possibly in combination with other information, the header 610 of the data packet for identifying the source device 120 is formatted as the payload data 650. Based on this format, the source device 120 may parse the payload data 650 to determine the user input that is received into a destination device 160.

[0092] Since the input category 624, in the example of FIG. 6 is 4 bits, sixteen different categories of input may be identified. One such category is the input format may be a universal input to indicate that the data for user input of operating data 650 formatted using universal information elements specified in the protocol executed by both the source device 120 and destination device 160. The format of a universal input as described in greater detail below, may use the universal information elements that enable the user of the device receiver 160 to interact with the source device 120 in the application layer.

[0093] Another such category can be input format command devices with HMI (HIDC), to indicate that the data user input operational data 650 is formatted based on the type of input device that is used to accept input. Examples of the types of devices include a keyboard, mouse, touch input device, a joystick, a camera, a capture device gestures (for example, the input device on the basis of the camera) and a remote control. Other types of categories I, which can be identified in the field 624 input category include format redirected input to indicate that the user data in the operation data 650 did not originate in a particular device receiver 160 or the operating system format, and the format voice command to indicate that the payload data 650 includes the voice instruction.

[0094] The field 625 may comprise a length of 16-bit field to indicate the length of data packet 600. Length, for example, can be specified in units of 8 bits. Since data packet 600 is parsed by a source device 120 words by 16 bits, the data packet 600 can be padded to an integer in 16 bits. Based on the length field contained in the 625 length, source device 120 may identify the end of the job data 650 (i.e., the end of data packet 600) and the beginning of a new, next packet data.

[0095] Various sizes of fields provided in the example of FIG. 6, are merely intended to illustrate, and it is intended that the fields may be implemented using different numbers of bits other than that shown in FIG. 6. Further, it is also assumed that the data packet header 610 may include not all fields are explained above, or may use additional fields not EXPLANATIONS above. In fact, the techniques of this disclosure may be flexible in terms of the actual format used for different data fields of packets.

[0096] After parsing the header of the data packet 610 to determine the format working data 650, the source device 120 can parse the operational data 650 to determine the user input command contained in the operational data 650 650 Operating data can be own header working data (the header 630 of operating data) indicating operating data content 650. Thus, source device 120 may parse a header 630 on the basis of operational data parsing the header of the data packet 610, and then parse the remaining payload data 650 on the basis of parsing the data header 630 workers.

[0097] If, for example, the field 624 input category data packet header 610 indicates that there is a universal input of operating data 650, the operating data 650 may be formatted as a universal input. Thus, source device 120 may parse the payload data 650 according to the format of the universal input. As part of a universal format input operating data 650 may include a sequence of one or more input events, each input event is an input event own header. Table 1 below identifies the fields which may be included in the header of the input.

Table 1PoleRazmer (octet) ZnachenieIdentifikator universal IE1Sm. 2Dlina2Dlina table the following fields in oktetahOpisaniePeremennyyPodrobnosti user input. See. Table

[0098] The identifier field (ID) of the universal input events (IE) identifying event identification data of the universal input to identify the type of input. IE Identifier field of universal, for example, may be one octet in length and may include the identification data selected from the following table 2. If, analogously to this example, the universal IE identifier field is 8 bits, 256 different types of inputs (identified 0- 255) may be identifiable, although not all necessarily required 256 identifiers associated type of input. Some of 256 may be reserved for future use in future versions of any protocol that is implemented by the device receiver 160 and source device 120. In Table 2, for example, 9-255 universal identifiers have associated IE input types, but they may be administered input types in the future.

[0099] The length field in the header identifies the length of an input event description field, whereas the description field includes information elements that describe a user input. Formatting description field may depend on the type of the input field identified by an identifier universal IE. Thus, source device 120 may parse the content description field based on the input type identified in the identifier field of universal IE. On the basis of the length of the header field input events, the source device 120 can determine the end of the event put into operational data 650 and the beginning of a new input event. As explained in more detail below, one user command can be described in the job data 650 as one or more input events.

[00100] Table 2 provides an example of input types, each of which has a corresponding universal identifier IE, which can be used to identify the type of input.

Table 2Identifikator universal IETip vvoda0Nazhatie the left mouse button / touch and uderzhivanie1Otpuskanie the left mouse button / touch and otpuskanie2Peremeschenie mouse / touch with peremescheniem3Nazhatie klavishi4Otpuskanie klavishi5Izmenenie masshtaba6Vertikalnaya prokrutka7Gorizontalnaya prokrutka8Povorot9-255Zarezervirovano

[00101] Description fields associated with each type of input may have a different format. Fields description of the event clicking the left mouse button / touch and retention event you release the left mouse button / touch and release, and events moving the mouse / touch with the movement, for example, may include information elements identified in Table 3 below, although other sizes may also be used in other examples.

Table 3PoleRazmer (octet) PrimechaniyaChislo pointers (N) 1The pointer events multitouch. When equal to 1, it indicates a traditional touch event vvoda.Dlya i = 1: N {ukazatelya1Identifikatsionny ID number of the pointer. The value is in [0, 1, ...] for the X-coordinate X2Koordinata events normalized regarding specified resolution video stream between the device and the receiver device istochnikom.Koordinata 2Koordinata Y Y} for events normalized regarding specified resolution video stream between the device and the receiver source device.

[00102] The number of indexes can identify the number of taps or mouse clicks associated with event input. Each index can have a unique identifier pointer. If, for example, multi-touch event includes a three-finger touch, the input event may have three pointers, each of which has a unique identifier pointer. Each pointer (ie, every touch of the fingers) can be appropriately coordinate X and coordinate Y, corresponding to the location where the touch occurred.

[00103] A user command can be described as a sequence of input events. For example, if holding three fingers is a command to close the application, holding three fingers may be described in the operational data 650 as a touch event and holding a three-pointer, touch event to the movement of three-pointers and touch events and release three pointers. Three pointer touching and holding events can have identical identifiers as pointers in three-pointers touch event to the movement and touch events and release. The source device 120 can interpret the combination of these three events as the input of the three fingers.

[00104] The fields describing key events or key release events, for example, may include information elements identified in the following Table 4.

Table 4PoleRazmer (octet) PrimechaniyaZarezervirovano1ZarezervirovanoKod 1 key (ASCII) 2Kod first key press events and key release. Basic / extended ASCII-code uses a more junior one byte. The older one byte is reserved for future ASCII-compatible code klavishi.Kod key 2 (ASCII) for the second 2Kod key press events and key release. Basic / extended ASCII-code uses a more junior one byte. The older one byte is reserved for future ASCII-compatible key code.

[00105] The description field events zoom, for example, may include information elements identified in the following Table 5.

Table 5PoleRazmer (octet) PrimechaniyaX2Etalonnaya X coordinate operations for zoom, normalized with respect to an agreed resolution of the video stream between the device and the receiver device istochnikom.Y2Etalonnaya Y coordinate operations for zoom, normalized with respect to an agreed resolution of the video stream between the device and the receiver device istochnikom.Tselaya of the rate change masshtaba1Chast unsigned integer factor of the rate change masshtabaDrobnaya masshtaba1Drobnaya change of the zoom factor

[00106] The description field events horizontal scrolling or vertical scrolling events, for example, may include information elements identified in the following Table 6.

Table 6PoleRazmer (octet) PrimechaniyaVelichina prokrutki2Chislo pixels to scroll, normalized to an agreed resolution of the video stream between the device receiver and source device. Negative numbers may indicate scrolling right, and a positive number can indicate scroll left

[00107] The above examples show some exemplary methods that can be formatted operational data for the category of the universal input. If the field is 624 category I data packet header 610 points to another category I, for example, redirect the user input, then the operating data 650 may have a different input format. Using the redirected user input device receiver 160 may receive data from a third-party user input device and send the input data from the source device 120 without interpreting user input data. Thus, source device 120 may parse the payload data according to the format 650 forwards user input. For example, the header 630 of operating data to operating data 650 may include a field to identify the device side, from which the user input. Field, for example, may include an address for an Internet Protocol (IP) external device, MAC-address of a domain name or some other identifier such. The source device 120 can parse the rest of the operating data based on the identifier of the device.

[00108] The device receiver 160 can negotiate with third-party device characteristics through a series of messages. The device receiver 160 may then transmit the unique identifier of the device into the source device 120 as part of establishing a communication session with the source device 120 as part of the matching characteristics. Alternatively, the device receiver 160 can transmit information describing the device side, a source device 120, and based on the information the source device 120 can determine a unique identifier for the device. Information describing the device side, for example, may include information to identify the device side and / or information to identify the characteristics of the device. Regardless of whether a unique identifier by the source device 120 or by the device receiver 160, when the destination device 160 transmits the data packets with the user input received from a third-party device, destination device 160 may include, for example, the unique identifier in the data packet title job data, so that the source device 120 can identify the source of the user input.

[00109] When the input category field 624 of the data packet header 610 indicates another input another category, for example, a voice command, the job data 650 may have yet another input format. For voice instruction operation information 650 may include the encoded audio. Codec for encoding and decoding audio of the voice instruction may be specified by the source device 120 and destination device 160 via the message sequence. To transmit the voice instruction timestamp field 626 may include the value of the sampling time speech. In this case, the flag 622 can be given a time stamp so that it indicates that the time stamp is present, but instead of a timestamp, as described above, the timestamp field 626 may include the value of the sampling time for the speech encoded audio data 650 workers.

[00110] In some examples, the voice command can be transmitted as a universal command, as described above, and in this case, the field 624 input category may be defined such that it identifies the format of universal commands, and one of the reserved identifiers universal IE may be assigned speech commands. If the voice command is transmitted as a universal command, the sampling rate of speech may be present in field 626 the timestamp of the data packet header 610 or may be present in the working data 650.

[00111] For the captured data, speech data of voice commands may be encapsulated in several ways. For example, data of voice commands may be encapsulated using RTP, which allows to provide the type of operation data to identify a codec and a timestamp, wherein the timestamp is used to identify the sampling frequency. RTP-data can be encapsulated using the universal format for user input as described above, with or without an optional timestamp. The device receiver 160 can transmit data to the universal input data are transferred speech commands, in the source device 120 using the TPC / IP.

[00112] As explained above, when the coordinates are included as part of the packet data, for example, packet 600 of data in the job data 650, e.g., coordinates can correspond to the coordinates scaled based on an agreed-resolution coordinates of the display window, normalized coordinates or coordinates associated with the display of the receiver. In some cases, additional information may be included in the data packet, or transmitted separately for use by the source device to normalize coordinates received in the data packet.

[00113] Regardless of the input to a particular category of data packet header data packet may be an application layer packet header and packet data can be transmitted over TCP / IP. TCP / IP can enable the receiver device 160 and source device 120 to implement the technology in the case of the retransmission packet loss. The data packet may be sent from the receiver device 160 to source device 120 to manage audio data or video data of the source device 120, or for other purposes, for example, in order to control an application running on the source device 120.

[00114] FIG. 7A is a block diagram of an exemplary method of matching characteristics between the receiver device and the source device. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in one or more flowcharts a method described herein.

[00115] The method of FIG. 7A includes receiving, by the device receiver 160 of the source device 120 of the first message (701). Message, for example, may contain a request for parameters. In response to the first message receiver device 160 may send a second message to the source device 120 (703). The second message, for example, may contain the answer to obtain parameters that identifies the categories of the first list of supported input and a plurality of first list of supported types, with each of the supported input categories from the first list of supported categories of input has an associated first list of supported types. Supported input category, for example, can correspond to identical category used to input the category field 624 of FIG. 6. The above table 2 is one example of the supported types of category-specific input (universal inputs in this example). The device receiver 160 may receive from the source device 120 a third message (705). The third message, for example, may contain a request to configure, with a request to configure identifies the communication port, a second list of supported categories of input and a plurality of second lists the supported types, each of the supported categories of input from the second list of supported categories of input has an associated second a list of supported types, and each of the supported types of the second list includes a subset of the first type of lists. Receiver device 160 may transmit to the source device 120 fourth message (707). The fourth message, for example, may contain the answer on the instructions of the parameters to confirm what types of second-lists activated. The device receiver 160 may receive from the source device 120 fifth message (709). The fifth message, for example, may comprise a second request for setting parameters, which indicates that the communication channel between the source device 120 and destination device 160 is activated. The communication link may for example comprise a user input return channel (UIBC). Receiver device 160 may transmit to the source device 120 sixth message (711). The sixth message, for example, may include a second response on the instructions of the parameters, which confirms receiving a second request to configure the device by the receiver 160.

[00116] FIG. 7B is a flowchart of an exemplary method of matching characteristics between the receiver device and the source device. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00117] The method of FIG. 7B includes transmission by the source device 120 to destination device 160 of the first message (702). The first message may for example comprise a request for obtaining parameters. The source device 120 may receive the second message from the receiver unit 160 (704). The second message, for example, may contain the answer to obtain parameters that identifies the categories of the first list of supported input and a plurality of first list of supported types, with each of the supported input categories from the first list of supported categories of input has an associated first list of supported types. The source device 120 may transmit to the destination device 160 third message (706). The third message, for example, may contain a request for assignment of parameters that identifies the port for communication, and the second list of supported categories of input and a plurality of second lists the supported types, each of Supported second input category from a list of supported input category has an associated second list of supported types, and each of the supported types of the second list includes a subset of the first type of lists. The source device 120 may receive from the device receiver 160 fourth message (708). The fourth message, for example, may contain the answer on the instructions of the parameters to confirm what types of second-lists activated. The source device 120 can transmit to the device receiver 160 fifth message (710). The fifth message, for example, may comprise a second request for setting parameters, which indicates that the communication channel between the source device 120 and destination device 160 is activated. The communication link may for example comprise a user input return channel (UIBC). The source device 120 may receive from the device receiver 160 sixth message (712). The sixth message, for example, may include a second response on the instructions of the parameters, which confirms receiving a second request to configure the device by the receiver 160.

[00118] FIG. 8A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00119] The method of FIG. 8A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (801). These user input may be received via a user input component of the wireless device receiver 160, such as, for example, the user input interface 376, shown with respect to a wireless device receiver 360. Additionally, receiver device 160 may classify the user input data, such as universal , redirect, or specific to the operating system. The device receiver 160 may then generate a data packet header based on the user input (803). The packet header data can be application-layer packet header. Data packet header may include, among other fields, a field to identify the category of the input data corresponding to the user input. Category I may comprise, for example, a format command or a universal input device HMI. The device receiver 160 can generate a further data packet (805), wherein the data packet contains the generated data packet header and payload data. In one example, operating data may include data received user input may identify one or more user commands. The device receiver 160 may then transmit the generated packet data (807) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown in FIG. 3. The device receiver 160 can transmit the data packet to TCP / IP.

[00120] FIG. 8B is a flowchart of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00121] The method of FIG. 8B includes receiving a data packet (802), the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in relation to FIG. 2. The source device 120 may then parse the data packet header (804) included in the data packet to determine the type of input associated with the user input data contained in the job data. The source device 120 can process the operating data on the basis of a specific category of input (806). The data packets are described with reference to FIG. 8A and 8B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data and applications in a source device.

[00122] FIG. 9A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00123] The method of FIG. 9A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (901). These user input may be received via a user input component of the wireless device receiver 160, such as, for example, the user input interface 376, shown with reference to FIG. 3. The receiver device 160 then may generate operational data (903), the payload data may describe the user input data. In one example, operating data may include data received user input may identify one or more user commands. The device receiver 160 can generate a further data packet (905), wherein the data packet comprises a data packet header and payload data generated. The device receiver 160 may then transmit the generated packet data (907) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise, for example, components that enable the transmission of data packets, such as the transport unit 333 and the wireless modem 334. The data packet may be transmitted to the wireless source device on TCP / IP.

[00124] FIG. 9B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00125] The method of FIG. 9B includes receiving a data packet from the receiver unit 360 (902), the data packet may comprise, inter alia, the data packet header and payload data. In one example, operating data may contain, for example, data describing the details of the user input, for example, the value of the input type. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown with reference to FIG. 2. The source device 120 may then parse the data packet (904) to determine the type of input value in the type field in the input job data. The source device 120 can process data describing details of a user input based on a particular type of input values ​​(906). The data packets are described with reference to FIG. 9A and 9B, in general, can take the form of data packets, as described with reference to FIG. 6.

[00126] FIG. 10A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00127] The method of FIG. 10A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (1001). These user input may be received via a user input component of the wireless device receiver 160, such as, for example, the user input interface 376, as shown with reference to FIG. 3. The receiver device 160 then may generate a packet header of data based on user input (1003). Data packet header may include, among other fields, the timestamp flag (for example, 1-bit field) to indicate the presence or not the time stamp field in the packet header data. Flag timestamp, for example, may include a "1" to indicate that the timestamp field is present, and may include a "0" to indicate that the timestamp field is not present. Timestamp field may be, for example, a 16-bit field containing a timestamp generated by the source device 120 and added to the video data prior to transmission. The device receiver 160 can generate a further data packet (1005), wherein the data packet contains the generated data packet header and payload data. In one example, operating data may include data received user input may identify one or more user commands. The device receiver 160 may then transmit the generated packet data (1007) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown in relation to FIG. 3. The data packet can be transmitted to the wireless source device for TCP / IP.

[00128] FIG. 10B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00129] The method of FIG. 10B includes receiving a data packet from a wireless device receiver 160 (1002), the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in relation to FIG. 2. The source device 120 may then parse the header of the data packet (1004), included in the data packet. The source device 120 can determine whether or not a timestamp field in the header of the data packet (1006). In one example, source device 120 may perform the determination based on a flag value of the timestamp included in the header of the data packet. If a data packet header includes a timestamp field, source device 120 may process the payload data based on the time stamp, which is in the timestamp field (1008). The data packets are described with reference to FIG. 10A and 10B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00130] FIG. 11A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00131] The method of FIG. 11A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (1101). These user input may be received via a user input component of the wireless device receiver 160, such as, for example, the user input interface 376, shown in relation to FIG. 3. The receiver device 160 then may generate a packet header of data based on user input (1103). Data packet header may include, among other fields, the timestamp field. Timestamp field may contain, for example, a 16-bit field containing a timestamp based on the multimedia data which are formed by a wireless source device 120 and transmitted to the wireless device receiver 160. The time stamp may be added to frame image data via wireless devices -source 120 to transmit to the wireless device receiver. Timestamp field, for example, may identify the timestamp associated with the frame image data displayed in the wireless device receiver 160 when the data captured by the user input. The device receiver 160 can generate a further data packet (1105), wherein the data packet contains the generated data packet header and payload data. In one example, operating data may include data received user input may identify one or more user commands. The device receiver 160 may then transmit the generated packet data (1107) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown in relation to FIG. 3. The data packet can be transmitted to the wireless source device for TCP / IP.

[00132] FIG. 11B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00133] The method of FIG. 11B includes receiving a data packet from a wireless device receiver, such as wireless device receiver 160 (1102), the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in relation to FIG. 2. The source device 120 may then identify the timestamp field in the header of the data packet (1104). The source device 120 may process the payload data based on the time stamp, which is in the timestamp field (1106). As part of processing operational data on the basis of a timestamp, a source device 120 may identify the frame image data displayed in the wireless device receiver when the received user input data, and to interpret the operational data based on the content of the frame. As part of processing operational data on the basis of a timestamp, a source device 120 may compare the timestamp with the current time stamp for the current video frame transmitted by the source device 120, and can execute the user input, as described in the working data in response to a time difference between the timestamp and current timestamp smaller than the threshold or not to execute the command the user input, as described in the working data in response to a time difference between the timestamp and current timestamp exceeds the threshold. The data packets are described with reference to FIG. 11A and 11B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00134] FIG. 12A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00135] The method of FIG. 12A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (1201). In one example, user input data may be data of voice commands that can be received via a user input component of the wireless device receiver 160, such as, for example, voice recognition module, included in a user input interface 376 of FIG. 3. The device receiver 160 can generate a packet header of data based on user input (1203). Receiver device 160 may also generate operating data (1205), the operational data may include data of voice commands. In one example, the operating data may also comprise received data, and user input may identify one or more user commands. The device receiver 160 can generate a further data packet (1207), wherein the data packet contains the generated data packet header and payload data. The device receiver 160 may then transmit the generated packet data (1209) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown in relation to FIG. 3. The data packet can be transmitted to the wireless source device for TCP / IP.

[00136] FIG. 12B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00137] The method of FIG. 12B includes receiving a data packet (1202), wherein the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data, such as voice data commands. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in relation to FIG. 2. The source device 120 may then parse the payload data (1204), included in the data packet to determine whether the operating data contain no data or voice commands. The data packets are described with reference to FIG. 12A and 12B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00138] FIG. 13A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00139] The method of FIG. 13A includes obtaining user input data at the wireless device, a receiver such as wireless device receiver 160 (1301). In one example, user input data may be a multi-touch gesture that can be accessed via a user input component of the wireless device receiver 160, such as, for example, UI 167 or the user input interface 376 of FIG. 3. In one example, multi-touch gesture may include a first touch input and second touch input. The device receiver 160 can generate a packet header of data based on user input (1303). Receiver device 160 may also generate operating data (1305), the operational data may associate a user input data for the first touch input event with the identifier of the first pointer and the user input data for the second event the touch input with the identifier of the second pointer. The device receiver 160 can generate a further data packet (1307), wherein the data packet contains the generated data packet header and payload data. The device receiver 160 may then transmit the generated packet data (1309) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown in relation to FIG. 3. The data packet can be transmitted to the wireless source device for TCP / IP.

[00140] FIG. 13B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00141] The method of FIG. 13B includes receiving a data packet (1302), the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data, such as multi-touch gesture. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in FIG. 2. The source device 120 may then parse the payload data (1304), included in the data packet, to identify a user input information included in the job data. In one example, the identified data may include user input data for the first touch input event with the identifier of the first pointer and the user input data for the second event the touch input with the identifier of the second pointer. The source device 120 can then interpret the data user input events for the first touch input data and user input events for the second touch input as multi-touch gestures (1306). The data packets are described with reference to FIG. 13A and 13B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00142] FIG. 14A is a block diagram of an exemplary method for transmitting user input data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00143] The method of FIG. 14A includes obtaining user input data at the wireless device receiver 360 from an external device (1401). In one example, an external device can be a third-party device connected to the destination device. The device receiver 160 can generate a packet header of data based on user input (1403). In one example, the header of a data packet may identify the user input data as the data of the redirected user input. Receiver device 160 may also generate operating data (1405), the operational data may comprise a user input data. The device receiver 160 can generate a further data packet (1407), the data packet may contain data generated by a packet header and payload data. The device receiver 160 may then transmit the generated packet data (1409) to the wireless source device (such as source device 120 of FIG. 1A or 220 of FIG. 2). The device receiver 160 can comprise the components that enable the transmission of data packets, including, for example, the transport unit 333 and the wireless modem 334 as shown with reference to FIG. 3. The data packet can be transmitted to the wireless source device for TCP / IP.

[00144] FIG. 14B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00145] The method of FIG. 14B includes receiving a data packet (1402), the data packet may comprise, inter alia, the data packet header and payload data. Operational data may include, for example, user input data, such as command directs a user input indicating that the user data is forwarded from the input of the device. The source device 120 may include communication components that enable the transmission of data packets, including, for example, the transport unit 233 and the wireless modem 234 as shown in relation to FIG. 2. The source device 120 may then parse the data packet header and may determine that the operating data contain redirects the user input (1404). Source device 120 may then parse the payload data (1406), included in the data packet to identify the identification data associated with the sides of the device corresponding to a user input command is redirected. The source device 120 may then process the operating data on the basis of the identity of the identified third-party device (1408). The data packets are described with reference to FIG. 14A and 14B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00146] FIG. 15A is a block diagram of an exemplary method for transmitting user data from the wireless device to the wireless receiver source device in accordance with this disclosure. The illustrated exemplary method may be performed by a receiver unit 160 (FIG. 1A) or 360 (FIG. 3). In some instances, the computer readable storage medium (e.g., memory 332) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 331) to perform one or more of the illustrated steps in a flow chart of a method .

[00147] The method of FIG. 15A includes obtaining user input data at the wireless device receiver (1501). These user input may be associated coordinate data. Associated coordinate data, for example, may correspond to the location of the mouse click event or event location touch input. The device receiver 160 may then normalize associates coordinate data to generate normalized coordinate data (1503). The device receiver 160 may then generate a data packet, which includes the normalized coordinate data (1505). Normalization coordinate data may include associated scaling the coordinate data based on the resolution ratio of the display window and the display resolution source, for example, the display 22 of the source device 120. The resolution of the display window may be determined by the device receiver 160, and the display resolution of the source device It can be taken from the source device 120. The destination device 160 may then transmit the data packet with normalized coordinates to the wireless source device 120 (1507). As part of the method of FIG. 15A, the device receiver 160 can also determine whether or not the associated coordinate data within a window display for the content received from a wireless source device, and, for example, to handle user input locally if associated coordinate data are outside of the display window, or otherwise normalized coordinates as described, if the input is within the display window.

[00148] FIG. 15B is a block diagram of an exemplary method for receiving user input data from the wireless device in the wireless receiver source unit in accordance with this disclosure. The illustrated exemplary method may be performed by the source device 120 (FIG. 1A) or 220 (FIG. 2). In some instances, the computer readable storage medium (e.g., memory 232) may store instructions, the modules or algorithms when executed instruct one or more processors (e.g., processor 231) to perform one or more of the illustrated steps in a flow chart of a method .

[00149] The method of FIG. 15B includes receiving a data packet at the wireless source device, wherein the data packet contains user input data associated with the coordinate data (1502). Associated coordinate data, for example, may correspond to the location of the mouse click event or event location touch input device receiver. The source device 120 can then normalize the associated coordinate data to generate normalized coordinate data (1504). The source device 120 can normalize the coordinate data by scaling the associated coordinate data based on the relative resolution of the display window and the display resolution of the source. The source device 120 may determine the display resolution of the source device and can accept the resolution of the display window of the receiver of the wireless device. The source device may then process the data packet based on the normalized coordinate data (1506). The data packets are described with reference to FIG. 15A and 15B, in general, can take the form of data packets, as described with reference to FIG. 6, and can be used to control audio and video data in the source device.

[00150] For ease of explanation, aspects of the disclosure are described separately with reference to FIG. 7-15. However, it is contemplated that these different aspects may be combined and used in conjunction with each other, not just separately. In general, the functionality and / or modules described herein may be implemented in one or both of the wireless source device and a wireless device receiver. Thus, the characteristics of the user interface described in this example can be used interchangeably between a wireless source device and the wireless device receiver.

[00151] The technology of this disclosure may be implemented in a wide range of devices or apparatus, including a wireless handset, and integrated circuit (IC) or a set of IC (i.e. chipset). All of the components, modules or units are provided to emphasize functional aspects and does not necessarily require realization by different hardware units.

[00152] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, any features described as modules, units or components may be implemented together in an integrated logic device or separately as discrete but interacting logic devices. When implemented in software, the techniques may be realized at least in part by a computer-readable medium containing instructions that when executed by a processor carry out one or more of the methods described above, the computer readable medium may comprise a material and a non-volatile computer-readable storage medium and may be part of a computer program product which may include packaging materials. Computer readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), nonvolatile random access memory (NVRAM), electrically erasable programmable read only memory (EEPROM), flash -Memory, magnetic or optical data storage media, etc. Additionally or alternatively, the techniques may be realized at least in part by a computer readable communication medium that carries or communicates code in the form of instructions or data structures and which can be accessed, read, or execute by a computer.

[00153] The code may be executed by one or more processors, such as one or more digital signal processors (DSP), general purpose microprocessors, application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other equivalent integrated or discrete logic circuitry . Accordingly, the term "processor" as used herein can mean any of the above structure or other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated in a combined video codec. Furthermore, the technology can be fully implemented in one or more circuits or logic elements.

[00154] Various aspects of the disclosure. These and other aspects are within the scope of the appended claims.