User input back channel for wireless displays

User input return channel for wireless displays

[0001] This application requires the rights of the following U.S. provisional applications:

[0002] U.S. Provisional Application No. 61/435, 194 filed on January 21, 2011; [0003] U.S. Provisional Application No. 61/447, 592 filed on February 28, 2011; [0004] In U.S. Provisional Application No. 61/44 & 312 filed on March 2, 2011; [0005] U.S. Provisional Application No. 61/450, 101 filed on March 7, 2011; [0006] On March 25, 2011 U.S. Provisional Application No. 61/467, 535 filed on March 25, 2011; [0007] U.S. Provisional Application No. 61/467, 543 filed on March 25, 2011; [0008] The United States filed on August 3, 2011 Provisional Application No. 61/514, 863; [0009] U.S. Provisional Application No. 61/544, 434 filed on October 7, 2011; [0010] The entire content of the above provisional application is incorporated herein by reference.

Technical field

[0011] The present disclosure relates to techniques for transmitting data between a wireless source device and a wireless sink device.

Background technique

[0012] A wireless display (WD) or Wi-Fi display (WFD) system includes a wireless source device and one or more wireless sink devices. The source device and each sink device can be a mobile device or a wired device with wireless communication capabilities. For example, one or more of the source device and the sink device may include a mobile phone, a portable computer with a wireless communication card, a personal digital assistant (PDA), a portable media player, or other such devices with wireless communication capabilities, including So-called "smart" phones and "smart" tablets or tablets, e-readers, or any type of wireless display, video game device, or other type of wireless communication device. One or more of the source device and the sink device may also include wired devices with communication capabilities, such as televisions, desktop computers, monitors, projectors, and the like.

[0013] The source device sends media data (such as audio video (AV) data) to one or more of the sink devices participating in a specific media sharing session. The media data can be played back on each of the local display of the source device and the display of the sink device. More precisely, each of the participating sink devices presents the received media data on its screen and audio device.

Summary of the invention

[0014] The present disclosure generally describes a system in which a wireless sink device can communicate with a wireless sink device. As part of the communication session, the wireless source device may send audio and video data to the wireless sink device, and the wireless sink device may send user input received at the wireless sink device back to the wireless source device. In this way, the user of the wireless sink device can control the wireless source device and can control the content transmitted from the wireless source device to the wireless sink device.

[0015] In an example, a method for transmitting user input data from a wireless sink device to a wireless source device includes: receiving a data packet including a packet header and payload data; and parsing the payload data to determine Whether the payload data includes voice command data.

[0016] In another example, a wireless sink device configured to send user input data to a wireless source device. So

The wireless sink device includes: a transmission unit, which is used to receive a data packet including a packet header and payload data; a memory, which stores instructions; one or more processors, which are configured to execute the instructions. After the instructions, the one or more processors cause the payload data to be parsed to determine whether the payload data includes voice command data.

[0017] In another example, a computer-readable storage medium storing instructions that, after the instructions are executed by one or more processors, causes the one or more processors to execute from a wireless sink device to a wireless The method by which the source device sends user input data. The method includes: receiving a data packet including a packet header and payload data; and parsing the payload data to determine whether the payload data includes voice command data.

[0018] In another example, a wireless sink device configured to send user input to a wireless source device. The wireless sink device includes: a module for receiving a data packet including a packet header and payload data; and a module for analyzing the payload data to determine whether the payload data includes voice command data.

[0019] In another example, a method for sending user input data from a wireless sink device to a wireless source device includes: obtaining voice command data at the wireless sink device; generating a data packet header; generating the voice command Data payload data; generating a data packet including the data packet header and the payload data; and sending the data packet to the wireless source device.

[0020] In another example, a wireless sink device is configured to send user input data to a wireless source device. The wireless sink device includes: a memory that stores instructions; one or more processors configured to execute the instructions, wherein, after executing the instructions, the one or more processors enable: The wireless sink device obtains voice command data; generates a data packet header; generates payload data including the voice command data; and generates a data packet including the data packet header and the payload data. The wireless sink device further includes: a transmission unit configured to send the data packet to the wireless source device.

[0021] In another example, a computer-readable storage medium that stores instructions, after the instructions are executed by one or more processors, causes the one or more processors to execute from a wireless sink device to a wireless The method by which the source device sends user input data. The method includes: obtaining voice command data at the wireless sink device; generating a data packet header; generating payload data including the voice command data; generating a data packet including the data packet header and the payload data ; Send the data packet to the wireless source device.

[0022] In another example, a wireless sink device is configured to send user input data to a wireless source device. The wireless sink device includes: a module for obtaining voice command data at the wireless sink device; a module for generating a data packet header; a module for generating payload data including the voice command data; and A module for generating a data packet including the data packet header and the payload data; and a module for sending the data packet to the wireless source device.

Description of the drawings

[0023] FIG. 1A is a block diagram showing an example of a source/sink system that can implement the technology of the present disclosure.

[0024] FIG. 1B is a block diagram showing an example of a source/sink system with two sink devices.

[0025] FIG. 2 shows an example of a source device that can implement the technology of the present disclosure.

[0026] FIG. 3 shows an example of a sink device that can implement the technology of the present disclosure.

[0027] FIG. 4 shows a block diagram of a transmitter system and a receiver system that can implement the technology of the present disclosure.

[0028] FIGS. 5A and 5B illustrate an example message transmission sequence for performing capability negotiation according to the technology of the present disclosure.

sequence.

[0029] FIG. 6 shows an example data packet that can be used to deliver user input data obtained at a sink device to a source device.

[0030] FIGS. 7A and 7B are flowcharts illustrating techniques of the present disclosure that can be used for capability negotiation between a source device and a sink device.

[0031] FIGS. 8A and 8B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets with user input data.

[0032] FIGS. 9A and 9B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets with user input data.

[0033] FIGS. 10A and 10B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets with time stamp information and user input data.

[0034] FIGS. 11A and 11B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets with time stamp information and user input data.

[0035] FIGS. 12A and 12B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets including voice commands.

[0036] FIGS. 13A and 13B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets with multi-touch user input commands.

[0037] FIGS. 14A and 14B are flowcharts illustrating techniques of the present disclosure that can be used to send and receive data packets with user input data forwarded from a third-party device.

[0038] FIGS. 15A and 15B are flowcharts illustrating techniques of the present disclosure that can be used to transmit and receive data packets.

Detailed ways

[0039] The present disclosure generally describes a system in which a wireless sink device can communicate with a wireless sink device. As part of the communication session, the wireless source device may send audio and video data to the wireless sink device, and the wireless sink device may send user input received at the wireless sink device back to the wireless source device. In this way, the user of the wireless sink device can control the wireless source device, and can control the content sent from the wireless source device to the wireless sink device.

[0040] FIG. 1A is a block diagram illustrating an exemplary source/sink system 100 that can implement one or more of the techniques of the present disclosure. As shown in FIG. 1A, the system 100 includes a source device 120 that communicates with a sink device 160 via a communication channel 150. The source device 120 may include a memory storing audio/video (A/V) data 121, a display 122, a speaker 123, an audio/video encoder 124 (also referred to as an encoder 124), an audio/video control module 125, and a transmitter /Receiver (TX/RX) unit 126. The sink device 160 may include a display 162, a speaker 163, an audio/video decoder 164 (also referred to as a decoder 164), a transmitter/receiver unit 166, a user input (UI) device 167, and a user input processing module (UIPM) 168_OThe components shown constitute only one example configuration of the source/sink system 100. Other configurations may include fewer components than those shown or may include additional components than those shown.

[0041] In the example of FIG. 1A, the source device 120 may display the video portion of the audio/video data 121 on the display 122, and may output the audio portion of the audio/video data 121 on the speaker 123. The audio/video data 121 can be stored locally on the source device 120, from an external storage medium (such as a file server, a hard disk drive, an external storage medium, etc.).

Storage, Blu-ray Disc, DVD, or other physical storage media) can be accessed, or can be streamed to the source device 120 via a network connection such as the Internet. In some cases, the audio/video data 121 may be captured in real time via the camera and microphone of the source device 120. The audio/video data 121 may include multimedia content such as movies, TV shows, or music, but may also include real-time content generated by the source device 120. For example, such real-time content may be generated by an application running on the source device 120, or captured video data (for example, as part of a video telephony session). As will be described in more detail, in some cases, such real-time content may include video frames of user input options that can be selected by the user. In some cases, the audio/video data 121 may include a combined video frame of different types of content, such as a video frame of a movie or TV show with user input options overlaid on the video frame.

[0042] In addition to locally presenting the audio/video data 121 via the display 122 and the speaker 123, the audio/video encoder 124 of the source device 120 may encode the audio/video data 121, and the transmitter/receiver unit 126 may The encoded data is sent to the sink device 160 on the communication channel 150. The transmitter/receiver unit 166 of the sink device 160 receives the encoded data, and the audio/video decoder 164 decodes the encoded data, and outputs the decoded data via the display 162 and the speaker 163. In this way, the audio and video data presented by the display 122 and the speaker 123 can be presented by the display 162 and the speaker 163 at the same time. The audio data and video data can be arranged in a frame, and when presented, the audio frame can be time synchronized with the video frame.

[0043] The audio/video encoder 124 and the audio/video decoder 164 can implement any number of audio and video compression standards, such as the ITU-T H.264 standard (or MPEG-4, part 10), advanced video coding (AVC) or the emerging High Efficiency Video Coding (HEVC) standard (sometimes called the H. 265 standard). Many other types of proprietary or standardized compression techniques can also be used. In general, the audio/video encoder 164 is configured to perform the reciprocal encoding operation of the audio/video encoder 124. Although not shown in FIG. 1A, in some aspects, both the A/V encoder 124 and the A/V decoder 164 may be integrated with the audio encoder and decoder, and may include appropriate MUX-DEMUX (multiplexing- Demultiplexing) unit or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams.

[0044] As will be described in more detail below, in addition to implementing the video compression standard as described above, the A/V encoder 124 may also perform other encoding functions. For example, before transmitting the A/V data 121 to the sink device 160, the A/V encoder 124 may add various types of metadata to the A/V data 121. In some cases, the A/V data 121 may be stored in the source device 120 or received at the source device 120 in an encoded form, so that no further compression by the A/V encoder 124 is required.

[0045] Although FIG. 1A shows a communication channel 150 that separately carries audio payload data and video payload data, it should be understood that, in some cases, the video payload data and the audio payload data may be common data. Part of the stream. If applicable, the MUX-DEMUX unit can follow the ITU Η.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP). Both the audio/video encoder 124 and the audio/video decoder 164 can be implemented as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete Logic device, software, hardware, firmware or any combination thereof. Each of the audio/video encoder 124 and the audio/video decoder 164 can be included in one or more encoders or decoders, and any one of them can be integrated as a combined encoder/decoder (CODEC) Part. Therefore, each of the source device 120 and the sink device 160 may include a dedicated machine configured to perform one or more of the techniques of the present disclosure.

[0046] The display 122 and the display 162 may include any of various video output devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, an organic light emitting diode (OLED) ) Display, or other types of display equipment. In these or other examples, both displays 122 and 162 may be

Emissive display or transmissive display. The displays 122 and 162 may also be touch displays, so that they are both input devices and display devices at the same time. This type of touch screen can be capacitive, resistive, or other types of touch panels that allow users to provide user input to their respective devices.

[0047] The speaker 123 may include any of various audio output devices, such as a headset, a single speaker system, a multi-speaker system, or a surround sound system. In addition, although the display 122 and the speaker 123 are shown as a part of the source device 120 and the display 162 and the speaker 163 are shown as a part of the sink device 160, the source device 120 and the sink device 160 may actually be a device system. As an example, the display 162 may be a television, the speaker 163 may be a surround sound system, and the decoder 164 may be a part of an outer box connected to the display 162 and the speaker 163 by wire or wirelessly. In other cases, the sink device 160 may be a single device such as a tablet computer or smart phone. In other cases, the source device 120 and the sink device 160 are similar devices, for example, both are smart phones, tablet computers, and so on. In this case, one device can operate as a source, and another device can operate as a sink. In subsequent communication sessions, these roles can even be reversed. In other cases, the source device may include a mobile device such as a smart phone, laptop, or tablet, and the sink device may include a more static device (for example, with an AC power cord), in this case, The source device can transmit audio and video data via the sink device for presentation to a larger group of people.

[0048] Both the transmitter/receiver unit 126 and the transmitter/receiver unit 166 may include various mixers, filters, amplifiers, and other components designed for signal modulation, as well as one or more antennas and design Other components for sending and receiving data. The communication channel 150 generally represents any suitable communication medium or a collection of different communication media for transmitting video data from the source device 120 to the sink device 160. The communication channel 150 is usually a relatively short-distance communication channel, similar to Wi-Fi, Bluetooth, etc. However, the communication channel 150 is not necessarily limited to this aspect, but may include any wireless or wired communication medium (such as a radio frequency (RF) spectrum or one or more physical transmission lines) or any combination of wireless and wired media. In other examples, the communication channel 150 may even form part of a packet-based network, such as a wired or wireless local area network, a wide area network, or a global network such as the Internet. In addition, the communication channel 150 can be used by the source device 120 and the sink device 160 to create a peer-to-peer link. The source device 120 and the sink device 160 may use, for example, those from IEEE802. 11 The standard communication protocol of the standard family communicates on the communication channel 150. For example, the source device 120 and the sink device 160 may communicate according to the Wi-Fi direct standard, so that the source device 120 and the sink device 160 directly communicate with each other without using a medium such as a wireless access point or a so-called hotspot. The source device 120 and the sink device 160 may also establish tunnel direct link establishment (TLDS) to avoid or reduce network congestion. Sometimes the technologies of the present disclosure can be described with reference to Wi-Fi, but it should be envisaged that various aspects of these technologies can also be compatible with other communication protocols. By way of example and not limitation, the wireless communication between the source device 120 and the sink device may use Orthogonal Frequency Division Multiplexing (OFDM) technology. Various other wireless communication technologies can also be used, including but not limited to time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA) or OFDM, FDMA, TDMA and/or CDMA. random combination. WiFi Direct and TDLS are designed to establish a relatively short-distance communication session. In this context, a relatively short distance may refer to, for example, less than 70 meters, although in a noisy or obstructed environment, the distance between devices may even be shorter, such as less than 35 meters.

[0049] In addition to decoding and presenting data received from the source device 120, the sink device 160 may also receive user input from the user input device 167. For example, the user input device 167 may be a keyboard, a mouse, a trackball or trackpad, a touch screen, a voice command recognition module, or any other such user input device. The UIPM formats the user input command received by the user input device 167 into a data packet structure that the source device 120 can interpret. The transmitter/receiver 166 transmits these data packets to the source device 120 on the communication channel 150. The transmitter/receiver unit 126 receives the data

Data is grouped, and the A/V control module 125 parses the data group to interpret the user input command received by the user input device 167. Based on the command received in the data packet, the A/V control module 125 can change the content to be encoded and transmitted. In this way, the user of the sink device 160 can remotely control the audio payload data and video payload data sent by the source device 120 without directly interacting with the source device 120. Examples of the types of commands that the user of the sink device 160 can send to the source device 120 include commands for rewinding, fast forwarding, pausing, and playing audio and video data, and commands for zooming, rotating, scrolling, and the like. For example, the user can also make a selection from a menu of options, and send the selection back to the source device 120.

[0050] In addition, the user of the sink device 160 can start and control applications on the source device 120. For example, the user of the sink device 160 can launch a photo editing application stored on the source device 120, and use the application to edit photos stored locally on the source device 120. The sink device 160 may present to the user a user experience that looks and feels like editing a picture locally on the sink device 160, but actually edits the picture on the source device 120. Using such a configuration, device users can use the capabilities of one device for several devices. For example, the source device 120 may be a smart phone with a large amount of memory and high-end processing capabilities. The user of the source device 120 can use the smart phone in all settings and situations normally used by the smart phone. However, when watching a movie, the user may wish to watch the movie on a device with a larger display screen. In this case, the sink device 160 may be a tablet computer or even a larger display device or TV. When wanting to send or reply to an email, the user may want to use a device with a keyboard, in this case, the sink device 160 may be a laptop computer. In the above two cases, although the user is interacting with the sink device, most of the processing can still be performed by the source device 120 (a smart phone in this example). In that particular operation In the context of operation, since most operations are performed by the source device 120, if the sink device 160 is required to perform the processing being performed by the source device 120, the sink device 160 can be a lower cost with relatively few resources. device of. In some examples, both the source device and the sink device can accept user input (such as touch screen commands), and the technology of the present disclosure can facilitate two-way communication by negotiating and or identifying the capabilities of the device in any given session Interactive.

[0051] In some configurations, the A/V control module 125 may be an operating system process being executed by the operating system of the source device 125. However, in other configurations, the A/V control module 125 may be a software process of an application running on the source device 120. In this configuration, the user input command may be interpreted by a software process, so that the user of the sink device 160 directly interacts with the application running on the source device 120 instead of the operating system running on the source device 120. By directly interacting with the application instead of the operating system, the user of the sink device 160 can have access to a command library local to the operating system that is not the source device 120. In addition, directly interacting with applications can make it easier for commands to be sent and processed by devices running on different platforms.

[0052] The source device 120 may respond to user input applied at the wireless sink device 160. In this interactive application setting, the user input applied at the wireless sink device 160 can be sent back to the wireless display source on the communication channel 150. In one example, a reverse channel architecture (also referred to as a user interface return channel (UIBC)) may be implemented to enable the sink device 160 to send user input applied at the sink device 160 to the source device 120. The reverse channel architecture may include upper-layer messages for transmitting user input and lower-layer frames for negotiating user interface capabilities at the sink device 160 and the source device 120. The UIBC may be located at the Internet Protocol (IP) transport layer between the sink device 160 and the source device 120. In this way, UIBC can be on top of the transport layer in the Open Systems Interconnection (OSI) communication model. In one example, OSI communication includes seven layers (1-Physical, 2-Data Link, 3-Network, 4-Transport, 5-Session, 6-Presentation, and 7-Application). In this example, above the transport layer refers to layers 5, 6, and 7. In order to improve reliable transmission and the sequential delivery of data packets containing user input data, UIBC can be configured to be used in other packet-based communication protocols (such as transmission control protocol).

/Internet Protocol (TCP/IP) or User Datagram Protocol (UDP)) runs on top. UDP and TCP can operate in parallel in the OSI layer architecture. TCP/IP can enable the sink device 160 and the source device 120 to implement retransmission technology in the case of packet loss.

[0053] In some cases, there may be a mismatch between the user input interfaces located at the source device 120 and the sink device 160. In order to solve the potential problems caused by this mismatch and improve a good user experience in these situations, user input may be performed between the source device 120 and the sink device 160 before establishing a communication session or at various times throughout the communication session. Interface capability negotiation. As part of the negotiation process, the source device 120 and the sink device 160 can reach an agreement on the negotiated screen resolution. When the sink device 160 transmits the coordinate data associated with the user input, the sink device 160 may scale the coordinate data obtained from the display 162 to match the negotiated screen resolution. In one example, if the sink device 160 has a resolution of 1280x720 and the source device 120 has a resolution of 1600x900, the device may, for example, use 1280x720 as its negotiated resolution. Although the resolution of the source device 120 or some other resolution can also be used, the negotiated resolution can be selected based on the resolution of the sink device 160. In the example of using a 1280x720 sink device, the sink device 160 can scale the obtained X coordinate by a factor of 1600/1280 before sending the coordinates to the source device 120, and similarly, the sink device 160 can send the coordinates to the source device 120. 120 Before sending the coordinates, 900/720 scales the obtained y coordinate. In other configurations, the source device 120 may scale the obtained coordinates to the negotiated resolution. The scaling may be based on whether the sink device 160 uses a higher resolution display than the source device 120 to increase or decrease the coordinate range, or vice versa.

[0054] In addition, in some instances, the resolution at the sink device 160 may change during the communication session, potentially causing a mismatch between the display 122 and the display 162. In order to improve user experience and ensure proper functions, the source/sink system 100 may implement a technology for reducing or preventing user interaction mismatch by implementing a technology for screen normalization. The display 122 of the source device 120 and the display 162 of the sink device 160 may have different resolutions and/or different screen aspect ratios. In addition, in some settings, the user of the sink device 160 may have the ability to adjust the size of the display window for the video data received from the source device 120, so that the slave device 160 is presented in a window covering less than all of the display 162 of the sink device 160. Video data received by the source device 120. In another example setting, the user of the sink device 160 may have the option of viewing content in a landscape mode or a portrait mode, each mode having unique coordinates and different screen aspect ratios. In these cases, the coordinates associated with the user input received at the sink device 160 (such as the coordinates where a mouse click or touch event occurred) may not be processed by the source device 120 without modifying the coordinates. Therefore, the technology of the present disclosure may include mapping the coordinates of the user input received at the sink device 160 to The coordinates associated with the source device 120. In this article, the mapping is also called normalization, and the mapping will be explained in more detail below. The mapping can be sink-based or source-based.

[0055] The user input received by the sink device 160 may be received by the UI module 167 (for example, at the driver layer) and passed to the operating system of the sink device 160. The operating system on the sink device 160 may receive the coordinates (Xs net, Ysink) associated with the user input on the display surface. In this example, (Xsm^sink) may be the coordinates of the display 162 where the mouse click or touch event occurs. The display window presented on the display 162 may have an X-coordinate length (! ") and a y-coordinate width (W_DW）_OThe display window may also have coordinates of the upper left corner (a^, b_DW）_oBased on L_dw, W_dwAnd the upper left coordinates (a^, bQ, can determine the part of the display 162 covered by the display window. For example, the upper right corner of the display window can be located at the coordinates (8^+1«, b^), and the lower left corner of the display window can be located at the coordinates ( a_DW+L_DW, b_DW+W_DW), and the lower right corner of the display window can be located at the coordinates (a_DW+L_DW, b_DW+W_DW) O If the input is received at the coordinates in the display window, the sink device 160 may input the input as a UIBC input

Line processing. In other words, if the following conditions are met, the input associated with the coordinate (Xsink, "sink) can be processed as UIBC input:

[0056] a_DW W Xsink W a_DW+L_DW （ 1）

[0057] b_DW W Ysink W b_DW+W_DW （2）

[0058] After determining that the user input is a UIBC input, the UIPM 168 may normalize the coordinates associated with the input before being sent to the source device 120. The input determined to be outside the display window may be processed locally by the sink device 160 as a non-UIBC input.

[0059] As mentioned above, the normalization of input coordinates can be source-based or sink-based. When realizing source-based normalization, the source device 120 may send the display resolution (Lsrc, Wsrc) supported by the display 122 to the sink device 160 together with the video data or independently of the video data. For example, the supported display resolution can be sent as part of the capability negotiation session or can be sent at another time during the communication session. The sink device 160 may determine the display resolution (L_sink, W_sink), the display window resolution (1«, %) of the window displaying the content received from the source device 120, and the coordinates of the upper left corner of the display window (a^, b_DW）_oAs mentioned above, when determining the coordinates corresponding to the user input (x_SINK, Y_SINK) When in the display window, the operating system of the sink device 160 can use the conversion function to map the coordinates (XSINK, YsINk) to the source coordinates (XSRC, y_SRC)°The example conversion function used to convert (XsiNK, Ysiffi) to (XSRC, YsRc) can be as follows:

<td>[0060][0061][0062]</td><td>Xsrc-(XsimC&amp;dw)* L_Src/L_dw) (3) ysRc⁼ (YsiNiCbpw) * (Wsrc/Wdw) (4) Therefore, when sending the coordinates corresponding to the received user input, the sink device 160 can send the_{SM (},</td>

The coordinates (XsRC, ySRC) input by the user received at "sink). As will be described in more detail below, for example, the coordinates (XsRc, ysRc) may be sent as part of a data packet used to send the user input received at the sink device 160 to the source device 120 in UIBC±. Throughout other parts of the disclosure describing the input coordinates as included in the data packet, as described above in the example where the source/sink system 100 implements sink-based normalization, those coordinates can be converted into source coordinates.

[0063] When the source/sink system 100 implements sink-based normalization, the user input determined to be UIBC input rather than local input (that is, in the display window instead of outside the display window), can be in the source device 120 Where the device is not at the host

Perform the above calculation at 160. To facilitate these calculations, the sink device 160 may send L to the source device 120_dw. W_dwThe value of and the position information of the display window (for example, a_DW&gt;b_DW), and the coordinates of (Xsink, Ysink). Using these sent values, the source device 120 can determine (xsrc, y_SEC) Value.

[0064] In other implementations of sink-based normalization, the sink device 160 may send user input coordinates describing the location within the display window where the user input event occurred instead of the location on the display 162 where the user input event occurred (^xdw»Ydw)°In such an implementation, the coordinates (XDW, yDW) can be sent to the source device 120 along with the value of (Ldw, %). Based on these received values, the source device 120 can determine (Xsrc, y_SEc):

[0065] ^XSRC^=XDW*(Lsrc/Ldw)(5)

[0066] ysRc=yDW*(Wsrc/Wdw)(6)

[0067] The sink device 160 may determine Xdw and y based on the following function_DW :

[0068] ^XDW^=XSINK^_aDW (7)

[0069] yDw⁼ysiNK^_b_DW（8）

[0070] When the present disclosure describes that coordinates associated with user input are sent, for example, in a data packet, these coordinates

The sending of may include sink-based or source-based normalization as described above, and/or may include any additional information necessary to perform sink-based or source-based normalization.

[0071] UIBC can be designed to transmit various types of user input data, including cross-platform user input data. For example, the source device 120 may run an iOS® operating system, while the sink device 160 runs another operating system such as Android® or Windows®. Regardless of the platform, the UIPM168 can encapsulate the received user input in a form understandable by the A/V control module 125. UIBC can support multiple different types of user input formats to allow many different types of source and sink devices to utilize the protocol, regardless of whether the source and sink devices operate on different platforms. A universal input format can be defined, and platform-specific input formats can be supported at the same time, thereby providing flexibility in the manner of transferring user input between the source device 120 and the sink device 160 through the UIBC.

[0072] In the example of FIG. 1A, the source device 120 may include a smart phone, a tablet computer, a laptop computer, a desktop computer, a Wi-Fi-enabled TV, or any other device capable of transmitting audio and video data. The sink device 160 may equally include a smart phone, a tablet computer, a laptop computer, a desktop computer, a Wi-Fi-enabled TV, or any other device capable of receiving audio and video data and user input data. In some cases, the sink device 160 may include a system of devices, such that the display 162, the speaker 163, the UI device 167, and the A/V encoder 164 are all separated but interoperable devices. The source device 120 may also be a system of devices, rather than a single device.

[0073] In this disclosure, the term source device is generally used to refer to a device that transmits audio/video data, and the term sink device is generally used to refer to a device that receives audio/video data from the source device. In many cases, the source device 120 and the sink device 160 may be similar or identical devices, with one device operating as the source and the other device operating as the sink . In addition, these roles can be reversed in different communication sessions. Therefore, the sink device in one communication session can become the source device in a subsequent communication session, or vice versa.

[0074] FIG. 1B is a block diagram illustrating an exemplary source/sink system 101 that can implement the techniques of the present disclosure. The source/sink system 101 includes a source device 120 and a sink device 160, and each of the source device 120 and the sink device 160 may run and operate in the manner described above with respect to FIG. 1A. The source/sink system 101 also includes a sink device 180. The sink device 180 may receive audio and video data from the source device 120 on the established UIBC in a similar manner to the sink device 160 described above, and send user commands to the source device 120. In some configurations, the sink device 160 and the sink device 180 may operate independently of each other, and the audio and video data output at the source device 120 may be output at the sink device 160 and the sink device 180 at the same time. In an alternative configuration, sink device 160 may be a primary sink device, and sink device 180 may be a secondary sink device. In this example configuration, the sink device 160 and the sink device 180 may be coupled, and the sink device 160 may display video data while the sink device 180 outputs corresponding audio data. In addition, in some configurations, the sink device 160 may only output the transmitted video data, and the sink device 180 may only output the transmitted audio data.

[0075] FIG. 2 is a block diagram showing an example of the source device 220. The source device 220 may be a device similar to the source device 120 in FIG. 1A, and may operate in the same manner as the source device 120. The source device 220 includes a local display 222, a local speaker 223, a processor 231, a memory 232, a transmission unit 233, and a wireless modem 234. As shown in FIG. 2, the source device 220 may include one or more processors (ie, the processor 231) that encodes and/or decodes A/V data for transmission, storage, and display. For example, A/V data can be stored at the memory 232. The memory 232 may store a complete A/V file, or may include a smaller buffer that stores only a part of the A/V file (eg, streamed from another device or source). The transmission unit 233 may process the encoded A/V data for network transmission. For example, the encoded A/V data may be processed by the processor 231 and encapsulated into a network access layer (NAL) unit by the transmission unit 233 for cross-network communication. Can be used by wireless modem 234

The NAL unit is sent to the wireless sink device by the network connection. The wireless modem 234 may be, for example, a Wi-Fi modem configured to implement one of the IEEE802.11 standard family.

[0076] The source device 220 can also perform local processing and display of A/V data. Specifically, the display processor 235 may process the video data to be displayed on the local display 222, and the audio processor 236 may process the audio data for output on the speaker 223.

[0077] As described above with reference to the source device 120 of FIG. 1A, the source device 220 may also receive user input from the sink device. In this way, the wireless modem 234 of the source device 220 receives an encapsulated data packet such as a NAL unit, and sends the encapsulated data unit to the transmission unit 233 for decapsulation. For example, the transmission unit 233 may extract a data packet from the NAL unit, and the processor 231 may parse the data packet to extract a user input command. Based on the user input command, the processor 231 can adjust the encoded A/V data sent by the source device 220 to the sink device. In this way, the functions described above with reference to the A/V control module 125 of FIG. 1A can be fully or partially implemented by the processor 231.

[0078] The processor 231 of FIG. 2 generally represents any one of various processors, including but not limited to one or more digital signal processors (DSP), general-purpose microprocessors, application-specific integrated circuits (ASICs), field Programmable logic array (FPGA), other equivalent integrated or discrete logic circuits, or some combination thereof. The memory 232 of FIG. 2 may include any of a variety of volatile or non-volatile memories, including but not limited to random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), only Read memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc. The memory 232 may include a computer-readable storage medium for storing audio/video data and other types of data. The memory 232 may additionally store instructions and program codes executed by the processor 231 as part of executing various technologies described in this disclosure.

[0079] FIG. 3 shows an example of a sink device 360. The sink device 360 may be a device similar to the sink device 160 in FIG. 1A, and may operate in the same manner as the sink device 160. The sink device 360 includes one or more processors (ie, the processor 331), a memory 332, a transmission unit 333, a wireless modem 334, a display processor 335, a local display 362, an audio processor 336, a speaker 363, and a user input interface 376 . The sink device 360 receives the encapsulated data unit sent from the source device at the wireless modem 334. The wireless modem 334 may be, for example, a Wi-Fi modem configured to implement one or more standards in the IEEE802.11 standard family. The transmission unit 333 may decapsulate the encapsulated data unit. For example, the transmission unit 333 may extract encoded video data from the packaged data unit, and send the encoded A/V data to the processor 331 for decoding and presentation for output. The display processor 335 may process the decoded video data to be displayed on the local display 362, and the audio processor 336 may process the decoded audio data for output on the speaker 363.

[0080] In addition to presenting audio and video data, the wireless sink device 360 may also receive user input data through the user input interface 376. The user input interface 376 may represent any one of multiple user input devices, including but not limited to: touch display interface, keyboard, mouse, voice command module, gesture capture device (for example, with camera-based input capture capability), or multiple Any other device in the user input device. The user input received through the user input interface 376 may be processed by the processor 331. This processing may include generating a data packet including the received user input command in accordance with the techniques described in this disclosure. Once generated, the transmission unit 333 can process these data packets for network transmission to the wireless source device on the UIBC.

[0081] The processor 331 of FIG. 3 may include one or more of a wide range of processors, such as one or more digital signal processors (DSP), general-purpose microprocessors, application-specific integrated circuits (ASICs), Programming logic array

(FPGA), other equivalent integrated or discrete logic circuits, or some combination thereof. The memory 332 of FIG. 3 may include any of various volatile or non-volatile memories, including but not limited to: random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), Read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc. The memory 232 may include a computer-readable storage medium for storing audio/video data and other types of data. The memory 332 may also store instructions and program codes executed by the processor 331 as part of executing various technologies described in this disclosure.

[0082] FIG. 4 shows a block diagram of an example transmitter system 410 and a receiver system 450, which can be used by the transmitter/receiver 126 and the transmitter/receiver 166 in FIG. 1A. To communicate on the communication channel 150. At the transmitter system 410, a transmit (TX) data processor 414 is provided with service data for multiple data streams from a data source 412. Each data stream can be sent on the corresponding transmitting antenna. The TX data processor 414 formats, encodes, and interleaves the service data of each data stream based on the specific coding scheme selected for each data stream.

[0083] Orthogonal Frequency Division Multiplexing (OFDM) technology may be used to multiplex the coded data of each data stream with pilot data. Various other wireless communication technologies may also be used, including but not limited to time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA, and/or CDMA.

[0084] Consistent with FIG. 4, the pilot data is usually a known data pattern that is processed in a known manner, and can be used at the receiver system to estimate the channel response. It can then be based on the specific modulation scheme selected for each data stream (for example, Binary Phase Shift Keying (BPSK), Quadrature Phase Shift Keying (QPSK), M-PSK, or M-QAM (Quadrature Amplitude Modulation)) , Where M can be a scene of 2) to modulate (for example, symbol mapping) the encoded data of the multiplexed pilot and data stream to provide modulation symbols. The data rate, coding, and modulation of each data stream can be determined by instructions executed by the processor 430, which can be coupled with the memory 432.

[0085] Then, the modulation symbols of the data stream are provided to the TX MIMO processor 420, and the TX MIMO processor 420 may further process the modulation symbols (for example, for OFDM). Then, the TX MIMO processor 420 may Transmitter (TMTR) 422a to 422t provide Ν_τA stream of modulation symbols. In certain aspects, the TX MIMO processor 420 applies beamforming weights to the symbols of the data stream and the antenna that transmits the symbols.

[0086] Each transmitter 422 may receive and process the corresponding symbol stream to provide one or more analog signals, and further adjust (eg, amplify, filter, and up-convert) the analog signals to provide suitable transmission on the MIMO channel The modulated signal. Then, from Ν_τAntennas 424a to 424t transmit N from transmitters 422a to 422t_τA modulated signal.

[0087] At the receiver system 450, the transmitted modulated signal is received through Nr antennas 452a to 452r, and the received signal from each antenna 452 is provided to a corresponding receiver (RCVR) 454a to 454r. The receiver 454 conditions (eg, filters, amplifies, and downconverts) the respective received signal, digitizes the conditioned signal to provide samples, and further processes the samples to provide a corresponding "received" symbol stream.

[0088] Then, the receive (RX) data processor 460 may receive and process the Nr received symbol streams from the Nr receivers 454 based on a specific receiver processing technique to provide Nτ "detected" symbol streams. Then, the RX data processor 460 may demodulate, deinterleave, and decode each detected symbol stream to restore the service data of the data stream. The processing performed by the RX data processor 460 is complementary to the processing performed by the TX MIMO processor 420 and the TX data processor 414 at the transmitter system 410.

[0089] The processor 470, which may be coupled with the memory 472, periodically determines which precoding matrix to use. Reverse chain

The road message may include various types of information related to the communication link and/or the received data stream. Then, the reverse link message is processed by TX data processor 438 (TX data processor 438 also receives service data of multiple data streams from data source 436), modulated by modulator 480, and processed by transmitters 454a to 454r. It is adjusted and sent back to the transmitter system 410.

[0090] At the transmitter system 410, the modulated signal from the receiver system 450 is received by the antenna 424, adjusted by the receiver 422, demodulated by the demodulator 440, and processed by the RX data processor 442 , To extract the reverse link message sent by the receiver system 450. Then, the processor 430 determines which precoding matrix to use to determine the beamforming weight, and then processes the extracted message.

[0091] FIG. 5A is a block diagram illustrating an example message transmission sequence between a source device 520 and a sink device 560 as part of a capability negotiation session. Capability negotiation can be performed as part of a larger communication session establishment process between the source device 520 and the sink device 560. For example, Wi-Fi Direct or TDLS can be used as the basic connectivity standard to establish the session. After establishing a Wi-Fi Direct or TDLS session, the sink device 560 may initiate a TCP connection with the source device 520. As part of establishing a TCP connection, a control port running Real Time Streaming Protocol (RTSP) can be established to manage the communication session between the source device 520 and the sink device 560.

[0092] The source device 520 may generally operate in the same manner as described above for the source device 120 of FIG. 1A, and the sink device 560 may generally operate in the same manner as described above for the sink device 160 of FIG. 1A. To operate. After the source device 520 and the sink device 560 establish connectivity, the source device 520 and the sink device 560 may determine the parameter set for their subsequent communication sessions as part of the capability negotiation exchange.

[0093] The source device 520 and the sink device 560 may negotiate capabilities through a series of messages. These messages may be real-time streaming protocol (RTSP) messages, for example. At any stage of the negotiation, the receiver of the RTSP request message can respond with an RTSP response that includes the RTSP status code in addition to RTSP 0K. In this case, a different set of parameters can be used to retry the message exchange, or The capability negotiation session can be ended.

[0094] The source device 520 may send a first message (RTSP selection request message) to the sink device 560 in order to determine the set of RTSP methods supported by the sink device 560. After receiving the first message from the source device 520, the sink device 560 may respond with a second message (RTSP selection response message), which lists the RTSP methods supported by the sink 560. The second message may also include the RTSP OK status code.

[0095] After sending the second message to the source device 520, the sink device 560 may send a third message (RTSP selection request message) in order to determine the set of RTSP methods supported by the source device 520. After receiving the third message from the sink device 560, the source device 520 may respond with a fourth message (RTSP selection response message), which lists the RTSP methods supported by the source device 520. The fourth message may also include the RTSP OK status code.

[0096] After sending the fourth message, the source device 520 may send a fifth message (RTSP obtain_parameter request message) to specify the list of capabilities that the source device 520 is interested in. The sink device 560 may respond with the sixth message (RTSP obtain_parameter response message). The sixth message may contain the RTSP status code. If the RTSP status code is OK, the sixth message may also include response parameters for the parameters supported by the sink device 560 specified in the fifth message. The sink device 560 can ignore the parameters that are not supported by the sink device 560 in the fifth message.

[0097] Based on the sixth message, the source 520 may determine an optimal parameter set for the communication session, and may send a seventh message (RTSP setting-parameter request message) to the sink device 560. This seventh message may contain a parameter set to be used during the communication session between the source device 520 and the sink device 560. The seventh message may include wfd-presentation-url describing the universal resource identifier (URI) to be used in the RTSP establishment request, so as to establish

The communication session. The wfd-presentation-url specifies the URI that the sink device 560 can use for subsequent messages during the session establishment exchange. The values of wfd-urlO and wfd-urll specified in this parameter may correspond to the values of rtp-portO and rtp-portl in wfd-client-rtp-ports in the seventh message. In this case, RTP generally refers to a real-time protocol that can run on top of UDP.

[0098] After receiving the seventh message, the sink device 560 may respond with an eighth message with an RTSP status code indicating whether the setting of the parameter as specified in the seventh message is successful. As mentioned above, in different sessions, the roles or the source device and the sink device can be reversed or changed. In some cases, the sequence of messages to establish a communication session may define a device operating as a source and a device operating as a sink.

[0099] FIG. 5B is a block diagram showing another example message transmission sequence between the source device 560 and the sink device 520 as part of a capability negotiation session. The message transmission sequence of FIG. 5B is intended to provide a more detailed view of the transmission sequence described above with respect to FIG. 5A. In FIG. 5B, the message "lb. Obtained-Parameter Response" shows an example of a message identifying a list of supported input categories (for example, general and HIDC) and multiple lists of supported input types. Each supported input category in the list of supported input categories has an associated list of supported types (for example, generic_cap_list and hidc_cap_list). In FIG. 5B, the message "2a. Settings-Parameter Request" is an example of a second message identifying a second list of supported input categories (for example, universal and HIDC) and a plurality of second lists of supported types. Each supported input category in the second list of supported input categories has an associated second list of supported types (for example, generic_cap_list and hidc_cap_list). Message lb. Obtained-The parameter response identifies the input category and input type supported by the sink device 560. The message "2a. Setting_Parameter Request" identifies the input categories and input types supported by the source device 520, but it may not be all input categories supported by the source device 520 And a comprehensive list of input types. Instead, the message "2a. Setting_Parameter Request" may only identify those input categories and input types that are identified in the message "lb. Obtain_Parameter Response" as supported by the sink device 560. In this way, the input categories and input types identified in the message "2a. Setting_Parameter Request" may constitute a subset of the input categories and input types identified in the message "lb. Obtain-Parameter Response".

[0100] FIG. 6 is a conceptual diagram showing one example of a data packet that can be generated by a sink device and transmitted to a source device. The various aspects of the data packet 600 will be explained with reference to FIG. 1A, but the discussed techniques can be applied to other types of source/sink systems. The data packet 600 may include a data packet header 610, and the data packet header 610 is followed by payload data 650. The payload data 650 may additionally include one or more payload headers (eg, payload header 630). For example, the data packet 600 may be sent from the sink device 160 of FIG. 1A to the source device 120, so that the user of the sink device 160 can control the audio/video data sent by the source device 120. In this case, the payload data 650 may include user input data received at the sink device 160. For example, the payload data 650 may identify one or more user commands. The sink device 160 may receive the one or more user commands, and may generate a data packet header 610 and payload data 650 based on the received commands. Based on the content of the data packet header 610 of the data packet 600, the source device 120 may parse the payload data 650 to identify the user input data received at the sink device 160. Based on the user input data contained in the payload data 650, the source device 120 may The audio and video data sent from the source device 120 to the sink device 160 is changed in some way.

[0101] As used in this disclosure, the terms "parse" and "parse" generally refer to the process of analyzing a bitstream to extract data from the bitstream. Once extracted, the data can be processed by the source device 120, for example. For example, extracting data can include identifying how the data in the bitstream is formatted. As will be described in more detail below, the data packet header 610 may define a standardization known to both the source device 120 and the sink device 160

format. However, the payload data 650 can be formatted in one of many possible ways. By parsing the data packet header 610, the source device 120 can determine how the payload data 650 is formatted, and thus, the source device 120 can parse the payload data 650 to extract one or more from the payload data 650 Multiple users enter commands. This can provide flexibility in the different types of payload data supported in source-sink communication. As will be described in more detail below, the payload data 650 may also include one or more payload headers such as the payload header 630. In these cases, the source device 120 may parse the data packet header 610 to determine the format of the payload header 630, and then parse the payload header 630 to determine the format of the remainder of the payload data 650.

[0102] FIG. 620 is a conceptual description of how the data packet header 610 may be formatted. The numbers 0-15 in row 615 are intended to identify the bit positions in the data packet header 610, and are not intended to actually represent the information contained in the data packet header 610. The data packet header 610 includes a version field 621, a timestamp flag 622, a reserved field 623, an input category field 624, a length field 625, and an optional timestamp field 626.

[0103] In the example of FIG. 6, the version field 621 is a 3-bit field, which may indicate the version of a specific communication protocol implemented by the sink device 160. The value in the version field 621 can inform the source device 120 how to parse the remaining part of the data packet header 610 and how to parse the payload data 650. In the example of FIG. 6, the version field 621 is a 3-bit field, which can realize a unique identifier for 8 different versions. In other examples, more or fewer bits may be dedicated to the version field 621.

[0104] In the example of FIG. 6, the time stamp flag (T) 622 is a 1-bit field, which indicates whether the time stamp field 626 is present in the data packet header 610. The timestamp field 626 is a 16-bit field that contains a timestamp based on the multimedia data generated by the source device 120 and sent to the sink device 160. For example, the timestamp may be the sequence value assigned to the frame by the source device 120 before the video frame is sent to the sink device 160. The timestamp flag 622 may, for example, include "1" for indicating that the timestamp field 626 exists, and may include "0" for indicating that the timestamp field 626 does not exist. After parsing the data packet header 610 and determining that the timestamp field 626 exists, the source device 120 may process the timestamp included in the timestamp field 626. After parsing the data packet header 610 and determining that the timestamp field 626 does not exist, since there is no timestamp field in the data packet header 610, the source device 120 can start to analyze the payload data 650 after parsing the length field 625 Analyze.

[0105] If present, the timestamp field 626 may include a timestamp for identifying the video data frame displayed at the wireless sink device 160 when the user input data of the payload data 650 is obtained. For example, before the source device 120 sends the video frame to the sink device 160, the time stamp may have been added to the video frame by the source device 120. Therefore, the source device 120 may generate a video frame and embed a time stamp in the video data of the frame (for example, as metadata). The source device 120 may send a video frame with a time stamp to the sink device 160, and the sink device 160 may display the video frame. When the video frame is displayed by the sink device 160, the sink device 160 may receive a user command from the user. When the sink device 160 generates a data packet to transmit the user command to the source device 120, the sink device 160 may include the timestamp of the frame displayed by the sink device 160 when the user command is received in the timestamp field 626.

[0106] After receiving the data packet 600 with the timestamp field 626 present in the header, the wireless source device 120 can recognize the video frame displayed at the sink device 160 when the user input data of the payload data 650 is obtained, and The user input data can be processed based on the content of the frame identified by the time stamp. For example, if the user input data is a touch command applied to a touch display or a click of a mouse pointer, the source device 120 may determine the content of a frame displayed when the user applies a touch command to the display or clicks a mouse. In some cases, the content of the frame may be needed to

Appropriate processing of payload data. For example, user input based on user touch or mouse click may depend on what is being displayed on the display when the touch or click occurs. For example, touch or click can correspond to icons or menu options. In the event that the content of the display changes, the source device 120 can use the timestamp present in the timestamp field 626 to match the touch or click to the correct icon or menu option.

[0107] Additionally or alternatively, the source device 120 may compare the timestamp in the timestamp field 626 with the timestamp applied to the currently presented video frame. By comparing the timestamp in the timestamp field 626 with the current timestamp, the source device 120 can determine the round trip time. The round-trip time generally corresponds to the amount of time that has passed from the moment the source device 120 sends the frame to the moment the user input based on the frame is received back at the source device 120 from the sink device 160. The round-trip time can provide the source device 120 with an indication of the system delay, and if the round-trip time is greater than the threshold, the source device 120 can ignore the data contained in the payload data 650 under the assumption that the input command is applied to the outdated display frame. User input data in. When the round trip time is less than the threshold, the source device 120 can process the user input data and adjust the audio/video content sent in response to the user input data. The threshold can be programmable, and different types of devices (or different source-sink combinations) can be configured to define acceptable different thresholds for round trip time.

[0108] In the example of FIG. 6, the reserved field 623 is an 8-bit field, which does not include information used by the source device 120 when parsing the data packet header 610 and the payload data 650. However, future versions of the specific protocol (as identified in the version field 621) may use the reserved field 623. In this case, the source device 120 may use the information in the reserved field 623 to parse and parse the data packet header 610. /Or parse the payload data 650. The reserved field 623 combined with the version field 621 potentially provides the ability to extend and add features to the data packet format without fundamentally changing the format and features already in use.

[0109] In the example of FIG. 6, the input category field 624 is a 4-bit field, which is used to identify the input category of the user input data contained in the payload data 650. The sink device 160 may classify the user input data to determine the input category. For example, the classification of user input data can be based on the device from which the command is received or based on the attributes of the command itself. The value of the input category field 624 (possibly combined with other information in the data packet header 610) identifies to the source device 120 how the payload data 650 is formatted. Based on the formatting, the source device 120 may parse the payload data 650 to determine the user input received at the sink device 160.

[0110] In the example of FIG. 6, since the input category field 624 is 4 bits, 16 different input categories can be identified. One such input category may be a universal input format, which indicates that the user input data of the payload data 650 is formatted using universal information elements defined in the protocol executed by both the source device 120 and the sink device 160 . As will be described in more detail below, the universal input format may use universal information elements that allow the user of the sink device 160 to interact with the source device 120 at the application layer.

[0111] Another such input category may be a human interface device command (HIDC) format, which indicates that the user input data of the payload data 650 is formatted based on the type of input device used to receive the input data. Examples of types of devices include: keyboards, mice, touch input devices, joysticks, cameras, gesture capture devices (such as camera-based input devices), and remote controls. Other types of input categories that can be identified in the input category field 624 include: indicating that the user input data in the payload data 650 does not originate from the forwarding input format of the sink device 160 or an operating system-specific format, and indicating payload data 650 includes a voice command format for voice commands.

[0112] The length field 625 may include a 16-bit field for indicating the length of the data packet 600. For example, the length can be indicated in units of 8 bits. Since the data packet 600 is decoded in 16-bit words by the source device 120

Therefore, the data packet 600 can be filled to reach an integer multiple of 16 bits. Based on the length contained in the length field 625, the source device 120 can identify the end of the payload data 650 (that is, the end of the data packet 600) and the beginning of a new subsequent data packet.

[0113] The various sizes of the fields provided in the example of FIG. 6 are only intended to be explanatory, and it is intended that these fields can be implemented using a different number of bits than that shown in FIG. 6. In addition, it is also conceivable that the data packet header 610 may include fewer fields than all the fields discussed above or may use additional fields not discussed above. Of course, the technology of the present disclosure can be flexible in terms of the actual format used for each data field in the packet.

[0114] After parsing the data packet header 610 to determine the formatting of the payload data 650, the source device 120 may parse the payload data 650 to determine the user input command contained in the payload data 650. The payload data 650 may have its own payload header (payload header 630), which indicates the content of the payload data 650. In this way, the source device 120 can parse the payload header 630 based on the analysis of the data packet header 610, and then can parse the payload data 650 based on the analysis of the payload header 630.

[0115] For example, if the input category field 624 of the data packet header 610 indicates that there is a universal input in the payload data 650, the payload data 650 may have a universal input format. Therefore, the source device 120 can parse the payload data 650 according to the universal input format. As part of the universal input format, the payload data 650 may include a series of one or more input events, where each input event has its own input event header. Table 1 below identifies the fields that can be included in the input header.

[0116] Table 1

[0117]

<td>Field</td><td>Size (octets)</td><td>value</td>

<td>Universal IEID</td><td>1</td><td>See table 2</td>

<td>length</td><td>2</td><td>The length of the following field in octets</td>

<td>description</td><td>variable</td><td>The details entered by the user. See each table</td>

[0118]

[0119] The universal input event (IE) identification (ID) field identifies the universal input event identifier used to identify the input type. For example, the general IE ID field may be 1 octet in length, and may include an identification selected from Table 2 below. For example, in this example, if the universal IE ID field is 8 bits, 256 different types of input (identified as 0-255) can be identified, although not all 256 types of identification necessarily require associated input types. Some of these 256 can be reserved for future use in future versions of any protocol implemented by sink device 160 and source device 120. In Table 2, for example, the universal IE ID9-255 does not have an associated input type, but it can be assigned an input type in the future.

[0120] The length field in the input event header identifies the length of the description field, and the description field includes an information element describing the user input. The formatting of the description field may depend on the type of input identified in the general IE ID field. Therefore, the source device 120 can parse the content of the description field based on the input type identified in the universal IE ID field. Based on the length field of the input event header, the source device 120 can determine the end of one input event in the payload data 650 and the start of a new input event. As will be explained in more detail below, a user command can be described as one or more input events in the payload data 650.

[0121] Table 2 provides examples of input types, each input type has a corresponding general IE ID that can be used to identify the input type.

[0122] Table 2

[0123]

<td>Universal IE ID</td><td>Input type</td>

<td>0</td><td>Left mouse press/touch press</td>

<td>1</td><td>Left mouse up/touch up</td>

<td>2</td><td>Mouse movement/touch movement</td>

<td>3</td><td>Key press</td>

<td>4</td><td>Button up</td>

<td>5</td><td>Zoom</td>

<td>6</td><td>Vertical scroll</td>

<td>7</td><td>Scroll horizontally</td>

<td>8</td><td>Spin</td>

<td>9-255</td><td>Keep</td>

[0124]

[0125] The description field associated with each input type may have a different format. For example, the description fields of left mouse down/touch down event, left mouse up/touch up event, and mouse movement/touch movement may include the information elements identified in Table 3 below, although it may also be in other examples Use other formats.

[0126] Table 3

[0127]

<td>r segment</td><td>Size (octets)</td><td>Comment,</td>

<td>Number of pointers·|1: (Ν)</td><td>1</td><td>When the pointer count of the multi-touch action event is set to 1, it indicates the touch action event</td>

<td>For Ding i =!:Ν{</td><td></td><td></td>

<td>Pointer ID</td><td>1</td><td>The indicator of the pointer is J-[0, 1,...]</td>

<td>X Friend Standard</td><td>2</td><td>The X coordinate of the event normalized with reference to the negotiated resolution of the video stream between the sink device and the source device.</td>

<td>Y1 coordinate}</td><td>2</td><td>Refer to the negotiated resolution of the video stream between the sink device and the source device B to be normalized, converted: the coordinates of the event"</td>

[0128] The number of pointers may identify the number of touches or mouse clicks associated with the input event. Each pointer can have a unique pointer ID. For example, if the multi-touch event includes a three-finger touch, the input event can have three pointers, where each pointer has a unique pointer ID. Each pointer (that is, each finger touch) can have a pointer corresponding to where the touch occurred. X coordinate and y coordinate.

[0129] A single user command can be described as a series of input events. For example, if the three-finger swipe is a command to close the application, the three-finger swipe can be described in the payload data 650 as a touch down event with three pointers, a touch movement event with three pointers, and a touch movement event with three pointers. Pointer touch up event. The three pointers of the touch-down event may have the same pointer ID as the three pointers of the touch-move event and the touch-up event. The source device 120 may interpret the combination of these three input events as a three-finger swipe.

[0130] For example, the description field of a key press event or a key up event may include the information elements identified in Table 4 below.

[0131] Table 4

[0132]

<td>Field</td><td>Size 1 octet)</td><td>Comment,</td>

<td>Keep</td><td>1</td><td>Bao Zhao</td>

<td>Key code 1 (ASCII)</td><td>2</td><td>No.... The key code of the key press or lift event. The firewood/extended ASCII code uses the lower... bytes. Higher · bytes guarantee future ASCII-compatible key codes of nfJIJJ</td>

<td>Key code 2 (ASCID</td><td>2</td><td>The first·]button press...Bu Yi or raise the key code of the event. Kawaki/Extended ASCII uses the lower byte and the higher byte reserved for future ASCI-compatible key codes.</td>

[0133] For example, the description field of the zoom event may include the information elements identified in Table 5 below.

[0134] Table 5

[0135]

<td>Qin</td><td>Too small "(octets)</td><td>Notes:</td>

<td>X</td><td>2</td><td>Refer to the reference X standard of the scaling operation normalized by the negotiated resolution of the video stream between the sink device and the source device Z</td>

<td>Y</td><td>2</td><td>Refer to the reference Y coordinates of the scaling operation normalized with the negotiated resolution of the video stream between the sink device and the source device.</td>

<td>Integer multiples of scaling</td><td>1</td><td>The unsigned integer part of the multiple to be scaled</td>

<td>Play zoom II decimal multiples</td><td>1</td><td>The fractional part of the multiple to be zoomed</td>

[0136] For example, the description field of the horizontal scroll event or the vertical scroll event may include the information elements identified in Table 6 below.

[0137] Table 6

[0138]

<td>7 segments</td><td>Size (octets)</td><td>Annotation</td>

<td>Amount to scroll</td><td>2</td><td>Refer to the negotiated resolution of the video stream between the sink device and the source device to normalize... The number of pixels to be scrolled. Negative numbers can indicate scrolling to the right, and the stop number can indicate scrolling to the left.</td>

[0139] The above examples show some exemplary ways that payload data can be formatted for common input categories. If the input category field 624 of the data packet header 610 indicates a different input category (such as forwarded user input), the payload data 650 may have a different input format. In the case of forwarded user input, the sink device 160 may receive user input data from a third-party device, and forward the input to the source device 120 without interpreting the user input data. Thus, the source device 120 can parse the payload data 650 according to the forwarded user input format. For example, the payload header 630 of the payload data 650 may include a field for identifying the third-party device from which the user input is obtained. For example, this field may include the Internet Protocol (IP) address, MAC address, domain name, or some other such identifier of the third-party device. The source device 120 may parse the remaining part of the payload data based on the identifier of the third-party device.

[0140] The sink device 160 may negotiate capabilities with a third-party device via a series of messages. Then, as part of the capability negotiation process, the sink device 160 may send the unique identifier of the third-party device to the source device 120 as part of establishing a communication session with the source device 120. Alternatively, the sink device 160 may send information describing the third-party device to the source device 120, and based on this information, the source device 120 may determine the unique identifier of the third-party device. For example, the information describing the third-party device may include information used to identify the third-party device and/or information used to identify the capability of the third-party device. Regardless of whether the unique identifier is determined by the source device 120 or the sink device 160, when the sink device 160 transmits a data packet with user input obtained from a third-party device, the sink device 160 may include the unique identifier in the data packet Medium (for example, included in the payload header) so that the source device 120 can identify the source of the user input.

[0141] If the input category field 624 of the data packet header 610 indicates yet another different input category, such as a voice command, the payload data 650 may have yet another different input type. For voice commands, the payload data 650 may include encoded audio. The codec for encoding and decoding the audio of the voice command can be negotiated between the source device 120 and the sink device 160 via a series of messages. In order to send a voice command, the timestamp field 626 may include a voice sampling time value. In this case, the timestamp flag 622 may be set to indicate that there is a timestamp, instead of the timestamp as described above, the timestamp field 626 may include the voice sampling time value of the encoded audio for the payload data 650 .

[0142] In some examples, as described above, the voice command can be sent as a general command. In this case, the input category field 626 can be set to identify the general command format, and the reserved general IE ID can be One of them is assigned to the voice command. If the voice command is sent as a general command, the voice sampling rate may exist in the timestamp field 626 of the data packet header 610 or may exist in the payload data 650.

[0143] For the captured voice command data, the voice data can be encapsulated in a variety of ways. For example, RTP can be used to encapsulate voice command data, and RTP can provide a payload type to identify a codec and a time stamp, where the time stamp is used to identify the sampling rate. With or without an optional time stamp, the general user input format described above can be used to encapsulate the RTP data. The sink device 160 may use TCP/IP to send universal input data carrying voice command data to the source device 120.

[0144] As previously discussed, when the coordinates are included as part of a data packet (such as data packet 600) in, for example, the payload data 650, the coordinates may be the same as the coordinates scaled based on the negotiated resolution, the display window coordinates, Normalized coordinates, or coordinates corresponding to the coordinates associated with the sink display. In some cases, additional information may be included in the data packet or sent separately, so as to be used by the source device to normalize the coordinates received in the data packet.

[0145] Regardless of the input category of a specific data packet, the data packet header may be an application layer packet header, and may

To send data packets via TCP/IP. TCP/IP may enable the sink device 160 and the source device 120 to perform a retransmission technique in case of packet loss. Data packets may be sent from the sink device 160 to the source device 120 to control audio data or video data of the source device 120, or for other purposes (such as controlling an application running on the source device 120).

[0146] FIG. 7A is a flowchart of an example method of negotiating capabilities between a sink device and a source device. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules or algorithms, and when the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute One or more of the steps shown in one or more of the flowcharts described herein.

[0147] The method of FIG. 7A includes sink device 160 receiving a first message from source device 120 (701). For example, the message may include a request to obtain parameters. In response to the first message, the sink device 160 may send a second message to the source device 120 (703). For example, the second message may include an obtaining parameter response that identifies a first list of supported input categories and a plurality of first lists of supported types. Wherein, each supported input category in the first list of supported input categories has an associated first list of supported types. For example, the supported input categories may correspond to the same categories used in the input category field 624 of FIG. 6. Table 2 above represents an example of the supported types for a specific input category (generic input in this example). The sink device 160 may receive the third message from the source device 120 (705). For example, the third message may include a setting parameter request, where the setting parameter request identifies a port used for communication, a second list of supported input categories, and multiple second lists of supported types, where all Each supported input category in the second list of supported input categories has an associated second list of supported types, and each of the supported types in the second list includes the types in the first list Subset. Lodging The device 160 may send a fourth message to the source device 120 (707). For example, the fourth message may include a setting parameter response for confirming that the types in the second list have been activated. The sink device 160 may receive the fifth message from the source device 120 (709). For example, the fifth message may include a first setting parameter request indicating that the communication channel between the source device 120 and the sink device 160 has been activated. For example, the communication channel may include a user input return channel (UIBC). The sink device 160 may send a sixth message to the source device 120 (711). For example, the sixth message may include a second setting parameter response confirming the reception of the second setting parameter request by the sink device 160.

[0148] FIG. 7B is a flowchart of an example method of negotiating capabilities between a sink device and a source device. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0149] The method of FIG. 7B includes: the source device 120 sends a first message to the sink device 160 (702). For example, the first message may include a request for obtaining parameters. The source device 120 may receive the second message from the sink device 160 (704). For example, the second message may include an obtaining parameter response that identifies a first list of supported input categories and multiple first lists of supported types, wherein the first list of supported input categories is Each of the supported input categories has an associated first list of supported types. The source device 120 may send a third message to the sink device 160 (706). For example, the third message may include a setting parameter request that identifies a port used for communication, a second list of supported input categories, and multiple second lists of supported types, where the supported Each supported input category in the second list of input categories has an associated second list of supported types, and each supported type in the second list includes a subset of the types in the first list . The source device 120 may receive the fourth message from the sink device 160 (708). For example, the fourth message may include a type used to confirm that the class in the second list has been enabled.

Type of setting parameter response. The source device 120 may send a fifth message to the sink device 160 (710). For example, the fifth message may include a second setting parameter request indicating that the communication channel between the source device 120 and the sink device 160 has been activated. For example, the communication channel may include a user input return channel (UIBC). The source device 120 may receive the sixth message from the sink device 160 (712). For example, the sixth message may include a second setting parameter response confirming the reception of the second setting parameter request by the sink device 160.

[0150] FIG. 8A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0151] The method of FIG. 8A includes obtaining user input data at a wireless sink device (such as wireless sink device 160) (801). The user input data may be obtained through a user input component of the wireless sink device 160 (such as, for example, the user input interface 376 shown in conjunction with the wireless sink device 360). In addition, the sink device 160 may classify user input data as, for example, general, forwarded, or operating system specific. Then, the sink device 160 may generate a data packet header based on the user input data (803). The data packet header may be an application layer packet header. In addition to other fields, the data packet header may also include a field for identifying the input category corresponding to the user input data. For example, input categories can include universal input formats or human-machine interface device commands. The sink device 160 may also generate a data packet (805), where the data packet includes the generated data packet header and payload data. In one example, the payload data may include received user input data, and may identify one or more user commands. Then, the sink device 160 may send the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or the 220 of FIG. 2) (807). The sink device 160 may include components that allow data packets to be transmitted, for example, including the transmission unit 333 shown in FIG. 3 and the wireless Modem 334. The sink device 160 may transmit data packets through TCP/IP.

[0152] FIG. 8B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0153] The method of FIG. 8B includes receiving a data packet (802), where, among other things, the data packet may include a data packet header and payload data. The payload data may include user input data, for example. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown in FIG. 2. Then, the source device 120 may parse the data packet header included in the data packet (804 ) To determine the input category associated with the user input data contained in the payload data. The source device 120 may process the payload data based on the determined input category (806). The description with reference to FIGS. 8A and 8B The data packet can generally take the form of the data packet described with reference to FIG. 6, and can be used to control audio/video data and applications at the source device.

[0154] FIG. 9A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. Shown in the flow chart

One or more of the steps.

[0155] The method of FIG. 9A includes obtaining user input data at a wireless sink device (such as wireless sink device 160) (901). The user input data may be obtained through a user input component of the wireless sink device 160 (such as, for example, the user input interface 376 shown with reference to FIG. 3). Then, the sink device 160 may generate payload data (903), where the payload data may describe user input data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 may also generate a data packet (905), where the data packet includes a data packet header and the generated payload data. Then, the sink device 160 may send the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or the 220 of FIG. 2) (907). The sink device 160 may include components that allow data packets to be transmitted, such as, for example, a transmission unit 333 and a wireless modem 334. Data packets can be sent to the wireless source device via TCP/IP.

[0156] FIG. 9B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0157] The method of FIG. 9B includes receiving a data packet from the sink device 360 (902), where, among other things, the data packet may include a data packet header and payload data. In one example, the payload data may include, for example, data describing the details of the user input (such as input type values). The source device 120 may include communication components that allow data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 may parse the data packet (904) to determine the input type value in the input type field in the payload data. The source device 120 may process data describing the details of the user input based on the determined input type value (906). The data packet described with reference to FIGS. 9A and 9B may generally take the form of the data packet described with reference to FIG. 6.

[0158] FIG. 10A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0159] The method of FIG. 10A includes obtaining user input data (100) at a wireless sink device (such as the wireless sink device 160). The user input component of the wireless sink device 160 (such as, for example, the user shown in FIG. 3) Input interface 376) to obtain user input data. Then, the sink device 160 may generate a data packet header based on the user input (1003). In addition to other fields, the data packet header may include a time stamp flag (for example, a 1-bit field) for indicating whether a time stamp field is present in the data packet header. For example, the timestamp flag may include "1" indicating that the timestamp field exists, and may include "0" indicating that the timestamp field does not exist. For example, the timestamp field may be a 16-bit field that contains a timestamp generated by the source device 120 and added to the video data before transmission. The sink device 160 may also generate a data packet (1005), where the data packet includes the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. Then, the sink device 160 may send the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or the 220 of FIG. 2) (1007). The sink device 160 may include a component that allows data packets to be transmitted.

It includes, for example, the transmission unit 333 and the wireless modem 334 shown with reference to FIG. 3. Data packets can be sent to the wireless source device via TCP/IP.

[0160] FIG. 10B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0161] The method of FIG. 10B includes receiving a data packet (1002) from the wireless sink device 160, where, among other things, the data packet may include a data packet header and payload data. For example, the payload data may include user input data. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 may parse the data packet header included in the data packet (1004). The source device 120 may determine whether there is a timestamp field in the data packet header (1006). In one example, the source device 120 may make this determination based on the timestamp flag value included in the data packet header. If the data packet header includes a timestamp field, the source device 120 may process the payload data based on the timestamp in the timestamp field (1008). The data packets described with reference to FIGS. 10A and 10B may generally take the form of the data packets described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0162] FIG. 11A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method may be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0163] The method of FIG. 11A includes obtaining user input data (HO1) at a wireless sink device (such as the wireless sink device 160). The user input component of the wireless sink device 160 may pass through a user input component (such as, for example, the user shown with reference to FIG. 3). Input interface 376) to obtain the user input data. Then, the sink device 160 may generate a data packet header based on the user input (1103). In addition to other fields, the data packet header may include a timestamp field. For example, the timestamp field may include a 16-bit field containing a timestamp based on multimedia data generated by the wireless source device 120 and transmitted to the wireless sink device 160. The time stamp may be added to the video data frame by the wireless source device 120 before being sent to the wireless sink device. For example, the timestamp field may identify the timestamp associated with the video data frame displayed at the wireless sink device 160 when the user input data was captured. The sink device 160 may also generate a data packet (1105), where the data packet includes the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. Then, the sink device 160 may send the generated data component to a wireless source device (for example, the source device 120 in FIG. 1A or 220 in FIG. 2). Group (1107). The sink device 160 may include components that allow data packets to be transmitted, including, for example, the transmission unit 333 and the wireless modem 334 shown with reference to FIG. 3. Data packets can be sent to the wireless source device via TCP/IP.

[0164] FIG. 11B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0165] The method of FIG. 11B includes receiving a data packet (1102) from a wireless sink device (such as wireless sink device 160), where, among other things, the data packet may include a data packet header and payload data. For example, the payload data may include user input data. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 can identify the timestamp field in the data packet header (1104). The source device 120 may process the payload data based on the timestamp in the timestamp field (1106). As part of processing the payload data, the source device 120 may identify the video data frame displayed at the wireless sink device when the user input data is obtained based on the timestamp, and interpret the payload data based on the content of the frame. As part of processing the payload data based on the timestamp, the source device 120 may compare the timestamp with the current timestamp of the current video frame sent by the source device 120, and may respond to the timestamp and the current timestamp. The time difference between is less than the threshold to execute the user input command described in the payload data, or the payload is not executed in response to the time difference between the time stamp and the current time stamp being greater than the threshold User input commands described in the charge data. The data packet described with reference to FIGS. 11A and 11B may generally take the form of the data packet described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0166] FIG. 12A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0167] The method of FIG. 12A includes obtaining user input data at a wireless sink device (such as wireless sink device 160) (1201). In an example, the user input data may be voice command data, which may be obtained through a user input component of the wireless sink device 160 (such as, for example, the voice command recognition module included in the user input interface 376 in FIG. 3) Command data. The sink device 160 may generate a data packet header based on the user input (1203). The sink device 160 may also generate payload data (1205), where the payload data may include voice command data. In one example, the payload data may also include received user input data and may identify one or more user commands. The sink device 160 may also generate a data packet (1207), where the data packet includes the generated data packet header and payload data. Then, the sink device 160 may send the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or the 220 of FIG. 2) (1209). The sink device 160 may include components that allow data packets to be transmitted, including, for example, the transmission unit 333 and the wireless modem 334 shown with reference to FIG. 3. Data packets can be sent to the wireless source device via TCP/IP.

[0168] FIG. 12B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0169] The method of FIG. 12B includes receiving a data packet (1202), where, among other things, the data packet may include a data packet header and payload data. For example, the payload data may include user input data such as voice command data. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 may parse the payload data included in the data packet (1204) to determine whether the payload data includes voice command data. Participate

The data packets described with reference to FIGS. 12A and 12B may generally take the form of the data packets described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0170] FIG. 13A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0171] The method of FIG. 13A includes obtaining user input data at a wireless sink device (such as wireless sink device 160) (1301). In one example, the user input data may be a multi-touch gesture, which may be obtained through a user input component of the wireless sink device 160 (such as, for example, the UI 167 or the user input interface 376 in FIG. 3). In an example, the multi-touch gesture may include a first touch input and a second touch input. The sink device 160 may generate a data packet header based on the user input (1303). The sink device 160 may also generate payload data (1305), where the payload data may associate the user input data for the first touch input event with the first pointer identification, and combine the user input data for the second touch input event. Associated with the second pointer identification. The sink device 160 may also generate a data packet (1307), where the data packet includes the generated data packet header and payload data. Then, the sink device 160 may send the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or 220 of FIG. 2) (1309). The sink device 160 may include components that allow data packets to be transmitted, including, for example, the transmission unit 333 and the wireless modem 334 shown with reference to FIG. 3. Data packets can be sent to the wireless source device via TCP/IP.

[0172] FIG. 13B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0173] The method of FIG. 13B includes receiving a data packet (1302), where, among other aspects, the data packet may include a data packet header and payload data. For example, the payload data may include user input data such as multi-touch gestures. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 may parse the payload data included in the data packet (1304) to identify the user input data included in the payload data. In one example, the identified data may include user input data for a first touch input event with a first pointer identification and user input data for a second touch input event with a second pointer identification. Then, the source device 120 may interpret the user input data for the first touch input event and the user input data for the second touch input event as a multi-touch gesture (1306). The data packets described with reference to FIGS. 13A and 13B may generally take the form of the data packets described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0174] FIG. 14A is a flowchart of an example method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0175] The method of FIG. 14A includes obtaining user input data from an external device at the wireless sink device 360 (1401). in

In an example, the external device may be a third-party device connected to the sink device. The sink device 160 may generate a data packet header based on the user input (1403). In one example, the data packet header may identify the user input data as forwarded user input data. The sink device 160 may also generate payload data (1405), where the payload data may include user input data. The sink device 160 may also generate a data packet (1407), where the data packet includes the generated data packet header and payload data. Then, the sink device 160 may transmit the generated data packet to a wireless source device (for example, the source device 120 of FIG. 1A or the 220 of FIG. 2) (1409). The sink device 160 may include components that allow data packets to be transmitted, including, for example, the transmission unit 333 and the wireless modem 334 shown with reference to FIG. 3. Data packets can be sent to the wireless source device via TCP/IP.

[0176] FIG. 14B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0177] The method of FIG. 14B includes receiving a data packet (1402), where, among other things, the data packet may include a data packet header and payload data. For example, the payload data may include user input data such as a forwarded user input command that indicates that the user input data is forwarded from a third-party device. The source device 120 may include a communication component that allows data packets to be transmitted, including, for example, the transmission unit 233 and the wireless modem 234 shown with reference to FIG. 2. Then, the source device 120 may parse the data packet header (1404) and may determine that the payload data includes a forwarded user input command (1404). Then, the source device 120 may parse the payload data included in the data packet (1406) to identify the identification associated with the third-party device corresponding to the forwarded user input command. Then, the source device 120 may process the payload data based on the identification of the identified third-party device (1408). The data packets described with reference to FIGS. 14A and 14B may generally take the form of the data packets described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0178] FIG. 15A is a flowchart of an example method of transmitting user data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated example method can be executed by sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, a computer-readable storage medium (for example, the memory 332) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 331) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0179] The method of FIG. 15A includes obtaining user input data at a wireless sink device (1501). The user input data may have associated coordinate data. For example, the associated coordinate data may correspond to the position of a mouse click event or the position of a touch event. Then, the sink device 160 may normalize the associated coordinate data to generate normalized coordinate data (1503). Then, the sink device 160 may generate a data packet including the normalized coordinate data (1505). Normalizing the coordinate data may include scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source's display (such as the display 22 of the source device 120). The resolution of the display window may be determined by the sink device 160, and the resolution of the display of the source device may be received from the source device 120. Then, the sink device 160 may send a data packet with normalized coordinates to the wireless source device 120 (1507). As part of the method of FIG. 15A, the sink device 160 may also determine whether the associated coordinate data is within the display window for the content received from the wireless source device, and for example, if the associated coordinate data is outside the display window. Process the user input locally, or otherwise normalize the coordinates as described if the input is within the display window.

[0180] FIG. 15B is a flowchart of an example method of receiving user input data from a wireless sink device at a wireless source device according to the present disclosure. The example method shown can be executed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, a computer-readable storage medium (for example, the memory 232) may store instructions, modules, or algorithms. When the instructions, modules or algorithms are executed, one or more processors (for example, the processor 231) execute the instructions, modules, or algorithms. One or more of the steps shown in the flowchart.

[0181] The method of FIG. 15B includes receiving a data packet at a wireless source device, wherein the data packet includes user input data associated with coordinate data (1502). For example, the associated coordinate data may correspond to the location of the mouse click event or the location of the touch event at the sink device. Then, the source device 120 may normalize the associated coordinate data to generate normalized coordinate data (1504). The source device 120 may normalize the coordinate data by scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source's display. The source device 120 may determine the resolution of the display of the source device, and may receive the resolution of the display window from the wireless sink device. Then, the source device may process the data packet based on the normalized coordinate data (1506). The data packets described with reference to FIGS. 15A and 15B may generally take the form of the data packets described with reference to FIG. 6, and may be used to control audio/video data at the source device.

[0182] For simplicity of explanation, various aspects of the present disclosure are individually described with reference to FIGS. 7-15. However, it should be envisaged that these various aspects can be combined with each other and used in conjunction with each other rather than just used alone. Generally, the functions and/or modules described herein can be implemented in both the wireless source device and the wireless sink device. In this way, the user interface functions described in the current example can be used interchangeably between the wireless source device and the wireless sink device.

[0183] Various devices and devices including wireless handheld devices and integrated circuits (ICs) or a set of ICs (ie, chipsets) can be used to implement the technology of the present disclosure. Any component, module or unit described is provided to emphasize the functional aspects, and it is not necessarily required to be implemented by different hardware units.

[0184] Accordingly, hardware, software, firmware, or any combination thereof can be used to implement the techniques described herein. If implemented by hardware, any features described as modules, units or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable medium, which includes instructions that, when executed in a processor, perform one or more of the above methods. The computer-readable medium may include a tangible and non-transitory computer-readable storage medium, and may form a part of a computer program product, and the computer program product may include packaging materials. The computer-readable storage medium may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable Programming read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, these technologies can be implemented at least in part by a computer-readable communication medium that carries communication codes in the form of instructions or data structures and can be accessed, read, and/or executed by a computer.

[0185] The code can be composed of one or more digital signal processors (DSP), general-purpose microprocessors, application specific integrated circuits (ASIC), field programmable logic arrays (FPGA), or other equivalent integrated or discrete logic circuits. One or more processors to execute. Accordingly, as used herein, the term "processor" may refer to any of the foregoing structure or any other structure suitable for implementing the technology described herein. Furthermore, in some aspects, it may be The functions described in this article are provided in a dedicated software module or hardware module configured to perform encoding and decoding or incorporated into a combined video codec. In addition, one or more circuits or logic units can be used to fully implement these technologies.

[0186] Various aspects of the present disclosure have been described. These and other aspects are within the scope of the following claims.