User input back channel for wireless displays
Abstract
Problem to be solved.To control a user of a wireless sink device to control a wireless source device and control the contents transmitted from the wireless source device to the wireless sink device.
Solution.A user input data is acquired in a wireless sink device, coordinate data associated with the input is acquired, and normalized coordinate data is generated. After generating a data packet with normalized coordinate data, the data packet is sent to the wireless source device. [Selection diagram] Fig. 1A

Term
Projected expiry 6 August 2035.
- Priority
- Filed
- Published
- Today
- Projected expiry
26 claims: 10 independent, 16 dependent
- 1ワイヤレスシンクデバイスからワイヤレスソースデバイスにユーザデータを送信する方法であって、前記方法は、 前記ワイヤレスシンクデバイスにおいてユーザ入力データを取得することであって、前記ユーザ入力データが、関連付けられた座標データを有する、取得することと、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データを備えるデータパケットを生成することと、 ワイヤレスソースデバイスに前記データパケットを送信することとを備える、方法。
- 2前記関連付けられた座標データが、前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウ内にあるかどうかを決定することをさらに備える、請求項1に記載の方法。
- 3前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウの解像度を決定することと、 前記ソースデバイスから前記ソースデバイスのディスプレイの解像度のインジケーションを受信することとをさらに備える、請求項1に記載の方法。
- 4前記座標データを正規化することが、前記ディスプレイウィンドウの前記解像度と前記ソースの前記ディスプレイの前記解像度との比に基づいて、前記関連付けられた座標データをスケーリングすることを含む、請求項3に記載の方法。
- 5前記関連付けられた座標データがマウスクリックイベントのロケーションに対応する、請求項1に記載の方法。
- 6前記関連付けられた座標データがタッチイベントのロケーションに対応する、請求項1に記載の方法。
- 7ワイヤレスソースデバイスにユーザデータを送信するためのワイヤレスシンクデバイスであって、前記ワイヤレスシンクデバイスは、 命令を記憶するメモリと、 前記命令を実行するように構成された1つまたは複数のプロセッサであって、前記命令を実行すると、前記1つまたは複数のプロセッサが、 前記ワイヤレスシンクデバイスにおいてユーザ入力データを取得することであって、前記ユーザ入力データが、関連付けられた座標データを有する、取得することと、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データを備えるデータパケットを生成することと を行わせる、1つまたは複数のプロセッサと、 ワイヤレスソースデバイスに前記データパケットを送信するためのトランスポートユニットとを備える、ワイヤレスシンクデバイス。
- 8前記命令を実行すると、前記1つまたは複数のプロセッサは、 前記関連付けられた座標データが、前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウ内にあるかどうかを決定することをさらに行わせる、請求項7に記載のワイヤレスシンクデバイス。
- 9前記命令を実行すると、前記1つまたは複数のプロセッサが、 前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウの解像度を決定することと、 前記ソースデバイスから前記ソースデバイスのディスプレイの解像度のインジケーションを受信することとをさらに行わせる、請求項7に記載のワイヤレスシンクデバイス。
- 10前記座標データを正規化することが、前記ディスプレイウィンドウの前記解像度と前記ソースの前記ディスプレイの前記解像度との比に基づいて、前記関連付けられた座標データをスケーリングすることを含む、請求項9に記載のワイヤレスシンクデバイス。
- 11前記関連付けられた座標データがマウスクリックイベントのロケーションに対応する、請求項7に記載のワイヤレスシンクデバイス。
- 12前記関連付けられた座標データがタッチイベントのロケーションに対応する、請求項7に記載のワイヤレスシンクデバイス。
- 131つまたは複数のプロセッサによって実行されると、ワイヤレスシンクデバイスからワイヤレスソースデバイスにユーザデータを送信する方法を実行することを前記1つまたは複数のプロセッサに行わせる、命令を記憶するコンピュータ可読記憶媒体であって、前記方法は、 前記ワイヤレスシンクデバイスにおいてユーザ入力データを取得することであって、前記ユーザ入力データが、関連付けられた座標データを有する、取得することと、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データを備えるデータパケットを生成することと、 ワイヤレスソースデバイスに前記データパケットを送信することとを備える、コンピュータ可読記憶媒体。
- 14ワイヤレスソースデバイスにユーザデータを送信するためのワイヤレスシンクデバイスであって、前記ワイヤレスシンクデバイスは、 前記ワイヤレスシンクデバイスにおいてユーザ入力データを取得するための手段であって、前記ユーザ入力データが、関連付けられた座標データを有する、取得するための手段と、 正規化座標データを生成するために、前記関連付けられた座標データを正規化するための手段と、 前記正規化座標データを備えるデータパケットを生成するための手段と、 ワイヤレスソースデバイスに前記データパケットを送信するための手段とを備える、ワイヤレスシンクデバイス。
- 15ワイヤレスソースデバイスにおいてワイヤレスシンクデバイスからユーザデータを受信する方法であって、前記方法は、 前記ワイヤレスソースデバイスにおいてデータパケットを受信することであって、前記データパケットが、関連付けられた座標データをもつユーザ入力データを備える、受信することと、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データに基づいて前記データパケットを処理することとを備える、方法。
- 16前記ワイヤレスシンクデバイスから、前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウの解像度と前記ディスプレイウィンドウについてのロケーション情報とを受信することと、 前記ソースデバイスのディスプレイの解像度を決定することとをさらに備える、請求項15に記載の方法。
- 17前記座標データを正規化することが、前記ディスプレイウィンドウの前記解像度と前記ソースの前記ディスプレイの前記解像度との比に基づいて、前記関連付けられた座標データをスケーリングすることを含む、請求項16に記載の方法。
- 18前記関連付けられた座標データがマウスクリックイベントのロケーションに対応する、請求項15に記載の方法。
- 19前記関連付けられた座標データがタッチイベントのロケーションに対応する、請求項15に記載の方法。
- 20ワイヤレスシンクデバイスからユーザデータを受信するためのワイヤレスソースデバイスであって、前記ワイヤレスソースデバイスは、 前記ワイヤレスソースデバイスにおいてデータパケットを受信するためのトランスポートユニットであって、前記データパケットが、関連付けられた座標データをもつユーザ入力データを備える、トランスポートユニットと、 命令を記憶するメモリと、 前記命令を実行するように構成された1つまたは複数のプロセッサであって、前記命令を実行すると、前記1つまたは複数のプロセッサが、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データに基づいて前記データパケットを処理することと を行わせる、1つまたは複数のプロセッサとを備える、ワイヤレスソースデバイス。
- 21前記命令を実行すると、前記1つまたは複数のプロセッサが、 前記ワイヤレスシンクデバイスから、前記ワイヤレスソースデバイスから受信されているコンテンツのためのディスプレイウィンドウの解像度と前記ディスプレイウィンドウについてのロケーション情報とを受信することと、 前記ソースデバイスのディスプレイの解像度を決定することとをさらに行わせる、請求項20に記載のワイヤレスソースデバイス。
- 22前記座標データを正規化することが、前記ディスプレイウィンドウの前記解像度と前記ソースの前記ディスプレイの前記解像度との比に基づいて、前記関連付けられた座標データをスケーリングすることを含む、請求項21に記載のワイヤレスソースデバイス。
- 23前記関連付けられた座標データがマウスクリックイベントのロケーションに対応する、請求項20に記載のワイヤレスソースデバイス。
- 24前記関連付けられた座標データがタッチイベントのロケーションに対応する、請求項20に記載のワイヤレスソースデバイス。
- 251つまたは複数のプロセッサによって実行されると、ワイヤレスソースデバイスにおいてワイヤレスシンクデバイスからユーザデータを受信する方法を実行することを前記1つまたは複数のプロセッサに行わせる、命令を記憶するコンピュータ可読記憶媒体であって、前記方法は、 前記ワイヤレスソースデバイスにおいてデータパケットを受信することであって、前記データパケットが、関連付けられた座標データをもつユーザ入力データを備える、受信することと、 正規化座標データを生成するために、前記関連付けられた座標データを正規化することと、 前記正規化座標データに基づいて前記データパケットを処理することとを備える、コンピュータ可読記憶媒体。
- 26ワイヤレスシンクデバイスからユーザデータを受信するためのワイヤレスソースデバイスであって、前記ワイヤレスソースデバイスは、 前記ワイヤレスソースデバイスにおいてデータパケットを受信するための手段であって、前記データパケットが、関連付けられた座標データをもつユーザ入力データを備える、受信するための手段と、 正規化座標データを生成するために、前記関連付けられた座標データを正規化するための手段と、 前記正規化座標データに基づいて前記データパケットを処理するための手段とを備える、ワイヤレスソースデバイス。
Independent claims26
140 paragraphs, as filed
0001The entire contents of this application are incorporated herein by reference. US Provisional Application No. 61 / 435,194, filed January 21, 2011, US Provisional Application No. 61 / 447,592, filed February 28, 2011, US Provisional Application No. 61 / 448,312, filed March 2, 2011, US Provisional Application No. 61 / 450,101, filed March 7, 2011, US Provisional Application No. 61 / 467,535, filed March 25, 2011, US Provisional Application No. 61 / 467,543, filed March 25, 2011, US Provisional Application No. 61 / 514,863 filed on August 3, 2011, and US Provisional Application No. 61 / 544,440 filed on October 7, 2011 Claim the interests of.
0002The present disclosure relates to techniques for transmitting data between a wireless source device and a wireless sink device.
0003A wireless display (WD) or Wi-Fi® display (WFD) system includes a wireless source device and one or more wireless sink devices. Each of the source device and the sink device can be either a mobile device or a wired device having wireless communication capability. One or more of the source and sink devices are, for example, mobile phones, portable computers with wireless communication cards, personal digital assistants (PDAs), portable media players, or so-called "smart" phones and "smart". It may include pads or tablets, electronic readers, or other such devices with wireless communication capabilities, including any type of wireless display, video game device, or other types of wireless communication devices. One or more of the source and sink devices may also include wired devices such as televisions, desktop computers, monitors and projectors that include communication capabilities.
0004The source device sends media data, such as audio-video (AV) data, to one or more of the sink devices participating in a particular media sharing session. Media data can be played on both the local display of the source device and each of the displays of the sink device. More specifically, each of the participating sink devices renders the received media data on its screen and audio equipment.
0005The present disclosure generally describes a system in which a wireless sink device can communicate with the wireless sink device. As part of a communication session, the wireless source device can send audio and video data to the wireless sink device, which can send the user input received by the wireless sink device to the wireless source device. it can. In this way, the user of the wireless sink device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sink device.
0006In one example, the method of transmitting user data from a wireless sync device to a wireless source device is to acquire user input data in the wireless sync device, the user input data having associated coordinate data. And to normalize the associated coordinate data to generate the normalized coordinate data, to generate a data packet with the normalized coordinate data, and to send the data packet to the wireless source device. Including.
0007In another example, the wireless sync device for sending user data to the wireless source device is a memory that stores the instructions and one or more processors configured to execute the instructions and execute the instructions. Then, one or more processors acquire the user input data in the wireless sink device, and the user input data has the associated coordinate data, acquires the user input data, and generates the normalized coordinate data. To send a data packet to one or more processors and a wireless source device to normalize the associated coordinate data and generate a data packet with the normalized coordinate data. Including the transport unit of.
0008In another example, when a computer-readable storage medium is run by one or more processors, it performs to one or more processors how to send user data from a wireless sink device to a wireless source device. Let, memorize the command. The above method is to acquire user input data in a wireless sync device, the user input data has associated coordinate data, is associated with the acquisition, and to generate normalized coordinate data. It includes normalizing the coordinate data, generating a data packet with the normalized coordinate data, and sending the data packet to the wireless source device.
0009In another example, the wireless sync device for transmitting user data to the wireless source device is a means for retrieving user input data in the wireless sync device, the user input data having associated coordinate data. , A means to obtain, a means to normalize the associated coordinate data to generate normalized coordinate data, a means to generate a data packet with normalized coordinate data, and a wireless source. Includes means for sending data packets to the device.
0010In another example, the method of receiving user data from a wireless sink device on a wireless source device is to receive a data packet on the wireless source device, where the data packet contains user input data with associated coordinate data. It comprises receiving, normalizing the associated coordinate data to generate normalized coordinate data, and processing the data packet based on the normalized coordinate data.
0011In another example, the wireless sink device for sending user data to the wireless source device is a transport unit for receiving data packets on the wireless source device, and the data packets have associated coordinate data. A transport unit with user input data, a memory for storing instructions, and one or more processors configured to execute the instructions, and when the instructions are executed, the one or more processors Includes one or more processors that allow the associated coordinate data to be normalized and the data packets to be processed based on the normalized coordinate data in order to generate the normalized coordinate data.
0012In another example, when the computer-readable storage medium is run by one or more processors, the wireless source device performs the method of receiving user data from the wireless sink device on the one or more processors. Let, memorize the command. The method is to receive a data packet on a wireless source device, the data packet comprising user input data with associated coordinate data, to receive and to generate normalized coordinate data. It includes normalizing the associated coordinate data and processing data packets based on the normalized coordinate data.
0013In another example, a wireless sink device for sending user data to a wireless source device is a means for receiving a data packet on the wireless source device, and the data packet is user input with associated coordinate data. Means for receiving data, means for normalizing the associated coordinate data to generate normalized coordinate data, and means for processing data packets based on the normalized coordinate data. And include.
0014<figref num="1A">The block diagram which shows an example of the source / sink system which can implement the technique of this disclosure.</figref><figref num="1B">A block diagram showing an example of a source / sink system with two sink devices.</figref><figref num="2">The figure which shows an example of the source device which can implement the technique of this disclosure.</figref><figref num="3">The figure which shows an example of the sink device which can implement the technique of this disclosure.</figref><figref num="4">The block diagram of the transmitter system and the receiver system which can implement the technique of this disclosure.</figref><figref num="5A">The figure which shows the exemplary message forwarding sequence for performing functional negotiation according to the technique of this disclosure.</figref><figref num="5B">The figure which shows the exemplary message forwarding sequence for performing functional negotiation according to the technique of this disclosure.</figref><figref num="6">The figure which shows the exemplary data packet which can be used to deliver the user input data acquired in a sink device to a source device.</figref><figref num="7A">A flowchart illustrating the techniques of the present disclosure that can be used for functional negotiation between a source device and a sink device.</figref><figref num="7B">A flowchart illustrating the techniques of the present disclosure that can be used for functional negotiation between a source device and a sink device.</figref><figref num="8A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data.</figref><figref num="8B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data.</figref><figref num="9A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data.</figref><figref num="9B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data.</figref><figref num="10A">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="10B">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="11A">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="11B">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="12A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets containing voice commands.</figref><figref num="12B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets containing voice commands.</figref><figref num="13A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with multi-touch user input commands.</figref><figref num="13B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with multi-touch user input commands.</figref><figref num="14A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data forwarded from a third party device.</figref><figref num="14B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data forwarded from a third party device.</figref><figref num="15A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets.</figref><figref num="15B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets.</figref>
0015The present disclosure generally describes a system in which a wireless sink device can communicate with the wireless sink device. As part of a communication session, the wireless source device can send audio and video data to the wireless sink device, which can send the user input received by the wireless sink device to the wireless source device. it can. In this way, the user of the wireless sink device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sink device.
0016FIG. 1A is a block diagram showing an exemplary source / sync system 100 that may implement one or more of the techniques of the present disclosure. As shown in FIG. 1A, the system 100 includes a source device 120 that communicates with the sink device 160 over the communication channel 150. The source device 120 includes a memory for storing audio / video (A / V) data 121, a display 122, a speaker 123, an audio / video encoder 124 (also called an encoder 124), an audio / video control module 125, and the like. It may include a transmitter / receiver (TX / RX) unit 126. The sink device 160 includes a display 162, a speaker 163, an audio / video decoder 164 (also called a decoder 164), a transmitter / receiver unit 166, a user input (UI) device 167, and a user input processing module (UIPM). : user input processing It can include module) 168 and. The illustrated components form only one exemplary configuration for the Source / Sync System 100. Other configurations may include fewer components than the components shown, or may include additional components other than those shown.
0017In the example of FIG. 1A, the source device 120 can display the video portion of the audio / video data 121 on the display 122 and output the audio portion of the audio / video data 121 on the speaker 123. Can the audio / video data 121 be stored locally on the source device 120 and accessed from an external storage medium such as a file server, hard drive, external memory, Blu-ray Disc, DVD, or other physical storage medium? Or it can be streamed to the source device 120 over a network connection such as the Internet. In some cases, audio / video data 121 may be captured in real time via the camera and microphone of the source device 120. The audio / video data 121 may include multimedia content, such as a movie, television program, or music, but may also include real-time content generated by the source device 120. Such real-time content can be, for example, video data generated by an application running on the source device 120 or captured, for example, as part of a video telephony session. As described in more detail, such real-time content may, in some cases, include video frames of user input options available for the user to select. In some cases, audio / video data 121 may include video frames that are a combination of different types of content, such as video frames for movies or TV shows with user input options overlaid on the frames of the video.
0018In addition to locally rendering the audio / video data 121 through the display 122 and the speaker 123, the audio / video encoder 124 of the source device 120 can encode the audio / video data 121 and the transmitter. / The receiver unit 126 can transmit the encoded data to the sink device 160 via the communication channel 150. The transmitter / receiver unit 166 of the sink device 160 receives the encoded data, and the audio / video decoder 164 decodes the encoded data and transfers the decoded data to the display 162 and the speaker 163. Output via. In this way, the audio and video data rendered by the display 122 and the speaker 123 can be rendered simultaneously by the display 162 and the speaker 163. Audio data and video data can be composed of frames, which can be time synchronized with the video frame when rendered.
0019The audio / video encoder 124 and audio / video decoder 164 are alternatives to the new ITU-T H.264 standard, sometimes referred to as MPEG-4, Part10, Advanced Video Coding (AVC), or H.265 standard. Any number of audio and video compression standards can be implemented, including high efficiency video coding (HEVC) standards. Many other types of proprietary or standardized compression techniques can also be used. Generally, the audio / video decoder 164 is configured to perform the reverse coding operation of the audio / video encoder 124. Although not shown in FIG. 1A, in some embodiments, the A / V encoder 124 and A / V decoder 164 can be integrated with the audio encoder and decoder, respectively, and are suitable MUX-DEMUX units, or Other hardware and software can be included to handle both audio and video encoding in common or separate data streams.
0020As described in more detail below, the A / V encoder 124 can perform other coding functions in addition to implementing the video compression standard described above. For example, the A / V encoder 124 can add various types of metadata to the A / V data 121 before the A / V data 121 is sent to the sink device 160. In some cases, the A / V data 121 may be stored in or received in the source device 120 in encoded form and therefore does not require further compression by the A / V encoder 124. is there.
0021Figure 1A shows that the communication channel 150 carries the audio payload data and the video payload data separately, but in some cases the video payload data and the audio payload data are part of a common data stream. Please understand that it can be. If applicable, the MUX-DEMUX unit is ITU It may comply with the H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP). The audio / video encoder 124 and audio / video decoder 164 are each one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software. , Hardware, firmware, or any combination thereof. Each of the audio / video encoder 124 and the audio / video decoder 164 may be included in one or more encoders or decoders, both of which may be integrated as part of a composite encoder / decoder (codec). Thus, each of the source device 120 and the sink device 160 may be equipped with a dedicated machine configured to perform one or more of the techniques of the present disclosure.
0022Display 122 and Display 162 have a variety of video outputs, including cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, light emitting diode (LED) displays, organic light emitting diode (OLED) displays, or other types of display devices. It may be equipped with any of the devices. In these or other examples, the displays 122 and 162 can be emissive displays or transmissive displays, respectively. The display 122 and display 162 can also be touch displays, such that they are both input and display devices at the same time. Such a touch display can be a capacitive, resistive, or other type of touch panel that allows the user to provide user input to each device.
0023Speaker 123 may include any of a variety of audio output devices, such as headphones, single speaker systems, multi-speaker systems, or surround sound systems. Further, although the display 122 and the speaker 123 are shown as part of the source device 120 and the display 162 and the speaker 163 are shown as part of the sink device 160, the source device 120 and the sink device 160 are actually plural. It can be a system of devices. As an example, the display 162 can be a television, the speaker 163 can be a surround sound system, and the decoder 164 can be part of an external box that is wired or wirelessly connected to the display 162 and the speaker 163. In another example, the sink device 160 can be a single device, such as a tablet computer or smartphone. In yet other cases, the source device 120 and the sink device 160 are similar devices, for example, both are smartphones, tablet computers, and so on. In this case, one device can act as a source and the other can act as a sink. These roles may be reversed in subsequent communication sessions. In yet other cases, the source device can be equipped with a mobile device such as a smartphone, laptop or tablet computer, and the sink device can be equipped with a more stationary device (eg, using an AC power cord). , In which case, the source device may deliver audio and video data for presentation to a large crowd via the sink device.
0024The transmitter / receiver unit 126 and the transmitter / receiver unit 166 are each a variety of mixers, filters, amplifiers, and other components designed for signal modulation, as well as one or more antennas, and It may include other components designed to send and receive data. The communication channel 150 generally represents any communication medium suitable for transmitting video data from the source device 120 to the sink device 160, or a collection of various communication media. The communication channel 150 is usually a relatively short-range communication channel similar to Wi-Fi, Bluetooth (registered trademark), and the like. However, the communication channel 150 is not necessarily limited in this regard, with any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or with wireless and wired media. Can be any combination of. In another example, the communication channel 150 may even form part of a packet-based network, such as a wired or wireless local area network, a wide area network, or a global network such as the Internet. In addition, the communication channel 150 can be used by the source device 120 and the sink device 160 to create peer-to-peer links. Source device 120 and sink device 160 are IEEE802. It is possible to communicate through communication channel 150 using communication protocols such as standards from the 11 standard family. The source device 120 and sink device 160 comply with the Wi-Fi Direct standard so that, for example, the source device 120 and sink device 160 communicate directly with each other without the use of intermediates such as wireless access points or so-called hotspots. Can communicate. The source device 120 and sink device 160 can also establish a tunneled direct link setup (TLDS) to avoid or reduce network congestion. Although the techniques of the present disclosure are sometimes described with respect to Wi-Fi, it is contemplated that aspects of these techniques may be compatible with other communication protocols. As an example, but not limited to, wireless communication between the source device 120 and the sink device may utilize orthogonal frequency division multiplexing (OFDM) techniques. Includes, but is not limited to, time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. A wide variety of other wireless communication techniques can also be used. WiFi® Direct and TDLS are intended to set up relatively short distance communication sessions. The relatively short distance here can refer to less than 70 meters, for example, but in noisy or obstructed environments, the distance between devices can be even shorter, such as less than 35 meters. tunneled direct link setup) can be established. Although the techniques of the present disclosure are sometimes described with respect to Wi-Fi, it is contemplated that aspects of these techniques may also be compatible with other communication protocols. As an example, but not a limitation, wireless communication between the source device 120 and the sink device may utilize orthogonal frequency division multiplexing (OFDM) techniques. Includes, but is not limited to, time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. A wide variety of other wireless communication techniques can also be used. WiFi® Direct and TDLS are intended to set up relatively short distance communication sessions. The relatively short distance here can refer to less than 70 meters, for example, but in noisy or obstructed environments, the distance between devices can be even shorter, such as less than 35 meters. tunneled direct link setup) can be established. Although the techniques of the present disclosure are sometimes described with respect to Wi-Fi, it is contemplated that aspects of these techniques may be compatible with other communication protocols. As an example, but not a limitation, wireless communication between the source device 120 and the sink device may utilize orthogonal frequency division multiplexing (OFDM) techniques. Includes, but is not limited to, time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. A wide variety of other wireless communication techniques can also be used. WiFi® Direct and TDLS are intended to set up relatively short distance communication sessions. The relatively short distance here can refer to less than 70 meters, for example, but in noisy or obstructed environments, the distance between devices can be even shorter, such as less than 35 meters.
0025In addition to decoding and rendering the data received from the source device 120, the sink device 160 can also receive user input from the user input device 167. The user input device 167 can be, for example, a keyboard, mouse, trackball or trackpad, touch screen, voice command recognition module, or other such user input device. The UIPM168 formats the user input commands received by the user input device 167 into a data packet structure that can be interpreted by the source device 120. Such data packets are transmitted by the transmitter / receiver 166 to the source device 120 over the communication channel 150. The transmitter / receiver unit 126 receives the data packet, and the A / V control module 125 parses the data packet to interpret the user input command received by the user input device 167. Based on the commands received in the data packet, the A / V control module 125 can modify the encoded and transmitted content. In this way, the user of the sink device 160 can control the audio and video payload data transmitted by the source device 120 remotely and without interacting directly with the source device 120. Examples of the types of commands that a user of sink device 160 can send to source device 120 are commands for rewinding, fast-forwarding, pausing, and playing audio and video data, as well as for zooming, rotating, and scrolling. Includes commands such as. The user may also make a selection, for example from a menu of options, and send the selection to the source device 120.
0026In addition, the user of sink device 160 may be able to launch and control applications on source device 120. For example, a user of sync device 160 may be able to launch a photo editing application stored on source device 120 and use that application to edit photos stored locally on source device 120. The sink device 160 provides the user with a user experience where the photo is actually edited on the source device 120, but the photo appears and feels locally edited on the sink device 160. Can be done. Using such a configuration, device users may be able to take advantage of the capabilities of one device for use with several devices. For example, the source device 120 can be a smartphone with a large amount of memory and high-end processing power. The user of the source device 120 may use the smartphone in all settings and situations in which the smartphone is normally used. However, when watching a movie, the user may wish to watch the movie on a device with a larger display screen, in which case the sink device 160 may be a tablet computer or a larger display device or television. When wanting to send or respond to e-mail, the user may wish to use a device with a keyboard, in which case the sink device 160 can be a laptop. In both cases, the user is interacting with the sink device, but most of the processing can still be performed by the source device 120 (smartphone in this example). In this particular operating context, most of the processing is done by the source device 120, so the sink device 160 is more than if the sink device 160 is required to do the processing that is being done by the source device 120. Less It can be a lower cost device with a source. In some examples, both source and sink devices may be able to receive user input (such as touch screen commands), and the techniques of the present disclosure show the functionality of those devices in a given session. Two-way dialogue can be facilitated by negotiating and / or identifying.
0027In some configurations, the A / V control module 125 can be an operating system process running by the operating system of the source device 125. However, in other configurations, the A / V control module 125 can be the software process of the application running on the source device 120. In such a configuration, user input commands can be interpreted by a software process, so that the user of the sink device 160 is the source device as opposed to the operating system running on the source device 120. You are interacting directly with an application running on 120. By interacting directly with the application as opposed to the operating system, users of sink device 160 may have access to a library of commands that are not native to the operating system of source device 120. In addition, direct interaction with the application may allow commands to be more easily transmitted and processed by devices running on different platforms.
0028The source device 120 can respond to user input applied in the wireless sink device 160. In such an interactive application configuration, the user input applied in the wireless sync device 160 may be sent back to the wireless display source through the communication channel 150. In one example, the user interface back (UIBC) is used to allow the sink device 160 to send the user input applied at the sink device 160 to the source device 120. A reverse channel architecture, also known as channel), can be implemented. The reverse channel architecture may include a higher layer message for transporting user input and a lower layer frame for negotiating user interface functionality on the sink device 160 and the source device 120. The UIBC can reside on the Internet Protocol (IP) transport layer between the sink device 160 and the source device 120. In this way, UIBC can be above the transport layer in the Open Systems Interconnection (OSI) communication model. In one example, OSI communication includes seven layers: 1-physical, 2-datalink, 3-network, 4-transport, 5-session, 6-presentation, and 7-application. In this example, above the transport layer refers to Layer 5, Layer 6, and Layer 7. To facilitate the reliable transmission and sequential delivery of data packets containing user-input data, UIBC may include Transmission Control Protocol / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP), etc. Can be configured to operate on the packet-based communication protocol of. UDP and TCP can operate in parallel in the OSI layer architecture. TCP / IP allows sink device 160 and source device 120 to implement retransmission techniques in the event of packet loss.
0029In some cases, there may be a mismatch between the user input interface located on the source device 120 and the user input interface located on the sink device 160. In order to resolve the potential problems posed by such mismatches and promote a good user experience under such circumstances, user input interface feature negotiations may be performed prior to establishing a communication session or in communication. It can occur between the source device 120 and the sink device 160 at various times throughout the session. As part of this negotiation process, the source device 120 and the sink device 160 may agree on the negotiated screen resolution. When the sync device 160 transmits the coordinate data associated with the user input, the sync device 160 can scale the coordinate data obtained from the display 162 to match the negotiated screen resolution. In one example, if the sink device 160 has a resolution of 1280 x 720 and the source device 120 has a resolution of 1600 x 900, then those devices may use, for example, 1280 x 720 as their negotiated resolution. The negotiated resolution may be selected based on the resolution of the sink device 160, but the resolution of the source device 120 or some other resolution may also be used. In the example where a 1280x720 sink device is used, the sink device 160 can scale the obtained x-coordinates by 1600/1280 times before sending them to the source device 120, as well. The sink device 160 can scale the obtained y-coordinates by 900/720 before sending them to the source device 120. In other configurations, the source device 120 can scale the acquired coordinates to the negotiated resolution. The scaling is done by the sink device 160.
0030Moreover, in some cases, the resolution at the sink device 160 will fluctuate during the communication session, which can lead to a mismatch between the display 122 and the display 162. To improve the user experience and ensure proper functionality, Source / Sync System 100 implements techniques to reduce or prevent user interaction mismatches by implementing techniques for screen normalization. obtain. The display 122 of the source device 120 and the display 162 of the sink device 160 may have different resolutions and / or different aspect ratios. In addition, in some settings, the user of the sync device 160 will source the video data received from the source device 120 so that it is rendered in a window that covers a smaller portion of the sink device 160's display 162. It may have the ability to resize the display window for video data received from device 120. In another exemplary setting, the user of sink device 160 is in landscape mode. You may have the option to view the content in either mode or portrait mode, each of which has unique coordinates and different aspect ratios. In such situations, the coordinates associated with the user input received on the sink device 160, such as the coordinates on which the mouse click or touch event occurs, cannot be processed by the source device 120 without modification to those coordinates. Sometimes. Thus, the techniques of the present disclosure may include mapping the coordinates of user input received on the sink device 160 to the coordinates associated with the source device 120. This mapping, also referred to herein as normalization, can be either sink-based or source-based, as described in more detail below.
0031User input received by the sink device 160 may be received by the UI module 167 and passed to the operating system of the sink device 160, for example at the driver level. The operating system on the sink device 160 has the coordinates (x) associated with where the user input took place on the display surface.<sub>SINK</sub>, y<sub>SINK</sub>) Can be received. In this example, (x<sub>SINK</sub>, y<sub>SINK</sub>) Can be the coordinates of the display 162 where the mouse click or touch event took place. The display window rendered on display 162 has an x-coordinate length (L) that describes the size of the display window.<sub>DW</sub>) And y coordinate width (W)<sub>DW</sub>) And can have. The display window also has upper left corner coordinates (a) that describe the location of the display window.<sub>DW</sub>, b<sub>DW</sub>) Can also be. L<sub>DW</sub>, W<sub>DW</sub>, And upper left coordinates (a<sub>DW</sub>, b<sub>DW</sub>), The portion of the display 162 covered by the display window can be determined. For example, the upper right corner of the display window is the coordinates (a<sub>DW</sub>+ L<sub>DW</sub>, b<sub>DW</sub>), The lower left corner of the display window is the coordinates (a)<sub>DW</sub>, b<sub>DW</sub>+ W<sub>DW</sub>), The lower right corner of the display window is the coordinates (a)<sub>DW</sub>+ L<sub>DW</sub>, b<sub>DW</sub>+ W<sub>DW</sub>) Can be located. When the input is received at the coordinates in the display window, the sink device 160 can process the input as a UIBC input. In other words, the associated coordinates (x)<sub>SINK</sub>, y<sub>SINK</sub>An input with) can be treated as a UIBC input if the following conditions are met:<maths num="1"><img id="000003" he="16" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="2"><img id="000004" he="15" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths>
0032After determining that the user input is a UIBC input, the coordinates associated with that input can be normalized by UIPM168 before being sent to the source device 120. Input determined to be outside the display window may be processed locally by the sink device 160 as non-UIBC input.
0033As mentioned above, the input coordinate normalization can be either source-based or sync-based. When implementing sync-based normalization, the source device 120 has a supported display resolution (L) for display 122.<sub>SRC</sub>, W<sub>SRC</sub>) Can be sent to the sink device 160 with or separate from the video data. The supported display resolutions can be transmitted, for example, as part of a functional negotiation session or at another time during a communication session. The sink device 160 has a display resolution (L) for the display 162.<sub>SINK</sub>, W<sub>SINK</sub>) And the display window resolution (L) for the window displaying the content received from the source device 120<sub>DW</sub>, W<sub>DW</sub>) And the upper left corner coordinates for the display window (a)<sub>DW</sub>, b<sub>DW</sub>) And can be determined. As explained above, the coordinates (x) corresponding to the user input<sub>SINK</sub>, y<sub>SINK</sub>When) is determined to be in the display window, the operating system of the sink device 160 uses a transform function to coordinate (x).<sub>SINK</sub>, y<sub>SINK</sub>) To source coordinates (x<sub>SRC</sub>, y<sub>SRC</sub>) Can be mapped. (x<sub>SINK</sub>, y<sub>SINK</sub>) To (x<sub>SRC</sub>, y<sub>SRC</sub>An exemplary conversion function for converting to) can be:<maths num="3"><img id="000005" he="14" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="4"><img id="000006" he="15" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths>
0034Therefore, when transmitting the coordinates corresponding to the received user input, the sink device 160 is (x).<sub>SINK</sub>, y<sub>SINK</sub>) Coordinates for user input received in (x)<sub>SRC</sub>, y<sub>SRC</sub>) Can be sent. Coordinates (x), as described in more detail below.<sub>SRC</sub>, y<sub>SRC</sub>) May be transmitted, for example, as part of a data packet used to transmit user input received at sink device 160 to source device 120 through UIBC. Throughout the rest of the disclosure, where the input coordinates are described as being contained within the data packet, those coordinates are as described above in the case where the source / sync system 100 implements sink-based normalization. Can be converted to source coordinates.
0035When the Source / Sync System 100 implements source-based normalization, the above calculation is for user input that is determined to be UIBC input rather than local input (ie, inside the display window instead of outside the display window). Can be performed on the source device 120 instead of the sink device 160. To facilitate such calculations, the sink device 160 is L.<sub>DW</sub>, W<sub>DW</sub>Values for, as well as location information about the display window (eg a)<sub>DW</sub>, b<sub>DW</sub>), And (x<sub>SINK</sub>, y<sub>SINK</sub>) Can be sent to the source device 120. Using these transmitted values, the source device 120 follows Equations 3 and 4 above (x).<sub>SRC</sub>, y<sub>SRC</sub>) Can be determined.
0036In another implementation of sync-based normalization, the sink device 160 describes user input where in the display window the user input event occurs, rather than where it occurs on the display 162. Coordinates for (x<sub>DW</sub>, y<sub>DW</sub>) Can be sent. In such an implementation, the coordinates (x)<sub>DW</sub>, y<sub>DW</sub>) Is (L<sub>DW</sub>, W<sub>DW</sub>Can be sent to the source device with a value for). Based on these received values, the source device 120 follows the following conversion function (x)<sub>SRC</sub>, y<sub>SRC</sub>) Can be determined.<maths num="5"><img id="000007" he="15" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="6"><img id="000008" he="16" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths>
0037The sink device 160 is x based on the following function<sub>DW</sub>And y<sub>DW</sub>Can be determined.<maths num="7"><img id="000009" he="15" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths><maths num="8"><img id="000010" he="14" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></maths>
0038When the present disclosure describes transmitting coordinates associated with user input, eg, in a data packet, the transmission of these coordinates may include sink-based or source-based normalization as described above. And / or may contain any additional information needed to perform sink-based or source-based normalization.
0039UIBC can be designed to transport various types of user input data, including cross-platform user input data. For example, the source device 120 can run the iOS® operating system, while the sink device 160 runs another operating system, such as Android® or Windows®. Regardless of the platform, the UIPM168 can encapsulate the received user input in a form understandable to the A / V control module 125. UIBC supports several different types of user input formats to allow many different types of source and sink devices to utilize the protocol regardless of whether they run on different platforms. obtain. General input formats can be defined and both platform-specific input formats can be supported, thus the flexibility of how UIBC can communicate between source device 120 and sink device 160. Is brought.
0040In the example of Figure 1A, the source device 120 can include a smartphone, tablet computer, laptop computer, desktop computer, Wi-Fi enabled television, or any other device capable of transmitting audio and video data. .. The sync device 160 can also be a smartphone, tablet computer, laptop computer, desktop computer, Wi-Fi enabled television, or any other device that can receive audio and video data and receive user input data. Can be prepared. In some cases, the sink device 160 is such that the display 162, speaker 163, UI device 167, and A / V encoder 164 are all separate but interoperable devices. It may include a system of devices. Similarly, the source device 120 can be a system of multiple devices rather than a single device.
0041In the present disclosure, the term source device is generally used to refer to a device that is transmitting audio / video data, and the term sink device is generally used to refer to a device that is receiving audio / video data from a source device. Used to point to. In many cases, the source device 120 and the sink device 160 can be similar or identical devices, one device acting as a source and the other acting as a sink. Moreover, these roles can be reversed in different communication sessions. Thus, a sink device in one communication session can be a source device in subsequent communication sessions and vice versa.
0042FIG. 1B is a block diagram showing an exemplary source / sync system 101 that may implement the techniques of the present disclosure. The source / sink system 101 includes a source device 120 and a sink device 160, each of which may function and operate in the manner described above for FIG. 1A. The source / sink system 101 further includes a sink device 180. In a manner similar to the sink device 160 described above, the sink device 180 may receive audio and video data from the source device 120 and send user commands to the source device 120 through an established UIBC. In some configurations, the sink device 160 and the sink device 180 can operate independently of each other, and the audio and video data output by the source device 120 is output simultaneously by the sink device 160 and the sink device 180. Can be done. In an alternative configuration, the sink device 160 can be the primary sink device and the sink device 180 can be the secondary sink device. In such an exemplary configuration, the sink device 160 and the sink device 180 can be combined, the sink device 160 can display video data, and the sink device 180 outputs the corresponding audio data. Further, in some configurations, the sink device 160 can output only the transmitted video data and the sink device 180 outputs only the transmitted audio data.
0043FIG. 2 is a block diagram showing an example of the source device 220. The source device 220 can be a device similar to the source device 120 in FIG. 1A and can operate in the same way as the source device 120. The source device 220 includes a local display 222, a local speaker 223, a processor 231, a memory 232, a transport unit 233, and a wireless modem 234. As shown in FIG. 2, the source device 220 may include one or more processors (ie, processor 231) that encode and / or decode A / V data for transport, storage, and display. A / V data can be stored, for example, in memory 232. Memory 232 can store the entire A / V file, or can include, for example, a smaller buffer that simply stores a portion of the A / V file, streamed from another device or source. .. Transport unit 233 can process encoded A / V data for network transport. For example, encoded A / V data can be processed by processor 231 and encapsulated in a network access layer (NAL) unit by transport unit 233 for communication over the network. The NAL unit can be sent to the wireless sink device over a network connection by wireless modem 234. The wireless modem 234 can be, for example, a Wi-Fi modem configured to implement one of the IEEE 802.11 standard families.
0044In addition, the source device 220 can process and display A / V data locally. Specifically, the display processor 235 can process the video data to be displayed on the local display 222, and the audio processor 236 can process the audio data for output on the speaker 223. it can.
0045As described above for the source device 120 of FIG. 1A, the source device 220 may also receive user input commands from the sink device. In this way, the wireless modem 234 of the source device 220 receives the encapsulated data packet, such as the NAL unit, and sends the encapsulated data unit to the transport unit 233 for decapsulation. For example, transport unit 233 can extract data packets from the NAL unit, and processor 231 can parse the data packets to extract user input commands. Based on user input commands, processor 231 can adjust the encoded A / V data transmitted by the source device 220 to the sink device. In this way, the functionality described above with respect to the A / V control module 125 of FIG. 1A may be fully or partially implemented by processor 231.
0046Processor 231 in FIG. 2 generally includes, but is not limited to, one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), and field programmable logic arrays (FPGAs). Represents any of a wide variety of processors, or any combination thereof, including other equivalent integrated circuits or discrete logic circuits. The memory 232 in FIG. 2 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), It may have any of a wide variety of volatile or non-volatile memory, including electrically erasable programmable read-only memory (EEPROM®), flash memory, and the like. The memory 232 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 232 may further store instructions and program code executed by processor 231 as part of performing the various techniques described herein.
0047Figure 3 shows an example of the sink device 360. The sink device 360 can be a device similar to the sink device 160 in FIG. 1A and can operate in the same way as the sink device 160. The sink device 360 includes one or more processors (ie, processor 331), memory 332, transport unit 333, wireless modem 334, display processor 335, local display 362, audio processor 336, and speaker 363. And the user input interface 376. The sink device 360 receives the encapsulated data unit sent from the source device in the wireless modem 334. The wireless modem 334 can be, for example, a Wi-Fi modem configured to implement one or more standards from the IEEE 802.11 standard family. The transport unit 333 can decapsulate the encapsulated data unit. For example, the transport unit 333 may extract encoded video data from the encapsulated data unit and send the encoded A / V data to processor 331 for decoding and rendering for output. it can. The display processor 335 can process the decoded video data as it is displayed on the local display 362, and the audio processor 336 can process the decoded audio data for output on the speaker 363.
0048In addition to rendering audio and video data, the wireless sync device 360 can also receive user input data through user input interface 376. The user input interface 376 includes, but is not limited to, a touch display interface, a keyboard, a mouse, a voice command module, and several user inputs, including, for example, a gesture capture device (with camera-based input capture capability). It can represent any of the devices, or any other user input device of some user input devices. User input received through user input interface 376 may be processed by processor 331. This process may include generating a data packet containing the received user input command according to the techniques described herein. Once generated, transport unit 333 may process its data packets for network transport to wireless source devices through UIBC.
0049Processor 331 in Figure 3 is one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), other equivalent integrated circuits or discrete logic circuits. It may have one or more of a wide range of processors, or any combination thereof. The memory 332 in FIG. 3 is not limited to these, but includes, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), and non-volatile random access memory (NVRAM). It can have either a wide variety of volatile or non-volatile memory, including electrically erasable programmable read-only memory (EEPROM), flash memory, and so on. The memory 232 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 332 can further store instructions and program code executed by processor 331 as part of performing the various techniques described in the present disclosure.
0050FIG. 4 shows a block diagram of an exemplary transmitter system 410 and receiver system 450 that can be used by transmitter / receiver 126 and transmitter / receiver 166 of FIG. 1A to communicate through communication channel 150. Shown. In transmitter system 410, traffic data for some data streams is provided from data source 412 to transmit (TX) data processor 414. Each data stream may be transmitted through its own transmitting antenna. The TX data processor 414 formats, codes, and interleaves the traffic data for each data stream based on the specific coding scheme selected for that data stream.
0051The coded data for each data stream can be multiplexed with pilot data using Orthogonal Frequency Division Multiplexing (OFDM) techniques. Includes, but is not limited to, time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. A wide variety of other wireless communication techniques can also be used.
0052Consistent with FIG. 4, pilot data is a known data pattern that is typically processed in a known way and can be used in receiver systems to estimate channel response. The multiplexed pilot and coded data for each data stream is then selected for the particular modulation scheme (eg, 2-phase shift keying (BPSK), 4-phase shift) for that data stream to provide modulation symbols. Modulated (eg, symbol mapping) based on keying (QPSK), M-PSK, or M-QAM (quadrature keying), where M can be a power of 2. The data rate, coding, and modulation for each data stream can be determined by instructions executed by processor 430, which can be coupled to memory 432.
0053A modulation symbol for the data stream is then given to the TX MIMO processor 420, which can further process that modulation symbol (for example, for OFDM). The TX MIMO processor 420 then<sub>T</sub>N modulated symbol streams<sub>T</sub>It can be given to multiple transmitters (TMTR) 422a ~ 422t. In some embodiments, the TX MIMO processor 420 applies beamforming weights to a symbol in the data stream and to the antenna from which the symbol is transmitted.
0054Each transmitter 422 receives and processes its own symbol stream to provide one or more analog signals and further tunes (eg, amplifies, filters, and upconverts) those analog signals. Therefore, it is possible to provide a modulated signal suitable for transmission through a MIMO channel. Then N from transmitters 422a ~ 422t<sub>T</sub>Each of the modulated signals is N<sub>T</sub>It is transmitted from the antennas 424a to 424t.
0055In the receiver system 450, the transmitted modulated signal is N<sub>R</sub>Received by the antennas 452a to 452r, the received signal from each antenna 452 is given to the respective receiver (RCVR) 454a to 454r. The receiver 454 tunes (eg, filters, amplifies, and downconverts) each received signal, digitizes the tuned signal to provide samples, and further processes those samples to accommodate the corresponding " Provides a "receive" symbol stream.
0056The receive (RX) data processor 460 is then N<sub>R</sub>Receivers 454 to N<sub>R</sub>Receives a received symbol stream and processes it based on a particular receiver processing technique, N<sub>T</sub>Provides a "detection" symbol stream. The RX data processor 460 then demodulates, deinterleaves, and decodes each detected symbol stream to restore traffic data about the data stream. The processing by the RX data processor 460 is complementary to the processing performed by the TX MIMO processor 420 and the TX data processor 414 in the transmitter system 410.
0057Processor 470, which can be combined with memory 472, periodically determines which precoding matrix should be used. The reverse link message may contain various types of information about the communication link and / or the received data stream. The reverse link message is then processed by the TX data processor 438, which also receives traffic data for some data streams from the data source 436, modulated by the modulator 480, tuned by transmitters 454a-454r, and transmitted. It is sent back to the machine system 410.
0058In transmitter system 410, the modulated signal from receiver system 450 is received by antenna 424, tuned by receiver 422, demodulated by demodulator 440, processed by RX data processor 442, and received by receiver system 450. The reverse link message sent by is extracted. Processor 430 then determines which precoding matrix should be used to determine the beamforming weights, and then processes the extracted message.
0059FIG. 5A is a block diagram showing an exemplary message forwarding sequence between the source device 520 and the sink device 560 as part of a functional negotiation session. Functional negotiation can occur as part of the larger communication session establishment process between the source device 520 and the sink device 560. This session can be established, for example, by Wi-Fi Direct or TDLS as the underlying connectivity standard. After establishing a Wi-Fi Direct or TDLS session, the sink device 560 can initiate a TCP connection with the source device 520. As part of establishing a TCP connection, a control port is established that runs the real time streaming protocol (RTSP) to manage communication sessions between the source device 520 and the sink device 560. be able to.
0060The source device 520 may generally operate in the same manner as described above for the source device 120 in FIG. 1A, and the sink device 560 may generally operate in the same manner as described above for the sink device 160 in FIG. 1A. Can work. After the source device 520 and sink device 560 establish a connection, the source device 520 and sink device 560 are a set of parameters that should be used for subsequent communication sessions of those devices as part of a functional negotiation exchange. Can be determined.
0061The source device 520 and sink device 560 can negotiate functionality through a sequence of messages. Those messages can be, for example, Real Time Streaming Protocol (RTSP) messages. At any stage of negotiation, the recipient of the RTSP request message can respond with an RTSP response that contains an RTSP status code other than RTSP OK, in which case the message exchange will be retried with a different set of parameters. Can be, or the feature negotiation session can be terminated.
0062The source device 520 can send a first message (RTSP OPTIONS request message) to the sink device 560 to determine the set of RTSP methods supported by the sink device 560. Upon receiving the first message from the source device 520, the sink device 560 can respond with a second message (RTSP OPTIONS response message) listing the RTSP methods supported by the sink 560. The second message may also include the RTSP OK status code.
0063After sending the second message to the source device 520, the sink device 560 can send a third message (RTSP OPTIONS request message) to determine the set of RTSP methods supported by the source device 520. Upon receiving the third message from the sink device 560, the source device 520 can respond with a fourth message (RTSP OPTIONS response message) listing the RTSP methods supported by the source device 520. Also, the fourth message can include the RTSP OK status code.
0064After sending the fourth message, the source device 520 can send a fifth message (RTSP GET_PARAMETER request message) to specify a list of features of interest to the source device 520. The sink device 560 can respond with a sixth message (RTSP GET_PARAMETER response message). The sixth message may contain the RTSP status code. If the RTSP status code is OK, the sixth message can also include response parameters to the parameters specified in the fifth message, supported by sink device 560. The sink device 560 can ignore the parameters in the fifth message that the sink device 560 does not support.
0065Based on the sixth message, the source 520 can determine the optimal set of parameters to be used for the communication session and can send the seventh message (RTSP SET_PARAMETER request message) to the sink device 560. it can. The seventh message may include a set of parameters to be used during the communication session between the source device 520 and the sink device 560. The seventh message is the Universal Resource Identifier (URI) that should be used in the RTSP Setup request to set up the communication session. Can include a wfd-presentation-url that describes the Identifier). wfd-presentation-url specifies a URI that sink device 560 can use for later messages during a session establishment exchange. The wfd-url0 and wfd-url1 values specified in this parameter can correspond to the rtp-port0 and rtp-port1 values in wfd-client-rtp-ports in the seventh message. .. RTP in this case generally refers to a real-time protocol that can run on top of UDP.
0066Upon receiving the seventh message, the sink device 560 can respond with an eighth message with an RTSP status code indicating whether the parameters specified in the seventh message were successfully set. .. As mentioned above, the roles of the source device and the sink device can be reversed or changed in different sessions. The order of messages that set up a communication session can, in some cases, define a device that acts as a source and a device that acts as a sink.
0067FIG. 5B is a block diagram showing another exemplary message forwarding sequence between the source device 560 and the sink device 520 as part of a functional negotiation session. The message forwarding sequence of FIG. 5B is intended to provide a more detailed diagram of the forwarding sequence described above for FIG. 5A. In Figure 5B, the message "1b.GET_PARAMETER response" shows an example of a message that identifies a list of supported input categories (eg, general and HIDC) from a list of supported input types. List of Supported Input Categories Each of the supported input categories has an associated list of supported types (eg generic_cap_list and hidc_cap_list). In Figure 5B, the message "2a.SET_PARAMETER request" identifies a second list of supported input categories (eg, general and HIDC) from a second list of supported types. This is an example of a message. Second List of Supported Input Categories Each of the supported input categories has an associated second list of supported types (eg generic_cap_list and hidc_cap_list). The message "1b.GET_PARAMETER response" identifies the input categories and input types supported by the sink device 560. The message "2a.SET_PARAMETER request" identifies the input categories and input types supported by the source device 520, but it is not a comprehensive list of all input categories and input types supported by the source device 520. Sometimes. Instead, message "2a.SET_PARAMETER request" is assumed to be supported by sink device 560 and message "1b. Only the input categories and input types identified in the GET_PARAMETER response can be identified. In this way, the input categories and input types identified in the message "2a.SET_PARAMETER request" may constitute a subset of the input categories and input types identified in the message "1b.GET_PARAMETER response".
0068FIG. 6 is a conceptual diagram showing an example of a data packet that can be generated by a sink device and transmitted to a source device. Aspects of the data packet 600 will be described with reference to FIG. 1A, but the techniques described may be applicable to additional types of source / sink systems. The data packet 600 may include a data packet header 610 followed by payload data 650. Payload data 650 may further include one or more payload headers (eg, payload header 630). The data packet 600 may be transmitted from the sink device 160 of FIG. 1A to the source device 120, for example, so that the user of the sink device 160 can control the audio / video data transmitted by the source device 120. In such cases, the payload data 650 may include user input data received at the sink device 160. Payload data 650 may identify, for example, one or more user commands. The sink device 160 can receive one or more user commands and can generate data packet headers 610 and payload data 650 based on the received commands. Based on the content of the data packet header 610 of the data packet 600, the source device 120 can parse the payload data 650 to identify the user input data received by the sink device 160. Based on the user input data contained in the payload data 650, the source device 120 may somehow modify the audio and video data transmitted from the source device 120 to the sink device 160.
0069As used in this disclosure, the terms "parse" and "persing" generally refer to the process of analyzing a bitstream to extract data from the bitstream. Once extracted, the data can be processed, for example, by the source device 120. Extracting the data can include, for example, identifying how the information in the bitstream is formatted. As described in more detail below, the data packet header 610 may define a standardized format known to both the source device 120 and the sink device 160. However, the payload data 650 can be formatted in one of many possible ways. By parsing the data packet header 610, the source device 120 can determine how the payload data 650 is formatted, so the source device 120 can be one or more users from the payload data 650. Payload data 650 can be parsed to extract input commands. This can provide flexibility in terms of different types of payload data that may be supported in source sink communication. As described in more detail below, the payload data 650 may also include one or more payload headers, such as the payload header 630. In such a case, the source device 120 parses the data packet header 610 to determine the format for the payload header 630 and then the payload header 630 to determine the format for the rest of the payload data 650. obtain.
0070FIG. 620 is a conceptual diagram of how the data packet header 610 can be formatted. The numbers 0 to 15 in line 615 are intended to identify the bit locations within the data packet header 610 and are not intended to actually represent the information contained within the data packet header 610. The data packet header 610 includes a version field 621, a time stamp flag 622, a reserved field 623, an input category field 624, a length field 625, and an optional time stamp field 626.
0071In the example of FIG. 6, version field 621 is a 3-bit field that may indicate the version of a particular communication protocol implemented by sink device 160. The value in the version field 621 may inform the source device 120 how to parse the rest of the data packet header 610 and how to parse the payload data 650. In the example of Figure 6, version field 621 is a 3-bit field that allows for unique identifiers for eight different versions. In other examples, more or less bits may be dedicated to version field 621.
0072In the example of FIG. 6, the time stamp flag (T) 622 is a 1-bit field indicating whether or not the time stamp field 626 is present in the data packet header 610. Timestamp field 626 is a 16-bit field that contains a time stamp based on multimedia data generated by the source device 120 and sent to the sink device 160. The time stamp can be, for example, a continuous value assigned by the source device 120 to the frame of the video before the frame of the video is transmitted to the sink device 160. The time stamp flag 622 may include, for example, a "1" to indicate that the time stamp field 626 is present, and a "0" to indicate that the time stamp field 626 does not exist. it can. After parsing the data packet header 610 and determining that the time stamp field 626 is present, the source device 120 can process the time stamp contained in the time stamp field 626. If the data packet header 610 is parsed and it is determined that the timestamp field 626 does not exist, the source device 120 will parse the length field 625 and then the payload data 650 because the timestamp field does not exist in the data packet header 610. You can start parsing.
0073If present, the time stamp field 626 can include a time stamp to identify a frame of video data that was displayed on the wireless sink device 160 when the user input data of the payload data 650 was retrieved. The time stamp may be added to the video frame by the source device 120, for example, before the source device 120 sends the video frame to the sink device 160. Therefore, the source device 120 can generate a frame of video and embed a time stamp in the video data of the frame, for example as metadata. The source device 120 can send the video frame to the sink device 160 along with the time stamp, and the sink device 160 can display the frame of the video. The sink device 160 can receive user commands from the user while the frame of the video is displayed by the sink device 160. When the sink device 160 generates a data packet to forward the user command to the source device 120, the sink device 160 time stamps the time stamp of the frame displayed by the sink device 160 when the user command is received. off may be included in the field 626.
0074When the timestamp field 626 receives the data packet 600 present in the header, the wireless source device 120 identifies the frame of video displayed on the sink device 160 when the user input data of the payload data 650 is retrieved. User input data can be processed based on the content of the frame identified by the time stamp. For example, if the user input data is a touch command applied to the touch display or a mouse pointer click, the source device 120 is displayed when the user applies the touch command to the display or clicks the mouse. The content of the frame can be determined. In some cases, frame content may be required to properly process the payload data. For example, user input based on user touch or mouse click can depend on what was shown on the display at the time of touch or click. Touches or clicks may correspond to, for example, icons or menu options. In the case where the content of the display is changing, the time stamp present in the time stamp field 626 can be used by the source device 120 to match the touch or click to the correct icon or menu option.
0075The source device 120 may, additionally or optionally, compare the time stamp in the time stamp field 626 with the time stamp applied to the currently rendered frame of the video. By comparing the time stamp of the time stamp field 626 with the current time stamp, the source device 120 can determine the round trip time. Round-trip time generally corresponds to the amount of time elapsed from the time a frame was transmitted by the source device 120 to the time user input based on that frame was received back from the sink device 160 to the source device 120. The round trip time can give the source device 120 an indication of system latency, and if the round trip time is greater than the threshold, the source device 120 will assume that the input command was applied to the old display frame. Therefore, the user input data contained in the payload data 650 can be ignored. When the round trip time is less than the threshold, the source device 120 may process the user input data and adjust the audio / video content being transmitted in response to the user input data. Thresholds can be programmable and different types of devices (or different source sink combinations) can be configured to define different thresholds for acceptable round trip times.
0076In the example of FIG. 6, reserved field 623 is an 8-bit field that does not contain the information used by source 120 when parsing data packet header 610 and payload data 650. However, a future version of a particular protocol (identified in version field 621) may utilize reserved field 623, in which case the source device 120 is to parse the data packet header 610, and / Alternatively, the information in reserved field 623 may be used to parse payload data 650. Reserved field 623, along with version field 621, potentially provides the ability to extend features and add features to the data packet format without radically changing the format and features already in use.
0077In the example of FIG. 6, the input category field 624 is a 4-bit field for identifying the input category for the user input data contained in the payload data 650. The sink device 160 may categorize user input data to determine the input category. Categorizing user input data can be based, for example, on the device on which the command was received, or on the properties of the command itself. The value of the input category field 624 identifies to the source device 120 how the payload data 650 is formatted, possibly along with other information in the data packet header 610. Based on this formatting, the source device 120 can parse the payload data 650 to determine the user input received by the sink device 160.
0078In the example of FIG. 6, since the input category 624 is 4 bits, it is possible that 16 different input categories can be identified. One such input category is that the user input data in payload data 650 is formatted using the general information elements defined in the protocol running by both the source device 120 and the sink device 160. It can be a general input format to indicate. The general input format may utilize general information elements that allow the user of the sink device 160 to interact with the source device 120 at the application level, as described in more detail below.
0079Another such input category is the human interface device command (to indicate that the user input data in payload data 650 is formatted based on the type of input device used to receive the input data. HIDC: human interface device command) format is possible. Examples of device types include keyboards, mice, touch input devices, joysticks, cameras, gesture capture devices (such as camera-based input devices), and remote control devices. Other types of input categories that can be identified in the input category field 624 are the forwarding input format, or operating system specific format, and payload data to indicate that the user data in the payload data 650 is not exiting the sink device 160. Includes a voice command format to indicate that the 650 contains voice commands.
0080The length field 625 may include a 16-bit field to indicate the length of the data packet 600. The length can be expressed in units of 8 bits, for example. When the data packet 600 is parsed by the source device 120 with a 16-bit word, the data packet 600 can be padded to a 16-bit integer. Based on the length contained in the length field 625, the source device 120 can distinguish between the end of payload data 650 (ie, the end of data packet 600) and the start of a new subsequent data packet.
0081The various sizes of the fields given in the example in Figure 6 are for illustration purposes only, and it is intended that those fields can be implemented using a different number of bits than those shown in Figure 6. Has been done. It is also contemplated that the data packet header 610 may contain fewer fields than all the fields described above, or may use additional fields not described above. In fact, the techniques of the present disclosure can be flexible in terms of the actual format used for the various data fields of the packet.
0082After parsing the data packet header 610 to determine the formatting of the payload data 650, the source device 120 can parse the payload data 650 to determine the user input commands contained in the payload data 650. .. The payload data 650 may have its own payload header (payload header 630) indicating the contents of the payload data 650. In this way, the source device 120 may parse the payload header 630 based on the parsing of the data packet header 610 and then the remaining payload data 650 based on the parsing of the payload header 630.
0083For example, if the input category field 624 of the data packet header 610 indicates that a general input is present in the payload data 650, the payload data 650 may have a general input format. Therefore, the source device 120 can parse the payload data 650 according to the general input format. As part of the general input format, the payload data 650 can include a series of one or more input events, where each input event has its own input event header. Table 1 below identifies the fields that can be included in the input header.<tables num="1"><img id="000011" he="38" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0084The General Input Event (IE) Identification Field identifies the general input event identification information for identifying the input type. The general IE ID field can be, for example, one octet in length and may contain identification information selected from Table 2 below. If the general IE ID field is 8-bit, as in this example, 256 different types of inputs (identified as 0-255) can be identifiable, but not all 256 identities are associated. It does not require the input type given. Some of the above 256 may be reserved for future use with future versions of any protocol implemented by the sink device 160 and the source device 120. In Table 2, for example, general IE IDs 9-255 do not have an associated input type, but may be assigned an input type in the future.
0085The length field in the input event header identifies the length of the descriptive field, and the descriptive field contains an information element that describes the user input. The formatting of the descriptive field can depend on the type of input identified in the general IE ID field. Therefore, the source device 120 may parse the contents of the descriptive field based on the input type identified in the general IE ID field. Based on the length field of the input event header, the source device 120 can determine the end of one input event and the start of a new input event in the payload data 650. As described in more detail below, a user command can be described as one or more input events in payload data 650.
0086Table 2 gives an example of an input type, each with a corresponding general IE ID that can be used to identify the input type.<tables num="2"><img id="000012" he="76" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0087The descriptive fields associated with each input type can have different formats. The left mouse down / touchdown event, left mouse up / touch up event, and mouse move / touch move event description fields may contain, for example, the information elements identified in Table 3 below, but in other examples. Other formats may be used.<tables num="3"><img id="000013" he="108" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0088The number of pointers can identify the number of touches or mouse clicks associated with an input event. Each pointer can have a unique pointer ID. For example, if the multi-touch event contains a three-finger touch, the input event can have three pointers, each with a unique pointer ID. Each pointer (ie, the touch of each finger) can have a corresponding x and y coordinate that corresponds to where the touch was made.
0089A single user command can be described as a series of input events. For example, if a three-finger swipe is a command to close an application, a three-finger swipe uses a touchdown event with three pointers, a touch movement event with three pointers, and a three pointers. It can be described in the payload data 650 as the touchup event that was there. The three pointers for a touchdown event can have the same pointer ID as the three pointers for a touch movement event and a touchup event. The source device 120 can interpret the combination of those three input events as a three-finger swipe.
0090The descriptive fields for key-down or key-up events can contain, for example, the information elements identified in Table 4 below.<tables num="4"><img id="000014" he="69" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0091The zoom event description field may contain, for example, the information elements identified in Table 5 below.<tables num="5"><img id="000015" he="86" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0092The description fields for horizontal or vertical scrolling events may include, for example, the information elements identified in Table 6 below.<tables num="6"><img id="000016" he="48" wi="169" file="JP2016021754A_D0001.tif" img-format="tif" img-content="drawing" /></tables>
0093The above example shows some exemplary ways in which payload data can be formatted in the case of general input categories. Payload data 650 may have different input formats if the input category field 624 of the data packet header 610 indicates a different input category, such as forwarded user input. For forwarded user input, the sink device 160 may receive user input data from a third party device and forward the input to the source device 120 without interpreting the user input data. Therefore, the source device 120 can parse the payload data 650 according to the forwarded user input format. For example, the payload header 630 of the payload data 650 may include a field for the user input to identify a third party device obtained from it. The field may include, for example, the Internet Protocol (IP) address, MAC address, domain name, or any other such identifier of a third party device. The source device 120 can parse the rest of the payload data based on the identifier of the third party device.
0094The sink device 160 can negotiate features with a third party device via a series of messages. The sink device 160 can then send a unique identifier for the third party device to the source device 120 as part of establishing a communication session with the source device 120 as part of the functional negotiation process. Alternatively, the sink device 160 may send information describing the third party device to the source device 120, based on which information the source device 120 can determine a unique identifier for the third party device. .. The information describing the third party device may include, for example, information for identifying the third party device and / or information for identifying the function of the third party device. When the sink device 160 sends a data packet with user input obtained from a third party device, whether the unique identifier is determined by the source device 120 or the sink device 160, the sink device 160 Can include a unique identifier in the data packet, for example in the payload header, so that the source device 120 can identify the origin of the user input.
0095If the input category field 624 of the data packet header 610 indicates a different input category, such as a voice command, the payload data 650 may have a different input format. For voice commands, the payload data 650 may include coded audio. A codec for encoding and decoding the audio of a voice command can be negotiated between the source device 120 and the sink device 160 via a series of messages. To send a voice command, the timestamp field 626 may include a voice sampling time value. In such cases, the time stamp flag 622 may be set to indicate the presence of a time stamp, but instead of the time stamp described above, the time stamp field 626 may be set for the encoded audio of the payload data 650. Can include audio sampling time values of.
0096In some examples, the voice command may be sent as a general command as described above, in which case the input category field 624 may be set to identify the general command format and of the reserved general IE ID. One of them can be assigned to a voice command. If the voice command is sent as a general command, the voice sampling rate can be present in the timestamp field 626 of the data packet header 610 or in the payload data 650.
0097For captured voice command data, the voice data can be encapsulated in multiple ways. For example, voice command data can be encapsulated using RTP, which can provide a payload type to identify the codec and timestamp, and that timestamp is used to identify the sampling rate. .. RTP data can be encapsulated using the general user input format described above with or without optional time stamps. The sink device 160 can use TPC / IP to send general input data that carries voice command data to the source device 120.
0098As described earlier, when coordinates are included as part of a data packet, such as data packet 600, in, for example, payload data 650, the coordinates are negotiated resolution, display window coordinates, normalized coordinates, or It may correspond to coordinates scaled based on the coordinates associated with the sync display. In some cases, additional information may be included in the data packet or transmitted separately for use by the source device to normalize the coordinates received in the data packet.
0099Regardless of the input category for a particular data packet, the data packet header can be an application layer packet header and the data packet can be sent over TCP / IP. TCP / IP may allow sink device 160 and source device 120 to perform retransmission techniques in case of packet loss. Data packets are sent from sink device 160 to source device 120 for other purposes, such as to control audio or video data on source device 120, or to control applications running on source device 120. Can be.
0100FIG. 7A is a flowchart of an exemplary method of negotiating functionality between a sink device and a source device. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more of the steps shown in one or more of the flowcharts described herein. Can store instructions, modules, or algorithms that cause one or more processors (eg, processor 331) to perform.
0101The method of FIG. 7A involves the sink device 160 receiving a first message from the source device 120 (701). The message may include, for example, a get parameter request. In response to the first message, the sink device 160 may send a second message to the source device 120 (703). The second message is, for example, a get parameter that identifies the first list of supported input categories and multiple first lists of supported types. response) can be included and each of the supported input categories in the first list of supported input categories has an associated first list of supported types. The supported input categories may correspond, for example, to the same categories used for the input category field 624 in FIG. Table 2 above represents an example of the supported types for a particular input category (general input in this example). The sink device 160 may receive a third message from the source device 120 (705). The third message is, for example, a set parameter Request) can be included, and the parameterization request identifies the port for communication, the second list of supported input categories, and multiple second lists of supported types, and the supported inputs. Each of the supported input categories in the second list of categories has an associated second list of supported types, and each of the supported types in the second list is in the first list. Contains a subset of types. The sink device 160 may send a fourth message to the source device 120 (707). The fourth message may include, for example, a set parameter response to confirm that the type in the second list has been enabled. The sink device 160 may receive a fifth message from the source device 120 (709). The fifth message may include, for example, a second parameter setting request indicating that the communication channel between the source device 120 and the sink device 160 has been enabled. The communication channel is, for example, a user input back (UIBC). channel) can be provided. The sink device 160 may send a sixth message to the source device 120 (711). The sixth message may include, for example, a second parameter setting response confirming receipt of the second parameter setting request by the sink device 160.
0102FIG. 7B is a flowchart of an exemplary method of negotiating functionality between a sink device and a source device. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the steps shown in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0103The method of FIG. 7B involves the source device 120 sending a first message to the sink device 160 (702). The first message can include, for example, a parameter acquisition request. The source device 120 may receive a second message from the sink device 160 (704). The second message can include, for example, a parameter acquisition response that identifies the first list of supported input categories from multiple first lists of supported types, and the second of the supported input categories. Each of the supported input categories in one list has an associated first list of supported types. The source device 120 may send a third message to the sink device 160 (706). The third message can include, for example, a parameter setting request that identifies the port for communication, the second list of supported input categories, and multiple second lists of supported types. , Each of the supported input categories in the second list of supported input categories has an associated second list of supported types, and each of the supported types in the second list Contains a subset of the types in the first list. The source device 120 may receive a fourth message from the sink device 160 (708). The fourth message can include, for example, a parameter setting response to confirm that the type in the second list has been enabled. The source device 120 may send a fifth message to the sink device 160 (710). The fifth message can include, for example, a second parameter setting request indicating that the communication channel between the source device 120 and the sink device 160 has been enabled. The communication channel may include, for example, a user input back channel (UIBC). The source device 120 may receive a sixth message from the sink device 160 (712). The sixth message is, for example, the sink
0104FIG. 8A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0105The method of FIG. 8A involves acquiring user input data in a wireless sink device such as the wireless sink device 160 (801). User input data can be obtained through the user input component of the wireless sync device 160, for example, the user input interface 376 shown for the wireless sync device 360. Additionally, the sink device 160 may categorize user input data as, for example, general, forwarded, or operating system specific. The sink device 160 may then generate a data packet header based on user input data (803). The data packet header can be an application layer packet header. The data packet header may include, among other fields, a field for identifying an input category corresponding to user input data. The input category may include, for example, a general input format or a human interface device command. The sink device 160 further generates a data packet (805), which contains the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 may then send the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (807). The sink device 160 may include components that allow the transfer of data packets, including, for example, a transport unit 333 and a wireless modem 334 as shown in FIG. The sink device 160 may forward data packets over TCP / IP.
0106FIG. 8B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0107The method of FIG. 8B involves receiving a data packet (802), which may include, in particular, a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. The source device 120 may then parse the data packet header contained in the data packet to determine the input category associated with the user input data contained in the payload data (804). Source device 120 may process payload data based on the determined input category (806). The data packets described with respect to FIGS. 8A and 8B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data and applications in the source device.
0108FIG. 9A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0109The method of FIG. 9A involves acquiring user input data in a wireless sink device such as the wireless sink device 160 (901). User input data can be obtained through the user input component of the wireless sink device 160, for example, the user input interface 376 shown with reference to FIG. The sink device 160 then generates payload data (903), which may describe user input data. In one example, the payload data can include received user input data and can identify one or more user commands. The sink device 160 may further generate a data packet (905), the data packet comprising a data packet header and the generated payload data. The sink device 160 may then transmit the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (907). The sink device 160 may include components that allow the transfer of data packets, such as the transport unit 333 and the wireless modem 334. Data packets can be sent to wireless source devices over TCP / IP.
0110FIG. 9B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0111The method of FIG. 9B involves receiving a data packet from the sink device 360 (902), which may include, in particular, a data packet header and payload data. In one example, the payload data may include data that describes user input details, such as input type values. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. The source device 120 may then parse the data packet to determine the input type value in the input type field in the payload data (904). The source device 120 may process data describing the details of user input based on the determined input type value (906). The data packets described with respect to FIGS. 9A and 9B may generally take the form of the data packets described with respect to FIG.
0112FIG. 10A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0113The method of FIG. 10A involves acquiring user input data (1001) on a wireless sink device such as the wireless sink device 160. User input data can be obtained through the user input component of the wireless sink device 160, for example, the user input interface 376 as shown with reference to FIG. The sink device 160 can then generate a data packet header based on user input (1003). The data packet header may include, among other things, a timestamp flag (eg, a 1-bit field) to indicate whether the timestamp field is present in the data packet header. The time stamp flag can include, for example, a "1" to indicate that the time stamp field exists and a "0" to indicate that the time stamp field does not exist. The time stamp field can be, for example, a 16-bit field containing a time stamp generated by the source device 120 and can be added to the video data prior to transmission. The sink device 160 further generates a data packet (1005), which contains the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 may then transmit the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1007). The sink device 160 may include components that allow the transfer of data packets, including, for example, transport unit 333 and wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
0114FIG. 10B is a flowchart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0115The method of FIG. 10B involves receiving a data packet from the wireless sync device 160 (1002), which may include, in particular, a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. The source device 120 can then parse the data packet header contained in the data packet (1004). The source device 120 may determine if the timestamp field is present in the data packet header (1006). In one example, the source device 120 can make a decision based on the timestamp flag value contained in the data packet header. If the data packet header contains a timestamp field, the source device 120 may process the payload data based on the timestamp in the timestamp field (1008). The data packets described with respect to FIGS. 10A and 10B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0116FIG. 11A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0117The method of FIG. 11A involves acquiring user input data (1101) on a wireless sink device such as the wireless sink device 160. User input data can be obtained through the user input component of the wireless sink device 160, for example, the user input interface 376 shown with reference to FIG. The sink device 160 can then generate a data packet header based on user input (1103). The data packet header can include a time stamp field, among other fields. The time stamp field may include, for example, a 16-bit field containing a time stamp based on multimedia data generated by the wireless source device 120 and transmitted to the wireless sync device 160. The time stamp may have been added to the frame of the video data by the wireless source device 120 before it was sent to the wireless sink device. The time stamp field may identify, for example, the time stamp associated with a frame of video data displayed on the wireless sync device 160 when user input data is captured. The sink device 160 further generates a data packet (1105), and the data packet contains the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 may then transmit the generated data packet to a wireless source device (eg, source device 120 in FIG. 1A or source device 220 in FIG. 2) (1107). The sink device 160 may include components that allow the transfer of data packets, including, for example, transport unit 333 and wireless modem 334, as shown with reference to FIG. Data packets are wireless source device via TCP / IP
0118FIG. 11B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0119The method of FIG. 11B involves receiving a data packet from a wireless sync device such as the wireless sync device 160 (1102), which may specifically include a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. The source device 120 can then identify the timestamp field in the data packet header (1104). Source device 120 may process payload data based on the time stamp in the time stamp field (1106). Based on the time stamp described above, as part of processing the payload data, the source device 120 identifies a frame of video data that is displayed on the wireless sync device when the user input data is retrieved and of that frame. Payload data can be interpreted based on the content. As part of processing the payload data based on the time stamp, the source device 120 can compare the above time stamp with the current time stamp for the current frame of the video being transmitted by the source device 120. , Execute the user input command described in the payload data in response that the time difference between the above time stamp and the current time stamp is less than the threshold, or, the above time stamp and the current time stamp In response to the time difference between and being greater than the threshold, the user input command described in the payload data can not be executed. The data packets described with respect to FIGS. 11A and 11B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0120FIG. 12A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0121The method of FIG. 12A involves acquiring user input data (1201) on a wireless sink device such as the wireless sink device 160. In one example, the user input data can be voice command data, which is obtained through the user input component of the wireless sync device 160, for example, the voice command recognition module included in the user input interface 376 in FIG. obtain. The sink device 160 may generate a data packet header based on user input (1203). The sink device 160 can also generate payload data (1205), which may include voice command data. In one example, the payload data may also include received user input data to identify one or more user commands. The sink device 160 further generates a data packet (1207), which comprises the generated data packet header and payload data. The sink device 160 may then transmit the generated data packet to a wireless source device (eg, source device 120 in FIG. 1A or source device 220 in FIG. 2) (1209). The sink device 160 may include components that allow the transfer of data packets, including, for example, transport unit 333 and wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
0122FIG. 12B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0123The method of FIG. 12B involves receiving a data packet (1202), which may include, in particular, a data packet header and payload data. The payload data may include user input data such as voice command data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. The source device 120 may then parse the payload data contained in the data packet to determine if the payload data contains voice command data (1204). The data packets described with respect to FIGS. 12A and 12B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0124FIG. 13A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0125The method of FIG. 13A involves acquiring user input data (1301) in a wireless sink device such as the wireless sink device 160. In one example, the user input data can be a multi-touch gesture, which can be obtained through a user input component of the wireless sync device 160, for example UI 167 or user input interface 376 in FIG. In one example, a multi-touch gesture may include a first touch input and a second touch input. The sink device 160 may generate a data packet header based on user input (1303). The sink device 160 may also generate payload data (1305), where the payload data associates the user input data for the first touch input event with the first pointer identification information and for the second touch input event. User input data can be associated with the second pointer identification information. The sink device 160 may further generate a data packet (1307), which contains the generated data packet header and payload data. The sink device 160 may then send the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1309). The sink device 160 may include components that allow the transfer of data packets, including, for example, transport unit 333 and wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
0126FIG. 13B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0127The method of FIG. 13B involves receiving a data packet (1302), which may include, in particular, a data packet header and payload data. The payload data may include user input data such as, for example, multi-touch gestures. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown in FIG. The source device 120 may then parse the payload data contained in the data packet to identify the user input data contained in the payload data (1304). In one example, the identified data is the user input data for the first touch input event using the first pointer identification information and the user input for the second touch input event using the second pointer identification information. May include data. The source device 120 can then interpret the user input data for the first touch input event and the user input data for the second touch input event as multi-touch gestures (1306). The data packets described with respect to FIGS. 13A and 13B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0128FIG. 14A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0129The method of FIG. 14A involves retrieving user input data from an external device on the wireless sink device 360 (1401). In one example, the external device can be a third-party device connected to the sink device. The sink device 160 may generate a data packet header based on user input (1403). In one example, the data packet header may identify the user input data as forwarded user input data. The sink device 160 also generates payload data (1405), which may include user input data. The sink device 160 may further generate a data packet (1407), which may include the generated data packet header and payload data. The sink device 160 may then transmit the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1409). The sink device 160 may include components that allow the transfer of data packets, including, for example, transport unit 333 and wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
0130FIG. 14B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0131The method of FIG. 14B involves receiving a data packet (1402), which may include, in particular, a data packet header and payload data. The payload data may include user input data, for example, a forwarded user input command indicating that the user input data has been forwarded from a third party device. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown with reference to FIG. Source device 120 may then parse the data packet header and determine that the payload data contains forwarded user input commands (1404). The source device 120 may then parse the payload data contained in the data packet to identify the identity associated with the third party device corresponding to the forwarded user input command (1406). The source device 120 may then process the payload data based on the identified identity of the third party device (1408). The data packets described with respect to FIGS. 14A and 14B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0132FIG. 15A is a flowchart of an exemplary method of transmitting user data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. , Processor 331), can store instructions, modules, or algorithms.
0133The method of FIG. 15A involves acquiring user input data (1501) in a wireless sink device. User input data may have associated coordinate data. The associated coordinate data may correspond to, for example, the location of a mouse click event or the location of a touch event. The sink device 160 can then normalize the associated coordinate data to generate the normalized coordinate data (1503). The sink device 160 can then generate a data packet containing normalized coordinate data (1505). Normalizing the coordinate data can include scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source display, such as display 22 of the source device 120. The resolution of the display window can be determined by the sink device 160, and the display resolution of the source device can be received from the source device 120. The sink device 160 may then send a data packet with normalized coordinates to the wireless source device 120 (1507). As part of the method in Figure 15A, the sync device 160 also determines if the associated coordinate data is in the display window for the content being received from the wireless source device, eg, associated. If the coordinate data is outside the display window, the user input can be processed locally, and if the input is inside the display window, the coordinates can be normalized as described above.
0134FIG. 15B is a flow chart of an exemplary method of receiving user input data from a wireless sink device in a wireless source device according to the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. , Processor 231) may store instructions, modules, or algorithms.
0135The method of FIG. 15B involves receiving a data packet on a wireless source device, which comprises user input data with associated coordinate data (1502). The associated coordinate data may correspond to, for example, the location of a mouse click event or the location of a touch event on the sink device. Source device 120 can then normalize the associated coordinate data to generate normalized coordinate data (1504). The source device 120 can normalize the coordinate data by scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source display. The source device 120 can determine the display resolution of the source device and can receive the resolution of the display window from the wireless sink device. The source device can then process the data packet based on the normalized coordinate data (1506). The data packets described with respect to FIGS. 15A and 15B can generally take the form of the data packets described with respect to FIG. 6 and can be used to control audio / video data in the source device.
0136For the sake of brevity, aspects of the present disclosure have been described separately with reference to FIGS. 7-15. However, it is contemplated that these various aspects may be combined and used in relation to each other, not just separately. In general, the features and / or modules described herein can be implemented in either or both wireless source devices and wireless sink devices. In this way, the user interface features described in this example can be used interchangeably between wireless source devices and wireless sink devices.
0137The techniques of the present disclosure can be implemented in a wide variety of devices or devices, including wireless handsets and integrated circuits (ICs) or sets of ICs (ie, chipsets). Although any component, module or unit has been described and presented to emphasize functional aspects, any of these components, modules or units do not necessarily require implementation by different hardware units.
0138Therefore, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the features described as modules, units or components can be implemented together in an integrated logical device or separately as a separate but interoperable logical device. When implemented in software, these techniques may be at least partially realized by a computer-readable medium with instructions that, when executed on a processor, performs one or more of the methods described above. The computer-readable medium may comprise a tangible, non-transitory computer-readable storage medium and may form part of a computer program product that may contain packaging material. Computer-readable storage media include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable read-only memory (EEPROM). ), Flash memory, magnetic or optical data storage medium, etc. The technique may, in addition or as an alternative, be at least partially realized by a computer-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.
0139The code is one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrations. It can be performed by a circuit or a discrete logic circuit. Thus, the term "processor" as used herein may refer to either the structure described above or any other structure suitable for implementing the techniques described herein. Moreover, in some embodiments, the functionality described herein may be provided within a dedicated software or hardware module configured for encoding and decoding, or may be incorporated into a composite video codec. .. Also, the technique may be fully implemented in one or more circuits or logic elements.
0140Various aspects of the present disclosure have been described. These and other aspects are within the scope of the following claims.
41 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| JP2001282673A | Cites | Japan | Search report |
| JP2001282673A | Cites | Japan | Search report |
| JP2005108211A | Cites | Japan | Search report |
| JP2005108211A | Cites | Japan | Search report |
| JP2006172423A | Cites | Japan | Search report |
| WO2010062617A1 | Cites | World Intellectual Property Organization (WIPO) | Search report |
| WO2011003897A2 | Cites | World Intellectual Property Organization (WIPO) | Search report |
| WO2011003897A2 | Cites | World Intellectual Property Organization (WIPO) | Search report |
168 members in 20 offices
Priority claims18
| Document | Office | Kind | Date |
|---|---|---|---|
| 61435194 | United States of America | – | |
| 201161435194 | United States of America | P | |
| 61447592 | United States of America | – | |
| 201161447592 | United States of America | P | |
| 61448312 | United States of America | – | |
| 201161448312 | United States of America | P | |
| 61450101 | United States of America | – | |
| 201161450101 | United States of America | P | |
| 61467535 | United States of America | – | |
| 61467543 | United States of America | – | |
| 201161467535 | United States of America | P | |
| 201161467543 | United States of America | P | |
| 61514863 | United States of America | – | |
| 201161514863 | United States of America | P | |
| 61544440 | United States of America | – | |
| 201161544440 | United States of America | P | |
| 13344424 | United States of America | – | |
| 201213344424 | United States of America | A |
Members168
| Document | Office | Kind | |
|---|---|---|---|
| CA2824287A1 | Canada | A1 | |
| CA2824559A1 | Canada | A1 | |
| CA2824563A1 | Canada | A1 | |
| CA2824567A1 | Canada | A1 | |
| WO2012100186A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100189A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100191A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100193A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100197A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100201A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100204A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100214A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100218A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2013002949A1 | United States of America | A1 | |
| US2013003621A1 | United States of America | A1 | |
| US2013003622A1 | United States of America | A1 | |
| US2013003623A1 | United States of America | A1 | |
| US2013003624A1 | United States of America | A1 | |
| US2013009873A1 | United States of America | A1 | |
| US2013009887A1 | United States of America | A1 | |
| US2013009996A1 | United States of America | A1 | |
| US2013013318A1 | United States of America | A1 | |
| AU2012207073A1 | Australia | A1 | |
| AU2012207127A1 | Australia | A1 | |
| AU2012207129A1 | Australia | A1 | |
| AU2012207133A1 | Australia | A1 | |
| PH12013501451A1 | Philippines | A1 | |
| PH12013501482A1 | Philippines | A1 | |
| PH12013501483A1 | Philippines | A1 | |
| SG191367A1 | Singapore | A1 | |
| SG191377A1 | Singapore | A1 | |
| SG191763A1 | Singapore | A1 | |
| SG191765A1 | Singapore | A1 | |
| KR20130115370A | Republic of Korea | A | |
| KR20130115371A | Republic of Korea | A | |
| KR20130118958A | Republic of Korea | A | |
| CN103384995A | China | A | |
| CN103392160A | China | A | |
| CN103392161A | China | A | |
| CN103392325A | China | A | |
| CN103392326A | China | A | |
| CN103392359A | China | A | |
| CN103403649A | China | A | |
| CN103404104A | China | A | |
| CN103404114A | China | A | |
| KR20130126968A | Republic of Korea | A | |
| KR20130126969A | Republic of Korea | A | |
| KR20130126970A | Republic of Korea | A | |
| KR20130126971A | Republic of Korea | A | |
| KR20130126972A | Republic of Korea | A | |
| KR20130126973A | Republic of Korea | A | |
| EP2666069A1 | European Patent Office (EPO) | A1 | |
| EP2666073A1 | European Patent Office (EPO) | A1 | |
| EP2666074A1 | European Patent Office (EPO) | A1 | |
| EP2666274A1 | European Patent Office (EPO) | A1 | |
| EP2666275A1 | European Patent Office (EPO) | A1 | |
| EP2666276A1 | European Patent Office (EPO) | A1 | |
| EP2666277A1 | European Patent Office (EPO) | A1 | |
| EP2666278A1 | European Patent Office (EPO) | A1 | |
| EP2666323A1 | European Patent Office (EPO) | A1 | |
| JP2014506082A | Japan | A | |
| US8677029B2 | United States of America | B2 | |
| JP2014508995A | Japan | A | |
| HK1188005A | Hong Kong, China | A | |
| HK1188005A1 | Hong Kong, China | A1 | |
| JP2014509475A | Japan | A | |
| JP2014509476A | Japan | A | |
| JP2014510434A | Japan | A | |
| JP2014510435A | Japan | A | |
| ZA201305997B | South Africa | B | |
| JP2014510961A | Japan | A | |
| JP2014511582A | Japan | A | |
| JP2014511583A | Japan | A | |
| ZA201306271B | South Africa | B | |
| UA107151C2 | Ukraine | C2 | |
| US8964783B2 | United States of America | B2 | |
| RU2013138718A | Russian Federation | A | |
| RU2013138723A | Russian Federation | A | |
| RU2013138748A | Russian Federation | A | |
| RU2013138750A | Russian Federation | A | |
| KR101503386B1 | Republic of Korea | B1 | |
| ZA201305995B | South Africa | B | |
| JP5694568B2 | Japan | B2 | |
| AU2012207073B2 | Australia | B2 | |
| JP5714726B2 | Japan | B2 | |
| AU2012207127B2 | Australia | B2 | |
| US9065876B2 | United States of America | B2 | |
| KR101533753B1 | Republic of Korea | B1 | |
| UA109176C2 | Ukraine | C2 | |
| AU2012207129B2 | Australia | B2 | |
| AU2012207133B2 | Australia | B2 | |
| UA109928C2 | Ukraine | C2 | |
| RU2567378C2 | Russian Federation | C2 | |
| JP5815741B2 | Japan | B2 | |
| ZA201404660B | South Africa | B | |
| JP5826860B2 | Japan | B2 | |
| JP5826861B2 | Japan | B2 | |
| JP2015222953A | Japan | A | |
| KR101572977B1 | Republic of Korea | B1 | |
| RU2571595C2 | Russian Federation | C2 |
6 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Decision of refusalJAPANESE INTERMEDIATE CODE: A02A02 | A02 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 |
Numbers
- Publication
- 2016021754
- Application
- 156158
Titles2
- Japanese
- ワイヤレスディスプレイのためのユーザ入力バックチャネル
- English
- User input back channel for wireless display
Classification
- CPC, 11
- H04M1/72412
- H04L65/756
- H04M2250/06
- H04L65/65
- G06F3/041
- G06F3/14
- G06F2203/04104
- G09G2370/16
- G09G2370/10
- H04L65/613
- H04W84/12
- IPC, 2
- H04N21 437
- H04L47 43