User input back channel for wireless displays
Abstract
As part of a communication session, the wireless source device can send audio and video data to the wireless sync device, and the wireless sync device sends the user input data received by the wireless sync device to the wireless source device. Can be done. In this way, the user of the wireless sync device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sync device. The user input data transmitted by the wireless sync device can be input data captured by a third party device and forwarded to the wireless source device.

Term
5.3 yearsto projected expiry
Projected expiry 20 January 2032, counted from filing; an application has no term until it is granted.
- Priority
- Filed
- Published
- Today
- Projected expiry
54 claims: 13 independent, 41 dependent
- 1ワイヤレスシンクデバイスからワイヤレスソースデバイスにユーザ入力データを送信する方法であって、前記方法は、 外部デバイスからユーザ入力データを取得することと、 データパケットヘッダを生成することであって、前記データパケットヘッダが、前記ユーザ入力データをフォワーディングされたユーザ入力データとして識別する、データパケットヘッダを生成することと、 前記ユーザ入力データを備えるペイロードデータを生成することと、 前記データパケットヘッダと前記ペイロードデータとを備えるデータパケットを生成することと、 前記ワイヤレスソースデバイスに前記データパケットを送信することとを備える、方法。
- 2一連のメッセージを介して、前記ワイヤレスシンクデバイスと、サードパーティデバイスの機能をネゴシエートすることをさらに備える、請求項1に記載の方法。
- 3前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスシンクデバイスから前記ワイヤレスソースデバイスに前記サードパーティデバイスの識別子を送信することをさらに備える、請求項1に記載の方法。
- 4前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスから前記サードパーティデバイスの識別子を受信することをさらに備える、請求項1に記載の方法。
- 5前記データパケットヘッダが、ユーザ入力カテゴリーを識別するためのフィールドを備え、前記フィールドの値は、前記ペイロードデータが前記フォワーディングされたユーザ入力データを備えることを示すように設定される、請求項1に記載の方法。
- 6前記ペイロードデータが前記サードパーティデバイスの識別子を備える、請求項1に記載の方法。
- 7前記サードパーティデバイスの前記識別子が、前記サードパーティデバイスのIPアドレスと、前記サードパーティデバイスのドメイン名とからなるグループから選択される、請求項1に記載の方法。
- 8前記識別子が、前記ワイヤレスソースデバイスによって生成され、前記ワイヤレスシンクデバイスに送信される、請求項1に記載の方法。
- 9前記外部デバイスが別のワイヤレスシンクデバイスである、請求項1に記載の方法。
- 10前記外部デバイスが、前記ワイヤレスシンクデバイスに通信可能に結合された入力デバイスである、請求項1に記載の方法。
- 11前記データパケットヘッダがアプリケーションレイヤパケットヘッダである、請求項1に記載の方法。
- 12前記データパケットが、前記ワイヤレスソースデバイスのオーディオデータまたはビデオデータを制御するためのものである、請求項1に記載の方法。
- 13前記データパケットがTCP/IPを介して送信される、請求項1に記載の方法。
- 14ワイヤレスソースデバイスにユーザ入力データを送信するように構成されたワイヤレスシンクデバイスであって、前記ワイヤレスシンクデバイスは、 命令を記憶するメモリと、 前記命令を実行するように構成された1つまたは複数のプロセッサであって、前記命令の実行時に、前記1つまたは複数のプロセッサが、 外部デバイスからユーザ入力データを取得することと、 データパケットヘッダを生成することであって、前記データパケットヘッダが、前記ユーザ入力データをフォワーディングされたユーザ入力データとして識別する、データパケットヘッダを生成することと、 前記ユーザ入力データを備えるペイロードデータを生成することと、 前記データパケットヘッダと前記ペイロードデータとを備えるデータパケットを生成することとを行わせる、1つまたは複数のプロセッサと、 前記ワイヤレスソースデバイスに前記データパケットを送信するためのトランスポートユニットとを備える、ワイヤレスシンクデバイス。
- 15前記命令の実行時に、前記1つまたは複数のプロセッサが、 一連のメッセージを介して、前記ワイヤレスシンクデバイスと、サードパーティデバイスの機能をネゴシエートすることをさらに行わせる、請求項14に記載のワイヤレスシンクデバイス。
- 16前記命令の実行時に、前記1つまたは複数のプロセッサが、 前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスシンクデバイスから前記ワイヤレスソースデバイスに前記サードパーティデバイスの識別子を送信することをさらに行わせる、請求項14に記載のワイヤレスシンクデバイス。
- 17前記命令の実行時に、前記1つまたは複数のプロセッサが、 前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスから前記サードパーティデバイスの識別子を受信することをさらに行わせる、請求項14に記載のワイヤレスシンクデバイス。
- 18前記データパケットヘッダが、ユーザ入力カテゴリーを識別するためのフィールドを備え、前記フィールドの値は、前記ペイロードデータが前記フォワーディングされたユーザ入力データを備えることを示すように設定される、請求項14に記載のワイヤレスシンクデバイス。
- 19前記ペイロードデータが前記サードパーティデバイスの識別子を備える、請求項14に記載のワイヤレスシンクデバイス。
- 20前記サードパーティデバイスの前記識別子が、前記サードパーティデバイスのIPアドレスと、前記サードパーティデバイスのドメイン名とからなるグループから選択される、請求項14に記載のワイヤレスシンクデバイス。
- 21前記識別子が、前記ワイヤレスソースデバイスによって生成され、前記ワイヤレスシンクデバイスに送信される、請求項14に記載のワイヤレスシンクデバイス。
- 22前記外部デバイスが別のワイヤレスシンクデバイスである、請求項14に記載のワイヤレスシンクデバイス。
- 23前記外部デバイスが、前記ワイヤレスシンクデバイスに通信可能に結合された入力デバイスである、請求項14に記載のワイヤレスシンクデバイス。
- 24前記データパケットヘッダがアプリケーションレイヤパケットヘッダである、請求項14に記載のワイヤレスシンクデバイス。
- 25前記データパケットが、前記ワイヤレスソースデバイスのオーディオデータまたはビデオデータを制御するためのものである、請求項14に記載のワイヤレスシンクデバイス。
- 26前記データパケットがTCP/IPを介して送信される、請求項14に記載のワイヤレスシンクデバイス。
- 271つまたは複数のプロセッサによって実行されると、ワイヤレスシンクデバイスからワイヤレスソースデバイスにユーザ入力データを送信する方法を実行することを前記1つまたは複数のプロセッサに行わせる、命令を記憶するコンピュータ可読記憶媒体であって、前記方法は、 外部デバイスからユーザ入力データを取得することと、 データパケットヘッダを生成することであって、前記データパケットヘッダが、前記ユーザ入力データをフォワーディングされたユーザ入力データとして識別する、データパケットヘッダを生成することと、 前記ユーザ入力データを備えるペイロードデータを生成することと、 前記データパケットヘッダと前記ペイロードデータとを備えるデータパケットを生成することと、 前記ワイヤレスソースデバイスに前記データパケットを送信することとを備える、コンピュータ可読記憶媒体。
- 28ワイヤレスソースデバイスにユーザ入力データを送信するように構成されたワイヤレスシンクデバイスであって、前記ワイヤレスシンクデバイスは、 外部デバイスからユーザ入力データを取得するための手段と、 データパケットヘッダを生成するための手段であって、前記データパケットヘッダが、前記ユーザ入力データをフォワーディングされたユーザ入力データとして識別する、データパケットヘッダを生成するための手段と、 前記ユーザ入力データを備えるペイロードデータを生成するための手段と、 前記データパケットヘッダと前記ペイロードデータとを備えるデータパケットを生成するための手段と、 前記ワイヤレスソースデバイスに前記データパケットを送信するための手段とを備える、ワイヤレスシンクデバイス。
- 29ワイヤレスソースデバイスにおいてワイヤレスシンクデバイスからユーザ入力データを受信する方法であって、前記方法は、 ワイヤレスシンクデバイスからデータパケットヘッダとペイロードデータとを備えるデータパケットを受信することと、 前記ペイロードデータがフォワーディングされたユーザ入力コマンドを備えることを判断するために前記データパケットヘッダをパースすることと、 サードパーティデバイスの識別情報を識別するために前記ペイロードデータをパースすることと、 前記サードパーティデバイスの前記識別情報に基づいて前記ペイロードデータを処理することとを備える、方法。
- 30一連のメッセージを介して、前記ワイヤレスソースデバイスと、前記サードパーティデバイスの機能をネゴシエートすることをさらに備える、請求項29に記載の方法。
- 31前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスに前記サードパーティデバイスの識別子を送信することをさらに備える、請求項29に記載の方法。
- 32前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスから前記サードパーティデバイスの識別子を受信することをさらに備える、請求項29に記載の方法。
- 33前記データパケットヘッダが、ユーザ入力カテゴリーを識別するためのフィールドを備え、前記フィールドの値は、前記ペイロードデータが前記フォワーディングされたユーザ入力データを備えることを示すように設定される、請求項29に記載の方法。
- 34前記サードパーティデバイスの前記識別子が、前記サードパーティデバイスのIPアドレスと、前記サードパーティデバイスのドメイン名とからなるグループから選択される、請求項29に記載の方法。
- 35前記識別子が、前記ワイヤレスソースデバイスによって生成され、前記ワイヤレスシンクデバイスに送信される、請求項29に記載の方法。
- 36外部デバイスが別のワイヤレスシンクデバイスである、請求項29に記載の方法。
- 37前記外部デバイスが、前記ワイヤレスシンクデバイスに通信可能に結合された入力デバイスである、請求項29に記載の方法。
- 38前記データパケットヘッダがアプリケーションレイヤパケットヘッダである、請求項29に記載の方法。
- 39前記データパケットが、前記ワイヤレスソースデバイスのオーディオデータまたはビデオデータを制御するためのものである、請求項29に記載の方法。
- 40前記データパケットがTCP/IPを介して送信される、請求項29に記載の方法。
- 41ワイヤレスシンクデバイスからユーザ入力データを受信するように構成されたワイヤレスソースデバイスであって、前記ワイヤレスソースデバイスは、 ワイヤレスシンクデバイスからデータパケットヘッダとペイロードデータとを備えるデータパケットを受信するように構成されたトランスポートユニットと、 命令を記憶するメモリと、 前記命令を実行するように構成された1つまたは複数のプロセッサであって、前記命令の実行時に、前記1つまたは複数のプロセッサが、 前記ペイロードデータがフォワーディングされたユーザ入力コマンドを備えることを判断するために前記データパケットヘッダをパースすることと、 サードパーティデバイスの識別情報を識別するために前記ペイロードデータをパースすることと、 前記サードパーティデバイスの前記識別情報に基づいて前記ペイロードデータを処理することとを行わせる、1つまたは複数のプロセッサとを備える、ワイヤレスソースデバイス。
- 42前記命令の実行時に、前記1つまたは複数のプロセッサが、 一連のメッセージを介して、前記ワイヤレスソースデバイスと、前記サードパーティデバイスの機能をネゴシエートすることを行わせる、請求項41に記載のワイヤレスソースデバイス。
- 43前記命令の実行時に、前記1つまたは複数のプロセッサが、 前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスに対する前記サードパーティデバイスの識別子を受信することを行わせる、請求項41に記載のワイヤレスソースデバイス。
- 44前記命令の実行時に、前記1つまたは複数のプロセッサが、 前記ワイヤレスソースデバイスと前記ワイヤレスシンクデバイスとの間の通信セッションを確立することの一部として、前記ワイヤレスソースデバイスから前記サードパーティデバイスの識別子を受信することをさらに行わせる、請求項41に記載のワイヤレスシンクデバイス。
- 45前記データパケットヘッダが、ユーザ入力カテゴリーを識別するためのフィールドを備え、前記フィールドの値は、前記ペイロードデータが前記フォワーディングされたユーザ入力データを備えることを示すように設定される、請求項41に記載のワイヤレスソースデバイス。
- 46前記サードパーティデバイスの前記識別子が、前記サードパーティデバイスのIPアドレスと、前記サードパーティデバイスのドメイン名とからなるグループから選択される、請求項41に記載のワイヤレスソースデバイス。
- 47前記識別子が、前記ワイヤレスソースデバイスによって生成され、前記ワイヤレスシンクデバイスに送信される、請求項41に記載のワイヤレスソースデバイス。
- 48外部デバイスが別のワイヤレスシンクデバイスである、請求項41に記載のワイヤレスソースデバイス。
- 49前記外部デバイスが、前記ワイヤレスシンクデバイスに通信可能に結合された入力デバイスである、請求項41に記載のワイヤレスソースデバイス。
- 50前記データパケットヘッダがアプリケーションレイヤパケットヘッダである、請求項41に記載のワイヤレスソースデバイス。
- 51前記データパケットが、前記ワイヤレスソースデバイスのオーディオデータまたはビデオデータを制御するためのものである、請求項41に記載のワイヤレスソースデバイス。
- 52前記データパケットがTCP/IPを介して送信される、請求項41に記載のワイヤレスソースデバイス。
- 531つまたは複数のプロセッサによって実行されると、ワイヤレスソースデバイスにおいてワイヤレスシンクデバイスからユーザ入力データを受信する方法を実行することを前記1つまたは複数のプロセッサに行わせる、命令を記憶するコンピュータ可読記憶媒体であって、前記方法は、 ワイヤレスシンクデバイスからデータパケットヘッダとペイロードデータとを備えるデータパケットを受信することと、 前記ペイロードデータがフォワーディングされたユーザ入力コマンドを備えることを判断するために前記データパケットヘッダをパースすることと、 サードパーティデバイスの識別情報を識別するために前記ペイロードデータをパースすることと、 前記サードパーティデバイスの前記識別情報に基づいて前記ペイロードデータを処理することとを備える、コンピュータ可読記憶媒体。
- 54ワイヤレスシンクデバイスからユーザ入力データを受信するように構成されたワイヤレスソースデバイスであって、前記ワイヤレスソースデバイスは、 ワイヤレスシンクデバイスからデータパケットヘッダとペイロードデータとを備えるデータパケットを受信するための手段と、 前記ペイロードデータがフォワーディングされたユーザ入力コマンドを備えることを判断するために前記データパケットヘッダをパースするための手段と、 サードパーティデバイスの識別情報を識別するために前記ペイロードデータをパースするための手段と、 前記サードパーティデバイスの前記識別情報に基づいて前記ペイロードデータを処理するための手段とを備える、ワイヤレスソースデバイス。
Independent claims54
140 paragraphs, as filed
This application is incorporated herein by reference in its entirety, US Provisional Application Nos. 61 / 435,194 filed on January 21, 2011, and US Provisional Application filed on February 28, 2011. 61 / 447,592, US Provisional Application No. 61 / 448,312 filed March 2, 2011, US Provisional Application No. 61 / 450,101 filed March 7, 2011, March 25, 2011 US Provisional Application No. 61 / 467,535 filed in, US Provisional Application No. 61 / 467,543 filed on March 25, 2011, US Provisional Application No. 61 / 514,863 filed on August 3, 2011. , And claims the interests of US Provisional Application No. 61 / 544,470 filed on October 7, 2011.
The present disclosure relates to techniques for transmitting data between a wireless source device and a wireless sync device.
A wireless display (WD) or Wi-Fi Display (WFD) system includes a wireless source device and one or more wireless sink devices. Each of the source device and the sink device can be either a mobile device or a wired device having a wireless communication function. One or more of the source and sink devices are, for example, mobile phones, portable computers with wireless communication cards, personal digital assistants (PDAs), portable media players, or so-called "smart" phones and "smart" pads. Alternatively, it may include other such devices with wireless communication capabilities, including tablets, electronic readers, or any type of wireless display, video game device, or other types of wireless communication devices. One or more of the source and sink devices may also include wired devices such as televisions, desktop computers, monitors, projectors, etc. that include communication capabilities.
The source device sends media data, such as audio-video (AV) data, to one or more of the sink devices participating in a particular media sharing session. Media data can be played on both the local display of the source device and each of the displays of the sink device. More specifically, each of the participating sync devices renders the received media data on its screen and audio equipment.
The present disclosure generally describes a system in which a wireless sync device can communicate with the wireless sync device. As part of a communication session, the wireless source device can send audio and video data to the wireless sync device, which can send the user input received by the wireless sync device to the wireless source device. it can. In this way, the user of the wireless sync device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sync device.
In one example, the method of sending user input data from a wireless sink device to a wireless source device is to obtain user input data from an external device and generate a data packet header, where the data packet header is the user input. Generating a data packet header that identifies the data as forwarded user input data, generating payload data with user input data, and generating a data packet with a data packet header and payload data. Includes sending data packets to wireless source devices.
In another example, the wireless sync device is configured to send user input data to the wireless source device. The wireless sync device is a memory that stores instructions and one or more processors configured to execute the instructions, and when the instructions are executed, the one or more processors input user data from an external device. Acquiring data and generating a data packet header, the data packet header identifies the user input data as forwarded user input data, generates a data packet header, and generates the user input data. Includes one or more processors that allow the generation of payload data to be provided and the generation of data packets with data packet headers and payload data. The wireless sync device also includes a transport unit for sending data packets to wireless source devices.
In another example, when a computer-readable storage medium is run by one or more processors, it performs the method of sending user input data from a wireless sink device to a wireless source device to one or more processors. Memorize the instructions to be made. The above method is to acquire user input data from an external device and generate a data packet header, wherein the data packet header identifies the user input data as forwarded user input data. It includes generating, generating payload data with user input data, generating a data packet with a data packet header and payload data, and sending the data packet to a wireless source device.
In another example, the wireless sync device is configured to send user input data to the wireless source device. The wireless sync device is a means for acquiring user input data from an external device and a means for generating a data packet header, and the data packet header identifies the user input data as forwarded user input data. Means for generating data packet headers, means for generating payload data with user input data, means for generating data packets with data packet headers and payload data, and wireless source devices. Includes means for sending data packets to.
In another example, the method of receiving user input data from a wireless sync device on a wireless source device is to receive a data packet with a data packet header and payload data from the wireless sync device and the user to whom the payload data has been forwarded. Parsing the data packet header to determine that it has an input command, parsing the payload data to identify the identity of the third-party device, and parsing the payload data based on the identity of the third-party device. Includes processing.
In another example, the wireless source device is configured to receive user input data from the wireless sync device. The wireless source device includes a transport unit configured to receive a data packet with a data packet header and payload data from a wireless sink device. The wireless source device is a memory that stores instructions and one or more processors that are configured to execute the instructions, and when the instructions are executed, one or more processors forward the payload data. Parsing the data packet header to determine that it has a user input command, parsing the payload data to identify the identity of the third-party device, and the payload based on the identity of the third-party device. It also includes one or more processors that allow it to process and do data.
In another example, when a computer-readable storage medium is run by one or more processors, the wireless source device performs the method of receiving user input data from a wireless sink device on one or more processors. Memorize the instructions to be made. The above method is to receive a data packet with a data packet header and payload data from a wireless sink device and to parse the data packet header to determine that the payload data has forwarded user input commands. Includes parsing payload data to identify third-party device identities and processing payload data based on third-party device identities.
In another example, the wireless source device is configured to receive user input data from the wireless sync device. The wireless source device provides a means for receiving a data packet with a data packet header and payload data from a wireless sink device and a data packet header to determine that the payload data comprises a forwarded user input command. It includes means for parsing, means for parsing the payload data to identify the identity of the third party device, and means for processing the payload data based on the identity of the third party device.
<figref num="1A">The block diagram which shows an example of the source / sink system which can implement the technique of this disclosure.</figref><figref num="1B">A block diagram showing an example of a source / sync system with two sync devices.</figref><figref num="2">The figure which shows an example of the source device which can implement the technique of this disclosure.</figref><figref num="3">The figure which shows an example of the sink device which can implement the technique of this disclosure.</figref><figref num="4">The block diagram of the transmitter system and the receiver system which can implement the technique of this disclosure.</figref><figref num="5A">The figure which shows the exemplary message transfer sequence for performing functional negotiation according to the technique of this disclosure.</figref><figref num="5B">The figure which shows the exemplary message transfer sequence for performing functional negotiation according to the technique of this disclosure.</figref><figref num="6">The figure which shows the exemplary data packet which can be used to deliver the user input data acquired in a sink device to a source device.</figref><figref num="7A">A flowchart illustrating the techniques of the present disclosure that can be used for functional negotiation between a source device and a sink device.</figref><figref num="7B">A flowchart illustrating the techniques of the present disclosure that can be used for functional negotiation between a source device and a sink device.</figref><figref num="8A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data.</figref><figref num="8B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data.</figref><figref num="9A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data.</figref><figref num="9B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user-input data.</figref><figref num="10A">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="10B">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="11A">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="11B">A flowchart illustrating a technique of the present disclosure that can be used to send and receive data packets with time stamp information and user input data.</figref><figref num="12A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets containing voice commands.</figref><figref num="12B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets containing voice commands.</figref><figref num="13A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with multi-touch user input commands.</figref><figref num="13B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with multi-touch user input commands.</figref><figref num="14A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data forwarded from a third party device.</figref><figref num="14B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets with user input data forwarded from a third party device.</figref><figref num="15A">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets.</figref><figref num="15B">A flowchart illustrating the techniques of the present disclosure that can be used to send and receive data packets.</figref>
The present disclosure generally describes a system in which a wireless sync device can communicate with the wireless sync device. As part of a communication session, the wireless source device can send audio and video data to the wireless sync device, which can send the user input received by the wireless sync device to the wireless source device. it can. In this way, the user of the wireless sync device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sync device.
FIG. 1A is a block diagram showing an exemplary source / sync system 100 that may implement one or more of the techniques of the present disclosure. As shown in FIG. 1A, the system 100 includes a source device 120 that communicates with the sink device 160 over the communication channel 150. The source device 120 includes a memory for storing audio / video (A / V) data 121, a display 122, a speaker 123, an audio / video encoder 124 (also called an encoder 124), an audio / video control module 125, and the like. It may include a transmitter / receiver (TX / RX) unit 126. The sink device 160 includes a display 162, a speaker 163, an audio / video decoder 164 (also called a decoder 164), a transmitter / receiver unit 166, a user input (UI) device 167, and a user input processing module (UIPM). : user input processing can include module) 168 and. The illustrated components form only one exemplary configuration for the Source / Sync System 100. Other components may contain fewer components than the components shown, or may include additional components other than those shown.
In the example of FIG. 1A, the source device 120 can display the video portion of the audio / video data 121 on the display 122 and output the audio portion of the audio / video data 121 on the speaker 123. Can the audio / video data 121 be stored locally on the source device 120 and accessed from an external storage medium such as a file server, hard drive, external memory, Blu-ray Disc, DVD, or other physical storage medium? Or it can be streamed to the source device 120 via a network connection such as the Internet. In some cases, audio / video data 121 may be captured in real time via the camera and microphone of the source device 120. The audio / video data 121 may include multimedia content such as video, television programs, or music, but may also include real-time content generated by the source device 120. Such real-time content can be, for example, video data generated by an application running on the source device 120 or captured, for example, as part of a video telephony session. As described in more detail, such real-time content may, in some cases, include video frames of user input options available for the user to select. In some cases, the audio / video data 121 may include a video frame that is a combination of different types of content, such as a video frame of a video or TV show with a user input option overlaid on the frame of the video.
In addition to locally rendering the audio / video data 121 through the display 122 and the speaker 123, the audio / video encoder 124 of the source device 120 can encode the audio / video data 121 and the transmitter / The receiver unit 126 can transmit the encoded data to the sink device 160 via the communication channel 150. The transmitter / receiver unit 166 of the sink device 160 receives the encoded data, the audio / video decoder 164 decodes the encoded data, and the decoded data is passed through the display 162 and the speaker 163. And output. In this way, the audio and video data rendered by the display 122 and the speaker 123 can be rendered simultaneously by the display 162 and the speaker 163. Audio data and video data can be composed of frames, which can be time synchronized with the video frame when rendered.
The audio / video encoder 124 and audio / video decoder 164 are alternatives to the new ITU-T H.264 standard called MPEG-4, Part10, Advanced Video Coding (AVC), or the H.265 standard. Any number of audio and video compression standards can be implemented, including high efficiency video coding (HEVC) standards. Many other types of proprietary or standardized compression techniques can also be used. Generally, the audio / video decoder 164 is configured to perform the reverse coding operation of the audio / video encoder 124. Although not shown in FIG. 1A, in some embodiments, the A / V encoder 124 and A / V decoder 164 may be integrated with an audio encoder and decoder, respectively, and may be a suitable MUX-DEMUX unit, or other hardware. It can handle both audio and video encoding in a common data stream or separate data streams, including hardware and software.
As described in more detail below, the A / V encoder 124 may perform other coding functions in addition to implementing the video compression standard as described above. For example, the A / V encoder 124 may add various types of metadata to the A / V data 121 before the A / V data 121 is transmitted to the sink device 160. In some cases, the A / V data 121 may be stored in or received in the source device 120 in encoded form and therefore does not require further compression by the A / V encoder 124. is there.
Figure 1A shows that the communication channel 150 carries the audio payload data and the video payload data separately, but in some cases the video payload data and the audio payload data are part of a common data stream. Please understand that it can be. If applicable, the MUX-DEMUX unit is ITU It may comply with the H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP). The audio / video encoder 124 and audio / video decoder 164 are each one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software. , Hardware, firmware, or any combination thereof. Each of the audio / video encoder 124 and the audio / video decoder 164 may be included in one or more encoders or decoders, both of which may be integrated as part of a composite encoder / decoder (codec). Thus, each of the source device 120 and the sink device 160 may be equipped with a dedicated machine configured to perform one or more of the techniques of the present disclosure.
The displays 122 and 162 have a variety of video outputs, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, light emitting diode (LED) displays, organic light emitting diode (OLED) displays, or other types of display devices. It may be equipped with any of the devices. In these or other examples, the displays 122 and 162 can be emissive displays or transmissive displays, respectively. The display 122 and display 162 can also be touch displays, such that they are both input and display devices at the same time. Such a touch display can be a capacitive, resistive, or other type of touch panel that allows the user to provide user input to each device.
Speaker 123 may include any of a variety of audio output devices, such as headphones, single speaker systems, multi-speaker systems, or surround sound systems. Further, the display 122 and the speaker 123 are shown as part of the source device 120, and the display 162 and the speaker 163 are shown as part of the sink device 160, but the source device 120 and the sink device 160 are actually plural. It can be a system of devices. As an example, the display 162 can be a television, the speaker 163 can be a surround sound system, and the decoder 164 can be part of an external box that is wired or wirelessly connected to the display 162 and the speaker 163. In another example, the sink device 160 can be a single device, such as a tablet computer or smartphone. In yet other cases, the source device 120 and the sink device 160 are similar devices, for example, both are smartphones, tablet computers, and so on. In this case, one device can act as a source and the other as a sink. These roles may even be reversed in subsequent communication sessions. In yet other cases, the source device may include a mobile device, such as a smartphone, laptop or tablet computer, and the sink device may include a more fixed device (eg, using an AC power cord), in which case the source. The device may deliver audio and video data for presentation to a large crowd via a sync device.
The transmitter / receiver unit 126 and the transmitter / receiver unit 166, respectively, include various mixers, filters, amplifiers, and other components designed for signal modulation, as well as one or more antennas. And may include other components designed to transmit and receive data. The communication channel 150 generally represents any communication medium suitable for transmitting video data from the source device 120 to the sink device 160, or a collection of various communication media. The communication channel 150 is usually a relatively short-range communication channel similar to Wi-Fi, Bluetooth (registered trademark), and the like. However, the communication channel 150 is not necessarily limited in this regard, and may be any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or a wireless medium and a wired medium. Can have any combination with. In another example, the communication channel 150 may even form part of a packet-based network, such as a wired or wireless local area network, a wide area network, or a global network such as the Internet. In addition, the communication channel 150 can be used by the source device 120 and the sink device 160 to create peer-to-peer links. The source device 120 and sink device 160 may communicate over the communication channel 150 using communication protocols such as standards from the IEEE 802.11 standard family. The source device 120 and sink device 160 comply with the Wi-Fi Direct standard so that, for example, the source device 120 and sink device 160 communicate directly with each other without the use of intermediates such as wireless access points or so-called hotspots. Can communicate. Source device 120 and sink device 160 are also tunneled direct link setups (TLDS:) to avoid or reduce network congestion. tunneled direct link setup) can be established. Although the techniques of the present disclosure are sometimes described with respect to Wi-Fi, it is contemplated that aspects of these techniques may also be compatible with other communication protocols. As an example, but not a limitation, wireless communication between the source device 120 and the sink device may utilize orthogonal frequency division multiplexing (OFDM) techniques. A wide variety of others, including, but not limited to, time division multiple access (TDMA), frequency division multiple connection (FDMA), code division multiple connection (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. Wireless communication techniques can also be used. WiFi Direct and TDLS aim to set up relatively short-range communication sessions. In this context, relatively short distances can refer, for example, less than 70 meters, but in noisy or obstructive environments, distances between devices can be even shorter, such as less than 35 meters. Direct and TDLS aim to set up relatively short-range communication sessions. In this context, relatively short distances can refer, for example, less than 70 meters, but in noisy or obstructive environments, distances between devices can be even shorter, such as less than 35 meters. Direct and TDLS aim to set up relatively short-range communication sessions. In this context, relatively short distances can refer, for example, less than 70 meters, but in noisy or obstructive environments, distances between devices can be even shorter, such as less than 35 meters.
In addition to decoding and rendering the data received from the source device 120, the sink device 160 can also receive user input from the user input device 167. The user input device 167 can be, for example, a keyboard, mouse, trackball or trackpad, touch screen, voice command recognition module, or other such user input device. The UIPM168 formats the user input commands received by the user input device 167 into a data packet structure that can be interpreted by the source device 120. Such data packets are transmitted by the transmitter / receiver 166 to the source device 120 over the communication channel 150. The transmitter / receiver unit 126 receives the data packet, and the A / V control module 125 parses the data packet to interpret the user input command received by the user input device 167. Based on the command received in the data packet, the A / V control module 125 can modify the coded and transmitted content. In this way, the user of the sink device 160 can control the audio payload data and the video payload data transmitted by the source device 120 remotely and without interacting directly with the source device 120. Examples of the types of commands that a user of sync device 160 can send to source device 120 are commands for rewinding, fast-forwarding, interrupting, and playing audio and video data, as well as for zooming, rotating, and scrolling. Includes commands and more. The user may also make a selection, for example from a menu of options, and send the selection to the source device 120.
In addition, the user of sync device 160 may be able to launch and control applications on source device 120. For example, a user of sync device 160 may launch a photo editing application stored on source device 120 and use that application to edit photos stored locally on source device 120. The sink device 160 can provide the user with a user experience in which the photo appears and feels locally edited on the sink device 160 while the photo is actually being edited on the source device 120. With such a configuration, device users may be able to take advantage of the capabilities of one device for use with several devices. For example, the source device 120 can be a smartphone with a large amount of memory and high-end processing power. The user of the source device 120 may use the smartphone in all settings and situations in which the smartphone is commonly used. However, when watching a video, the user may wish to watch the video on a device with a larger display screen, in which case the sync device 160 may be a tablet computer or a larger display device or television. When wishing to send or respond to e-mail, the user may wish to use a device with a keyboard, in which case the sync device 160 may be a laptop. In both cases, the user is interacting with the sink device, but most of the processing can still be performed by the source device 120 (smartphone in this example). In this particular operating context, most of the processing is done by the source device 120, so the sink device 160 is more than if the sink device 160 is asked to do the processing that is being done by the source device 120. , Using a few resources, yo It can be a low cost device. In some examples, both source and sink devices may be able to receive user input (such as touch screen commands), and the techniques of the present disclosure show the functionality of those devices in a given session. By negotiating and / or identifying, two-way dialogue may be possible.
In some configurations, the A / V control module 125 can be an operating system process running by the operating system of the source device 125. However, in other configurations, the A / V control module 125 can be the software process of the application running on the source device 120. In such a configuration, user input commands can be interpreted by a software process so that the user of sink device 160 runs on source device 120 as opposed to the operating system running on source device 120. You are interacting directly with the application you are using. By interacting directly with the application as opposed to the operating system, users of sink device 160 may have access to a library of commands that are not native to the operating system of source device 120. In addition, by interacting directly with the application, commands may be more easily transmitted and processed by devices running on different platforms.
The source device 120 can respond to user input applied in the wireless sync device 160. In such an interactive application configuration, the user input applied in the wireless sync device 160 may be sent to the wireless display source via the communication channel 150. In one example, the user interface back (UIBC) is used to allow the sink device 160 to send the user input applied at the sink device 160 to the source device 120. A reverse channel architecture, also known as channel), can be implemented. The reverse channel architecture may include a higher layer message for transporting user input and a lower layer frame for negotiating user interface functionality on the sink device 160 and the source device 120. The UIBC may reside on top of the Internet Protocol (IP) transport layer between the sink device 160 and the source device 120. In this way, UIBC is obtained located above the transport layer in the Open Systems Interconnection (OSI) communications model Ru. In one example, OSI communication includes seven layers: 1-Physical, 2-Datalink, 3-Network, 4-Transport, 5-Session, 6-Presentation, and 7-Application. In this example, above the transport layer refers to layers 5, 6, and 7. To facilitate the reliable transmission and sequential delivery of data packets containing user-entered data, UIBC uses other packets, such as Transmission Control Protocol / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP). It can be configured to run on top of the base communication protocol. UDP and TCP can operate in parallel in the OSI layer architecture. TCP / IP may allow sink device 160 and source device 120 to implement retransmission techniques in the event of packet loss.
In some cases, there may be a discrepancy between the user input interface located on the source device 120 and the user input interface located on the sink device 160. In order to resolve the potential problems posed by such discrepancies and promote a good user experience under such circumstances, user input interface feature negotiation is performed prior to establishing a communication session or in communication. It can occur between the source device 120 and the sink device 160 at various times throughout the session. As part of this negotiation process, the source device 120 and the sink device 160 can agree on the negotiated screen resolution. When the sync device 160 transmits the coordinate data associated with the user input, the sync device 160 can scale the coordinate data obtained from the display 162 to match the negotiated screen resolution. In one example, if the sink device 160 has a resolution of 1280 x 720 and the source device 120 has a resolution of 1600 x 900, then those devices may use, for example, 1280 x 720 as their negotiated resolution. The negotiated resolution may be selected based on the resolution of the sink device 160, but the resolution of the source device 120 or some other resolution may also be used. In the example where a 1280x720 sink device is used, the sink device 160 can scale the obtained x-coordinates at a ratio of 1600/1280 before sending them to the source device 120, as well. In addition, the sink device 160 can scale the obtained y-coordinates by 900/720 before sending them to the source device 120. In other configurations, the source device 120 can scale the acquired coordinates to the negotiated resolution. Its scaling is sync device 1
Moreover, in some cases, the resolution at the sink device 160 can fluctuate during the communication session, potentially resulting in a discrepancy between the display 122 and the display 162. To improve the user experience and ensure proper functionality, Source / Sync System 100 implements techniques to reduce or prevent user interaction discrepancies by implementing techniques for screen normalization. obtain. The display 122 of the source device 120 and the display 162 of the sink device 160 may have different resolutions and / or different aspect ratios. In addition, in some settings, the user of the sync device 160 will be able to render the video data received from the source device 120 in a window that covers a smaller portion of the sink device 160's display 162. It may have the ability to resize the display window for video data received from device 120. In another exemplary setting, the user of sync device 160 may have the option to view content in either landscape mode or portrait mode, each of which has unique coordinates and different aspect ratios. Have. In such situations, the coordinates associated with the user input received on the sink device 160, such as the coordinates of where the mouse click or touch event occurs, are processed by the source device 120 without modification to those coordinates. It may not be possible. Thus, the techniques of the present disclosure may include mapping the coordinates of user input received on the sink device 160 to the coordinates associated with the source device 120. This mapping, also referred to herein as normalization, can be either sink-based or source-based, as described in more detail below.
User input received by the sink device 160 may be received by the UI module 167 and passed to the operating system of the sink device 160, for example at the driver level. The operating system on the sink device 160 has the coordinates (x) associated with where the user input took place on the display surface.<sub>SINK</sub>, y<sub>SINK</sub>) Can be received. In this example, (x<sub>SINK</sub>, y<sub>SINK</sub>) Can be the coordinates of the display 162 where the mouse click or touch event took place. The display window rendered on the display 162 has an x-coordinate length (L) that describes the size of the display window.<sub>DW</sub>) And y coordinate width (W)<sub>DW</sub>) And may have. The display window is the upper left corner coordinate (a) that describes the location of the display window.<sub>DW</sub>, b<sub>DW</sub>) May also have. L<sub>DW</sub>, W<sub>DW</sub>, And upper left coordinates (a<sub>DW</sub>, b<sub>DW</sub>), The portion of the display 162 covered by the display window can be determined. For example, the upper right corner of the display window is the coordinates (a<sub>DW</sub>+ L<sub>DW</sub>, b<sub>DW</sub>), The lower left corner of the display window is the coordinates (a)<sub>DW</sub>, b<sub>DW</sub>+ W<sub>DW</sub>), The lower right corner of the display window is the coordinates (a)<sub>DW</sub>+ L<sub>DW</sub>, b<sub>DW</sub>+ W<sub>DW</sub>) Can be located. When the input is received at the coordinates in the display window, the sync device 160 can process the input as a UIBC input. In other words, the associated coordinates (x)<sub>SINK</sub>, y<sub>SINK</sub>The input in) can be processed as a UIBC input if the following conditions are met:<maths num="1"><img file="JP2014511582A_D0001.tif" /></maths><maths num="2"><img file="JP2014511582A_D0002.tif" /></maths>
After determining that the user input is a UIBC input, the coordinates associated with that input can be normalized by UIPM168 before being sent to the source device 120. Input determined to be outside the display window may be processed locally by the sync device 160 as non-UIBC input.
As mentioned above, the normalization of input coordinates can be either source-based or sync-based. When implementing sync-based normalization, the source device 120 has a supported display resolution (L) for display 122.<sub>SRC</sub>, W<sub>SRC</sub>) Can be sent to the sync device 160 with or without video data. The supported display resolutions can be transmitted, for example, as part of a functional negotiation session or at another time during a communication session. The sync device 160 has a display resolution (L) for the display 162.<sub>SINK</sub>, W<sub>SINK</sub>) And the display window resolution (L) for the window displaying the content received from the source device 120.<sub>DW</sub>, W<sub>DW</sub>) And the upper left corner coordinates for the display window (a)<sub>DW</sub>, b<sub>DW</sub>) And can be judged. As explained above, the coordinates (x) corresponding to the user input<sub>SINK</sub>, y<sub>SINK</sub>When it is determined that) is in the display window, the operating system of the sink device 160 uses a transform function to determine the coordinates (x).<sub>SINK</sub>, y<sub>SINK</sub>) To source coordinates (x<sub>SRC</sub>, y<sub>SRC</sub>) Can be mapped. (x<sub>SINK</sub>, y<sub>SINK</sub>) To (x<sub>SRC</sub>, y<sub>SRC</sub>An exemplary conversion function for converting to) can be:<maths num="3"><img file="JP2014511582A_D0003.tif" /></maths><maths num="4"><img file="JP2014511582A_D0004.tif" /></maths>
Therefore, when transmitting the coordinates corresponding to the received user input, the sink device 160 is (x).<sub>SINK</sub>, y<sub>SINK</sub>) Coordinates for user input received in (x)<sub>SRC</sub>, y<sub>SRC</sub>) Can be sent. Coordinates (x), as described in more detail below.<sub>SRC</sub>, y<sub>SRC</sub>) Can be transmitted, for example, as part of a data packet used to transmit user input received on the sink device 160 to the source device 120 via the UIBC. Throughout the rest of the disclosure, where the input coordinates are described as being contained within the data packet, those coordinates are as described above in the case where the source / sync system 100 implements sync-based normalization. Can be converted to source coordinates.
When Source / Sync System 100 implements source-based normalization, user input determined to be UIBC input as opposed to local input (ie, inside the display window as opposed to outside the display window). For, the above calculation can be performed on the source device 120 instead of the sink device 160. To enable such calculations, the sink device 160 is L.<sub>DW</sub>, W<sub>DW</sub>Values for, as well as location information about the display window (eg a)<sub>DW</sub>, b<sub>DW</sub>), And (x<sub>SINK</sub>, y<sub>SINK</sub>) Can be sent to the source device 120. Using these transmitted values, the source device 120 follows Equations 3 and 4 above (x).<sub>SRC</sub>, y<sub>SRC</sub>) Can be determined.
In another implementation of sync-based normalization, the sink device 160 describes where in the display window the user input event occurs, as opposed to where the user input event occurs on the display 162. Coordinates for user input (x<sub>DW</sub>, y<sub>DW</sub>) Can be sent. In such an implementation, the coordinates (x)<sub>DW</sub>, y<sub>DW</sub>) Is (L<sub>DW</sub>, W<sub>DW</sub>Can be sent 120 to the source device with a value for). Based on these received values, the source device 120 follows the following conversion function (x)<sub>SRC</sub>, y<sub>SRC</sub>) Can be judged.<maths num="5"><img file="JP2014511582A_D0005.tif" /></maths><maths num="6"><img file="JP2014511582A_D0006.tif" /></maths>
The sink device 160 is x based on the following function<sub>DW</sub>And y<sub>DW</sub>Can be judged.<maths num="7"><img file="JP2014511582A_D0007.tif" /></maths><maths num="8"><img file="JP2014511582A_D0008.tif" /></maths>
In the present disclosure, for example, when describing transmitting the coordinates associated with a user input in a data packet, the transmission of these coordinates may include sink-based or source-based normalization as described above. , And / or may contain any additional information needed to perform sink-based or source-based normalization.
UIBC can be designed to transport various types of user input data, including cross-platform user input data. For example, the source device 120 may run an iOS® operating system, and the sink device 160 may run another operating system, such as Android® or Windows®. Regardless of the platform, the UIPM168 can encapsulate the received user input in a form understandable to the A / V control module 125. Several different types of users allow them to leverage the protocol, regardless of whether many different types of source and sink devices run on different platforms. The input format may be supported by UIBC. General input formats can be defined and both platform-specific input formats can be supported, thus giving the flexibility of how user input can be communicated between the source device 120 and the sink device 160 by UIBC.
In the example of FIG. 1A, the source device 120 may include a smartphone, tablet computer, laptop computer, desktop computer, Wi-Fi capable television, or other device capable of transmitting audio and video data. The sync device 160 can also receive smartphones, tablet computers, laptop computers, desktop computers, Wi-Fi enabled televisions, or other devices capable of receiving audio and video data and receiving user input data. Can be prepared. In some cases, the sink device 160 is such that the display 162, speaker 163, UI device 167, and A / V encoder 164 are all separate but interoperable devices. It may include a system of devices. Similarly, the source device 120 can be a system of multiple devices rather than a single device.
In the present disclosure, the term source device is used generally to refer to a device transmitting audio / video data, and the term sink device is generally used to refer to a device receiving audio / video data from a source device. Used to point to. In many cases, the source device 120 and the sink device 160 can be similar or equivalent devices, one device acting as the source and the other acting as the sink. Moreover, these roles can be reversed in different communication sessions. Thus, a sink device in one communication session can be a source device in subsequent communication sessions, and vice versa.
FIG. 1B is a block diagram showing an exemplary source / sync system 101 that may implement the techniques of the present disclosure. The source / sink system 101 includes a source device 120 and a sink device 160, each of which may function and operate in the manner described above for FIG. 1A. The source / sync system 101 further includes a sink device 180. In a manner similar to the sink device 160 described above, the sink device 180 may receive audio and video data from the source device 120 and send user commands to the source device 120 via the established UIBC. In some configurations, the sink device 160 and the sink device 180 can operate independently of each other, and the audio and video data output by the source device 120 is output simultaneously by the sink device 160 and the sink device 180. obtain. In an alternative configuration, the sync device 160 can be a primary sync device and the sync device 180 can be a secondary sync device. In such an exemplary configuration, the sink device 160 and the sink device 180 can be combined, the sink device 160 can display video data, and the sink device 180 outputs the corresponding audio data. Further, in some configurations, the sync device 160 may output only the transmitted video data and the sync device 180 may output only the transmitted audio data.
FIG. 2 is a block diagram showing an example of the source device 220. The source device 220 can be a device similar to the source device 120 in FIG. 1A and can operate in the same way as the source device 120. The source device 220 includes a local display 222, a local speaker 223, a processor 231, a memory 232, a transport unit 233, and a wireless modem 234. As shown in FIG. 2, the source device 220 may include one or more processors (ie, processor 231) that encode and / or decode A / V data for transport, storage, and display. A / V data can be stored, for example, in memory 232. Memory 232 may include an entire A / V file, or may include, for example, a smaller buffer that stores only a portion of the A / V file, streamed from another device or source. Transport unit 233 may process encoded A / V data for network transport. For example, the encoded A / V data can be processed by processor 231 and encapsulated in a network access layer (NAL) unit by transport unit 233 for communication over the network. The NAL unit can be sent by the wireless modem 234 to the wireless sync device over a network connection. The wireless modem 234 can be, for example, a Wi-Fi modem configured to implement one of the IEEE 802.11 standard families.
The source device 220 may also process and display A / V data locally. In particular, the display processor 235 may process the video data to be displayed on the local display 222, and the audio processor 236 may process the audio data for output on the speaker 223.
As described above for the source device 120 of FIG. 1A, the source device 220 may also receive user input commands from the sink device. In this way, the wireless modem 234 of the source device 220 receives the encapsulated data packet, such as the NAL unit, and sends the encapsulated data unit to the transport unit 233 for decapsulation. For example, transport unit 233 can extract data packets from the NAL unit, and processor 231 can parse the data packets to extract user input commands. Based on the user input command, the processor 231 can adjust the encoded A / V data transmitted by the source device 220 to the sink device. In this way, the functionality described above for the A / V control module 125 of FIG. 1A may be fully or partially implemented by processor 231.
Processor 231 in Figure 2 is generally, but not limited to, one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), and other equivalents. Represents any of a wide variety of processors, or any combination thereof, including integrated circuits or discrete logic circuits. The memory 232 in FIG. 2 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It can have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM), flash memory, and so on. The memory 232 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 232 may further store instructions and program code executed by processor 231 as part of performing the various techniques described herein.
Figure 3 shows an example of the sync device 360. The sink device 360 can be a device similar to the sink device 160 in FIG. 1A and can operate in the same way as the sink device 160. The sink device 360 includes one or more processors (ie, processor 331), memory 332, transport unit 333, wireless modem 334, display processor 335, local display 362, audio processor 336, and speaker 363. And the user input interface 376. The sink device 360 receives the encapsulated data unit sent from the source device in the wireless modem 334. The wireless modem 334 can be, for example, a Wi-Fi modem configured to implement one or more standards from the IEEE 802.11 standard family. The transport unit 333 can decapsulate the encapsulated data unit. For example, the transport unit 333 may extract encoded video data from the encapsulated data unit and send the encoded A / V data to processor 331 for decoding and rendering for output. The display processor 335 may process the decoded video data as it is displayed on the local display 362, and the audio processor 336 may process the decoded audio data for output on the speaker 363.
In addition to rendering audio and video data, the wireless sync device 360 can also receive user input data through user input interface 376. The user input interface 376 is one of several user input devices, including, but not limited to, a touch display interface, a keyboard, a mouse, a voice command module, and a gesture capture device (eg, with camera-based input capture capabilities). , Or other user input devices of some user input devices. User input received through user input interface 376 may be processed by processor 331. This process may include generating a data packet containing the received user input command according to the techniques described herein. Once generated, transport unit 333 may process its data packets for network transport to wireless source devices via UIBC.
Processor 331 in Figure 3 is one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), other equivalent integrated circuits or discrete logic circuits. It may have one or more of a wide range of processors, or any combination thereof. The memory 332 in FIG. 3 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It can have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM), flash memory, and so on. The memory 232 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 332 may further store instructions and program code executed by processor 331 as part of performing the various techniques described herein.
FIG. 4 shows an exemplary transmitter system 410 and receiver system 450 that can be used by the transmitter / receiver 126 and transmitter / receiver 166 of FIG. 1A to communicate over communication channel 150. A block diagram is shown. In transmitter system 410, traffic data for several data streams is fed from data source 412 to transmit (TX) data processor 414. Each data stream may be transmitted via its own transmitting antenna. The TX data processor 414 formats, codes, and interleaves the traffic data for each data stream based on the particular coding scheme selected for that data stream.
The coded data in each data stream can be multiplexed with pilot data using Orthogonal Frequency Division Multiplexing (OFDM) techniques. A wide variety of others, including, but not limited to, time division multiple access (TDMA), frequency division multiple connection (FDMA), code division multiple connection (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. Wireless communication techniques can also be used.
According to FIG. 4, the pilot data is typically a known data pattern processed in a known manner and can be used in the receiver system to estimate the channel response. The multiplexed pilot and coded data of each data stream is then given a particular modulation scheme (eg, 2-phase shift keying (BPSK), 4-phase shift) selected for that data stream to give a modulation symbol. Modulated (eg, symbol mapping) based on keying (QPSK), M-PSK, or M-QAM (quadrature keying), where M can be a power of 2. The data rate, coding, and modulation of each data stream can be determined by instructions executed by processor 430, which can be coupled to memory 432.
A modulation symbol for the data stream is then given to the TX MIMO processor 420, which may further process that modulation symbol (for example, for OFDM). The TX MIMO processor 420 is then N<sub>T</sub>N modulation symbol streams<sub>T</sub>It can be given to multiple transmitters (TMTR) 422a ~ 422t. In some embodiments, the TX MIMO processor 420 applies beamforming weights to the symbol of the data stream and to the antenna at the source of the symbol.
Each transmitter 422 receives and processes its own symbol stream to give it one or more analog signals, and then tunes (eg, amplifies, filters, and upconverts) those analog signals. , A modulated signal suitable for transmission via a MIMO channel can be provided. Then N from transmitters 422a ~ 422t<sub>T</sub>Each modulated signal is N<sub>T</sub>It is transmitted from the antennas 424a to 424t.
In the receiver system 450, the transmitted modulated signal is N<sub>R</sub>It is received by the antennas 452a to 452r, and the received signal from each antenna 452 is given to the respective receivers (RCVR) 454a to 454r. The receiver 454 tunes (eg, filters, amplifies, and downconverts) each received signal, digitizes the tuned signal, gives a sample, and further processes the sample to make a corresponding " Gives a "receive" symbol stream.
The receive (RX) data processor 460 is then N<sub>R</sub>Receivers 454 to N<sub>R</sub>Receives a received symbol stream, processes it based on a particular receiver processing technique, and N<sub>T</sub>Gives a "detection" symbol stream. The RX data processor 460 then demodulates, deinterleaves, and decodes each detected symbol stream to restore the traffic data in the data stream. The processing by the RX data processor 460 complements the processing performed by the TX MIMO processor 420 and the TX data processor 414 in the transmitter system 410.
Processor 470, which can be combined with memory 472, periodically determines which precoding matrix should be used. A reverse link message can contain various types of information about communication links and / or received data streams. The reverse link message is then processed by the TX data processor 438, which also receives the traffic data of some data streams from the data source 436, modulated by the modulator 480, tuned by the transmitters 454a-454r, and the transmitter. Returned to system 410.
In transmitter system 410, the modulated signal from receiver system 450 is received by antenna 424, tuned by receiver 422, demodulated by demodulator 440, processed by RX data processor 442, and received by receiver system 450. The reverse link message sent by is extracted. Processor 430 then determines which precoding matrix should be used to determine the beamforming weights, and then processes the extracted message.
FIG. 5A is a block diagram showing an exemplary message transfer sequence between the source device 520 and the sink device 560 as part of a functional negotiation session. Functional negotiation can occur as part of the larger communication session establishment process between the source device 520 and the sink device 560. This session can be established, for example, with Wi-Fi Direct or TDLS as the underlying connectivity standard. After establishing a Wi-Fi Direct or TDLS session, the sink device 560 can initiate a TCP connection with the source device 520. As part of establishing a TCP connection, a control port can be established running a real time streaming protocol (RTSP) to manage communication sessions between the source device 520 and the sink device 560. ..
The source device 520 may generally operate in the same manner as described above for the source device 120 in FIG. 1A, and the sink device 560 may generally operate in the same manner as described above for the sink device 160 in FIG. 1A. Can work. After the source device 520 and sink device 560 establish connectivity, the source device 520 and sink device 560 are parameters that should be used for subsequent communication sessions of those devices as part of a functional negotiation exchange. You can judge the set.
Source device 520 and sink device 560 can negotiate functionality through a sequence of messages. Those messages can be, for example, Real Time Streaming Protocol (RTSP) messages. At any stage of that negotiation, the recipient of the RTSP request message may respond with an RTSP response that contains an RTSP status code other than RTSP OK, in which case the message exchange may be retried with a different set of parameters. Alternatively, the functional negotiation session can be terminated.
The source device 520 can send a first message (RTSP OPTIONS request message) to the sink device 560 to determine the set of RTSP methods supported by the sink device 560. Upon receiving the first message from the source device 520, the sink device 560 can respond with a second message (RTSP OPTIONS response message) listing the RTSP methods supported by the sink 560. Also, the second message may contain the RTSP OK status code.
After sending the second message to the source device 520, the sink device 560 can send a third message (RTSP OPTIONS request message) to determine the set of RTSP methods supported by the source device 520. Upon receiving the third message from the sink device 560, the source device 520 can respond with a fourth message (RTSP OPTIONS response message) listing the RTSP methods supported by the source device 520. Also, the fourth message can include the RTSP OK status code.
After sending the fourth message, the source device 520 can send a fifth message (RTSP GET_PARAMETER request message) to specify a list of features related to the source device 520. The sink device 560 can respond with a sixth message (RTSP GET_PARAMETER response message). The sixth message may contain the RTSP status code. If the RTSP status code is OK, the sixth message can also include response parameters to the parameters specified in the fifth message, supported by the sink device 560. The sink device 560 can ignore the parameters in the fifth message that the sink device 560 does not support.
Based on the sixth message, the source 520 can determine the optimal set of parameters to be used for the communication session and can send a seventh message (RTSP SET_PARAMETER request message) to the sink device 560. it can. The seventh message may contain a set of parameters that should be used during the communication session between the source device 520 and the sink device 560. The seventh message is the Universal Resource Identifier (URI) that should be used in the RTSP Setup request to set up the communication session. Can include a wfd-presentation-url that describes the Identifier). wfd-presentation-url specifies a URI that the sink device 560 can use for later messages during a session establishment exchange. The wfd-url0 and wfd-url1 values specified in this parameter can correspond to the rtp-port0 and rtp-port1 values in wfd-client-rtp-ports in the seventh message. .. RTP in this case generally refers to a real-time protocol that can run on top of UDP.
Upon receiving the seventh message, the sink device 560 can respond with an eighth message with an RTSP status code indicating whether the parameters specified in the seventh message were successfully set. .. As mentioned above, the roles of the source device and the sink device can be reversed or changed in different sessions. The order of messages that set up a communication session can, in some cases, define a device that acts as a source and a device that acts as a sink.
FIG. 5B is a block diagram showing another exemplary message transfer sequence between the source device 560 and the sink device 520 as part of a functional negotiation session. The message forwarding sequence of FIG. 5B is intended to provide a more detailed diagram of the forwarding sequence described above for FIG. 5A. In Figure 5B, the message "1b.GET_PARAMETER response" shows an example of a message that identifies a list of supported input categories (eg, general and HIDC) from a list of supported input types. List of Supported Input Categories Each of the supported input categories has an associated list of supported types (eg generic_cap_list and hidc_cap_list). In Figure 5B, the message "2a.SET_PARAMETER request" identifies a second list of supported input categories (eg, general and HIDC) from a second list of supported types. This is an example of a message. Second List of Supported Input Categories Each of the supported input categories has an associated second list of supported types (eg generic_cap_list and hidc_cap_list). The message "1b.GET_PARAMETER response" identifies the input categories and input types supported by the sink device 560. The message "2a.SET_PARAMETER request" identifies the input categories and input types supported by the source device 520, but it is not a comprehensive list of all input categories and input types supported by the source device 520. Sometimes. Instead, message "2a.SET_PARAMETER request" is assumed to be supported by sync device 560 and message "1b. Only the input category and input type identified in the GET_PARAMETER response can be identified. In this way, the input category and input type identified in the message "2a.SET_PARAMETER request" may constitute a subset of the input category and input type identified in the message "1b.GET_PARAMETER response".
FIG. 6 is a conceptual diagram showing an example of a data packet that can be generated by a sink device and sent to a source device. Although aspects of the data packet 600 will be described with reference to FIG. 1A, the techniques described may be applicable to additional types of source / sink systems. The data packet 600 may include a data packet header 610, followed by payload data 650. Payload data 650 may further include one or more payload headers (eg, payload header 630). The data packet 600 may be transmitted from the sink device 160 in FIG. 1A to the source device 120 so that, for example, the user of the sink device 160 can control the audio / video data transmitted by the source device 120. In such cases, the payload data 650 may include user input data received on the sink device 160. Payload data 650 may identify, for example, one or more user commands. The sink device 160 can receive one or more user commands and can generate data packet header 610 and payload data 650 based on the received commands. Based on the content of the data packet header 610 of the data packet 600, the source device 120 can parse the payload data 650 to identify the user input data received by the sink device 160. Based on the user input data contained in the payload data 650, the source device 120 may somehow modify the audio and video data transmitted from the source device 120 to the sink device 160.
As used in this disclosure, the terms "parse" and "persing" generally refer to the process of analyzing a bitstream to extract data from the bitstream. Once extracted, the data can be processed, for example, by the source device 120. Extracting the data can include, for example, identifying how the information in the bitstream is formatted. As described in more detail below, the data packet header 610 may define a standardized format known to both the source device 120 and the sink device 160. However, the payload data 650 can be formatted in one of many possible ways. By parsing the data packet header 610, the source device 120 can determine how the payload data 650 is formatted, so the source device 120 can be one or more users from the payload data 650. Payload data 650 can be parsed to extract input commands. This can provide flexibility in terms of different types of payload data that may be supported in source sink communication. As described in more detail below, the payload data 650 may also include one or more payload headers, such as the payload header 630. In such a case, the source device 120 parses the data packet header 610 to determine the format for the payload header 630 and then the payload header 630 to determine the format for the rest of the payload data 650. obtain.
FIG. 620 is a conceptual diagram of how the data packet header 610 can be formatted. Numbers 0-15 on line 615 are intended to identify the bit locations within the data packet header 610 and are not intended to actually represent the information contained within the data packet header 610. The data packet header 610 includes a version field 621, a time stamp flag 622, a reserved field 623, an input category field 624, a length field 625, and an optional time stamp field 626.
In the example of FIG. 6, version field 621 is a 3-bit field that may indicate the version of a particular communication protocol implemented by the sink device 160. The value in version field 621 may inform the source device 120 how to parse the rest of the data packet header 610 and how to parse the payload data 650. In the example in Figure 6, version field 621 is a 3-bit field that will allow unique identifiers for eight different versions. In other examples, more or less bits may be dedicated to version field 621.
In the example of FIG. 6, the time stamp flag (T) 622 is a 1-bit field indicating whether or not the time stamp field 626 is present in the data packet header 610. Timestamp field 626 is a 16-bit field that contains a time stamp based on multimedia data generated by the source device 120 and sent to the sink device 160. The time stamp can be, for example, a continuous value assigned by the source device 120 to the frame of the video before it is transmitted to the sink device 160. The time stamp flag 622 may include, for example, a "1" to indicate that the time stamp field 626 exists and a "0" to indicate that the time stamp field 626 does not exist. If the data packet header 610 is parsed and it is determined that the time stamp field 626 exists, the source device 120 can process the time stamp contained in the time stamp field 626. If it parses the data packet header 610 and determines that the timestamp field 626 does not exist, the source device 120 will parse the length field 625 and then the payload data 650 because the timestamp field does not exist in the data packet header 610. You can start parsing.
If present, the time stamp field 626 can include a time stamp to identify a frame of video data that was displayed on the wireless sync device 160 when the user input data of the payload data 650 was retrieved. The time stamp may be added to the video frame by the source device 120, for example, before the source device 120 sends the video frame to the sink device 160. Thus, the source device 120 may generate a frame of video and embed a time stamp in the video data of that frame, for example as metadata. The source device 120 can send the video frame to the sink device 160 along with the time stamp, and the sink device 160 can display the frame of the video. The sink device 160 can receive user commands from the user while the frame of the video is displayed by the sync device 160. When the sink device 160 generates a data packet to forward the user command to the source device 120, the sink device 160 time stamps the time stamp of the frame displayed by the sink device 160 when the user command is received. Can be included in field 626.
When the timestamp field 626 receives the data packet 600 present in the header, the wireless source device 120 identifies the frame of the video displayed on the sink device 160 when the user input data for the payload data 650 is retrieved. , User input data can be processed based on the content of the frame identified by the time stamp. For example, if the user input data is a touch command applied to the touch display or a mouse pointer click, the source device 120 is displayed when the user applies the touch command to the display or clicks the mouse. You can determine the content of the frame you are in. In some cases, the content of the frame may be required to properly process the payload data. For example, user input based on user touch or mouse click may depend on what was shown on the display at the time of touch or click. Touches or clicks may correspond to, for example, icons or menu options. In the case where the content of the display is changing, the time stamp present in the time stamp field 626 can be used by the source device 120 to match the touch or click to the correct icon or menu option.
The source device 120 may, in addition or as an alternative, compare the time stamp in the time stamp field 626 with the time stamp applied to the currently rendered frame of the video. By comparing the time stamp in the time stamp field 626 with the current time stamp, the source device 120 can determine the round trip time. Round-trip time generally corresponds to the amount of time elapsed from the time a frame was transmitted by the source device 120 to the time user input based on that frame was received back from the sink device 160 to the source device 120. .. The round trip time can give the source device 120 an indication of system latency, and if the round trip time is greater than the threshold, the source device 120 will assume that the input command was applied to the old display frame. Therefore, the user input data contained in the payload data 650 can be ignored. When the round trip time is less than the threshold, the source device 120 may process the user input data and adjust the audio / video content being transmitted in response to the user input data. Thresholds can be programmable and different types of devices (or different source sync combinations) can be configured to define different thresholds for acceptable round trip times.
In the example of FIG. 6, reserved field 623 is an 8-bit field that does not contain the information used by source 120 when parsing data packet header 610 and payload data 650. However, a future version of a particular protocol (identified in version field 621) may utilize reserved field 623, in which case the source device 120 is to parse the data packet header 610 and / or The information in reserved field 623 can be used to parse payload data 650. Reserved field 623, along with version field 621, has the potential to extend features and add features to the data packet format without essentially changing the format and features already in use. Give to.
In the example of FIG. 6, the input category field 624 is a 4-bit field for identifying the input category for the user input data contained in the payload data 650. The sink device 160 may categorize user input data to determine the input category. Categorizing user input data can be based, for example, on the device on which the command was received, or on the nature of the command itself. The value of the input category field 624 identifies to the source device 120 how the payload data 650 is formatted, optionally along with other information in the data packet header 610. Based on this formatting, the source device 120 can parse the payload data 650 to determine the user input received by the sink device 160.
In the example of FIG. 6, since the input category 624 is 4 bits, 16 different input categories can be identified in some cases. One such input category is that the user input data in payload data 650 is formatted using the general information elements defined in the protocol running by both the source device 120 and the sink device 160. Can be a general input format to indicate. The general input format may utilize general information elements that allow the user of the sink device 160 to interact with the source device 120 at the application level, as described in more detail below.
Another such input category is the human interface device command (to indicate that the user input data in payload data 650 is formatted based on the type of input device used to receive the input data. HIDC: human interface device command) format is possible. Examples of device types include keyboards, mice, touch input devices, joysticks, cameras, gesture capture devices (such as camera-based input devices), and remote control devices. Other types of input categories that can be identified in the input category field 624 are the forwarding input format, or operating system specific format, and payload to indicate that the user data in the payload data 650 did not originate on the sink device 160. Includes a voice command format to indicate that data 650 contains voice commands.
The length field 625 may include a 16-bit field to indicate the length of the data packet 600. The length can be expressed in units of 8 bits, for example. When the data packet 600 is parsed by the source device 120 with a 16-bit word, the data packet 600 can be padded to a 16-bit integer. Based on the length contained in the length field 625, the source device 120 can distinguish between the end of payload data 650 (ie, the end of data packet 600) and the start of a new subsequent data packet.
The various sizes of the fields given in the example of FIG. 6 are for illustration purposes only, and those fields may be implemented using a different number of bits than those shown in FIG. It is also contemplated that the data packet header 610 may contain fewer fields than all the fields described above, or may use additional fields not described above. In fact, the techniques of the present disclosure can be flexible in terms of the actual format used for the various data fields of the packet.
After parsing the data packet header 610 to determine the formatting of the payload data 650, the source device 120 can parse the payload data 650 to determine the user input commands contained in the payload data 650. .. The payload data 650 may have its own payload header (payload header 630) indicating the contents of the payload data 650. In this way, the source device 120 may parse the payload header 630 based on the parsing of the data packet header 610 and then the remaining payload data 650 based on the parsing of the payload header 630.
For example, if the input category field 624 of the data packet header 610 indicates that a general input is present in the payload data 650, the payload data 650 may have a general input format. Therefore, the source device 120 can parse the payload data 650 according to the general input format. As part of the general input format, the payload data 650 can include a series of one or more input events, where each input event has its own input event header. Table 1 below identifies the fields that can be included in the input header.<tables num="1"><img file="JP2014511582A_D0009.tif" /></tables>
The General Input Event (IE) Identification field identifies the General Input Event Identification to identify the input type. The general IE ID field can be, for example, one octet in length and may contain identification information selected from Table 2 below. If the general IE ID field is 8-bit, as in this example, 256 different types of inputs (identified as 0-255) can be identifiable, but not all 256 identities are necessarily identifiable. It does not always require the associated input type. Some of the above 256 may be reserved for future use of future versions of any protocol implemented by the sink device 160 and the source device 120. In Table 2, for example, general IE IDs 9-255 do not have an associated input type, but may be assigned an input type in the future.
The length field in the input event header identifies the length of the descriptive field, and the descriptive field contains an information element that describes the user input. The formatting of the descriptive field can depend on the type of input identified in the general IE ID field. Therefore, the source device 120 may parse the contents of the descriptive field based on the input type identified in the general IE ID field. Based on the length field of the input event header, the source device 120 can determine the end of one input event in the payload data 650 and the start of a new input event. As described in more detail below, a user command can be described as one or more input events in payload data 650.
Table 2 gives an example of an input type, each with a corresponding general IE ID that can be used to identify the input type.<tables num="2"><img file="JP2014511582A_D0010.tif" /></tables>
The descriptive fields associated with each input type can have different formats. The left mouse down / touchdown event, left mouse up / touch up event, and mouse move / touch move event descriptive fields may contain, for example, the information elements identified in Table 3 below, but other Other formats may be used in the example.<tables num="3"><img file="JP2014511582A_D0011.tif" /></tables>
The number of pointers can identify the number of touches or mouse clicks associated with an input event. Each pointer can have a unique pointer ID. For example, if the multi-touch event contains a three-finger touch, each input event may have three pointers, each with a unique pointer ID. Each pointer (ie, each finger touch) may have a corresponding x-coordinate and y-coordinate corresponding to where the touch was made.
A single user command can be described as a series of input events. For example, if a three-finger swipe is a command to close an application, a three-finger swipe uses a touchdown event with three pointers, a touch move event with three pointers, and a three pointers. It can be described in the payload data 650 as the touch-up event that was there. The three pointers for a touchdown event can have the same pointer ID as the three pointers for a touch movement event and a touchup event. The source device 120 can interpret the combination of those three input events as a three-finger swipe.
The descriptive field for a key-down event or key-up event may contain, for example, the information elements identified in Table 4 below.<tables num="4"><img file="JP2014511582A_D0012.tif" /></tables>
The zoom event description field may contain, for example, the information elements identified in Table 5 below.<tables num="5"><img file="JP2014511582A_D0013.tif" /></tables>
The description fields for horizontal or vertical scrolling events may include, for example, the information elements identified in Table 6 below.<tables num="6"><img file="JP2014511582A_D0014.tif" /></tables>
The above example shows some exemplary ways in which payload data can be formatted in the case of general input categories. Payload data 650 may have different input formats if the input category field 624 of the data packet header 610 indicates a different input category, such as forwarded user input. For forwarded user input, the sink device 160 may receive user input data from a third party device and forward the input to the source device 120 without interpreting the user input data. Therefore, the source device 120 can parse the payload data 650 according to the forwarded user input format. For example, the payload header 630 of the payload data 650 may include a field for the user input to identify a third party device obtained from it. The field may include, for example, the Internet Protocol (IP) address, MAC address, domain name, or any other such identifier of a third party device. The source device 120 can parse the rest of the payload data based on the identifier of the third party device.
The sink device 160 can negotiate features with a third party device via a series of messages. The sink device 160 can then send a unique identifier for the third party device to the source device 120 as part of establishing a communication session with the source device 120 as part of the functional negotiation process. Alternatively, the sink device 160 may send information describing the third party device to the source device 120, from which the source device 120 can determine a unique identifier for the third party device. The information describing the third party device may include, for example, information for identifying the third party device and / or information for identifying the function of the third party device. When the sink device 160 sends a data packet with user input obtained from a third party device, whether the unique identifier is determined by the source device 120 or the sink device 160, the sink device 160 Can include a unique identifier in the data packet, eg, in the payload header, so that the source device 120 can identify the origin of the user input.
If the input category field 624 of the data packet header 610 indicates a different input category, such as a voice command, the payload data 650 may have a different input format. For voice commands, the payload data 650 may include coded audio. A codec for encoding and decoding the audio of a voice command can be negotiated between the source device 120 and the sink device 160 via a series of messages. To send a voice command, the timestamp field 626 may include a voice sampling time value. In such cases, the time stamp flag 622 may be set to indicate that a time stamp is present, but instead of the time stamp described above, the time stamp field 626 is for the encoded audio of the payload data 650. Can include audio sampling time values of.
In some examples, the voice command may be sent as a general command as described above, in which case the input category field 624 may be set to identify the general command format and of the reserved general IE ID. One of them can be assigned to a voice command. If the voice command is sent as a general command, the voice sampling rate can be present in the timestamp field 626 of the data packet header 610 or in the payload data 650.
For captured voice command data, the voice data can be encapsulated in multiple ways. For example, voice command data can be encapsulated using RTP, which can be given a payload type to identify the codec and timestamp, which timestamp is used to identify the sampling rate. RTP data can be encapsulated using the general user input format described above, with or without optional time stamps. The sink device 160 can use TPC / IP to send general input data that carries voice command data to the source device 120.
As mentioned earlier, when coordinates are included as part of a data packet, such as data packet 600, in, for example, payload data 650, the coordinates are negotiated resolution, display window coordinates, normalized coordinates, or It may correspond to coordinates scaled based on the coordinates associated with the sync display. In some cases, additional information may be included in the data packet or transmitted separately for use by the source device to normalize the coordinates received in the data packet.
Regardless of the input category for a particular data packet, the data packet header can be an application layer packet header and the data packet can be sent over TCP / IP. TCP / IP may allow sink device 160 and source device 120 to perform retransmission techniques in the event of packet loss. Data packets are sent from sink device 160 to source device 120 for other purposes, such as to control audio or video data on source device 120, or to control applications running on source device 120. Can be sent.
FIG. 7A is a flowchart of an exemplary method of negotiating functionality between a sink device and a source device. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more of the illustrated steps in one or more of the flowcharts described herein. It may store instructions, modules, or algorithms that cause one or more processors (eg, processor 331) to do what they want to do.
The method of FIG. 7A involves the sink device 160 receiving a first message from the source device 120 (701). The message may include, for example, a get parameter request. In response to the first message, sink device 160 sends a second message to source device 120 (703). The second message is, for example, a get parameter that identifies the first list of supported input categories from the first list of supported types. Response) may be provided and each of the supported input categories in the first list of supported input categories has an associated first list of supported types. The supported input categories may correspond to, for example, the same categories used for input category field 624 in FIG. Table 2 above represents an example of the supported types for a particular input category (general input in this example). The sink device 160 receives a third message from the source device 120 (705). The third message is, for example, a set parameter A request) can be provided, the parameter setting request identifies the port for communication, the second list of supported input categories, and multiple second lists of supported types, and the supported inputs. Each of the supported input categories in the second list of categories has an associated second list of supported types, and each of the supported types in the second list is in the first list. Contains a subset of types. The sink device 160 sends a fourth message to the source device 120 (707). The fourth message may include, for example, a set parameter response to confirm that the type in the second list has been enabled. The sink device 160 receives a fifth message from the source device 120 (709). The fifth message may include, for example, a second parameter setting request indicating that the communication channel between the source device 120 and the sink device 160 has been enabled. The communication channel is, for example, a user input back (UIBC). channel) can be provided. The sink device 160 sends a sixth message to the source device 120 (711). The sixth message may include, for example, a second parameter setting response confirming receipt of the second parameter setting request by the sink device 160.
FIG. 7B is a flowchart of an exemplary method of negotiating functionality between a sink device and a source device. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 7B involves the source device 120 sending a first message to the sink device 160 (702). The first message may include, for example, a parameter acquisition request. The source device 120 receives a second message from the sink device 160 (704). The second message may include, for example, a parameter acquisition response that identifies the first list of supported input categories from a plurality of first lists of supported types, and the first of the supported input categories. Each of the supported input categories of the list has an associated first list of supported types. The source device 120 sends a third message to the sink device 160 (706). The third message may, for example, include and support a parameter setting request that identifies a port for communication, a second list of supported input categories, and multiple second lists of supported types. Each of the supported input categories in the second list of input categories to be made has an associated second list of supported types, and each of the supported types in the second list is the first. Contains a subset of the types of lists in. The source device 120 receives a fourth message from the sink device 160 (708). The fourth message may include, for example, a parameter setting response to confirm that the type in the second list has been enabled. The source device 120 sends a fifth message to the sink device 160 (710). The fifth message may include, for example, a second parameter setting request indicating that the communication channel between the source device 120 and the sink device 160 has been enabled. The communication channel may include, for example, a user input back channel (UIBC). Source device 120 receives a sixth message from sink device 160 (712). The sixth message confirms, for example, the receipt of the second parameter setting request by the sink device 160.
FIG. 8A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 8A involves acquiring user input data in a wireless sync device such as the wireless sync device 160 (801). User input data can be obtained through the user input component of the wireless sync device 160, for example, the user input interface 376 shown for the wireless sync device 360. In addition, the sink device 160 may categorize user input data as, for example, general, forwarded, or operating system specific. The sink device 160 then generates a data packet header based on user input data (803). The data packet header can be an application layer packet header. The data packet header may include, among the fields, a field for identifying an input category corresponding to user input data. The input category may include, for example, a general input format or a human interface device command. The sink device 160 further generates a data packet (805), which comprises the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (807). The sink device 160 may include components that allow the transfer of data packets, including, for example, a transport unit 333 and a wireless modem 334, as shown in FIG. The sink device 160 may forward data packets over TCP / IP.
FIG. 8B is a flowchart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 8B comprises receiving a data packet (802), which may comprise, in particular, a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. The source device 120 then parses the data packet header contained in the data packet to determine the input category associated with the user input data contained in the payload data (804). Source device 120 processes payload data based on the determined input category (806). The data packets described with reference to FIGS. 8A and 8B can generally take the form of the data packets described with reference to FIG. 6 and are used to control audio / video data and applications in the source device. obtain.
FIG. 9A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 9A involves acquiring user input data in a wireless sync device such as the wireless sync device 160 (901). User input data can be obtained through the user input component of the wireless sync device 160, for example, the user input interface 376 shown with reference to FIG. The sink device 160 then generates payload data (903), which may describe user input data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 further generates a data packet (905), which comprises a data packet header and the generated payload data. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (907). The sink device 160 may include components that allow the transfer of data packets, such as transport unit 333 and wireless modem 334. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 9B is a flow chart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 9B comprises receiving a data packet from the sink device 360 (902), which may include, in particular, a data packet header and payload data. In one example, the payload data may include data that describes user input details, such as input type values. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. The source device 120 then parses the data packet to determine the input type value in the input type field in the payload data (904). The source device 120 processes data describing the details of the user input based on the determined input type value (906). The data packet described with reference to FIGS. 9A and 9B may generally take the form of the data packet described with reference to FIG.
FIG. 10A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 10A involves acquiring user input data (1001) on a wireless sync device such as the wireless sync device 160. User input data can be obtained through the user input component of the wireless sync device 160, for example, the user input interface 376 as shown with reference to FIG. The sink device 160 then generates a data packet header based on user input (1003). The data packet header may include, among other fields, a time stamp flag (eg, a 1-bit field) to indicate whether the time stamp field is present in the data packet header. The time stamp flag may include, for example, a "1" to indicate that the time stamp field exists and a "0" to indicate that the time stamp field does not exist. The time stamp field is, for example, a 16-bit field containing a time stamp generated by the source device 120 and can be added to the video data prior to transmission. The sink device 160 further generates a data packet (1005), which comprises the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1007). The sink device 160 may include components that allow the transfer of data packets, including, for example, the transport unit 333 and the wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 10B is a flowchart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 10B involves receiving a data packet from the wireless sync device 160 (1002), which may include, in particular, a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. Source device 120 then parses the data packet header contained in the data packet (1004). Source device 120 determines if the timestamp field is present in the data packet header (1006). In one example, the source device 120 may make a decision based on the timestamp flag value contained in the data packet header. If the data packet header contains a timestamp field, the source device 120 processes the payload data based on the timestamp in the timestamp field (1008). The data packet described with reference to FIGS. 10A and 10B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
FIG. 11A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 11A involves acquiring user input data (1101) on a wireless sync device such as the wireless sync device 160. User input data can be obtained through the user input component of the wireless sync device 160, for example, the user input interface 376 shown with reference to FIG. The sink device 160 then generates a data packet header based on user input (1103). The data packet header may include a time stamp field among the fields. The time stamp field may include, for example, a 16-bit field containing a time stamp based on multimedia data generated by the wireless source device 120 and transmitted to the wireless sync device 160. The time stamp may have been added to the frame of the video data by the wireless source device 120 before it was sent to the wireless sync device. The time stamp field may identify, for example, the time stamp associated with a frame of video data displayed on the wireless sync device 160 when user input data is captured. The sink device 160 further generates a data packet (1105), which comprises the generated data packet header and payload data. In one example, the payload data may include received user input data and may identify one or more user commands. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in FIG. 1A or source device 220 in FIG. 2) (1107). The sink device 160 may include components that allow the transfer of data packets, including, for example, the transport unit 333 and the wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 11B is a flow chart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 11B involves receiving a data packet from a wireless sync device such as the wireless sync device 160 (1102), the data packet may specifically include a data packet header and payload data. The payload data may include, for example, user input data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. Source device 120 then identifies the timestamp field in the data packet header (1104). Source device 120 processes payload data based on the time stamp in the time stamp field (1106). Based on the above time stamps, as part of processing the payload data, the source device 120 identifies a frame of video data that is displayed on the wireless sync device when the user input data is retrieved and that frame. The payload data can be interpreted based on the contents of. As part of processing the payload data based on the time stamp, the source device 120 compares the time stamp with the current time stamp for the current frame of video being transmitted by the source device 120. Obtained, in response to the time difference between the above time stamp and the current time stamp being less than the threshold, execute the user input command described in the payload data, or execute the above time stamp and the current time stamp. In response to the time difference from the time stamp being greater than the threshold, the user input command described in the payload data may not be executed. The data packet described with reference to FIGS. 11A and 11B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
FIG. 12A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 12A involves acquiring user input data (1201) on a wireless sync device such as the wireless sync device 160. In one example, the user input data can be voice command data, which is obtained through the user input component of the wireless sync device 160, for example, the voice command recognition module included in the user input interface 376 in FIG. Can be done. The sink device 160 generates a data packet header based on user input (1203). The sink device 160 also generates payload data (1205), which may include voice command data. In one example, the payload data may also include received user input data to identify one or more user commands. The sink device 160 further generates a data packet (1207), which comprises the generated data packet header and payload data. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1209). The sink device 160 may include components that allow the transfer of data packets, including, for example, the transport unit 333 and the wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 12B is a flowchart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 12B involves receiving a data packet (1202), which may include, in particular, a data packet header and payload data. The payload data may include user input data such as voice command data. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. The source device 120 then parses the payload data contained in the data packet to determine if the payload data comprises voice command data (1204). The data packet described with reference to FIGS. 12A and 12B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
FIG. 13A is a flowchart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 13A involves acquiring user input data (1301) on a wireless sync device such as the wireless sync device 160. In one example, the user input data can be a multi-touch gesture, which can be obtained through a user input component of the wireless sync device 160, for example, UI 167 or the user input interface 376 of FIG. In one example, a multi-touch gesture may include a first touch input and a second touch input. The sink device 160 generates a data packet header based on user input (1303). The sink device 160 also generates payload data (1305), which associates user input data for the first touch input event with the first pointer identification information and user input for the second touch input event. The data can be associated with a second pointer identification. The sink device 160 further generates a data packet (1307), which comprises the generated data packet header and payload data. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1309). The sink device 160 may include components that allow the transfer of data packets, including, for example, the transport unit 333 and the wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 13B is a flowchart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 13B involves receiving a data packet (1302), which may include, in particular, a data packet header and payload data. The payload data may include user input data such as, for example, multi-touch gestures. The source device 120 may include communication components that allow the transfer of data packets, including, for example, transport unit 233 and wireless modem 234, as shown in FIG. The source device 120 then parses the payload data contained in the data packet to identify the user input data contained in the payload data (1304). In one example, the identified data is the user input data for the first touch input event using the first pointer identification information and the user input for the second touch input event using the second pointer identification information. May include data. The source device 120 then interprets the user input data for the first touch input event and the user input data for the second touch input event as multi-touch gestures (1306). The data packet described with reference to FIGS. 13A and 13B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
FIG. 14A is a flow chart of an exemplary method of transmitting user input data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 14A involves retrieving user input data from an external device on the wireless sync device 360 (1401). In one example, the external device can be a third-party device connected to the sink device. The sink device 160 generates a data packet header based on user input (1403). In one example, the data packet header may identify the user input data as forwarded user input data. The sink device 160 also generates payload data (1405), which may include user input data. The sink device 160 further generates a data packet (1407), which may include the generated data packet header and payload data. The sink device 160 then sends the generated data packet to a wireless source device (eg, source device 120 in Figure 1A or source device 220 in Figure 2) (1409). The sink device 160 may include components that allow the transfer of data packets, including, for example, the transport unit 333 and the wireless modem 334, as shown with reference to FIG. Data packets can be sent to wireless source devices over TCP / IP.
FIG. 14B is a flow chart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 14B involves receiving a data packet (1402), which may include, in particular, a data packet header and payload data. The payload data may include user input data, for example, a forwarded user input command indicating that the user input data has been forwarded from a third party device. The source device 120 may include communication components that allow the transfer of data packets, including, for example, the transport unit 233 and the wireless modem 234, as shown with respect to FIG. The source device 120 then parses the data packet header and determines that the payload data comprises a forwarded user input command (1404). Source device 120 then parses the payload data contained in the data packet to identify the identity associated with the third-party device that corresponds to the forwarded user input command (1406). The source device 120 then processes the payload data based on the identified identity of the third party device (1408). The data packet described with reference to FIGS. 14A and 14B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
FIG. 15A is a flowchart of an exemplary method of transmitting user data from a wireless sink device to a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by a sink device 160 (FIG. 1A) or 360 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331).
The method of FIG. 15A involves acquiring user input data (1501) in a wireless sync device. User input data may have associated coordinate data. The associated coordinate data may correspond to, for example, the location of a mouse click event or the location of a touch event. The sink device 160 then normalizes the associated coordinate data to generate the normalized coordinate data (1503). The sink device 160 then generates a data packet containing normalized coordinate data (1505). Normalizing the coordinate data can include scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source display, such as display 22 of the source device 120. The resolution of the display window can be determined by the sink device 160 and the display resolution of the source device can be received from the source device 120. The sink device 160 then sends a data packet with normalized coordinates to the wireless source device 120 (1507). As part of the method in Figure 15A, the sync device 160 also determines if the associated coordinate data is in the display window for the content being received from the wireless source device, eg, associated. If the coordinate data is outside the display window, the user input can be processed locally, or if the input is inside the display window, the coordinates can be normalized as described.
FIG. 15B is a flow chart of an exemplary method of receiving user input data from a wireless sync device in a wireless source device in accordance with the present disclosure. The illustrated exemplary method can be performed by source device 120 (FIG. 1A) or 220 (FIG. 2). In some examples, when a computer-readable storage medium (eg, memory 232) is executed, one or more processors (eg, memory 232) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 231).
The method of FIG. 15B involves receiving a data packet on a wireless source device, which comprises user input data with associated coordinate data (1502). The associated coordinate data may correspond to, for example, the location of a mouse click event or the location of a touch event on the sink device. Source device 120 then normalizes the associated coordinate data to generate normalized coordinate data (1504). The source device 120 can normalize the coordinate data by scaling the associated coordinate data based on the ratio of the resolution of the display window to the resolution of the source display. The source device 120 can determine the display resolution of the source device and can receive the resolution of the display window from the wireless sync device. The source device then processes the data packet based on the normalized coordinate data (1506). The data packet described with reference to FIGS. 15A and 15B can generally take the form of the data packet described with reference to FIG. 6 and can be used to control audio / video data in the source device.
For the sake of brevity, aspects of the present disclosure have been described separately with reference to FIGS. 7-15. However, it is contemplated that these various aspects may be combined and used in relation to each other, not just separately. In general, the features and / or modules described herein can be implemented in either or both wireless source devices and wireless sync devices. In this way, the user interface features described in this example can be used interchangeably between wireless source devices and wireless sync devices.
The techniques of the present disclosure can be implemented in a wide variety of devices or devices, including wireless handsets and integrated circuits (ICs) or sets of ICs (ie, chipsets). Although any component, module or unit has been described and given to emphasize functional aspects, those optional components, modules or units do not necessarily require implementation by different hardware units. ..
Therefore, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the features described as modules, units or components can be implemented together in an integrated logical device or separately as a separate but interoperable logical device. When implemented in software, these techniques may be at least partially realized by a computer-readable medium with instructions that, when executed on a processor, performs one or more of the methods described above. The computer-readable medium may comprise a tangible, non-transitory computer-readable storage medium and may form part of a computer program product that may contain packaging material. Computer-readable storage media include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable read-only memory (EEPROM). , Flash memory, magnetic or optical data storage medium, etc. may be provided. The technique may, in addition or as an alternative, be at least partially realized by a computer-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.
The code can be one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrations. It can be performed by a circuit or a discrete logic circuit. Thus, the term "processor" as used herein may refer to either the aforementioned structure or any other structure suitable for implementing the techniques described herein. Further, in some embodiments, the functionality described herein may be provided within a dedicated software or hardware module configured for coding and decoding, or may be incorporated into a composite video codec. .. Also, the technique may be fully implemented in one or more circuits or logic elements.
Various aspects of the present disclosure have been described. These and other aspects fall within the scope of the following claims.
41 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US11562742B2 | Cited by | United States of America | Applicant |
| JP2020036382A | Cited by | Japan | Search report |
| JP2021099813A | Cited by | Japan | Search report |
| JP2008301249A | Cites | Japan | Search report |
| US2010257238A1 | Cites | United States of America | Search report |
| WO2012096546A2 | Cites | World Intellectual Property Organization (WIPO) | Search report |
168 members in 20 offices
Priority claims49
| Document | Office | Kind | Date |
|---|---|---|---|
| 201161435194 | United States of America | P | |
| 201161435194 | United States of America | P | |
| 61435194 | United States of America | – | |
| 201161447592 | United States of America | P | |
| 201161447592 | United States of America | P | |
| 61447592 | United States of America | – | |
| 201161448312 | United States of America | P | |
| 201161448312 | United States of America | P | |
| 61448312 | United States of America | – | |
| 201161450101 | United States of America | P | |
| 201161450101 | United States of America | P | |
| 61450101 | United States of America | – | |
| 201161467535 | United States of America | P | |
| 201161467535 | United States of America | P | |
| 201161467543 | United States of America | P | |
| 201161467543 | United States of America | P | |
| 61467535 | United States of America | – | |
| 61467543 | United States of America | – | |
| 201161514863 | United States of America | P | |
| 201161514863 | United States of America | P | |
| 61514863 | United States of America | – | |
| 201161544470 | United States of America | P | |
| 201161544470 | United States of America | P | |
| 61544470 | United States of America | – | |
| 13344253 | United States of America | – | |
| 201213344253 | United States of America | A | |
| 201213344253 | United States of America | A | |
| 2012022072 | United States of America | W | |
| 2012022072 | United States of America | W | |
| 2011435194 | – | – | – |
| 2011447592 | – | – | – |
| 2011448312 | – | – | – |
| 2011450101 | – | – | – |
| 2011467535 | – | – | – |
| 2011467543 | – | – | – |
| 2011514863 | – | – | – |
| 2011544470 | – | – | – |
| 2012344253 | – | – | – |
| 2012022072 | – | – | – |
| US201161435194P | – | – | – |
| US201161447592P | – | – | – |
| US201161448312P | – | – | – |
| US201161450101P | – | – | – |
| US201161467535P | – | – | – |
| US201161467543P | – | – | – |
| US201161514863P | – | – | – |
| US201161544470P | – | – | – |
| US201213344253 | – | – | – |
| WO2012US22072 | – | – | – |
Members168
| Document | Office | Kind | |
|---|---|---|---|
| CA2824287A1 | Canada | A1 | |
| CA2824559A1 | Canada | A1 | |
| CA2824563A1 | Canada | A1 | |
| CA2824567A1 | Canada | A1 | |
| WO2012100186A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100189A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100191A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100193A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100197A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100201A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100204A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100214A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012100218A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2013002949A1 | United States of America | A1 | |
| US2013003621A1 | United States of America | A1 | |
| US2013003622A1 | United States of America | A1 | |
| US2013003623A1 | United States of America | A1 | |
| US2013003624A1 | United States of America | A1 | |
| US2013009873A1 | United States of America | A1 | |
| US2013009887A1 | United States of America | A1 | |
| US2013009996A1 | United States of America | A1 | |
| US2013013318A1 | United States of America | A1 | |
| AU2012207073A1 | Australia | A1 | |
| AU2012207127A1 | Australia | A1 | |
| AU2012207129A1 | Australia | A1 | |
| AU2012207133A1 | Australia | A1 | |
| PH12013501451A1 | Philippines | A1 | |
| PH12013501482A1 | Philippines | A1 | |
| PH12013501483A1 | Philippines | A1 | |
| SG191367A1 | Singapore | A1 | |
| SG191377A1 | Singapore | A1 | |
| SG191763A1 | Singapore | A1 | |
| SG191765A1 | Singapore | A1 | |
| KR20130115370A | Republic of Korea | A | |
| KR20130115371A | Republic of Korea | A | |
| KR20130118958A | Republic of Korea | A | |
| CN103384995A | China | A | |
| CN103392160A | China | A | |
| CN103392161A | China | A | |
| CN103392325A | China | A | |
| CN103392326A | China | A | |
| CN103392359A | China | A | |
| CN103403649A | China | A | |
| CN103404104A | China | A | |
| CN103404114A | China | A | |
| KR20130126968A | Republic of Korea | A | |
| KR20130126969A | Republic of Korea | A | |
| KR20130126970A | Republic of Korea | A | |
| KR20130126971A | Republic of Korea | A | |
| KR20130126972A | Republic of Korea | A | |
| KR20130126973A | Republic of Korea | A | |
| EP2666069A1 | European Patent Office (EPO) | A1 | |
| EP2666073A1 | European Patent Office (EPO) | A1 | |
| EP2666074A1 | European Patent Office (EPO) | A1 | |
| EP2666274A1 | European Patent Office (EPO) | A1 | |
| EP2666275A1 | European Patent Office (EPO) | A1 | |
| EP2666276A1 | European Patent Office (EPO) | A1 | |
| EP2666277A1 | European Patent Office (EPO) | A1 | |
| EP2666278A1 | European Patent Office (EPO) | A1 | |
| EP2666323A1 | European Patent Office (EPO) | A1 | |
| JP2014506082A | Japan | A | |
| US8677029B2 | United States of America | B2 | |
| JP2014508995A | Japan | A | |
| HK1188005A | Hong Kong, China | A | |
| HK1188005A1 | Hong Kong, China | A1 | |
| JP2014509475A | Japan | A | |
| JP2014509476A | Japan | A | |
| JP2014510434A | Japan | A | |
| JP2014510435A | Japan | A | |
| ZA201305997B | South Africa | B | |
| JP2014510961A | Japan | A | |
| JP2014511582AThis record | Japan | A | |
| JP2014511583A | Japan | A | |
| ZA201306271B | South Africa | B | |
| UA107151C2 | Ukraine | C2 | |
| US8964783B2 | United States of America | B2 | |
| RU2013138718A | Russian Federation | A | |
| RU2013138723A | Russian Federation | A | |
| RU2013138748A | Russian Federation | A | |
| RU2013138750A | Russian Federation | A | |
| KR101503386B1 | Republic of Korea | B1 | |
| ZA201305995B | South Africa | B | |
| JP5694568B2 | Japan | B2 | |
| AU2012207073B2 | Australia | B2 | |
| JP5714726B2 | Japan | B2 | |
| AU2012207127B2 | Australia | B2 | |
| US9065876B2 | United States of America | B2 | |
| KR101533753B1 | Republic of Korea | B1 | |
| UA109176C2 | Ukraine | C2 | |
| AU2012207129B2 | Australia | B2 | |
| AU2012207133B2 | Australia | B2 | |
| UA109928C2 | Ukraine | C2 | |
| RU2567378C2 | Russian Federation | C2 | |
| JP5815741B2 | Japan | B2 | |
| ZA201404660B | South Africa | B | |
| JP5826860B2 | Japan | B2 | |
| JP5826861B2 | Japan | B2 | |
| JP2015222953A | Japan | A | |
| KR101572977B1 | Republic of Korea | B1 | |
| RU2571595C2 | Russian Federation | C2 |
18 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written permission of extension of timeJAPANESE INTERMEDIATE CODE: A602A602 | A602 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 |
Numbers
- Publication
- 2014511582
- Publication, DOCDB
- 2014511582
- Publication, EPODOC
- JP2014511582
- Application
- 2013550629
- Application, DOCDB
- 2013550629
- Application, EPODOC
- JP20130550629
Titles2
- Japanese
- ワイヤレスディスプレイのためのユーザ入力バックチャネル
- English
- User input back channel for wireless display
Classification
- CPC, 6
- H04L65/00
- H04L69/24
- H04W99/00
- H04L2012/5603
- H04W4/00
- H04W88/00
- IPC, 3
- H04L47 43
- H04Q9 00
- H04L29 06
Designated states5
- Regional, 4
- Zimbabwe
- Turkmenistan
- Türkiye
- Togo
- National, 1
- Viet Nam