User input device for wireless back channel

This application is the US Patent Provisional Application No. 1 entitled "WI-FI DISPLAY REVERSE CHANNEL SETUP AND INPUT DATA ENCAPSULATION" filed on February 4, 2011, the entire contents of which are incorporated herein by reference. Claims priority to 61 / 439,809 and US Patent Provisional Application No. 61 / 579,056, filed December 22, 2011, entitled "USER INPUT DEVICE FOR WIRELESS BACK CHANNEL".

The present disclosure relates to techniques for transmitting data between a wireless source device and a wireless sink device.

A wireless display (WD) or Wi-Fi® Display (WFD) system includes a source device and one or more sink devices. Each of the source device and the sink device may be either a mobile device or a wired device having a wireless communication function. For example, as mobile devices, one or more of the source and sink devices are mobile phones, portable computers with wireless communication cards, personal digital assistants (PDAs), portable media players, or so-called "smart" phones and It may include a "smart" pad or tablet, or other flash memory device with wireless communication capabilities, including other types of wireless communication devices. For example, as a wired device, one or more of the source and sink devices may include televisions, desktop computers, monitors, projectors, etc., including wireless communication capabilities.

The source device sends media data, such as audio-video (AV) data, to one or more of the sink devices participating in a particular media sharing session. Media data can be played on both the local display of the source device and the display of the sink device. More specifically, each of the participating sink devices renders the received media data to its screen and audio equipment.

The techniques of the present disclosure generally relate to communication between a wireless source device and a wireless sink device. More specifically, the present disclosure describes a technique that allows a wireless source device to act as an input device that controls rendered video data in a wireless sink device. For example, according to aspects of the present disclosure, a wireless source device can transmit video data to a wireless sink device for rendering and display. In addition, the wireless source device can send user input to control the rendered video data.

In one example, the wireless source device may include a touch display to provide user input to the wireless source device. That is, the touch display of a wireless source device can act as a touchpad or mousepad similar to a laptop touchpad. In this example, according to aspects of the present disclosure, the wireless source device can transmit video data to be rendered by the wireless sink device, as well as user input from the touch display. Thus, when a user makes a user input on the touch display of a wireless source device, the wireless source display may generate one or more events corresponding to the user input and send such an event to the wireless sink device. obtain. In addition, according to aspects of the present disclosure, the wireless source device can map the location of the user input on the wireless source device to the appropriate location on the wireless sink device.

In one example, aspects of the present disclosure relate to methods that include, by way of example, transmitting display data to the sink device for rendering to the sink device by the source device. The method also involves receiving instructions from the source device for user input at the first display location of the touch-sensitive screen associated with the source device, where the user input is at least part of the display data rendered on the sink device. Specify the modification of, and a part of the display data is rendered at the second display location of the sink device. The method was also rendered in the second display location by the source device, at least in part, based on mapping the first display location to the second display location in response to receiving instructions. Includes sending a second configuration data to modify some of the display data.

In another example, an aspect of the present disclosure comprises a memory that stores an instruction and one or more processors that are configured to execute the instruction, with one or more processors at the time of executing the instruction. : Sending display data to the sink device for rendering to the sink device; receiving user input instructions at the first display location of the touch-sensitive screen associated with the source device, where the user input Specify at least a modification of the display data rendered on the sync device so that some of the display data is rendered at the sink device's second display location; in response to receiving the display, the first Sends second configuration data to modify some of the display data rendered at the second display location, at least in part based on mapping the first display location to the second display location. Regarding wireless source devices that cause things.

In another example, an aspect of the present disclosure is that when executed by one or more processors of a wireless source device, to one or more processors: sending display data to the sink device for rendering to the sink device. Receiving the display of user input at the primary display location of the touch-sensitive screen associated with the wireless source device, where the user input specifies modification of at least some of the display data rendered on the sink device. At least part of mapping the first display location to the second display location in response to receiving the display, with some of the display data being rendered at the second display location on the sink device. With respect to a computer-readable storage medium that stores instructions that cause the transmission of second configuration data to modify some of the display data rendered at the second display location.

In another example, aspects of the disclosure are a means for transmitting display data to the sink device for rendering to the sink device; user input at the first display location of the touch-sensitive screen associated with the source device. A means for receiving instructions, where user input specifies modifications to at least some of the display data rendered on the sink device, and some of the display data is rendered at the second display location on the sink device. One of the display data rendered at the second display location, at least partially based on mapping the first display location to the second display location in response to receiving the display. It relates to a wireless source device that provides a means for transmitting a second configuration data to modify the part.

In another example, an aspect of the present disclosure is that the wireless sync device receives display data for rendering to the wireless sync device; the display rendered by the wireless sync device at the first location of the wireless sync device. Receiving configuration data to modify some of the data, with some of the display data being rendered at the second display location of the wireless source device; at least partially by the wireless sync device. The present invention relates to a method of rendering a part of display data at the first location based on the mapping between the first display location and the second display location.

In another example, an aspect of the present disclosure comprises a memory that stores an instruction and one or more processors that are configured to execute the instruction, with one or more processors at the time of executing the instruction. : The wireless sync device receives display data for rendering to the wireless sync device; the wireless sync device is configured to modify some of the display data rendered at the primary location of the wireless sync device. Receiving the instruction data, with some of the display data being rendered at the second display location of the wireless source device; by the wireless sync device, at least in part the first display location and the second With respect to a wireless sync device that causes the rendering of part of the display data at the first location based on the mapping to the display location.

In another example, an aspect of the present disclosure is that when executed by one or more processors of a wireless sync device, one or more processors: receive display data for rendering to the wireless sync device; wireless. Receiving configuration data to modify some of the display data rendered at the first location of the sink device, some of the display data being rendered at the second display location of the wireless source device. And; a computer that stores instructions that cause it to render some of the display data at the first location, at least in part, based on the mapping between the first display location and the second display location. Regarding readable storage media.

In another example, an aspect of the present disclosure is a means for receiving display data for rendering to a wireless sync device; to modify a portion of the display data rendered at the first location of the wireless sync device. A means for receiving configuration data for, in which some of the display data is rendered at the second display location of the wireless source device, with means; at least in part the first display location and the second. It relates to a wireless sync device that provides a means for rendering a portion of display data at a first location based on a mapping with a display location.

Details of one or more examples are given in the accompanying drawings and in the description below. Other features, objectives, and advantages will become apparent from its description and drawings, as well as the claims.

<figref num="1A">The block diagram which shows an example of the source / sink system which can implement the technique of this disclosure.</figref><figref num="1B">The block diagram which shows an example of the source / sink system which has two sink devices which can implement the technique of this disclosure.</figref><figref num="2">The block diagram which shows an example of the source device which can implement the technique of this disclosure.</figref><figref num="3">The block diagram which shows an example of the sink device which can implement the technique of this disclosure.</figref><figref num="4">The block diagram which shows an example of the transmitter system and the receiver system which can implement the technique of this disclosure.</figref><figref num="5">A conceptual diagram of a sink device and a source device that can implement the techniques of the present disclosure.</figref><figref num="6">A flow diagram illustrating an exemplary method that can allow a wireless source device to act as an input device for data rendered in a wireless sink device.</figref><figref num="7">A flow diagram illustrating an exemplary method that can allow a wireless sink device to render display data based on user input from a wireless source device.</figref>

A wireless display (WD) system may include a wireless display (WD) source and at least one wireless display sink. In addition, the wireless display system may include multiple wireless display sinks. A wireless display source may transmit AV data to a wireless display sync that can instantly render the received audio video (AV) data on the terminal device. In some examples, audio-video data comprises a series of frames (or images) that are played rapidly and continuously, and the frames can also contain related audio frames, but also when they do not. is there.

In some cases, the wireless display source may be prompted to respond to user input applied in the wireless display sync. In such interactive applications, user input made in the wireless display sync may be sent back to the wireless display source. In one example, a reverse channel structure may be implemented to allow the wireless display system to send the user input applied in the wireless display sync to the wireless display source. The reverse channel structure may include a higher layer message for transporting user input and a lower layer frame for negotiating user interface functionality in wireless display sinks and wireless display sources.

The wireless display reverse channel may reside on top of the Internet Protocol (IP) transport layer between the wireless display sink and the wireless display source. To facilitate reliable transmission to user input and ordered delivery, the wireless display reverse channel may be configured to run on top of TCP / IP. In some cases, there may be inconsistencies between the user input interface in the wireless display sink and the wireless display source. In order to solve the problems caused by such inconsistencies and promote a good user experience in such an environment, user input interface feature negotiations occur between the wireless display sink and the wireless display source.

The techniques of the present disclosure generally relate to communication between a wireless source device and a wireless sink device. More specifically, the present disclosure describes techniques that can deliver video data to a sink device and allow the wireless source device to act as an input device that controls the rendered video data in the wireless sink device. For example, according to aspects of the present disclosure, a wireless source device can transmit video data to a wireless sink device for rendering and display. In addition, the wireless source device can send user input to control the rendered video data.

In one example, the wireless source device may include a touch display to allow user input to the wireless source device. That is, the touch display of a wireless source device acts as a touchpad or mousepad that resembles a laptop touchpad. In this example, according to aspects of the present disclosure, the wireless source device can transmit video data to be rendered by the wireless sink device, as well as user input from the touch display. Thus, when a user makes a user input on the touch display of a wireless source device, the wireless source display may generate one or more events corresponding to the user input and send such an event to the wireless sink device. obtain. In addition, according to aspects of the present disclosure, the wireless source device can map the location of the user input on the wireless source device to the appropriate location on the wireless sink device.

FIG. 1A is a block diagram showing an exemplary source / sync system 100 that may implement the techniques of the present disclosure. As shown in FIG. 1A, the system 100 includes a source device 120 that communicates with the sink device 160 over the communication channel 150. The source device 120 includes a memory for storing audio / video (A / V) data 121, a display 122, a speaker 123, an audio / video encoder 124 (also called an encoder 124), an audio / video control module 125, and the like. It may include a transmitter / receiver (TX / RX) unit 126. The sink device 160 includes a display 162, a speaker 163, an audio / video decoder 164 (also called a decoder 164), a transmitter / receiver unit 166, a user input (UI) device 167, and a user input processing module (UIPM). ) 168 and can be included. The illustrated components constitute only one example, and other examples may include fewer or additional components than these illustrated components.

In the example of FIG. 1A, the source device 120 may display the video portion of the audio / video data 121 on the display 122 and output the audio portion of the audio / video data 121 to the speaker 122. The audio / video data 121 can be stored locally on the source device 120 and can be accessed from a file server, Blu-ray Disc®, or an external storage medium such as a DVD, or via a network connection such as the Internet. Can be streamed to 120. In some examples, audio / video data 121 may be captured in real time through the camera and microphone of the source device 120. The audio / video data 121 may include multimedia content such as movies, television programs, or music, but may also include real-time content generated by the source device 120. Such real-time content can be created, for example, by an application running on the source device 120. In one example, such real-time content may also include video frames of user input options available for the user to select. In some examples, audio / video data 121 may include video frames that are a combination of different types of content, such as video frames for movies or TV shows that have user input options overlaid on video frames.

In addition to rendering the audio / video data 121 locally through the display 122 and the speaker 123, the audio / video encoder 124 of the source device 120 can encode the audio / video data 121 and the transmitter / receiver unit. The 126 may transmit the encoded data to the sink device 160 via the communication channel 150. The transmitter / receiver unit 166 of the sink device 160 receives the encoded data, the audio / video decoder 164 decodes the encoded data, and outputs the decoded data via the display 162 and the speaker 163. In this way, audio and video data rendered by display 122 and speaker 12 can be rendered simultaneously by display 162 and speaker 163. The audio and video data may be organized within the frame, and the audio frame may be time synchronized with the video frame when rendered.

The audio / video encoder 124 and audio / video decoder 164 are also known as the ITU-T H.264 standard, also known as MPEG-4 Part 10 Advanced Video Coding (AVC), or the newly emerged, high, sometimes called the H.265 standard. Any number of audio and video compression standards, such as the efficiency video coding (HEVC) standard, can be implemented. Generally, the audio / video decoder 164 is configured to perform reciprocal coding operations on the audio / video encoder 124. Although not shown in FIG. 1A, in some embodiments, the A / V encoder 124 and the A / V decoder 164 may be integrated with the audio encoder and decoder, respectively, and the audio in a common or separate data stream. It may include a suitable MUX-DEMUX unit, or other hardware and software, to handle both the encoding of the video and the video.

FIG. 1A shows a communication channel 150 carrying audio payload data and video payload data separately, but in some examples the video payload data and audio payload data may be part of a common data stream. Please understand that. Where applicable, the MUX-DEMUX unit can comply with the ITU H.223 multiplexer protocol, or other protocols such as the User Datagram Protocol (UDP). The audio / video encoder 124 and audio / video decoder 164 are each one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software. , Hardware, firmware, or any combination thereof. Each of the audio / video encoder 124 and the audio / video decoder 164 may be included in one or more encoders or decoders, both of which may be integrated as part of a composite encoder / decoder (codec).

The displays 122 and 162 may comprise any of a variety of video output devices, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, organic light emitting diode (OLED) displays, or other types of display devices. .. Speaker 123 may include any of a variety of audio output devices, such as headphones, single speaker systems, multi-speaker systems, or surround sound systems. In addition, the display 122 and speaker 123 are shown as part of the source device 120, and the display 162 and speaker 163 are shown as part of the sink device 160, but the source device 120 and the sink device 160 are actually them. It may be a system of devices. As an example, the display 162 may be a television, the speaker 163 may be a surround sound system, and the decoder 164 may be part of an external box connected to the display 162 and the speaker 163 either wired or wirelessly. It may be there. In another example, the sink device 160 may be a single device such as a tablet computer or smartphone. In yet other cases, the source device 160 and the sink device 120 are similar devices, such as both smartphones, tablet computers, and the like. In this case, one device may act as a source and the other as a sink. These roles may be reversed in subsequent communication sessions.

The transmitter / receiver unit 126 and the transmitter / receiver unit 166, respectively, for transmitting and receiving various mixers, filters, amplifiers, and other components designed for signal modulation, as well as data. It may include one or more antennas and other components designed. The communication channel 150 generally represents any communication medium suitable for transmitting video data from the source device 120 to the sink device 160, or a collection of various communication media. The communication channel 150 is usually a relatively short-range communication channel similar to WiFi (registered trademark), Bluetooth (registered trademark) and the like. However, the communication channel 150 is not necessarily limited in this regard, with any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or with wireless and wired media. Can be any combination of. In another example, the communication channel 150 can also form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The source device 120 and sink device 160 may communicate over a communication channel that uses a communication protocol such as a standard according to the IEEE 802.11 standard family.

In addition to decoding and rendering the data received from the source device 120, the sink device 160 may also receive user input from the user input device 167. The user input device 167 may be, for example, a keyboard, mouse, trackball or trackpad, touch screen, voice command recognition module, or any other such user input device. The user input processing module 168 formats the user input command received by the user input device 167 into a data packet structure that can be interpreted by the source device 120. Such data packets are transmitted by the transmitter / receiver 166 to the source device 120 over the communication channel 150. The transmitter / receiver unit 126 receives the data packet and the A / V control module parses the data packet to interpret the user input command received by the user input device 167. Based on the command received in the data packet, the A / V control module 125 can modify the encoded and transmitted content. In this way, the user of the sink device 160 can control the audio payload data and the video payload data remotely transmitted by the source device 120 and does not interact directly with the source device 120. Examples of the types of commands that a user of sink device 160 can send to source device 120 are commands for rewinding, fast-forwarding, pausing, and playing audio-video data, as well as commands for zooming, rotating, scrolling, and so on. including. The user can also select, for example, from a menu of options and send the selection back to the source device 120.

The source device 120 may respond to user input applied in the sink device 160. In such an interactive application setting, user input applied on the sink device 160 may be sent back to the wireless display source via the communication channel 150. In one example, a reverse channel structure, also called a user interface back channel (UIBC), may be implemented to allow the sink device 160 to send user input applied to the sink device 160 to the source device 120. The reverse channel structure may include a higher layer message for transporting user input and a lower layer frame for negotiating user interface functionality on the sink device 160 and the source device 120. The UIBC may reside on the Internet Protocol (IP) transport layer between the sink device 160 and the source device 120. To facilitate the reliable transmission and orderly delivery of data packets containing user-entered data, UIBC uses other packet-based communications, such as Transmission Control Protocol / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP). It may be configured to run at the top of the protocol.

In some cases, there may be an inconsistency between the user input interface located on the source device 120 and the sink device 160. In order to solve the potential problems created by such inconsistencies and promote a good user experience in such an environment, the user input interface feature negotiations before establishing a communication session, source device 120 May occur between and the sink device 160.

UIBC can be designed to transport various types of user input data, including cross-platform user input data. For example, the source device 120 may run an iOS® operating system, while the sink device 160 may run another operating system, such as Android® or Windows®. Regardless of the platform, the UIPM 168 can encapsulate the received user input in a form that is understandable to the A / V control module 125. Several different types of user input formats may be supported by UIBC to allow many different types of source and sink devices to utilize the protocol. General-purpose input formats may be defined and platform-specific input formats may also be supported, thus providing flexibility in a way that UIBC can communicate between source device 120 and sink device 160. ..

In the example of Figure 1A, the source device 120 may include a smartphone, tablet computer, laptop computer, desktop computer, WiFi-enabled television, or any other device capable of transmitting audio and video data. The sync device 160 can also be a smartphone, tablet computer, laptop computer, desktop computer, WiFi-enabled television, or any other device that can receive audio and video data and receive user input data. Can be prepared. In some examples, the sink device 160 includes a system of devices such that the display 162, the speaker 163, the UI device 167, and the A / V encoder 164 are all separate but interoperable parts of the device. Sometimes. Similarly, the source device 120 may be a system of devices rather than a single device. In many cases, the source device 120 and the sink device 160 may be similar or equivalent devices in which one device acts as a source and the other device acts as a sink. Moreover, these roles may be reversed in different communication sessions.

An exemplary technique of the present disclosure may allow the source device 120 to act as an input device capable of controlling the video data rendered in the sink device 160. That is, for example, the source device 120 may transmit video data to the sink device 160 for rendering and display on the display 162. In this way, the video data on the source device 120 can be extended to the display on the sink device 160.

The source device 120 may also include a touch display. For example, the display 122 may be a touch display. That is, in some examples, the display 122 may be a capacitive, resistive, or other type of touch panel that allows the user to input user input to the source device 120. In this way, the display 122 of the source device 120 can act as a touchpad or mousepad that resembles a laptop touchpad. Thus, when the user makes a user input on the display 122, the source device 120 may generate one or more events corresponding to the user input.

In one example, one or more events may include parameters that further describe the characteristics of the event, such as the type of user input, the graphical object selected by the user input. Upon generating an event, the source device 120 may send one or more events to the sink device 160. In some examples, the source device 120 may use Real-Time Transport Protocol (RTP) and User Datagram Protocol (UDP) to send events to the sink device 160. In another example, the source device 120 may use Transmission Control Protocol (TCP) and Internet Protocol (IP) to send events. In either case, the sink device 160 may receive one or more events and may modify the video data rendered by the sink device 160 based on the events. For example, the location or appearance of a graphical pointing object (eg, mouse pointer) displayed on the sink device 160 may be generated or modified. In other cases, the source device 120 may modify the display data rendered in the source device 120, and data representing any modification to the display data may be transmitted by the source device 120 to the sink device 160.

In this way, the display 122 of the source device 120 can be used as a mouse pad that can operate in the sink device 160. Therefore, in one scenario, the source device 120 may be a smartphone that renders the display data. The display data can be extended to a sink device 160, which can be a larger display, such as an LED television. According to some aspects of the disclosure, the display 122 is designed to conserve power while the movement of the finger touched by the user may be tracked, as described in more detail with respect to FIG. 5 below. The display function may be turned off and the corresponding action may be rendered on the sink device 160. For example, display 122 may not display video data, but pointers may be displayed on sink device 160. In some examples, when the source device 120 acts as a user input device, it does not have a touch-sensitive display but can render the display data received from the source device 120. Touch interactivity on the sink device 160. To enable. In this way, the user interface function may be used interchangeably between the source device 120 and the sink device 160.

Aspects of the present disclosure further provide techniques for mapping the resolution of the wireless sink device 120 to the resolution of the wireless source device 160. Mapping the resolution of the wireless device allows the events generated on the source device 120 to be translated to the sink device 160. For example, when the source device 120 and the sink device 160 first communicate, the devices may exchange display resolution information.

Such a mapping corresponds to the relative position of the source device on display 122 on the display 162 of the sink device 160, regardless of whether the display 122 has a different resolution and / or size than the display 162. Allows you to. For example, if the user later makes a user input in the upper left corner of the display 122 of the source device 120, the source device 120 can generate an event. The event may specify a user-entered location on display 122 and the type of event. In response to generating an event, the source device 120 bases the location of display 162 on sink device 160, which corresponds to the location of display 122 on wireless source device 120, based on the resolution mapping between the source device and the sink device. Can be decided. Thus, if user input is made in the upper left corner of display 122, that input may be mapped to the upper left corner of display 162.

In some examples, the source device 120 may modify the display data rendered by the source device 120 based on mapping and user input. Data indicating the modification may then be transmitted by the source device 120 to the sink device 160 to update the display data rendered on the sink device 160. In an alternative example, the source device 120 may send an event to the sink device 160, which may modify the display data based on the event and the resolution mapping between the source device and the sink device.

In some examples, the display 122 of the source device 120 may act as both an input device and an output device. For example, the display 122 can act as an input device that uses a capacitive, resistive, or other coupling to determine when a user makes a touch gesture on the display 122. The display 122 can also act as an output device by displaying display data using a liquid crystal display (LCD) display format or other display format. In some aspects of the disclosure, when the source device 120 is used as an input device to modify the display data rendered on the sink device 160, the source device 120 disables the output device function of the display 122. can do. For example, the source device 120 can disable or turn off the LCD display of the touch display to conserve power while the capacitive coupling feature remains enabled.

In some examples, different user inputs may be received on both the source device 120 and the sink device 160. Therefore, synchronization issues can occur when events representing user input are transported from one device to another. For example, the display data rendered by the source device 120 may be sent to the sink device 160 for rendering. The user can make user input on the sink device 160, which may generate an event sent to the source device 120. Another user may make user input on the source device 120 before the source device 120 receives the event. In such an example, a time stamp may be given to each user input to synchronize the way the user input is applied to the display data rendered by the source device 120. For example, a first user input with an earlier time stamp may be added before a second user input with a later time stamp.

In another example, the location of the display data rendered by the source device 120 may have changed due to user input received on the source device 120 before the event was received from the sink device 160. .. Since the event generated on the sink device 160 was based on the previous location of the display data, the source device 120 could evaluate the time stamp associated with the event to further identify the updated location of the display data. it can. Based on the event, the source device 120 may modify the display data using the updated location of the graphical object. In this way, the techniques of the present disclosure provide synchronization of display data across multiple devices.

Thus, FIG. 1A transmits display data to the sink device 160 by the source device 120 for rendering to the sink device 160; a display rendered on the sink device 160 and rendered at a second display location on the sink device 160. A display of user input specifying at least some modification of the data is received by the source device 120 at the first display location of the touch-sensitive screen associated with the source device 120; in response to receiving the display. To modify some of the display data rendered at the second display location, based at least in part on mapping the first display location to the second display location, the second configuration data, An example of a source / sync system 100 that can be transmitted by the source device 120 is provided.

FIG. 1B is a block diagram showing an exemplary source / sync system 101 that may implement the techniques of the present disclosure. The source / sink system 101 includes a source device 120, a sink device 160A, and a sink device 160B (collectively, a sink device 160). In some examples, the source device 120 and the sink device 160 can function and operate in the manner described above with respect to FIG. 1A. That is, in a manner similar to that described for the sink device 160 shown in FIG. 1A, the sink devices 160A and 160B may receive audio and video data from the source device 120. For example, in some configurations, the audio and video output on the source device 120 may be output simultaneously on the sink device 160A and the sink device 160B. In addition, according to aspects of the present disclosure, user input made on the source device 120 may be transmitted to the sink device 160A and the sink device 160B.

Therefore, it should be understood that, in general, the techniques of the present disclosure can be extended to support multiple source and / or sink devices. That is, according to aspects of the present disclosure, the source device 120 is an input capable of controlling the video data rendered on the sink devices 160A, 160B, and any other sink device that has established a connection with the source device 120. Can work as a device.

Therefore, the source device 120 may perform mapping for each of the connected sink devices 160. For example, when the source device 120 first communicates with each of the sink devices 160, the devices may exchange display resolution information. Such mapping is such that the relative position of the source device 120 on the display is the respective display of the sink device 160, regardless of whether the display of the source device 120 has a different resolution and / or size than the display of the sink device 160. Allows you to correspond to the same relative position above.

In some examples, the source device 120 may modify the display data rendered by the source device 120 based on mapping and user input. Data indicating the modification may then be transmitted by the source device 120 to the sink device 160 to update the display data rendered on the sink device 160. In an alternative example, the source device 120 may send an event to the sink device 160, which may modify the display data based on the event and the resolution mapping of the source device and the sink device. Moreover, timestamps (or other methods of tracking user input) may be maintained to avoid synchronization issues, as described above with respect to FIG. 1A.

The sink device 160A and the sink device 160B are assigned the same number, but in some examples the sink device 160A and the sink device 160B may have different capabilities and / or perform different functions. Please understand that it is good. For example, in some configurations, the sink device 160A may be the primary sink device and the sink device 160B may be the secondary sink device. In such a configuration, the sink device 160A and the sink device 160B may be combined so that the sink device 160A can display video data while the sink device 160B can output the corresponding audio data.

FIG. 2 is a block diagram showing an example of a source device 220 capable of performing some of the techniques of the present disclosure. In some examples, the source device 220 may be configured similar to the source device 120 illustrated and described with respect to FIG. 1A. In the example shown in FIG. 2, the source device 220 includes a local display 222, a local speaker 223, one or more processors 231 and a memory 232, a transport module 233, and a wireless modem 234.

Processor 231 in Figure 2 generally includes, but is not limited to, one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), and other equivalents. Represents any of a wide variety of processors, including integrated or discrete logic circuits, or some combination thereof. The memory 232 in FIG. 2 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It may have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM®), flash memory, etc., and memory 232 stores audio / video data as well as other types of data. A computer-readable storage medium for this purpose may be provided. Memory 232 may additionally store instructions and program code executed by processor 231 as part of performing the various techniques described herein.

In operation, processor 231 of the source device 220 can encode and / or decode A / V data for transport, storage, and display. The A / V data may be stored in memory 232, for example. Memory 232 can contain a smaller buffer that can store all A / V files, or only a portion of an A / V file, for example streamed from another device or source. Transport module 233 can process encoded A / V data for network transport. For example, the encoded A / V data can be processed by the multimedia processor 231 and encapsulated within a network access layer (NAL) unit by the transport module 233 for communication over the network. The NAL unit may be transmitted by the wireless modem 234 to the wireless sink device over a network connection.

The source device 220 can also process and display A / V data locally. Specifically, the display processor 235 may process the video data to be displayed on the local display 222, and the audio processor 236 may process the audio data for output to the speaker 223.

In the example shown in FIG. 2, the source device 220 includes a touch display interface 237. In some examples, the touch display interface 237 may be a capacitive, resistive, or other type of touch panel that allows the user to perform user input. For example, the user can provide one or more touch gestures to the touch display interface 237, which can generate one or more events corresponding to user input.

In one example, one or more events may include parameters that further describe the characteristics of the event, such as the type of user input, the graphical object selected by the user input. After generating the event, the source device 220 can send one or more events to the sink device (for example, the sink device 160 shown in FIG. 1). That is, the source device 220 can encapsulate the event data, which can then be transmitted to the sink device via the WiFi modem 234. In some examples, the source device 220 may use Real Time Transport Protocol (RTP) and User Datagram Protocol (UDP) to send events to the sink device. In another example, the source device 220 may use Transmission Control Protocol (TCP) and Internet Protocol (IP) to send events.

In either case, the sink device may receive one or more events and modify the video data rendered on the sink device based on the events. For example, the location or appearance of a graphical pointing object (eg, a mouse pointer) displayed on a sink device may be generated or modified. In other cases, the source device 220 may modify the display data rendered in the source device 220, and data representing any modification to the display data may be transmitted by the source device 220 to the sink device.

As described above for the source device 120 of FIG. 1A, the source device 220 may also receive user input commands from the sink device. In this way, the wireless modem 234 of the source device 220 receives the encapsulated data packet, such as the NAL unit, and sends the encapsulated data unit to the transport unit 233 for decapsulation . For example, transport unit 233 can extract data packets from the NAL unit, and processor 231 can parse data packets to extract user input commands. Based on the user input command, processor 231 can adjust the encoded A / V data transmitted by the source device 220 to the sink device.

FIG. 3 shows an example of a sink device 360 that can perform some of the techniques of the present disclosure. In some examples, the sink device 360 may be configured similar to the sink device 160 illustrated and described with respect to FIG. 1A. In the example shown in FIG. 3, the sink device 360 includes one or more processors 331, memory 332, transport unit 333, wireless modem 334, display processor 335, local display 362, and audio processor 336. , Includes speaker 363 and touch display interface 337.

Processor 331 in Figure 3 is one or more diverse processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), and field programmable logic arrays (FPGAs). , Other equivalent integrated or discrete logic circuits, or some combination thereof. Memory 332 in FIG. 3 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It may have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM), flash memory, etc., and memory 232 is a computer for storing audio / video data as well as other types of data. A readable storage medium may be provided. Memory 332 may additionally store instructions and program code executed by processor 331 as part of performing the various techniques described herein.

According to aspects of the disclosure, the sink device 360 is an encapsulated data unit transmitted from a source device (eg, source device 120 (FIG. 1A) or source device 220 (FIG. 2)) in a wireless modem 334. Can be received. The transport unit 333 can decapsulate the encapsulated data unit. For example, transport unit 333 extracts encoded video data from the encapsulated data unit and sends the encoded A / V data to processor 331 to be decoded and rendered for output. can do. The display processor 335 may process the decoded video data to be displayed on the local display 362, and the audio processor 336 may process the decoded audio data for output to the speaker 363.

In addition to rendering audio and video data, the wireless sync device 360 may also receive user input via the touch display interface 337. The example in Figure 3 uses a touch display interface as an exemplary input device, but any other input device, such as a keyboard, mouse, or voice command module, fits the techniques of the present disclosure. User input received via the touch display interface 337 may be processed by processor 331. This process may include generating a data packet containing the received user input command by the technique described in the present disclosure. Once generated, transport module 333 may process data packets for network transport to wireless source devices via UIBC.

FIG. 4 is a block diagram of an exemplary transmitter system 410 and receiver system 450 that can be used by the transmitter / receiver 126 and transmitter / receiver 166 of FIG. 1A to communicate over communication channel 150. Is shown. In transmitter system 410, traffic data from several data streams is fed from data source 412 to transmit (TX) data processor 414. Each data stream can be transmitted via individual transmit antennas. The TX data processor 414 formats, encodes, and interleaves the traffic data in each data stream based on the particular coding scheme selected for that data stream.

The encoded data in each data stream can be multiplexed with pilot data using Orthogonal Frequency Division Multiplexing (OFDM) techniques. A wide variety, but not limited to, including time division multiple access (TDMA), frequency division multiple access (FDMA), code division multiple access (CDMA), or any combination of OFDM, FDMA, TDMA, and / or CDMA. Other wireless communication techniques can be used as well.

According to FIG. 4, the pilot data is typically a known data pattern that can be processed in a known manner and used in the receiver system to estimate the channel response. The multiplexed pilot and coded data of each data stream is then given a particular modulation scheme (eg, 2-phase shift keying (BPSK), 4-phase) selected for that data stream to provide modulation symbols. Modulated (eg, symbol mapping) based on shift keying (QPSK), M-PSK, or M-QAM (quadrature keying), where M is a power of 2. The data rate, encoding, and modulation of each data stream can be determined by instructions executed by processor 430, which can be coupled to memory 432.

A modulation symbol for the data stream is then given to the TX MIMO processor 420, which may further process that modulation symbol (for example, for OFDM). The TX MIMO processor 420 may then supply NT modulation symbol streams to NT transmitters (TMTR) 422a-422t. In certain embodiments, the TX MIMO processor 420 applies beamforming weights to the symbols of the data stream and the antenna to which the symbols are transmitted.

Each transmitter 422 receives and processes individual symbol streams to supply one or more analog signals and further tunes (eg, amplifies, filters, and upconverts) those analog signals. It supplies a modulated signal suitable for transmission over the MIMO channel. Next, the NT modulated signals from the transmitters 422a to 422t are transmitted from the NT antennas 424a to 424t, respectively.

In the receiver system 450, the transmitted modulated signal is received by NR antennas 452a to 452r, and the received signal from each antenna 452 is supplied to individual receivers (RCVR) 454a to 454r. The receiver 454 tunes (eg, filters, amplifies, and downconverts) the individual received signals, digitizes the tuned signals, feeds the samples, and further processes and responds to those samples. Supply the "receive" symbol stream.

The receive (RX) data processor 460 then receives and processes NR receive symbol streams from NR receivers 454 based on a particular receiver processing technique to produce NT "detection" symbol streams. give. The RX data processor 460 then demodulates, deinterleaves, and decodes each detected symbol stream to restore the traffic data in the data stream. The processing by the RX data processor 460 complements the processing performed by the TX MIMO processor 420 and the TX data processor 414 in the transmitter system 410.

Processor 470, which can be combined with memory 472, periodically determines which precoding matrix should be used. The reverse link message can include various types of information about the communication link and / or the received data stream. The reverse link message is then processed by the TX data processor 438, which also receives traffic data for some data streams from the data source 436, modulated by modulator 480, tuned by transmitters 454a-454r, and transmitter. Returned to system 410.

In transmitter system 410, the modulated signal from receiver system 450 is received by antenna 424, tuned by receiver 422, demodulated by demodulator 440, and extracts the reverse link message transmitted by receiver system 450. Processed by the RX data processor 442 to. Processor 430 then determines which precoding matrix should be used to determine the beamforming weights, and then processes the extracted message.

FIG. 5 is a conceptual diagram of a sink device and a source device that can implement the techniques of the present disclosure. Although FIG. 5 describes the source device 120 and the sink device 160 and (FIG. 1A), it is understood that the technique of FIG. 5 can be performed by a variety of devices, including other source and sink devices. I want to.

In the example shown in FIG. 5, the display 122 of the source device 120 is configured as a touch-sensitive display. That is, for example, the display 122 may be a capacitive, resistive, or other type of touch panel that allows the user to make user inputs to the source device 120. Thus, when the user 500 makes a user input on the display 122, the source device 120 may generate one or more events corresponding to the user input. In some examples, such user input is sometimes referred to as a "touch gesture." The touch gesture corresponds to the user 500 touching the display 122 at one or more points. That is, the display 122 may be designed to recognize touches at two or more points at the same time, and the feature is generally called "multi-touch". Multi-touch displays can individually identify and interpret touches in different areas of the screen. Thus, touching the display 122 simultaneously at two or more locations may provide the user with an additional way to interact with the source device 120 (eg, for a single touch).

As mentioned above, the exemplary techniques of the present disclosure may allow the source device 120 to act as an input device capable of controlling the video data rendered in the sink device 160. For example, the source device 120 may transmit video data to the sink device 160 for rendering and display on the display 162 (shown in FIG. 5 as a series of cubes 504). In this way, the video data on the source device 120 can be extended to the display on the sink device 160.

In addition, according to aspects of the present disclosure, the source device 120 may transmit one or more events to the sink device 160. For example, one or more events may contain parameters that further describe the characteristics of the event, such as the type of user input (eg, the type of touch gesture), the graphical object selected by the user input. In some examples, the source device 120 may use Real-Time Transport Protocol (RTP) and User Datagram Protocol (UDP) to send events to the sink device 160. In another example, the source device 120 may use Transmission Control Protocol (TCP) and Internet Protocol (IP) to send events to the sink device 160.

In either case, the sink device 160 may receive one or more events and may modify the video data rendered by the sink device 160 based on the events. In one example, the location or appearance of a graphical pointing object (eg, mouse pointer) 508 displayed on the sink device 160 may be generated or modified. In another example, the source device 120 may modify the display data rendered in the source device 120, and data representing any modification to the display data may be transmitted by the source device 120 to the sink device 160.

In this way, the display 122 of the source device 120 can be used as a mouse pad that can operate in the sink device 160. Therefore, in one scenario, the source device 120 may be a smartphone that renders the display data. The display data can be extended to a sink device 160, which can be a larger display, such as an LED television.

According to some aspects of the disclosure, the display function of the display 122 may be turned off to conserve power while the user's touch / finger movements may be tracked, and the corresponding action may be taken. It can be rendered on the sink device 160. In the example shown in FIG. 5, the user 500 makes a touch input from the lower left corner of the display 122 toward the relative center of the display 122, as shown by the dashed line 512 (for example, while the user 500 maintains contact with the display 122). Swipe touch gesture) can be performed. In this example, the graphical pointing object 508 displayed on display 162 of sink device 160 follows a path 516 substantially similar to the input made on source device 120, from the lower left corner of display 162 to the center of display 162. Can move towards. In addition, according to aspects of the present disclosure, the graphical pointing object 508 may not be visible on display 122 of source device 120, but may be visible on display 162 of sink device 160.

Aspects of the present disclosure further provide techniques for mapping the resolution of the wireless sink device 120 to the resolution of the wireless source device 160. By mapping the resolution of the wireless device, the events generated by the source device 120 can be translated to the sink device 160. For example, when the source device 120 and the sink device 160 first communicate, the devices may exchange display resolution information.

Such a mapping corresponds to the relative position of the source device on display 122 on the display 162 of the sink device 160, regardless of whether the display 122 has a different resolution and / or size than the display 162. Allows you to. That is, as described above, if the user 500 makes a touch input to the lower left corner of the display 122 of the source device 120, the input can also be made to the lower left corner of the display 162 of the sink device 160. In addition, if the user makes a swipe gesture on the display 122 towards the relative center of the display 122 (as shown by the dashed line 512), the swipe gesture is performed on the sink device 160 towards the center of the display 162. Can be done. Even though the display 122 has a different resolution than the display 162, the resolution of the source device 120 and the sink device 160 is such that when the user 500 reaches the relative center of the display 122, the graphical pointing object 508 is also on the display 162. It can be mapped to reach the relative center.

In some examples, the source device 120 may modify the display data rendered by the source device 120 based on mapping and user input. Data indicating the modification may then be transmitted by the source device 120 to the sink device 160 to update the display data rendered on the sink device 160. In an alternative example, the source device 120 may send an event to the sink device 160, which may modify the display data based on the event and the resolution mapping between the source device and the sink device.

FIG. 6 is a flow diagram illustrating an exemplary method that can allow a wireless source device to act as an input device. That is, according to the example shown in FIG. 6, the source device can control the video data rendered in the wireless sink device. The method shown in FIG. 6 describes the source device 120 and the sink device 160 for illustration purposes, but the method of FIG. 6 can be performed by a variety of devices, including other source and / or sink devices. Please understand that.

As shown in FIG. 6, the source device 120 can first send display data to the sink device 160 for rendering to the sink device 160 (600). The display data may include, for example, a series of frames (or images) that are played continuously at high speed in order to simulate the rendition of a full motion video. In some examples, the source device 120 can encode the display data before transmitting the display data to the sink device 160. In response to receiving the display data, the sink device 160 can render the display data. For example, the sink device 160 may decode the display data and may further display a visual representation of the display data on the display 162.

According to the technique of the present disclosure, which allows a wireless source device to act as an input device, the source device 120 receives a display of user input at the first display location of the touch-sensitive screen associated with the source device 120. Get (602). For example, a display of user input may include a user performing a gesture at a first display location on a touch-sensitive screen, the gesture being detectable by the source device 120.

User input can specify modifications to at least some of the display data rendered on the sink device 160. For example, a user may move a graphical pointing object from one location on display 162 to another on display 162. In another example, the user may select a graphical object on the display 162. A portion of the display data rendered on the sink device 160 may be rendered at the second display location on the sink device 160.

In some examples, the source device 120 responds by receiving a display of user input to modify a second display data that is rendered at the second display location of the sink device 160. Configuration data can be sent (604). In some examples, the second configuration data can be at least partially based on mapping the first display location to the second display location. For example, the mapping may be to map the first display resolution associated with the display 122 of the source device 120 to the second display resolution of the display 162 of the sink device 160.

FIG. 7 shows a flow diagram illustrating an exemplary method that allows a wireless sink device to render display data based on user input from a wireless source device. The method shown in FIG. 7 describes the source device 120 and the sink device 160 for illustration purposes, but the method of FIG. 7 can be performed by a variety of devices, including other source and / or sink devices. Please understand that.

As shown in FIG. 7, the sink device 160 may receive display data from the source device 120 for rendering to the sink device 160 (700). The display data may include, for example, a series of frames (or images) that are played continuously at high speed to simulate the representation of a full motion video. In some examples, the sink device 160 may receive encoded display data from the source device 120. Therefore, the sink device 160 can decode the display data and further allow the display 162 to display a visual representation of the display data.

According to aspects of the disclosure, the sink device 160 may also receive configuration data for modifying a portion of the display data rendered at the first location of the sink device 160 (702). Some of the display data may correspond to the data rendered at the second display location on the source device 120. That is, according to aspects of the present disclosure, the sink device 160 has a first location, at least in part, based on the mapping of the first display location on the sink device 160 from the second display location on the source device 120. A part of the display data can be rendered in (704).

In one or more examples, the features described may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, a function can be stored on a computer-readable medium as one or more instructions or codes, or transmitted over a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium is, for example, a computer-readable storage medium corresponding to a tangible medium such as a data storage medium or a communication medium including any medium that enables the transfer of a computer program from one place to another according to a communication protocol. Can include. In this way, the computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier. The data storage medium is any available that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for the implementation of the techniques described in this disclosure. It can be a medium. Computer program products may include computer-readable media.

As an example, but not limited to, such computer-readable storage media can be RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage device, flash memory, or instruction or data structure. Any other medium can be provided that can be used to store the desired program code of the form and can be accessed by a computer. Furthermore, it is reasonable to refer to any connection as a computer-readable medium. For example, instructions are sent from a website, server, or other remote source using coaxial cable, fiber optic cable, twist pair, digital subscriber line (DSL), or wireless technology such as infrared, wireless, and microwave. Where so, coaxial cables, fiber optic cables, twisted pairs, DSL, or wireless technologies such as infrared, wireless, and microwave are included in the definition of medium. However, it should be understood that computer-readable and data storage media do not include connections, carriers, signals, or other temporary media, but instead target non-temporary tangible storage media. The discs and discs used herein are compact discs (CDs), laser discs (registered trademarks) (discs), optical discs, and digital versatile discs (DVDs). ), Flop (registered trademark) disc (disk) and Blu-ray (registered trademark) disc (disc), the disc (disk) usually reproduces data magnetically, and the disc (disc) lasers the data. Reproduce optically. The above combinations should also be included within the scope of computer-readable media.

Instructions can be one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrations. It can be performed by a circuit or a discrete logic circuit. Thus, the term "processor" as used herein refers to either the aforementioned structure or any other structure suitable for implementing the techniques described herein. Further, in some embodiments, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a composite codec. .. Also, the technique may be fully implemented in one or more circuits or logic elements.

The techniques of the present disclosure can be implemented in a wide variety of devices or devices, including wireless handsets, integrated circuits (ICs) or sets of ICs (eg, chipsets). Although various components, modules, or units have been described in this disclosure to emphasize the functional aspects of devices configured to perform the disclosed techniques, these components, modules, or units are described. It does not necessarily have to be achieved by different hardware units. Rather, as described above, the various units are combined or interact with suitable software and / or firmware as codec hardware units, including one or more processors as described above. It can be realized by a set of hardware units.

Various examples have been described. These and other examples fall within the scope of the following claims.