Low latency wireless display for graphics

Priority claim US Provisional Application No. 61 / 439,690, entitled "LOW LATENCY WIRELESS DISPLAY FOR GRAPHICS USING MATCHED MEDIA PROCESSOR," filed February 4, 2011, in which the entire contents of this application are incorporated herein by reference. , And claims the interests of US Provisional Application No. 61 / 584,021 entitled "SOURCE ADAPTATION BASED ON SINK CAPABILITIES" filed on January 6, 2012.

The present disclosure relates to techniques for transmitting data between a wireless source device and a wireless sync device.

A wireless display (WD) or Wi-Fi display (WFD) system includes a wireless source device and one or more wireless sink devices. Each of the source device and the sink device can be either a mobile device or a wired device having a wireless communication function. One or more of the source and sink devices are, for example, mobile phones, portable computers with wireless communication cards, personal digital assistants (PDAs), portable media players, or so-called "smart" phones and "smart" pads. Or include tablets, e-readers, any of a wide variety of wireless displays or projectors, video game devices, or other such devices with wireless communication capabilities, including other types of wireless communication devices. One or more of the source and sink devices may also include wired devices such as televisions, desktop computers, monitors, projectors, etc., including communication capabilities.

The source device sends media data, such as audio-video (AV) data, to one or more of the sink devices participating in a particular media sharing session. Media data can be played on both the local display of the source device and each of the displays of the sink device. More specifically, each of the participating sink devices may render the received media data on its display screen and output an audio portion of the media data via an audio device.

This disclosure generally describes a system in which a wireless source device can communicate with a wireless sync device. As part of a communication session, the wireless source device can send audio and video data to the wireless sync device so that the wireless source device and the wireless sync device render the same audio and video data at substantially the same time. In addition, in some communication sessions, the wireless sync device can send the user input received by the wireless sync device to the wireless source device.

In one example, the method of sending video data from a wireless source device to a wireless sync device intercepts the video component before it is rendered on the wireless source device and generates metadata that describes the video component. This includes sending video components and metadata to wireless sync devices.

In another example, the wireless source device has a metadata encoder configured to intercept the video component prior to rendering on the wireless source device and to generate metadata that describes the video component. Includes a transport unit configured to send video components and metadata to wireless sync devices.

In another example, when a computer-readable storage medium is run by one or more processors, it performs to one or more processors the method of sending video data from a wireless source device to a wireless sync device. Memorize the command to be made. This method intercepts the video component before rendering it on the wireless source device, generates metadata that describes the video component, and sends the video component and metadata to the wireless sync device. Including.

In another example, the wireless source device is configured to send video data to the wireless sync device. A wireless source device provides a means for intercepting a video component prior to rendering on the wireless source device, a means for generating metadata that describes the video component, and a video component and metadata into a wireless sync device. Includes means for transmitting.

In another example, the way to receive video data from a wireless source device in a wireless sync is to receive the first type of video component data and the second type of video component data and metadata from the wireless source device. , Where the metadata identifies the position of the image data for the first video component with respect to the image data for the second video component, the first type of video component data and the second type of video component. Includes generating frames for video based on data and metadata.

In another example, a wireless sync device is a transport unit configured to receive first type video component data and second type video component data and metadata from a wireless source device. The metadata contains a transport unit that identifies the location of the image data about the first video component relative to the image data about the second video component, including the first type of video component data and the second type of video. Includes a metadata decoder configured to generate frames of video based on component data and metadata.

Details of one or more aspects of the present disclosure are given in the accompanying drawings and in the description below. Other features, objectives, and advantages of the present disclosure will become apparent from the description and drawings, as well as the claims.

<figref num="1A">A block diagram showing an example of a source / sync system that can implement the techniques of the present disclosure.</figref><figref num="1B">Block diagram showing an example of a source / sync system with two sync devices.</figref><figref num="2A">A block diagram showing an example of a source / sync system that can implement the techniques of the present disclosure.</figref><figref num="2B">A block diagram showing an example of a source / sync system that can implement the techniques of the present disclosure.</figref><figref num="3">The block diagram which shows the example of the source device which can implement the technique of this disclosure.</figref><figref num="4">The block diagram which shows the example of the sink device which can implement the technique of this disclosure.</figref><figref num="5">The block diagram of the transmitter system and the receiver system which can implement the technique of this disclosure.</figref><figref num="6A">A flowchart of an exemplary method of transmitting video data according to the present disclosure.</figref><figref num="6B">A flowchart of an exemplary method of receiving video data according to the present disclosure.</figref>

This disclosure describes a system in which a wireless source device can communicate with a wireless sync device. As part of a communication session, the wireless source device can send audio and video data to the wireless sync device so that the wireless source device and the wireless sync device render the same audio and video data at substantially the same time. In addition, in some communication sessions, the wireless sync device can send the user input received by the wireless sync device to the wireless source device. In this way, the user of the wireless sync device can control the wireless source device and control the content being transmitted from the wireless source device to the wireless sync device. The term "wireless" as used herein is generally used to refer to a device that communicates wirelessly, but the device may still have wires for other purposes, such as power.

Some wireless source devices have a pixel area (pixel) Send video data in domain). In some examples, this is where the wireless source device renders the pixel data, captures the framebuffer that stores the pixel data, encodes the pixel data, and wirelessly syncs the encoded pixel data. Means to send to the device. In such a configuration, a user application such as a 3D video game may generate video component data that should be converted to pixel data for local display and transmission to a wireless sync device. As an example, an application running on a wireless source device may generate graphics by making a call to an application program interface (API). The API can provide a standardized interface for a user application to communicate with another software component, such as an operating system graphics rendering application, and the user application can, for example, a graphics processing unit (GPU). ), Etc., can provide a standardized interface for communicating with hardware components via a driver. Based on the user application's API call, the graphics rendering application and GPU can generate pixel data for local display and for transmission to wireless sync devices.

According to the techniques of the present disclosure, wireless source devices can be configured to operate in the graphics domain in addition to the pixel domain. Therefore, the present disclosure generally describes wireless source devices and wireless sink devices that are configured to operate in multiple modes. One such mode can be a pixel area mode, or a pixel mode, as described above. In addition to pixel mode, according to the techniques of the present disclosure, wireless source and wireless sync devices can also operate in graphics area mode, also referred to herein as video component mode. Aspects of the present disclosure will be described with respect to pixel mode and video component mode for the sake of brevity. However, wireless source devices and wireless sink devices may implement the techniques of the present disclosure without utilizing defined modes of operation.

When operating in video component mode, the wireless source device intercepts the video component data, such as a graphics API call, and wirelessly syncs the video component data before the video component data is rendered on the wireless source device. Can be sent to. Wireless source devices can still render video component data for local display. The wireless sync device can generate pixel data based on the video component data received from the wireless source device so that the wireless source device and the wireless sync device render the same pixel data at about the same time. This allows a wireless source device operating in video component mode to send video component data to a wireless sink device instead of sending encoded pixel data, as described above. In addition, as part of operating in video component mode, the wireless source device may add metadata to the video component data to assist the wireless sync device in rendering the graphics data. By operating in video component mode, either with or as an alternative to pixel mode, the source / sink system reduces the consumption of system resources such as CPU, memory, and other hardware components. It may be possible, which in some cases improves the responsiveness of the system, improves performance on resource-constrained devices, and in some cases extends the battery life of battery-powered devices. obtain. Operating in video component mode can also reduce the amount of video-related data that needs to be sent from the source device to the sink device, which can also improve system performance.

FIG. 1A is a block diagram showing an exemplary source / sync system 100 that may implement one or more of the techniques of the present disclosure. As shown in FIG. 1A, the system 100 includes a source device 120 that communicates with the sink device 160 over the communication channel 150. The source device 120 includes a memory for storing audio / video (A / V) data 121, a display 122, a speaker 123, an audio / video encoder 124 (also called an encoder 124), an audio / video control module 125, and the like. It may include a transmitter / receiver (TX / RX) unit 126. The sink device 160 includes a display 162, a speaker 163, an audio / video decoder 164 (also called a decoder 164), a transmitter / receiver unit 166, a user input (UI) device 167, and a user input processing module (UIPM). : user input processing can include module) 168 and. The illustrated components form only one exemplary configuration for the Source / Sync System 100. Other components may contain fewer components than the components shown, or may include additional components.

In the example of FIG. 1A, the source device 120 can display the video portion of the audio / video data 121 on the display 122 and output the audio portion of the audio / video data 121 on the speaker 123. Can the audio / video data 121 be stored locally on the source device 120 and accessed from an external storage medium such as a file server, hard drive, external memory, Blu-ray Disc, DVD, or other physical storage medium? Or it can be streamed to the source device 120 via a network connection such as the Internet. In some cases, audio / video data 121 may be captured in real time via the camera and microphone of the source device 120. The audio / video data 121 may include multimedia content such as video, television programs, or music, but may also include real-time content generated by the source device 120. Such real-time content can be, for example, video data generated by an application running on the source device 120 or captured, for example, as part of a video telephony session. As described in more detail, such real-time content may, in some cases, include video frames of user input options available for the user to select. In some cases, the audio / video data 121 may include a video frame that is a combination of different types of content, such as a video frame of a video or TV show with user input options overlaid on the frame of the video.

In addition to locally rendering the audio / video data 121 through the display 122 and the speaker 123, the audio / video encoder 124 of the source device 120 can encode the audio / video data 121, the transmitter / receiver unit. The 126 may transmit the encoded data to the sink device 160 via the communication channel 150. The transmitter / receiver unit 166 of the sink device 160 receives the encoded data, the audio / video decoder 164 decodes the encoded data, and the decoded data is displayed on the display 162 and the speaker 163. Output via and. In this way, the audio and video data rendered by the display 122 and the speaker 123 can be rendered simultaneously by the display 162 and the speaker 163. In some modes of operation, the display 122 and speaker 123 are disabled during the communication session so that the wireless source device is transmitting audio and video data but is not rendering the audio and video data locally. ) Can be done. Audio data and video data can consist of frames, which can be time synchronized with the video frame when rendered. According to the techniques of the present disclosure, the video payload data transmitted over the communication channel 150 may include compressed or uncompressed pixel data, video component data with metadata, or any combination of both.

The audio / video encoder 124 and audio / video decoder 164 are alternatives to the new ITU-T H.264 standard called MPEG-4, Part10, Advanced Video Coding (AVC), or the H.265 standard. Any number of audio and video compression standards can be implemented, including high efficiency video coding (HEVC) standards. Many other types of proprietary or standardized compression techniques can also be used. Generally, the audio / video decoder 164 is configured to perform the reverse coding operation of the audio / video encoder 124. Although not shown in FIG. 1A, in some embodiments, the A / V encoder 124 and the A / V decoder 164 may be integrated with the audio encoder and decoder, respectively, in a common or separate data stream. It may include a suitable MUX-DEMUX unit, or other hardware and software, to handle both audio and video encoding.

As described in more detail below, the A / V encoder 124 may perform other encoding functions in addition to implementing the video compression standard as described above. For example, the A / V encoder 124 may add various types of metadata to the A / V data 121 before the A / V data 121 is transmitted to the sink device 160. In some cases, the A / V data 121 may be stored in or received in the source device 120 in encoded form and therefore does not require further compression by the A / V encoder 124. is there.

Further, when implementing the techniques of the present disclosure and operating in video component mode, the A / V encoder 124 may also intercept the video component data before it is converted to pixel data. The A / V encoder 124 can add metadata to the video component data and send the metadata and video component data to the wireless sync device 160 over the communication channel 150. The A / V decoder 164 of the sync device 160 can generate pixel data based on the received video component data and metadata.

To operate in video component mode, the wireless source device 120 and the wireless sync device 160 may have similar multimedia capabilities. The multimedia features of the wireless sync device 160 may be communicated to the wireless source device 120 as part of a feature negotiation exchange that takes place when the wireless source device 120 and the wireless sync device 160 establish a communication session. During the feature negotiation exchange, the wireless sync device 160 will have a list of supported graphics API types and versions, a list of supported specific instruction sets, a list of supported graphics textures, and a list of supported codecs. , And other such functional information (capability) information) can be given to the wireless source device 120. Based on the feature information received, the wireless source device 120 can determine whether the wireless sync device 160 possesses the multimedia features required to operate in video component mode. Alternatively, the wireless source device 120 may send a list of desired features to the wireless sync device 160, and based on that list, the wireless sync device 160 allows the wireless sync device 160 to operate in video component mode. You can determine if you have the required functionality. In cases where the wireless sync device 160 does not have the functionality required to operate in video component mode, the wireless source device 120 may transmit pixel data to the wireless sync device 160 as opposed to video component data.

Figure 1A shows that the communication channel 150 carries the audio payload data and the video payload data separately, but in some cases the video payload data and the audio payload data are part of a common data stream. possible. If applicable, the MUX-DEMUX unit is ITU It may comply with the H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP). The audio / video encoder 124 and audio / video decoder 164 are each one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software. , Hardware, firmware, or any combination thereof. Each of the audio / video encoder 124 and the audio / video decoder 164 may be included in one or more encoders or decoders, both of which may be integrated as part of a composite encoder / decoder (codec). Thus, each of the source device 120 and the sink device 160 may be equipped with specialized machines configured to perform one or more of the techniques of the present disclosure.

The displays 122 and 162 have a variety of video outputs, such as cathode ray tubes (CRTs), liquid crystal displays (LCDs), plasma displays, light emitting diode (LED) displays, organic light emitting diode (OLED) displays, or other types of display devices. It may be equipped with any of the devices. In these or other examples, the displays 122 and 162 can be emissive displays or transmissive displays, respectively. The display 122 and display 162 may also include touch displays such that they are both input and display devices at the same time. Such a touch display can be a capacitive, resistive, or other type of touch panel that allows the user to provide user input to the device in question.

Speaker 123 may include any of a variety of audio output devices, such as headphones, single speaker systems, multi-speaker systems, or surround sound systems. Further, the display 122 and the speaker 123 are shown as part of the source device 120, and the display 162 and the speaker 163 are shown as part of the sink device 160, but the source device 120 and the sink device 160 are actually plural. It can be a system of devices. As an example, the display 162 can be a television, the speaker 163 can be a surround sound system, and the decoder 164 can be part of an external box that is wired or wirelessly connected to the display 162 and the speaker 163. In another example, the sink device 160 can be a single device, such as a tablet computer or smartphone. In yet other cases, the source device 120 and the sink device 160 include similar devices, for example, both may be smartphones, tablet computers, and the like. In this case, one device can act as a source and the other as a sink. These roles may even be reversed in subsequent communication sessions. In yet other cases, the source device may include a mobile device, such as a smartphone, laptop or tablet computer, and the sink device may include a more fixed device (eg, a video display projector with an AC power cord), in which case. The source device may deliver audio and video data for presentation to a large crowd via a sink device.

The transmitter / receiver unit 126 and the transmitter / receiver unit 166 are each a variety of mixers, filters, amplifiers, and other components designed for signal modulation, as well as one or more antennas. And may include other components designed to transmit and receive data. The communication channel 150 generally represents any communication medium suitable for transmitting video data from the source device 120 to the sink device 160, or a collection of various communication media. The communication channel 150 is usually a relatively short-range communication channel similar to Wi-Fi, Bluetooth (registered trademark), and the like. However, the communication channel 150 is not necessarily limited in this regard, and may be any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or a wireless medium and a wired medium. Can have any combination with. In another example, the communication channel 150 may even form part of a packet-based network, such as a wired or wireless local area network, a wide area network, or a global network such as the Internet. In addition, the communication channel 150 can be used by the source device 120 and the sink device 160 to create peer-to-peer links. Source device 120 and sink device 160 are IEEE802. It is possible to communicate over communication channel 150 using communication protocols such as standards from the 11 standard family. The source device 120 and sink device 160 comply with the Wi-Fi Direct standard so that, for example, the source device 120 and sink device 160 communicate directly with each other without the use of intermediates such as wireless access points or so-called hotspots. Can communicate. Source device 120 and sink device 160 may also establish a tunneled direct link setup (TLDS) to avoid or reduce network congestion. Although the techniques of the present disclosure are sometimes described with respect to Wi-Fi, it is contemplated that aspects of these techniques may also be compatible with other communication protocols. As an example, but not a limitation, wireless communication between the source device 120 and the sink device may utilize orthogonal frequency division multiplexing (OFDM) techniques. A wide variety of others, including, but not limited to, time division multiple access (TDMA), frequency division multiple connection (FDMA), code division multiple connection (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. Wireless communication techniques can also be used. WiFi Direct and TDLS aim to set up relatively short-range communication sessions. A relatively short distance in this context may refer to, for example, less than about 70 meters, but in a noisy or obstructive environment, the distance between devices is less than about 35 meters, about. It can be even shorter, such as less than 20 meters, or even less than about 10 meters. Direct and TDLS aim to set up relatively short-range communication sessions. A relatively short distance in this context may refer to, for example, less than about 70 meters, but in a noisy or obstructive environment, the distance between devices is less than about 35 meters, about. It can be even shorter, such as less than 20 meters, or even less than about 10 meters. Direct and TDLS aim to set up relatively short-range communication sessions. A relatively short distance in this context may refer to, for example, less than about 70 meters, but in a noisy or obstructive environment, the distance between devices is less than about 35 meters, about. It can be even shorter, such as less than 20 meters, or even less than about 10 meters. Direct and TDLS aim to set up relatively short-range communication sessions. A relatively short distance in this context may refer to, for example, less than about 70 meters, but in a noisy or obstructive environment, the distance between devices is less than about 35 meters, about. It can be even shorter, such as less than 20 meters, or even less than about 10 meters. Direct and TDLS aim to set up relatively short-range communication sessions. A relatively short distance in this context may refer to, for example, less than about 70 meters, but in a noisy or obstructive environment, the distance between devices is less than about 35 meters, about. It can be even shorter, such as less than 20 meters, or even less than about 10 meters.

In addition to decoding and rendering the data received from the source device 120, the sink device 160 can also receive user input from the user input device 167. The user input device 167 can be, for example, a keyboard, mouse, trackball or trackpad, touch screen, voice command recognition module, or other such user input device. The UIPM168 formats the user input commands received by the user input device 167 into a data packet structure that can be interpreted by the source device 120. Such data packets are transmitted by the transmitter / receiver 166 to the source device 120 over the communication channel 150. The transmitter / receiver unit 126 receives the data packet, and the A / V control module 125 parses the data packet to interpret the user input command received by the user input device 167. Based on the commands received in the data packet, the A / V control module 125 can modify the coded and transmitted content. In this way, the user of the sink device 160 can control the audio and video payload data transmitted by the source device 120 remotely and without interacting directly with the source device 120. Examples of types of commands that a user of sync device 160 can send to source device 120 are commands for rewinding, fast-forwarding, pausing, and playing audio and video data, as well as for zooming, rotating, scrolling, and so on. Includes commands for. The user may also make a selection, for example from a menu of options, and return the selection to the source device 120.

In addition, the user of sync device 160 may be able to launch and control applications on source device 120. For example, a user of the sync device 160 may launch a photo editing application stored on the source device 120 and use that application to edit photos stored locally on the source device 120. The sync device 160 can provide the user with a user experience in which the photo appears and feels locally edited on the sync device 160 while the photo is actually being edited on the source device 120. With such a configuration, device users may be able to take advantage of the capabilities of one device for use with several devices. For example, the source device 120 can be a smartphone with a large amount of memory and high-end processing power. A user of the source device 120 may use the smartphone in all settings and situations in which the smartphone is commonly used. However, when watching a video, the user may wish to watch the video on a device with a larger display screen, in which case the sync device 160 may be a tablet computer or a larger display device or television. When wishing to send or respond to e-mail, the user may wish to use a device with a keyboard, in which case the sync device 160 may be a laptop. In both cases, the user is interacting with the sink device, but most of the processing can still be performed by the source device 120 (smartphone in this example). This particular operating environment In context), most of the processing is done by the source device 120, so the sink device 160 has fewer resources than if the sink device 160 was used to do the processing that is being done by the source device 120. It can be a lower cost device using. In some examples, both source and sink devices may be able to receive user input (such as touch screen commands), and the techniques of the present disclosure show the functionality of those devices in a given session. By negotiating and / or identifying, two-way dialogue may be possible.

In some configurations, the A / V control module 125 runs the operating system processes run by the operating system of the source device 125. However, in other configurations, the A / V control module 125 may include software processes for the application running on the source device 120. In such a configuration, user input commands can be interpreted by a software process so that the user of sink device 160 runs on source device 120 as opposed to the operating system running on source device 120. You are interacting directly with the application you are using. By interacting directly with the application as opposed to the operating system, users of sink device 160 may have access to a library of commands that are not native to the operating system of source device 120. In addition, by interacting directly with the application, commands may be more easily transmitted and processed by devices running on different platforms.

The source device 120 can respond to user input applied in the wireless sync device 160. In such an interactive application configuration, the user input applied in the wireless sync device 160 may be sent to the wireless display source via the communication channel 150. In one example, a reverse channel architecture, also known as a user interface back channel (UIBC), may be implemented to allow the sink device 160 to send the user input applied at the sink device 160 to the source device 120. The reverse channel architecture may include a higher layer message for transporting user input and a lower layer frame for negotiating user interface functionality on the sink device 160 and the source device 120. The UIBC may reside on top of the Internet Protocol (IP) transport layer between the sink device 160 and the source device 120. In this way, UIBC can be above the transport layer in the Open Systems Interconnection (OSI) communication model. In one example, OSI communication includes seven layers: 1-Physical, 2-Datalink, 3-Network, 4-Transport, 5-Session, 6-Presentation, and 7-Application. In this example, above the transport layer refers to layers 5, 6, and 7. To facilitate the reliable transmission and sequential delivery of data packets containing user-entered data, UIBC uses other packets, such as Transmission Control Protocol / Internet Protocol (TCP / IP) or User Datagram Protocol (UDP). It can be configured to run on top of the base communication protocol. UDP and TCP can operate in parallel in the OSI layer architecture. TCP / IP allows sink device 160 and source device 120 to implement retransmission techniques in case of packet loss.

In some cases, there may be a discrepancy between the user display located on the source device 120 and the user display located on the sink device 160. Resolve potential issues caused by such discrepancies and have a good user experience under such circumstances. To facilitate experience), as part of the functional negotiation exchange between the source device 120 and the sink device 160, the source device 120 and the sink device 160 can agree on the negotiated screen resolution. When the sync device 160 transmits the coordinate data associated with the user input, the sync device 160 can scale the coordinate data obtained from the display 162 to match the negotiated screen resolution. Similarly, when the wireless source device 120 sends video component data to the wireless sync device 160 with the specific resolution identified in the metadata, the wireless source device 120 negotiates the resolution contained in the metadata. Can be scaled to match the resolution. In one example, if the sink device 160 has a 1280 x 720 resolution and the source device 120 has a 1600 x 900 resolution, then those devices may use, for example, a 1280 x 720 resolution as their negotiated resolution. The negotiated resolution may be selected based on the resolution of the sink device 160, but the resolution of the source device 120 or some other resolution may also be used. In an example where a 1280 x 720 resolution sink device is used, the sink device 160 can scale the obtained x-coordinates at a ratio of 1600/1280 before sending them to the source device 120. Similarly, the sink device 160 can scale its obtained y-coordinates by 900/720 before sending them to the source device 120. In other configurations, the source device 120 can scale the acquired coordinates to the negotiated resolution. The scaling can increase or decrease the coordinate range based on whether the sink device 160 uses a higher resolution display than the source device 120 or vice versa.

Moreover, in some cases, the resolution at the sink device 160 can fluctuate during the communication session, potentially resulting in a discrepancy between the display 122 and the display 162. To improve the user experience and ensure proper functionality, the Source / Sync System 100 implements techniques to reduce or prevent user interaction discrepancies by implementing techniques for screen normalization. obtain. The display 122 of the source device 120 and the display 162 of the sink device 160 may have different resolutions and / or different aspect ratios. In addition, in some settings, the user of the sync device 160 will be able to render the video data received from the source device 120 in a window that covers a smaller portion of the sink device 160's display 162. It may have the ability to resize the display window for video data received from device 120. In another exemplary setting, the user of sync device 160 may have the option to view content in either landscape mode or portrait mode, each of which has unique coordinates and different aspect ratios. Have. In such situations, the coordinates associated with the user input received on the sink device 160, such as the coordinates about where the mouse click or touch event takes place, are processed by the source device 120 without modification to those coordinates. It may not be possible. Thus, the techniques of the present disclosure may include mapping the coordinates of user input received on the sink device 160 to the coordinates associated with the source device 120. This mapping, also referred to herein as normalization, can be either sink-based or source-based, as described in more detail below.

User input received by the sink device 160 may be received by the UI module 167 and passed to the operating system of the sink device 160, for example at the driver level. The operating system on the sink device 160 has the coordinates (x) associated with where the user input took place on the display surface._SINK, y_SINK) Can be received. In this example, (x_SINK, y_SINK) Can be the coordinates of the display 162 where the mouse click or touch event took place. The display window rendered on the display 162 has an x-coordinate length (L) that describes the size of the display window._DW) And y coordinate width (W)_DW) And may have. The display window is the upper left corner coordinate (a) that describes the location of the display window._DW, b_DW) May also have. L_DW, W_DW, And upper left coordinates (a_DW, b_DW), The portion of the display 162 covered by the display window can be determined. For example, the upper right corner of the display window is the coordinates (a_DW+ L_DW, b_DW), The lower left corner of the display window is the coordinates (a)_DW, b_DW+ W_DW), The lower right corner of the display window is the coordinates (a)_DW+ L_DW, b_DW+ W_DW) Can be located. When the input is received at the coordinates in the display window, the sync device 160 can process the input as a UIBC input. In other words, the associated coordinates (x)_SINK, y_SINKThe input in) can be processed as a UIBC input if the following conditions are met:

a_DW x_SINK a_DW + L_DW(1) b b_DW y_SINK b_DW + W_DW(2) After determining that the user input is a UIBC input, the coordinates associated with that input can be normalized by UIPM168 before being sent to the source device 120. Input determined to be outside the display window may be processed locally by the sync device 160 as non-UIBC input.

As mentioned above, the normalization of input coordinates can be either source-based or sync-based. When implementing sync-based normalization, the source device 120 has a supported display resolution (L) for display 122._SRC, W_SRC) Can be sent to the sync device 160 with or without video data. The supported display resolutions can be transmitted, for example, as part of a functional negotiation session or at another time during the communication session. The sync device 160 has a display resolution (L) for the display 162._SINK, W_SINK) And the display window resolution (L) for the window displaying the content received from the source device 120._DW, W_DW) And the upper left corner coordinates for the display window (a)_DW, b_DW) Can be determined. As explained above, the coordinates (x) corresponding to the user input_SINK, y_SINKWhen it is determined that) is in the display window, the operating system of the sink device 160 uses a transform function to determine the coordinates (x)._SINK, y_SINK) To source coordinates (x_SRC, y_SRC) Can be mapped. (x_SINK, y_SINK) To (x_SRC, y_SRCAn exemplary conversion function for converting to) can be:

x_SRC = (x_SINK --a_DW) * (L_SRC/ L_DW) (3) y_SRC = (y_SINK --b_DW) * (W_SRC/ W_DW) (Four) Therefore, when transmitting the coordinates corresponding to the received user input, the sink device 160 is (x)._SINK, y_SINK) Coordinates for user input received in (x)_SRC, y_SRC) Can be sent. Coordinates (x), as described in more detail below._SRC, y_SRC) May be transmitted, for example, as part of a data packet used to transmit user input received on the sink device 160 to the source device 120 via the UIBC. Throughout the rest of the disclosure, where the input coordinates are described as being contained within the data packet, those coordinates are as described above in the case where the source / sync system 100 implements sync-based normalization. Can be converted to source coordinates.

When Source / Sync System 100 implements source-based normalization, user input determined to be UIBC input as opposed to local input (ie, inside the display window as opposed to outside the display window). For, the above calculation can be performed on the source device 120 instead of the sink device 160. To enable such calculations, the sink device 160 is L._DW, W_DWValues for, as well as location information about the display window (eg, a_DW, b_DW), And (x_SINK, y_SINK) Can be sent to the source device 120. Using these transmitted values, the source device 120 follows Equations 3 and 4 above (x)._SRC, y_SRC) Can be determined.

In another implementation of sync-based normalization, the sink device 160 describes where in the display window the user input event occurs, as opposed to where the user input event occurs on the display 162. Coordinates for user input (x_DW, y_DW) Can be sent. In such an implementation, the coordinates (x)_DW, y_DW) Is (L_DW, W_DWCan be sent 120 to the source device with a value for). Based on these received values, the source device 120 follows the following conversion function (x)_SRC, y_SRC) Can be judged.

x_SRC = x_DW * (L_SRC/ L_DW) (Five) y_SRC = y_DW * (W_SRC/ W_DW) (6) The sink device 160 is x based on the following function_DWAnd y_DWCan be judged.

x_DW = x_SINK--a_DW(7) y_DW = y_SINK --b_DW(8) In the present disclosure, for example, when describing transmitting the coordinates associated with a user input in a data packet, the transmission of these coordinates may include sink-based or source-based normalization as described above. , And / or may contain any additional information desirable to perform sink-based or source-based normalization.

UIBC can be designed to transport various types of user input data, including cross-platform user input data. For example, the source device 120 may run an iOS® operating system, and the sink device 160 may run another operating system, such as Android® or Windows®. Regardless of the platform, the UIPM168 can encapsulate received user input in a manner understandable to the A / V control module 125. Several different types of users allow them to leverage the protocol, regardless of whether many different types of source and sink devices run on different platforms. The input format may be supported by UIBC. General input formats can be defined and both platform-specific input formats can be supported, thus giving the flexibility of how user input can be communicated between the source device 120 and the sink device 160 by UIBC.

In the example of FIG. 1A, the source device 120 may include a smartphone, tablet computer, laptop computer, desktop computer, Wi-Fi capable television, or other device capable of transmitting audio and video data. The sync device 160 can also receive smartphones, tablet computers, laptop computers, desktop computers, Wi-Fi enabled televisions, or other devices capable of receiving audio and video data and receiving user input data. Can be prepared. In some cases, the sink device 160 appears to be a display 162, a speaker 163, a UI device 167, and an A / V encoder 164, all of which are separate but interoperable devices. It may include a system of multiple devices. Similarly, the source device 120 can be a system of multiple devices rather than a single device.

In the present disclosure, the term source device is used generally to refer to a device transmitting audio / video data, and the term sink device is generally used to refer to a device receiving audio / video data from a source device. Used to point to. In many cases, the source device 120 and the sink device 160 can be similar or equivalent devices, one device acting as the source and the other acting as the sink. Moreover, these roles can be reversed in different communication sessions. Thus, a sink device in one communication session can be a source device in subsequent communication sessions and vice versa.

FIG. 1B is a block diagram showing an exemplary source / sync system 101 that may implement the techniques of the present disclosure. The source / sink system 101 includes a source device 120 and a sink device 160, each of which may function and operate in the manner described above for FIG. 1A. The source / sync system 101 further includes a sink device 180. In a manner similar to the sink device 160 described above, the sink device 180 may receive audio and video data from the source device 120 and send user commands to the source device 120 via the established UIBC. In some configurations, the sink device 160 and the sink device 180 can operate independently of each other, and the audio and video data output by the source device 120 is output simultaneously by the sink device 160 and the sink device 180. obtain. In an alternative configuration, the sync device 160 can be a primary sync device and the sync device 180 can be a secondary sync device. In such an exemplary configuration, the sink device 160 and the sink device 180 can be combined, the sink device 160 can display video data, and the sink device 180 outputs the corresponding audio data. Further, in some configurations, the sync device 160 may output only the transmitted video data and the sync device 180 may output only the transmitted audio data.

2A and 2B are block diagrams showing examples of source / sync systems that may implement the techniques of the present disclosure. The source / sync system of FIGS. 2A and 2B includes the source device 220 shown in FIG. 2A and the sink device 260 shown in FIG. 2B. Source device 220 and sink device 260 generally operate in the same way as source device 120 and sink device 160 in FIG. 1A, but FIGS. 2A and 2B highlight different components of the device. The source device 220 includes an application 272, a metadata encoder 274, a graphics synthesis module 276, a local display 222, a transport unit 233A, and a WiFi modem 234A. The sink device 260 includes a WiFi modem 233B, a transport unit 234B, a metadata decoder 275, a graphics synthesis module 277, and a display 262.

The source device 220 may execute one or more user applications (application 272) that generate video component data, such as the video component data 273A, 273B, and 273C shown in FIG. 2A. As described in more detail below, the metadata encoder 274 may intercept the video component data 273A-C prior to rendering by the graphics compositing module 276 and add metadata to the video component data 273A-C. The transport unit 233A may encapsulate the video component data 273A-C and the metadata and use the WiFi modem 234A to send the encapsulated data to the wireless sync device 260.

Video component data 273A-C can also be processed by the wireless source device 220 to generate pixel data. The graphics compositing module 276 may render pixel data for local display on display 222 based on video component data 273A-C. In this way, the graphics compositing module 276 is generally intended to represent all the graphics rendering resources available to the source device 220, which resources are hardware, such as general purpose processors and graphics processors. It can include hardware components and can also include software components such as supported APIs, graphics applications, graphics drivers, and video codecs.

When operating in video component mode, the metadata encoder 274 may intercept video components 273A-C before being rendered by the graphics compositing module to generate metadata related to the video component data. Video components 273A-C are generated by calls to graphics APIs, such as OpenGL commands or Microsoft DirectX commands, compressed video encoded by encoding standards such as H.264 or the emerging HEVC standard, operating systems or applications. It may represent compressed or uncompressed audio data, or other such data. In some examples, one or more of the video components 273A-C can also be pixel data, but not necessarily the full frame of pixel data. For example, one of the video components can be pixel data corresponding to an icon that should be an overlay compressed video. Although three video components (273A-C) are shown in Figures 2A and 2B, it is conceivable that less than three, and in some cases only one, video component may be used in some situations. Similarly, it is also contemplated that four or more video components may be used. Further, in the present disclosure, the video data components 273A-C are intended to represent different types of video component data, but not necessarily to represent only one component of each type. No no. For example, video component 273A may represent multiple OpenGL commands, video component 273C may represent multiple pixel data icons, and so on.

As an example of how video component data can be intercepted, application 272 may issue a command to the GPU driver to instruct the GPU to pull out one or more primitives for 3D graphics. This draw mand can be an OpenGL glDraw command, for example glDrawArrays or glDrawElements. When application 272 issues the glDraw command, the metadata encoder 274 can intercept the draw command and send the draw command along with the metadata to the wireless sync device 260. Other graphics commands, such as texture commands, can be intercepted as well. Commands for other graphics APIs can be intercepted as well.

As another example of how video component data can be intercepted, the metadata encoder 274 may monitor application 272 to detect whether application 272 initializes the video codec. Upon initialization of the video codec, the metadata encoder 274 can identify the data as compressed video data. As another example, when a media player application is launched to play a video clip on the source device 220, the media parser component may be initialized along with other components to build the playback pipeline. The metadata encoder 274 may implement a stub in the parser component to detect its initialization and intercept the video component data. In yet another example, the stub can be inserted into the synthesis engine on the wireless source 220. Stubs can be used to detect data flow and intercept video component data, if desired.

The metadata encoder 274 can generate metadata that describes how the video component data should be assembled to render the frame for display by the wireless sync device. Metadata can include, for example, data that identifies the screen location for video components 273A-C, as well as data that allows one or more of video components 273A-C to be synchronized with audio data. The metadata may identify the screen location by including the coordinates of the upper left point or other point about the window or the coordinates of other such locations of a portion of the image data generated from the video component. The metadata may also include resolution data that identifies the resolution of the source device. The screen coordinate data and resolution data contained in the metadata can be scaled by either the sink device or the source device in the manner described above.

Metadata can also indicate, for example, whether the image data produced by one video component should be before or after (ie, front or back) the image data produced by another video component, duplicate components. It can also include color blending information about or depth information about 3D graphics content.

In addition, the metadata generated by the metadata encoder 274 may include a frame identifier used to identify the frame, such as a time stamp. The frame identifier can be used, for example, by the wireless sync device 260 to assemble the video component data 273A-C as part of the correct frame, and the wireless source to determine which frame a particular user input is associated with. It can also be used by device 220. For example, the wireless source device 220 may include a time stamp in the metadata associated with the video component data. When the sink device 260 receives the user input, it syncs so that the source device 220 can process the user input in view of the frame displayed by the sync device 260 when the user input is received by the wireless sync device 260. The device 260 may identify the time stamp associated with the frame displayed when the user input is received and return the time stamp to the wireless source device 220.

The transport unit 233A can encapsulate the video component data and the metadata, and the WiFi modem 234A can send the encapsulated data to the sink device 260. The WiFi modem 234B can receive the encapsulated data and the transport unit 233B can decapsulate the encapsulated data. The functions of the transport unit 233A and the WiFi modem 234A are generally similar to those of the transport unit 333 and the WiFi modem 334 and will be described in more detail below with reference to FIG. Similarly, the functionality of the transport unit 234B and the WiFi modem 234B is generally similar to that of the transport unit 433 and the WiFi modem 434 and will be described in more detail below with reference to FIG.

The metadata decoder 275 may be configured to extract video component data (video component data 273A-C in this example) from the decapsulated data and the metadata generated by the metadata encoder 274. The graphics compositing module 277 can render pixel data based on video component data 273A-C for display on display 262. In this way, the graphics compositing module 277 is intended to represent, in general, all graphics rendering resources and features available to the sink device 260, which are hardware such as general purpose processors and graphics processors, for example. It can include hardware components and can also include software components such as supported APIs, graphics applications, graphics drivers, and video codecs. In some implementations, the features of the graphics synthesis module 276 of the source device 220 and the graphics synthesis module 277 of the sink device 260 are at least some similar features of the graphics synthesis module 276 and the graphics synthesis module 277. Can be selected to share, thus allowing the graphics compositing module 277 to render the video component data transmitted from the wireless source device 220.

As previously described, according to the techniques of the present disclosure, the wireless source device 220 may be configured to transmit video component data and metadata to the wireless sync device 260. Based on the video component data and metadata, the wireless sync device 260 may generate pixel data as part of rendering the video data given by the wireless source device 220. In some cases, transmitting video component data and metadata rather than pixel data may reduce the overall amount of data that needs to be transmitted from the source device 220 to the sink device 260. The wireless source device 220 can still generate pixel data for local display on the display 222, but the pixel data does not necessarily have to be transmitted to the wireless sync device 260. In addition, in some implementations, the wireless source device 220 and wireless sync device 260 may support multiple modes of operation, for example, in pixel mode, the wireless source device 220 transmits pixel data, and in video component mode, The wireless source device 260 transmits component video data. In video component mode, the wireless source device 220 may transmit pixel data in addition to the video component data and metadata described above.

The wireless sync device 260 receives video component data and metadata and can assemble the video component data into frames for display based on the metadata. In one example, the video component data 279B can be compressed video data coded using the H.264 coding standard. The wireless sync device 260 can decode the encoded video data and render the frame for display. For example, the graphics compositing module 277 may include an H.264 video decoder. In this way, the wireless sync device 260 and the wireless source device 220 may possess many or some of the same video processing power. For example, if the wireless source device 220 implements the H.264 codec, the wireless sync device 260 may also implement the H.264 codec.

In another example, the wireless source device 220 may send two video data components (eg, 273A and 273B) to the wireless sync device, where the video component 273B is for video or television programming in encoded format. Corresponding, video component 273A corresponds to the media player user interface element rendered by the application of wireless source device 220. The user interface element may include on-screen controls such as play, pause, and stop that indicate where the user should touch to control the playback of the video or television program of video component 273B, for example. The wireless source device 220 can intercept the video component data 273A and 273B, generate metadata describing the video components 273A and 273B, and send the video components 273A and 273B to the wireless sync device 260. Metadata, for example, the screen resolution information, such as the wireless sink device 260 can be scaled appropriately video components 273A and 273B shown , and the user interface elements of the video component 273A is applied to the correct frame of the video data of the video component 273B It may include synchronization information to obtain, location information indicating where user interface elements should be located, and where they should be on top of the video in video component 273B, and may include other such information. Based on the metadata, the graphics compositing module 227 can render a frame of video with the user interface element of video component 273A on top of the video generated by video component 273B.

FIG. 3 is a block diagram showing an example of the source device 320. The source device 320 can be a device similar to the source device 120 in FIG. 1A and the source device 220 in FIG. 2A and can operate in the same manner as described above. The source device 320 includes a local display 322, a local speaker 323, a processor 331, a memory 332, a transport unit 333, and a wireless modem 334. As shown in FIG. 3, the source device 320 may include one or more processors (ie, processor 331) that encode and / or decode A / V data for transport, storage, and display. A / V data can be stored, for example, in memory 332. Memory 332 can store the entire A / V file, or can include, for example, a smaller buffer that stores only a portion of the A / V file, streamed from another device or source. Transport unit 333 may process encoded A / V data for network transport. For example, the encoded A / V data can be processed by processor 331 and encapsulated in a network access layer (NAL) unit by transport unit 333 for communication over the network. The NAL unit can be sent to the wireless sync device over a network connection by the wireless modem 334. The wireless modem 334 can be, for example, a Wi-Fi modem configured to implement one of the IEEE 802.11 standard families.

The source device 320 may also process and display A / V data locally. In particular, the display processor 335 may process the video data as displayed on the local display 322, and the audio processor 336 may process the audio data for output on the speaker 323.

As described above for the source device 120 of FIG. 1A, the source device 320 may also receive user input commands from the sink device. In this way, the wireless modem 334 of the source device 320 receives the encapsulated data packet, such as the NAL unit, and sends the encapsulated data unit to the transport unit 333 for decapsulation. For example, transport unit 333 can extract data packets from the NAL unit, and processor 331 can parse the data packets to extract user input commands. Based on user input commands, processor 331 can adjust the encoded A / V data sent by the source device 320 to the sink device. In this way, the functionality described above for the A / V control module 125 of FIG. 1A may be fully or partially implemented by processor 331.

Processor 331 in Figure 3 is generally, but not limited to, one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), and other equivalents. Represents any of a wide variety of processors, or any combination thereof, including integrated circuits or discrete logic circuits. Processor 331 may represent, for example, a set of processors that include both a central processing unit and one or more instructional set processors (ASIPs), tuned for specific applications. The memory 332 in FIG. 3 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It can have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM), flash memory, and so on. The memory 332 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 332 may further store instructions and program code executed by processor 331 as part of performing the various techniques described herein.

FIG. 4 shows an example of the sink device 460. The sink device 460 can be a device similar to the sink device 160 in FIG. 1A and the sink device 260 in FIG. 2B and can operate in the same manner as described above. The sink device 460 includes one or more processors (ie, processor 431), memory 432, transport unit 433, wireless modem 434, display processor 435, local display 462, audio processor 436, and speakers. Includes 463 and user input interface 476. The sink device 460 receives the encapsulated data unit sent from the source device in the wireless modem 434. The wireless modem 434 can be, for example, a Wi-Fi modem configured to implement one or more standards from the IEEE 802.11 standard family. Transport unit 433 can decapsulate the encapsulated data unit. For example, transport unit 433 may extract encoded video data from the encapsulated data unit and send the encoded A / V data to processor 431 for decoding and rendering for output. The display processor 435 may process the decoded video data for display on the local display 462, and the audio processor 436 may process the decoded audio data for output on the speaker 463.

In addition to rendering audio and video data, the wireless sync device 460 can also receive user input data through user input interface 476. The user input interface 476 is any of several user input devices, including, but not limited to, a touch display interface, keyboard, mouse, voice command module, and gesture capture device (eg, with camera-based input capture capabilities). Or it can represent other user input devices of some user input devices. User input received through user input interface 476 may be processed by processor 431. This process may include generating a data packet containing the received user input command according to the techniques described herein. Once generated, transport unit 433 may process its data packets for network transport to wireless source devices via UIBC.

Processor 431 in Figure 4 includes one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), and other equivalent integrated or discrete logic circuits. It may have one or more of a wide range of processors, or any combination thereof. Processor 431 may represent, for example, a set of processors that include both a central processing unit and one or more instructional set processors (ASIPs), tuned for specific applications. The memory 432 in FIG. 4 is, but is not limited to, random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable. It can have either a wide variety of volatile or non-volatile memory, including read-only memory (EEPROM), flash memory, and so on. The memory 232 may include a computer-readable storage medium for storing audio / video data as well as other types of data. Memory 432 may further store instructions and program code executed by processor 431 as part of performing the various techniques described herein.

FIG. 5 shows an exemplary transmitter system 510 and receiver system 550 that can be used by the transmitter / receiver 126 and transmitter / receiver 166 of FIG. 1A to communicate over communication channel 150. A block diagram is shown. In transmitter system 510, traffic data for several data streams is fed from the data source 512 to the transmit (TX) data processor 514. Each data stream may be transmitted via the corresponding transmitting antenna. The TX data processor 514 formats, codes, and interleaves the traffic data in each data stream based on the particular coding scheme selected for that data stream.

The coded data in each data stream can be multiplexed with pilot data using Orthogonal Frequency Division Multiplexing (OFDM) techniques. A wide variety of others, including, but not limited to, time division multiple access (TDMA), frequency division multiple connection (FDMA), code division multiple connection (CDMA), or any combination of OFDM, FDMA, TDMA and / or CDMA. Wireless communication techniques can also be used.

According to FIG. 5, the pilot data is typically a known data pattern processed in a known manner and can be used in a receiver system to estimate the channel response. The multiplexed pilot and coded data of each data stream is then subjected to a particular modulation scheme (eg, 2-phase shift keying (BPSK), 4-phase shift) selected for that data stream to provide a modulation symbol. Modulation (eg, symbol mapping) is based on keying (QPSK), M-PSK, or M-QAM (quadrature keying), where M can be a power of 2. The data rate, coding, and modulation of each data stream can be determined by instructions executed by processor 530, which can be coupled to memory 532.

A modulation symbol for the data stream is then given to the TX MIMO processor 520, which may further process that modulation symbol (eg, for OFDM). The TX MIMO processor 520 is then N_TN modulation symbol streams_TIt can be given to multiple transmitters (TMTR) 522a ~ 522t. In some embodiments, the TX MIMO processor 520 applies beamforming weights to the symbols of the data stream and to the antenna at the source of the symbols.

Each transmitter 522 receives and processes the corresponding symbol stream to give one or more analog signals, and further tunes (eg, amplifies, filters, and upconverts) those analog signals. , A modulated signal suitable for transmission via a MIMO channel can be provided. Then N from transmitters 522a ~ 522t_TEach modulated signal is N_TIt is transmitted from the antennas 524a to 524t.

In the receiver system 550, the transmitted modulated signal is N_RIt is received by the antennas 552a to 552r, and the received signal from each antenna 552 is given to the corresponding receiver (RCVR) 554a to 554r. The receiver 554 tunes (eg, filters, amplifies, and downconverts) the received signal in question, digitizes the tuned signal, gives a sample, and further processes those samples to make the corresponding " Gives a "receive" symbol stream.

The receive (RX) data processor 560 is then N based on a particular receiver processing technique._RTransceivers 554 to N_RReceives and processes an incoming symbol stream, N_TGives a "detection" symbol stream. The RX data processor 560 then demodulates, deinterleaves, and decodes each detected symbol stream to restore the traffic data in the data stream. The processing by the RX data processor 560 complements the processing performed by the TX MIMO processor 520 and the TX data processor 514 in the transmitter system 510.

Processor 570, which can be combined with memory 572, periodically determines which precoding matrix should be used. A reverse link message can contain various types of information about communication links and / or received data streams. The reverse link message is then processed by the TX data processor 538, which also receives the traffic data of some data streams from the data source 536, modulated by the modulator 580, tuned by the transmitters 554a-554r, and the transmitter. Reply to system 510.

In transmitter system 510, the modulated signal from receiver system 550 is received by antenna 524, tuned by receiver 522, demodulated by demodulator 540, and extracts the reverse link message transmitted by receiver system 550. Processed by the RX data processor 542. Processor 530 then determines which precoding matrix should be used to determine the beamforming weights, and then processes the extracted message.

FIG. 6A is a flow chart illustrating an exemplary method of transmitting video data according to the present disclosure. The illustrated exemplary method can be performed by a source device, such as source device 120 (FIG. 1A), source device 220 (FIG. 2A), or source device 320 (FIG. 3). In some examples, when a computer-readable storage medium (eg, memory 332) is executed, one or more processors (eg, memory 332) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 331). As an example, the method will be described with reference to the wireless source device 220 of FIG. 2A.

The method of Figure 6A involves the metadata encoder 274 intercepting the video component prior to rendering on the source device 220 (601). Metadata encoder 274 generates metadata that describes the video component (603). Transport unit 233A can send video components and metadata to sink device 260.

FIG. 6B is a flowchart of an exemplary method of receiving video data according to the present disclosure. The illustrated exemplary method may be performed by a source device, such as source device 160 (FIG. 1A), source device 260 (FIG. 2B), or source device 460 (FIG. 4). In some examples, when a computer-readable storage medium (eg, memory 432) is executed, one or more processors (eg, memory 432) perform one or more of the illustrated steps in the flowchart. It may store instructions, modules, or algorithms to be performed by the processor 431). As an example, the method will be described with reference to the wireless source device 260 in FIG. 2B.

The method of FIG. 6B involves the transport unit 234B receiving the encapsulated data from the wireless source device 220 and decapsulating it (602). The metadata decoder 275 extracts the video component data and the metadata from the decapsulated data (604). Based on the metadata and the video component data, the graphics compositing module 277 generates pixel data for display (606). The pixel data can be, for example, a frame of video. The video component data may include, for example, a first type of video component data, a second type of video component data, and metadata. The metadata may identify the position of the image data for the first video component relative to the image data for the second video component. In one example, the first video component data may correspond to compressed audio / video files and the second video component data may correspond to pixel data for user interface control that should be overlaid on the video.

The techniques of the present disclosure can be implemented in a wide variety of devices or devices, including wireless handsets and integrated circuits (ICs) or sets of ICs (ie, chipsets). Although any component, module or unit has been described and given to emphasize functional aspects, those optional components, modules or units do not necessarily require implementation by different hardware units. ..

Thus, the techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the features described as modules, units or components can be implemented together in an integrated logical device or separately as a separate but interoperable logical device. When implemented in software, these techniques may be at least partially realized by a computer-readable medium with instructions that, when executed on a processor, performs one or more of the methods described above. A computer-readable medium is a non-transitory computer-readable storage. It can be equipped with (medium) and can form part of a computer program product that may contain packaging materials. Computer-readable storage media include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically erasable programmable read-only memory (EEPROM). , Flash memory, magnetic or optical data storage medium, etc. may be provided. The technique may, in addition or as an alternative, be at least partially realized by a computer-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer.

The code can be one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrations. It can be performed by a circuit or a discrete logic circuit. Thus, the term "processor" as used herein may refer to either the aforementioned structure or any other structure suitable for implementing the techniques described herein. Further, in some embodiments, the functionality described herein may be provided within a dedicated software or hardware module configured for coding and decoding, or may be incorporated into a composite video codec. .. Also, the technique may be fully implemented in one or more circuits or logic elements.

Various aspects of the present disclosure have been described. These and other aspects fall within the scope of the following claims.