Method and apparatus for outputting supplementary content from WFD

0001The present invention relates to wireless communication, and more particularly to a method and an apparatus for outputting auxiliary contents in WFD (WiFi (wireless fidelity) display).

0002Currently, the performance of mobile devices has improved remarkably as much as PCs (personal computers), but there is a limit to the size of the screen. In particular, since portability is important for smartphones, it is said that about 6 inches is a magino line with a screen size, and the 6-inch display is still a small screen from the user's point of view of enjoying multimedia contents.

0003Therefore, research is being conducted on technology that enables video on mobile devices to be viewed on a large-screen TV (television) or monitor. Such technology can be described by the term wireless display transmission technology. Wireless display transmission technology can be broadly divided into content transmission and mirroring (screen casting). Content transmission must be linked with a VOD (Video on Demand) service, rather than transmitting the screen of a mobile device as it is. Content transmission is a method of transmitting video as a signal, and mirroring is a method of sending a content file by streaming to a remote device and displaying it again on a large screen display such as a TV.

0004Mirroring (screen casting), as the name implies, is a method of simultaneously displaying the screen displayed on a mobile device as if it were illuminated by a mirror. When giving a presentation, it is similar to the method of projecting a computer screen onto a projector by connecting with a wired method such as D-sub (D-Subminiature, RGB), DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface). To do. The mirroring method has an advantage that the pixel information of the original screen can be transmitted wirelessly as it is without being dependent on a specific service in real time.

0005Wi-Fi Miracast is being researched as a wireless display transmission technology using Wi-Fi. Miracast is a wireless video transmission standard created by the WiFi alliance and is a wireless display transmission technology. Miracast is a type of mirroring (screen casting) technology that compresses the screen and audio, sends them via wireless LAN, decompresses them to a receiver made of a dongle or an integrated type, and displays them on the screen.

<p num="0006"> An object of the present invention is to provide a method for outputting auxiliary contents in WFD.</p><p num="0007"> Another object of the present invention is to provide a device for outputting auxiliary contents in WFD.</p>

<p num="0008"> The method of outputting auxiliary content in WFD (WiFi display) according to one aspect of the present invention for achieving the above-mentioned object of the present invention is a WFD for a WFD source to search for service capability in a WFD sink. A step of transmitting a service search request frame, a step of the WFD source receiving a WFD service search response frame from the WFD sink as a response to the WFD service search request frame, a step of the WFD source receiving the WFD service search response frame based on the WFD service search response frame. Steps to set up a WFD sync and WFD connection, RTSP (real time streaming) for the WFD sync and capability negotiation procedure after the WFD source has made the WFD connection. A step of sending an RTSP parameter request message based on protocol) to the WFD sink, a step of the WFD source sending an RTSP parameter response message to the WFD sink as a response to the RTSP parameter request message, and a step of the WFD source transmitting the capability negotiation. After the procedure, the WFD search response frame includes a step of transmitting a transmission stream in which a video stream containing video data of the first resolution and an auxiliary stream containing auxiliary data of the second resolution are overlaid and multiplexed to the WFD sink. , The second resolution-related information for determining the second resolution, and the RTSP parameter response frame contains the second resolution-related information.</p><p num="0009"> On the other hand, the auxiliary data includes subtitle data and user interface data, the information related to the second resolution includes information with respect to the native resolution of the WFD sink, and the first resolution is said. Determined based on the native resolution of the WFD source, the second resolution is determined based on the native resolution of the WFD sink.</p><p num="0010"> Further, the auxiliary data includes subtitle data and user interface data, and the information related to the second resolution includes information on the maximum resolution supported for the auxiliary data of the WFD sink, and the first resolution is included. The resolution of is determined based on the native resolution of the WFD source, and the second resolution is determined based on the information for the maximum resolution supported for the auxiliary data of the WFD sink.</p><p num="0011"> Further, the video data is rescaled by the WFD sink and output at a resolution different from the first resolution, and the auxiliary data is output at the second resolution without being rescaled by the WFD sink. To.</p><p num="0012"> Further, a step in which the WFD source receives user operation information by a user operation on the WFD sink from the WFD sink via UIBC (user input back channel), and an operation in which the WFD source corresponds to the user operation information. The user action information includes stream information, and the stream information is an overlay of the user action based on each of the video data and the auxiliary data output by the WFD sink. Contains identifying information that indicates on which of the images of.</p><p num="0013"> In a WFD source for transmitting auxiliary data in a WFD (WiFi display) according to another aspect of the present invention for achieving the above-mentioned object of the present invention, the WFD transmits and receives radio signals (RF (radio frequency)). The processor includes a unit and a processor operatively coupled to the RF unit, which transmits a WFD service search request frame to the WFD sink to search for service capability. A WFD service search response frame is received from the WFD sink as a response to the WFD service search request frame, a WFD connection is set with the WFD sink based on the WFD service search response frame, and after the WFD connection, the WFD RTSP (real time streaming) for sink and capability negotiation procedures An RTSP parameter request message based on protocol) is sent to the WFD sink, an RTSP parameter response message is sent to the WFD sink as a response to the RTSP parameter request message, and video data of the first resolution is transmitted after the capacity negotiation procedure. It is embodied to transmit a transmission stream in which a video stream including the video stream and an auxiliary stream containing auxiliary data of the second resolution are overlaid and multiplexed to the WFD sink, and the WFD search response frame is used for determining the second resolution. The RTSP parameter response frame includes the second resolution-related information of the above.</p><p num="0014"> On the other hand, the auxiliary data includes subtitle data and user interface data, the information related to the second resolution includes information with respect to the native resolution of the WFD sink, and the first resolution is said. Determined based on the native resolution of the WFD source, the second resolution is determined based on the native resolution of the WFD sink.</p><p num="0015"> Further, the auxiliary data includes subtitle data and user interface data, and the information related to the second resolution includes information on the maximum resolution supported for the auxiliary data of the WFD sink, and the first resolution is included. The resolution of is determined based on the native resolution of the WFD source, and the second resolution is determined based on the information for the maximum resolution supported for the auxiliary data of the WFD sink.</p><p num="0016"> Further, the video data is rescaled by the WFD sink and output at a resolution different from the first resolution, and the auxiliary data is output at the second resolution without being rescaled by the WFD sink. To.</p><p num="0017"> Further, the processor receives user operation information due to user operation on the WFD sink via UIBC (user input back channel) from the WFD sink, and executes an operation corresponding to the user operation information. The user operation information includes stream information, and the stream information is an image of a plurality of overlaid images based on each of the video data and the auxiliary data whose user operation is output by the WFD sink. Contains identifying information that dictates whether it will be performed on.</p>

<p num="0018"> When sending auxiliary content to a WFD sink with a WFD (WiFi display) source, it is possible to send an auxiliary stream containing auxiliary content with a resolution that the WFD sink can support. Therefore, the auxiliary content can be output on the display of the WFD sink adaptively according to the resolution of the WFD sink.</p>

0019<figref num="1">It is a conceptual diagram which shows the component of the Wai Pai Direct Service (WFDS) framework.</figref><figref num="2">It is a conceptual diagram which shows a WFD network.</figref><figref num="3">It is a conceptual diagram which shows a WFD session.</figref><figref num="4">It is a conceptual diagram which shows the WFD session setting method.</figref><figref num="5">It is a conceptual diagram which shows the network between a WFD source and a WFD sink.</figref><figref num="6">It is a conceptual diagram which shows the WFD capacity exchange and negotiation procedure.</figref><figref num="7">It is a conceptual diagram which shows the WFD session establishment procedure.</figref><figref num="8">It is a conceptual diagram which shows the method of transmitting and receiving a plurality of streams based on MPEG-TS which concerns on embodiment of this invention.</figref><figref num="9">It is a conceptual diagram which shows the problem of overlay multiplexing (overlay Mux) based on MPEG-TS which concerns on embodiment of this invention.</figref><figref num="10">It is a conceptual diagram which shows the problem of overlay multiplexing based on MPEG-TS which concerns on embodiment of this invention.</figref><figref num="11">It is a conceptual diagram which shows the setting method of the user interface and the subtitle which concerns on embodiment of this invention.</figref><figref num="12">It is a conceptual diagram which shows the transmission method of the user interface information and the subtitle information of the WFD source which concerns on embodiment of this invention.</figref><figref num="13">It is a conceptual diagram which shows the method of setting the optimum user interface and subtitle which concerns on embodiment of this invention.</figref><figref num="14">It is a conceptual diagram which shows the operation of the WFD source and the WFD sink based on UIBC which concerns on embodiment of this invention.</figref><figref num="15">It is a conceptual diagram which shows the operation of the WFD source and the WFD sink based on UIBC which concerns on embodiment of this invention.</figref><figref num="16">It is a conceptual diagram which shows the method of determining the resolution for the auxiliary stream which concerns on embodiment of this invention.</figref><figref num="17">It is a conceptual diagram which shows the method of determining the resolution for the auxiliary stream which concerns on embodiment of this invention.</figref><figref num="18">It is a conceptual diagram which shows the method of determining the resolution for the auxiliary stream which concerns on embodiment of this invention.</figref><figref num="19">It is a block diagram which shows the wireless apparatus to which the Example of this invention can be applied.</figref>

0020In the existing wireless LAN system, the operation between devices (AP and STA (station)) in the infrastructure BSS (basic service set) in which the access point (AP) functions as a hub is mainly defined. It was. The AP can be responsible for physical tier support for wireless / wired connectivity, routing for devices on the network, adding / removing devices to the network, and providing services. That is, in the existing wireless LAN system, the devices in the network are connected via the AP, and are not directly connected to each other.

0021The Wi-Fi Direct standard has been defined as a technology to support direct connectivity between devices. Wai Pai Direct is a direct communication technology that enables easy connection between devices (or STAs (stations)) without the access points basically required by existing wireless LAN systems. When Wai Pai Direct is used, it is possible to provide various services to users by setting connections between devices without a complicated setting process.

0022WFA (Wi-Fi Alliance) is a Wi-Fi Direct service that supports various services using Wi-Fi Direct links (for example, Send, Play, Display, Print, etc.). (WFDS) is being studied. According to WFDS, applications can be controlled or managed by a service platform called ASP (Application Service Platform).

0023WFDS-supported WFDS devices include devices that support wireless LAN systems such as display devices, printers, digital cameras, projectors and smartphones. The WFDS device can also include STAs and APs. WFDS devices in the WFDS network can be directly linked to each other.

0024FIG. 1 is a conceptual diagram showing the components of the Wai Pai Direct Services (WFDS) framework.

0025With reference to Figure 1, the WFDS framework can include Wi-Fi Direct tier 100, ASP120, service tier 140, and application tier 160.

0026The Wi-Fi Direct layer 100 is a MAC (medium access control) layer defined by the Wi-Fi Direct standard. Below the Wi-Fi Direct layer 100, a wireless connection can be configured with a physical layer (not shown) compatible with the Wi-Fi PHY. ASP (Application Service Platform) 120 is defined above the Wi-Fi Direct hierarchy 100.

0027ASP120 is a common shared platform, and session management, service instruction processing, and between ASPs between the application layer 160 above it and the Wi-Fi Direct layer 100 below it. Performs control and security functions.

0028Service hierarchy 140 is defined above ASP120. For example, service hierarchy 140 can support four basic services: Send, Play, Display, Print, and services defined by third-party applications. Not only that, service tier 140 can also support Wi-Fi Serial Bus (WSB), Wi-Fi Docking, or Neighbor Awareness Networking (NAN). ..

0029The application hierarchy 160 can provide a user interface (UI), express information in a human-recognizable form, and transmit user input to lower layers.

0030Hereinafter, in the examples of the present invention, a Wi-Fi (wireless fidelity) display (WFD) among WFDS will be disclosed more specifically.

0031The WFD standard was defined to transmit audio / video (AV) data between devices while satisfying high quality and low latency. Wi-Fi devices are connected to each other in a peer-to-peer manner through a WFD network (WFD session) to which the WFD standard is applied, without connecting to a home network, office network, or hot-spot network. be able to. Hereinafter, a device that transmits and receives data according to the WFD standard can be expressed by the term WFD device. WFD devices in a WFD network can search for information about the WFD device (eg, capability information) from each other, set up a WFD session, and then render the content through the WFD session.

0032A WFD session is a network between a source device that provides content and a sink device that receives and renders the content. The source device can also be referred to as the WFD source, and the sink device can also be referred to as the WFD sink. The WFD source can mirror the data that resides on the display (or screen) of the WFD source on the display of the WFD sink.

0033The WFD source and WFD sink can exchange the first sequence message with each other to perform device search and service search procedures. After the device and service discovery procedures between the WFD source and the WFD sink are complete, each of the WFD source and WFD sink can be assigned an internet protocol (IP) address. A transmission control protocol (TCP) connection has been established between the WFD source and the WFD sink, and since then, the real time streaming protocol (RTSP) and real time protocol (RTP) for the WFD source and WFD sink have been established. time protocol) The stack can be activated.

0034The capability negotiation procedure between the WFD source and the WFD sink is performed via RTSP, and while the capacity negotiation procedure is performed, the WFD source and the WFD sink receive RTSP-based messages (M (M). message) 1 to M4) can be exchanged. The WFD source and WFD sink can then exchange WFD session control messages. In addition, a data session via RTP can be established between the WFD source and the WFD sink. The WFD network can use the User Datagram Protocol (UDP) for data transport.

0035FIG. 2 is a conceptual diagram showing the WFD network.

0036Referring to FIG. 2, the WFD source 200 and the WFD sink 250 are WFD devices and can be connected based on WiFi-P2P.

0037Here, the WFD source 200 is a device that supports streaming of multimedia content via a WiFi P2P (peer to peer) link, and the WFD sink 250 receives multimedia content from the WFD source 200 via the P2P link. And means a device that performs steps to generate images and / or sounds. The procedure for producing images and / or sounds can be described in terms of rendering.

0038The WFD sink 250 can be divided into a primary sink and a secondary sink. In particular, the secondary sink can only render the audio payload when concatenated independently of the WFD source 200.

0039FIG. 3 is a conceptual diagram showing a WFD session.

0040The first in the upper part of Figure 3 is the audio-only session. The WFD source 300 can be linked to either the primary sink 305 or the secondary sink 310 via an audio-only session.

0041The second in the upper part of Figure 3 is a video-only session. The WFD source 320 can be concatenated with the primary sink 325.

0042The third upper part of FIG. 3 is an audio and video session, in which the WFD source 340 can be concatenated with the primary sink 345, similar to a video-only session.

0043The fourth in the upper part of Fig. 3 discloses the session connection by the Coupled WFD Sink operation. In Coupled WFD Sink operation, the primary sink 365 can render the video and the secondary sink 370 can render the audio respectively. Alternatively, Primary Sync 365 can render both video and audio.

0044Such a WFD session can be established after performing the procedure as shown in Figure 4 below.

0045FIG. 4 is a conceptual diagram showing a WFD session setting method.

0046See Figure 4, WFD Device Discovery S401, WFD Service Discovery S402, WFD Connection Setup S403, Capability Exchange and Negotiation. A WFD session can be set up after S404 has been executed.

0047Specifically, in the WFD device search procedure S401, the WFD source can search for a peer device for WFD, that is, a WFD sink, via the WFD device search procedure.

0048Beacon frames, probe request frames, probe response frames, and the like transmitted by the WFD source and WFD sink for WFD device search can include WFD IE (Information Element). Here, WFD IE is an information element including information related to WFD such as device type and device status.

0049The WFD source can send a probe request frame containing WFD IE to the WFD sink, and the WFD sink can send a probe response frame containing WFD IE in response to the probe request frame. If the WFD device is linked to an infrastructure AP and operates on a Wi-Fi P2P device, the probe request frame can contain WFD IE and P2P information elements. The probe response frame, which is the response to the probe request frame, is transmitted over the channel on which the probe request frame was received and can include both P2P IE and WFD IE.

0050Unreferenced WFD device search and related content is in accordance with the Wi-Fi Display Technical Specification and / or the Wi-Fi Peer-to-Peer (P2P) Technical Specification Wi-Fi Direct Service Addendum document. It also applies to the description.

0051In the WFD service search procedure S402, a search for the service capability between the WFD source and the WFD sink that executed the WFD device search can be executed. For example, if the WFD source sends a service search request frame that contains information about the WFD capability, the WFD sink can send a service search response frame that contains information about the WFD capability in response to the service search request frame. .. The WFD service search procedure is an optional procedure.

0052The probe request frame and probe response frame used in the WFD device search procedure for performing the WFD service search procedure can contain information indicating whether the WFD device is capable of supporting the service search procedure. ..

0053The WFD device that executed the WFD device search procedure and selectively the WFD service search procedure in the WFD connection setup procedure S403 can select the WFD device for the WFD connection setup. After the WFD device for WFD connection setup is selected by policy or user input, one of Wi-Fi P2P and TDLS (tunneled direct link service) connection method (connectivity scheme) for WFD connection. ) Can be used. The WFD device can determine the connection method based on the BSSID (basic service set identifier) subelement transmitted along with the preferred connectivity information and the WFD information element.

0054FIG. 5 is a conceptual diagram showing the network between the WFD source and the WFD sink.

0055The upper part of Fig. 5 discloses the connection between WFD source 500 and WFD sink 510 based on Wi-Fi P2P, and the lower part of Fig. 5 discloses the connection between WFD source 550 and WFD sink 560 based on TDLS link. Has been done.

0056As shown in the upper part of FIG. 5, the AP is common to or different from the WFD source 500 and the WFD sink 510. Alternatively, the AP may not exist. When making a WFD connection using a TDLS link, as shown in the lower part of Figure 5, the WFD source 550 and WFD sink 560 must maintain the same AP and connection.

0057WFD capacity exchange and negotiation procedures can be performed after the WFD connection setup procedure between WFD devices. Through WFD capability exchange and negotiation, WFD sources and WFD sinks can exchange at least one or more of the codecs they support, codec profile information, codec level information and resolution information with each other. .. WFD capacity exchange and negotiation can be executed by exchanging messages using RTSP (Real Time Streaming Protocol). You can also determine the parameter set that defines the audio / video payload between WFD sessions. The WFD capacity exchange and negotiation procedure can be performed by exchanging RTSP M1 to RTSP M4 messages, as shown in Figure 6 below.

0058After the WFD exchange and negotiation procedure, the WFD session establishment procedure can be performed.

0059FIG. 6 is a conceptual diagram showing the WFD capacity exchange and negotiation procedure.

0060Referring to FIG. 6, the WFD source can send an RTSP M1 request message to initiate the RSTP procedure and WFD capability negotiations (step S601).

0061The RTSP M1 request message can include an RTSP OPTIONS request to determine the RTSP methods set to support in the WFD sink. Upon receiving the RTSP M1 request message, the WFD sink can send an RTSP M1 response message listing the RTSP methods it supports (step S602).

0062The WFD sink can then send an RTSP M2 request message to determine which RTSP method set the WFD source supports (step S603).

0063When the RTSP M2 request message is received, the WFD source can respond with an RTSP M2 response message enumerating the RTSP methods it supports (step S604).

0064The WFD source can send an RTSP M3 request message (RTSP GET_PARAMETER request message) that specifies a list of WFD capabilities that it wants to know (step S605).

0065When the RTSP M3 request message is received, the WFD sink can respond with an RTSP M3 response message (RTSP GET_PARAMETER response message) (step S606).

0066Based on the RTSP M3 response message, the WFD source determines the optimal parameter set to be used during the WFD session and sends an RTSP M4 request message (RTSP SET_PARAMETER request message) containing the determined parameter set to the WFD sink. be able to.

0067A WFD sink that receives an RTSP M4 request message can send an RTSP M4 response message (RTSP SET_PARAMETER response message) (step S607).

0068FIG. 7 is a conceptual diagram showing the procedure for establishing a WFD session.

0069In Figure 7, the WFD source / WFD sink that performed the WFD capacity exchange and negotiation can establish a WFD session. Specifically, the WFD source can send an RTSP SET parameter request message (RTSP M5 Trigger SETUP request) to the WFD sink (S701).

0070The WFD sink can send an RTSP M5 response message in response to the RTSP SET parameter request message (step S702).

0071If the RTSP M5 message containing the trigger parameter setting (SETUP) is successfully exchanged, the WFD sink can send an RTSP SETUP request message (RTSP M6 request) to the WFD source (step S703).

0072When the RTSP M6 request message is received, the WFD source can respond with an RTSP SETUP response message (RTSP M6 response) (step S704).

0073Successful construction of an RTSP session can be directed through setting the status code of the RTSP M6 response message.

0074After a successful exchange of RTSP M6 messages, the WFD sink can send an RTSP PLAY request message to the source device to signal that it is ready to receive the RTP stream (step). S705), the WFD source can respond with an RTSP PLAY response message (step S706). The status code of the RTSP PLAY response message can indicate the successful establishment of a WFD session.

0075After the WFD session is established, the WFD source will update the RTSP M3 request message (RTSP GET_PARAMETER request message), AV (Audio / Video) format to acquire the ability for at least one RTSP parameter supported by the WFD sink. WFD sink sends RTSP PAUSE request message (RTSP M9 request message), RTSP M4 request message to set at least one RTSP parameter value corresponding to WFD session for capacity renegotiation between WFD source and WFD sink RTSP M5 request message triggering to, RTSP M12 request message instructing the WFD source to enter WFD standby mode, input types used in UIBC (user input back channel), input devices and other RTSP M14 request message or UIBC (user input back) for selecting parameters An RTSP M15 request message for enabling or disabling a channel) can be sent to the WFD sink. A WFD sink that receives the RTSP request message described above from a WFD source can respond with an RTSP response message.

0076The WFD sync then starts (or resumes) audio / video streaming with an RTSP M7 request message (RTSP PLAY request message), a pause in audio / video streaming sent from the WFD source to the WFD sync. RTSP M9 request message for (RTSP PAUSE request message), RTSP M10 request message for requesting the WFD source to change the audio rendering device, RTSP M11 request message for changing the active connector type , RTSP M12 request message indicating that the WFD sink has entered WFD standby mode, M13 request message requesting the WFD source to refresh the IDR (instantaneous decoding refresh), input types used in UIBC, input devices and others. RTSP for selecting parameters for M14 Request message or RTSP for enabling or disabling UIBC M15 Request messages etc. can be sent to the WFD source. A WFD source that receives the listed RTSP request messages from a WFD sink can respond with an RTSP response message.

0077Once the WFD session has been established and audio / video streaming has begun, the WFD source and WFD sink can proceed with audio / video streaming using codecs that both support in common. Interoperability between the two can be guaranteed by using a codec commonly supported by the WFD source and the WFD sink.

0078WFD communication is based on WFD IE, and the format of WFD IE can be defined as shown in Table 1 below.

0079<tables num="1"><img id="000003" he="107" wi="161" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0080It consists of an element ID field, a Length field, a WFD specific OUI field, an OUI type field indicating the type / version of WFD IE, and a WFD subelement field. WFD subelement field than has the format as shown in Table 2 below.

0081<tables num="2"><img id="000004" he="77" wi="161" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0082The sub-element ID can be defined as shown in Table 3 below.

0083<tables num="3"><img id="000005" he="111" wi="160" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0084With reference to Table 3, the one-octet subelement ID field can indicate what information the WFD subelement contains. Specifically, the values 0, 1, ..., 10 of the subelement ID field indicate that each of the subelements is WFD Device Information subelement, Associated BSSID subelement, WFD Audio Formats subelement, WFD Video Formats subelement, WFD 3D Video Formats subelement. , WFD Content Protection subelement, Coupled Sink Information subelement, WFD Extended Capability subelement, Local IP Address subelement, WFD Session Information subelement, Alternative MAC Address It can be shown that it is a subelement. Here, the WFD Device Information subelement can contain information necessary to determine whether to attempt pairing with the WFD device and session generation. The Associated BSSID subelement can be used to indicate the address of the currently associated AP. Each of the WFD Audio Formats subelement, WFD Video Formats subelement, and WFD 3D Video Formats subelement can be used to indicate the capabilities of a WFD device associated with audio, video, and 3D video. The WFD Content Protection subelement can convey information related to content protection methods, and the Coupled Sink Information subelement can convey information about the state of coupled sinks, MAC addresses, and so on. WFD Extended The Capability subelement can be used to convey a variety of capability information for other WFD devices, and the Local IP Address subelement can be used to convey an IP address to WFD peers during the TDLS setup process. The WFD Session Information subelement can contain information such as a list of WFD equipment information technicians in the WFD group. If the WFD connection method requires a different interface (eg, MAC address) than that used in device discovery, the Alternative MAC Address subelement can convey relevant information.

0085Hereinafter, in the embodiment of the present invention, a method of transmitting information for a user input (UI) and a subtitle (subtitle) via WFD (or miracast) is disclosed.

0086In particular, each of the UI images / subtitles is packet-sized as an elementary stream, generated by a separate PES (Packetized Elementary Stream), and then passed through the multiplexing part of MPEG2-TS (transport stream). It can be overlaid and multiplexed (MUXed) and transmitted to the video stream. In the embodiments of the present invention, a method for optimizing and setting the UI image / subtitle transmitted by the WFD source in the WFD sink is disclosed.

0087According to an embodiment of the present invention, the WFD source can recognize the native resolution of the WFD sync and transmit a UI stream and a subtitle stream having a resolution optimized for the WFD sync. Based on such a method, the WFD sink can output a UI image / subtitle image having an optimum size on the display of the WFD sink on the display of the WFD sink without deterioration of image quality.

0088Further, in the following embodiment of the present invention, when the UI image and the subtitle image are overlaid on the video image, the UIBC data (or UIBC body format) is transmitted by the WFD sync including the information related to the overlay. The method of doing so is disclosed. Information associated with overlays can include overlay indexes, stream indexes, and the like. Based on the overlay and related information contained in the UIBC data transmitted via UIBC, the WFD source can accurately recognize the intent of the user's actions to be performed on the WFD sink.

0089A single MPEG2-TS multiplexed stream transmitted on a WFD source contains multiple audio / video streams, a UI stream, and a subtitle stream, and a WFD sync is multiplexed on a single MPEG2-TS. The demultiplexed stream can be demultiplexed, and multiple different streams can be overlaid by a method called alpha-blending and output on the WFD sync display.

0090MPEG2-TS is an example of a transmit stream for overlay multiplexing multiple streams, where the transmit stream in another format (or structure) replaces MPEG2-TS as an audio stream / video stream. , UI stream, subtitle stream, etc. can also be used to multiplex and send.

0091FIG. 8 is a conceptual diagram showing a method of transmitting and receiving a plurality of streams based on MPEG-TS according to an embodiment of the present invention.

0092Referring to FIG. 8, the video stream 800, the audio stream 810, the subtitle stream 820, and the user interface stream 830 can be overlay-multiplexed and generated in MPEG2-TS840.

0093The video stream 800 may include video data, the audio stream 810 may include audio data, the subtitle stream 820 may include subtitle data, and the user interface stream 830 may contain data for the user interface.

0094The video stream 800, audio stream 810, subtitle stream 820, and user interface stream 830 are individual ESs (elementary streams), and each of the plurality of ESs has different formats, different resolutions, and different bit rates. be able to. Each of the plurality of ESs can be packetized and generated by each of the plurality of PESs, and each of the plurality of PESs can be overlay-multiplexed and generated by MPEG2-TS.

0095The WFD source can send the generated MPEG2-TS840 to the WFD sink based on WiFi Direct (WiFi P2P (peer to peer)).

0096The WFD sink can receive MPEG2-TS840 based on WiFi Direct. MPEG2-TS840 in WFD sync can be demultiplexed and re-separated into video stream 850, audio stream 860, subtitle stream 870, and user interface stream 880. The isolated video stream 850, audio stream 860, subtitle stream 870, and user interface stream 880 can be played based on alpha blending (or overlay).

0097FIG. 9 is a conceptual diagram showing a problem of overlay multiplexing (overlay Mux) based on MPEG-TS according to an embodiment of the present invention.

0098As mentioned above, when overlaying multiple ESs based on MPEG2-TS, each of the multiple ESs can have different resolutions from each other. For most WFD syncs, the resolution of the received video stream (or video data) / subtitle stream (or subtitle data) / user interface stream (or user interface data) is the native resolution of the WFD sync. If not, the resolution of the received video stream / subtitle stream / user interface stream can be rescaled (upscaled or downscaled) to match the native resolution.

0099Hereinafter, for convenience of explanation, the expression that the resolution of the video stream / subtitle stream / user interface stream is rescaled is used, but such an expression is included in the resolution / subtitle stream of the video data contained in the video stream. Subtitle data resolution / means that the resolution of the user interface data contained in the user interface stream is rescaled.

0100When the video stream is rescaled, the user does not have a great sense of refusal. However, if the user interface stream or subtitle stream is rescaled, the user can see the user interface image / subtitle image that is too large or too small on the WDF sink.

0101FIG. 9 discloses a case where the resolution of the received video stream / subtitle stream / user interface stream is upscaled to match the native resolution of the WFD sync.

0102Referring to FIG. 9, it is assumed that the native resolution of the WFD sink is 1920 × 1080 and the native resolution of the WFD source is 640 × 480. In such cases, the resolution of the video stream, subtitle stream or user interface stream contained in the MPEG2-TS containing multiple ESs transmitted by the WFD source is 640 x 480.

0103A WFD sink that receives a video stream, subtitle stream, and user interface stream with a resolution of 640 x 480 will have a video stream, subtitle stream, and user interface stream with a resolution of 640 x 480 in 1920, taking into account the native resolution of the WFD sink. Upscale ring 900 can be upscaled to × 1080.

0104In such cases, the video image can be output in the WFD sync without a large texture, but the user interface image and subtitle image will be too large. For example, WFD sources can take advantage of image subtitles and user interfaces that are relatively large relative to the screen ratio for user visibility and ease of use, and WFD sync considers such ratios. If the subtitle image and user interface image are upscaled 900 in consideration of the native resolution of the WFD sink without doing so, the subtitle image and user interface image will be too large. Not only that, but if the resolution of the subtitle image and the user interface image is significantly different from the resolution of the WFD sync display due to the upscale ring 900 of the subtitle image and the user interface image, the character readability will decrease and the user will feel more different. Will be.

0105FIG. 10 is a conceptual diagram showing a problem of overlay multiplexing based on MPEG-TS according to an embodiment of the present invention.

0106FIG. 10 discloses a case where the resolution of the received video stream / subtitle stream / user interface stream is downscaled 1000 to match the native resolution of the WFD sync.

0107Referring to FIG. 10, it is assumed that the native resolution of the WFD sink is 1920 × 1080 and the native resolution of the WFD source is 4096 × 2160. In such a case, the resolution of the video stream, subtitle stream, and user interface stream included in the MPEG2-TS including a plurality of ESs transmitted by the WFD source is 4096 × 2160.

0108A WFD sync that receives a video stream, subtitle stream, and user interface stream with a resolution of 4096 x 2160 will have a video stream, subtitle stream, and user interface stream with a resolution of 4096 x 2160 in 1920, taking into account the native resolution of the WFD sink. It can be downscaled 1000 to × 1080.

0109In such cases, the video stream can be output to the user with WFD sync without a large discomfort, but too small if the user interface and subtitles are the same. For example, WFD sources can take advantage of subtitles and user interfaces that are relatively small compared to screen proportions for user visibility and ease of use, and WFD syncs do not consider such proportions. If the subtitle and user interface are downscaled 1000 in consideration of the native resolution of the WFD sync, the subtitle and user interface will be too small.

0110That is, as shown in FIGS. 9 and 10, if the WFD source performs overlay encoding and sends ES / PES without considering the native resolution of the WFD sink, an unnecessarily large or small user on the WFD sink. Interface images and subtitle images can be output. In the embodiment of the present invention, a method for adjusting the size of the subtitle image / user interface image output superimposed on the screen according to the WFD sync is disclosed.

0111FIG. 11 is a conceptual diagram showing a method of setting a user interface and a subtitle according to an embodiment of the present invention.

0112Figure 11 discloses how the WFD source can obtain information about the native resolution of the WFD sink. As mentioned above, when the user interface and subtitle are transmitted as an overlay, a user interface and subtitle setting and display method optimized for the display of the WFD sink are required.

0113According to the embodiments of the present invention, the source WFD can perform overlay encoding based on the subtitle stream and the user interface stream that match the native resolution of the WFD sink during overlay encoding.

0114That is, the WFD sync can only rescale (upscale ring / downscale) the video stream received from the WFD source. Since the subtitle stream and user interface stream are sent by the WFD source taking into account the native resolution of the WFD sink, the WFD sink does not perform rescaling based on the native resolution for the subtitle stream and user interface stream.

0115That is, the WFD sink can receive the subtitle stream and the user interface stream that match the native resolution of the WFD sink from the WFD source and output them as they are without rescaling. Therefore, a subtitle image and a user interface image that adaptively match the resolution of the WFD sink can be provided to the WFD sink without loss of image quality for the subtitle and the user interface.

0116The WFD source can obtain information on the native resolution of the WFD sink in the WFD service search procedure and RTSP capability negotiation procedure.

0117Referring to FIG. 11, the WFD source can send the WFD service search request frame 1100 to the WFD sink. The WFD sink can send the WFD service search response frame 1120 as a response to the WFD service search request frame 1100. The WFD service search response frame 1120 can contain information for the native resolution of the WFD sink. For example, the WFD service search response frame 1120 can convey information about the native resolution of the WFD sync via the native resolution / refresh rate bitmap contained in the WFD Video Formats subelement.

0118Alternatively, the WFD source sends an RTSP M3 request message (or M3 RTSP GET PARAMETER Request) 1140, and the WFD sink sends an RTSP M3 response message (or M3 RTSP GET PARAMETER Response) 1160 in response to the RTSP M3 request message 1140. Can be sent. The RTSP M3 response message 1160 can then contain information about the native resolution of the WFD sink.

0119FIG. 12 is a conceptual diagram showing a method of transmitting user interface information and subtitle information of the WFD source according to the embodiment of the present invention.

0120With reference to Figure 12, the WFD source can recognize the native resolution of the WFD sink, and the WFD source has optimal user interface data and optimal subtitle data that match the resolution of the WFD sink with respect to the WFD sink. The ES for each of the above can be packetized, multiplexed, and sent to the WFD sink.

0121If the WFD source transmits a subtitle stream and user interface stream with the optimum resolution determined based on the native resolution of the WFD sink overlaid with the video stream, the WFD sink will display the subtitle and user interface on the display of the WFD sink. It can be optimized and output.

0122When such a method is used, the user can receive a subtitle stream and a user interface stream that match the native resolution of the WFD sync, which has a resolution independent of the resolution of the video stream (or video data). That is, the WFD sync can output the user interface and subtitles on the display without any loss of image quality.

0123To that end, the WFD source can include a set of user interface data and subtitle data with various resolutions / settings in the storage 1200. The WFD source can select the user interface and subtitle that are most suitable for the native resolution of the WFD source in the setting set of user interface data and subtitle data and send them to the WFD sink.

0124For example, the WFD source storage unit 1200 may store horizontal user interface information / horizontal subtitle information of various resolutions (2048 x N, 1920 x N, 1680 x N, 1280 x N, etc.). it can. In addition, the WFD source storage unit 1200 may store vertical user interface information / vertical subtitle information of various resolutions (N × 2048, N × 1920, N × 1680, N × 1280, etc.). it can.

0125The WFD source obtains information on the native resolution of the WFD sink as well as the preferred user interface and subtitle orientation on the WFD service search procedure and RTSP capability negotiation procedure, and information on the resolution and user interface and subtitle orientation. The optimal subtitle stream and user interface stream can be sent to the WFD sink based on.

0126FIG. 13 is a conceptual diagram showing a method of setting the optimum user interface and subtitle according to the embodiment of the present invention.

0127In FIG. 13, the WFD source can adjust the position of the user interface to fit the WFD sync based on the WFD source user interface data and subtitle data configuration set.

0128For example, as shown in FIG. 12, if the set of user interface data and subtitle data includes vertical user interface data and subtitle data and horizontal user interface data and subtitle data, then the WFD source and WFD sync Negotiation between them allows horizontal or vertical user interface data and subtitle data to be sent from the WFD source to the WFD sink.

0129Referring to FIG. 13, the WFD source has each of vertical or horizontal subtitle data / user interface data with a resolution determined based on the native resolution of the WFD sync independently of the resolution of the video stream to be encoded / streamed. It can be generated in a stream of, overlay-multiplexed with a video stream, and transmitted via MPEG2-TS.

0130The WFD sync can perform demultiplexing to MPEG2-TS and then rescale (upscale ring / downscale) only the video stream and output it on the display. In addition, the WFD sink can immediately output each of the subtitle stream and the user interface stream separated via demultiplexing to MPEG2-TS without a rescaling procedure.

0131FIG. 14 is a conceptual diagram showing the operation of the WFD source and the WFD sink based on the UIBC according to the embodiment of the present invention.

0132In FIG. 14, when the WFD source transmits a video stream and a subtitle stream user interface stream with overlay encoding, the input signal to the user's WFD sync is transmitted to the WFD source via UIBC (user input back channel). The method for doing so is disclosed. The UIBC is a channel for transmitting the user input information to the WFD source when the user input information based on the user behavior is input by the user interface existing in the WFD sink.

0133Hereinafter, problems in interpreting user input information will be disclosed in communication between the WFD sink and the WFD source via the existing UIBC.

0134Referring to FIG. 14, if the WFD source overlays each of the multiple user interface streams for each of two different user interfaces (UI1, UI2) on the MPEG2-TS, the WFD sync will be UI1 image 1400 anywhere on the display. And UI2 image 1450 can be output.

0135The WFD sink transmits the X and Y coordinates to the user input signal with reference to the display of the WFD sink via UIBC, and the WFD source accurately interprets the operation by the user input signal with reference to the display of the WFD sink. Can not do it. For example, after the user presses the left mouse button on the WFD sync display to set the click state and selects UI1 image 1400, then move the mouse to move the UI1 image 1400 to another position and press the left mouse button. It can be assumed that the click state is released by canceling.

0136In such cases, the WFD sync generates a Left mouse Down (10,200) signal (step S1410) indicating that a left mouse down signal is generated at position (10,200) on the display of the WFD sync. Generates a Mouse move (1700, 800) signal (step S1420) instructing the mouse to move to the (1700, 800) position on the WFD sync display, and a left mouse up signal at the (1710, 820) position. Left mouse Up (1710, 820) signals (step S1430) can be sequentially generated and sent to the WFD source to indicate that is generated.

0137However, where each of the UI1 image 1400 based on the user interface stream 1 for UI1 and the UI2 image 1450 based on the user interface stream 2 for UI2 is output on the display of the WFD sync, which is overlaid with the video stream. Follows the WFD sync's decision, so the WFD source cannot accurately interpret the user's intent by the user input signal. In addition, when a generic type UIBC packet is generated as in the existing case, the WFD source cannot accurately recognize which UI of UI1 and UI2 the user is acting on.

0138FIG. 15 is a conceptual diagram showing the operation of the WFD source and the WFD sink based on the UIBC according to the embodiment of the present invention.

0139FIG. 15 discloses a method for solving the problem of interpretation of user input information in the communication between the WFD sink and the WFD source via the existing UIBC disclosed in FIG. For UIBC-based operation, the WFD sink provides existing coordinate information and additional information (overlay index or MPEG2-TS stream identifier, stream type, description tag ( (description tag) etc.) can be sent to the WFD source via UIBC to convey user input information.

0140Referring to FIG. 15, the WFD source is currently the user through the WFD sink based on the stream information contained within the UIBC generic input body of the UIBC message sent by the WFD sink. You can get information about what kind of action was performed on the UI.

0141Specifically, the UIBC comprehensive input body can include the configurations shown in Table 4 below.

0142<tables num="4"><img id="000006" he="106" wi="170" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0143In Table 4 above, stream information can include fields such as Table 5 below.

0144<tables num="5"><img id="000007" he="146" wi="161" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0145That is, the WFD source can acquire the information for the UI specified by the user in the WFD sink based on the overlay index information included in the stream information field. In addition, the WFD source can acquire the information for the UI specified by the user based on the stream ID information and the stream type information included in the stream information field.

0146The description tag understands the user input signal (or user input information) generated by the user's actions for each of the UI image / subtitle image / video image indicated by each of the multiple overlay IDs (or stream IDs). Can include detailed information for For example, the descriptive tag can include information about the X and Y coordinates possible in the image output on the display of the WFD sink corresponding to the overlay ID. For example, the X-coordinate and Y-coordinate values included in the description tag can correspond to the horizontal / vertical size of the UI image output by the WFD sync. For example, if the size of the UI1 image (UI image 1) 1500 is 400 x 200 and the user generates a user input signal on the UI image 1 1500 on the WFD sync display, the X coordinates that can be generated, The Y coordinate can be in the range from (0, 0) to the maximum (400, 200).

0147As shown in FIG. 15, the WFD sink can send the stream information including the stream information in the UIBC comprehensive input body. The WFD source can determine on which UI image the WFD sync ran based on the stream information. For example, the WFD sync left mouse down and click on specific partial coordinates (390, 200) on UI image 1 1500 (step S1510), relative to the specific partial coordinates (step S1510) based on the video stream based video stream. You can move the mouse to 1700, 800) (step S1520) and perform a Mouse up operation at specific position coordinates (200, 100) on UI image 2 1550 (step S1530).

0148In this way, the WFD sink can propagate coordinates relative to each overlay image (or stream to be overlaid) (eg, a UI image) that is not coordinate-value transmission relative to the entire screen. Therefore, if the position of each overlay UI is changed on the display of the WFD sink, the WFD source will accurately perform the user action by the user input signal based on the stream information in the UIBC comprehensive input body transmitted from the WFD sink. Can be interpreted (or determined).

0149Further, in the embodiment of the present invention, when the user interface and subtitle are transmitted based on overlay multiplexing, the following method is used for setting and outputting the UI and subtitle optimized for the display of WFD sync. Can be UI and subtitles can be referred to as auxiliary contents, and streams for UI and subtitles can be referred to as auxiliary contents. Compare the resolution of the auxiliary content (auxiliary stream) that can be supported by the WFD source with the native resolution (or panel resolution) of the WFD sync, and the one with the lower resolution is the auxiliary content (auxiliary stream) that is output by the WFD sync. Can be set with the resolution of.

0150As a formula, Resolution for auxiliary contents in WFD sink = Minimum {Source device (WFD source)'s maximum resolution, sink device (Or, it can be expressed in native resolution (Sink device (WFD sink)'s native resolution)}.

0151For example, if the maximum resolution of the WFD source is 2048 x 1080 (2k), the native resolution of the WFD sink is 3840 x 2160 (or 4096 x 2160) (4k), and the maximum resolution of the sink device is 2k. The resolution of the auxiliary content output by WFD Sync is 2k.

0152As another example, if the maximum resolution of the WFD source is 4k, the native resolution of the WFD sync is 4k, and the maximum resolution of the sink device is 2k, the resolution of the auxiliary content output by the WFD sync is 4k. is there.

0153Further, in the embodiment of the present invention, when the user interface and subtitle are transmitted based on overlay multiplexing, the following method is used for setting and outputting the UI and subtitle optimized for the display of WFD sync. Can be

0154Compare the maximum resolution that can be supported for the auxiliary content (or auxiliary stream) of the WFD source with the maximum resolution that can be supported for the auxiliary content (or auxiliary stream) of the WFD sync, and the lower of the two resolutions is WFD. The maximum resolution for the auxiliary content (or auxiliary stream) output by the sync.

0155As a formula, Resolution for auxiliary streams in WFD sink = Minimum {Source device (WFD source)'s maximum resolution of auxiliary streams and the maximum resolution of auxiliary streams of the sink device (or WFD sink) (Sink device (WFD sink)'s maximum resolution of auxiliary streams)}.

0156For example, if the WFD source has a maximum supportable resolution of 2048 x 1080 (2k) for the auxiliary stream and the WFD sink has a maximum supportable resolution of 3840 x 2160 (4k) for the auxiliary stream, the WFD The resolution of the auxiliary content output by the sync is 2k.

0157As another example, if the WFD source has a maximum supportable resolution of 4k for the auxiliary stream and the WFD sync has a maximum supportable resolution of 2k for the auxiliary stream, the auxiliary content output by the WFD sink. The resolution of is 4k.

0158FIG. 16 is a conceptual diagram showing a method of determining the resolution for the auxiliary stream according to the embodiment of the present invention.

0159Referring to FIG. 16, the WFD source is based on WFD2 service discovery and RTSP capability negotiation for information on the auxiliary stream formats that the WFD sink can support / for the maximum resolution of the auxiliary stream that can be supported. Information can be obtained.

0160For example, the WFD source can send the WFD2 service search request frame 1600 to the WFD sink. The WFD sink can send the WFD2 service search response frame 1610 to the WFD source as a response to the WFD2 service search request frame 1600. The WFD2 service search response frame 1610 of the WFD sink can include auxiliary stream formats subelements. Auxiliary stream format sub-elements can include auxiliary stream codec bitmaps, reference overlay resolution bitmaps. Auxiliary stream codec bitmaps can contain information about auxiliary stream formats that WFD sinks can support, and reference overlay resolution bitmaps can contain information about the maximum resolution of auxiliary streams that WFD sinks can support.

0161Also, information on the auxiliary stream formats supported by the WFD sink based on the RTSP M3 request message (M3 RTSP GET PARAMETER Request) 1620 / RTSP M3 response message (M3 RTSP GET PARAMETER Response) 1630 / maximum of the auxiliary streams that can be supported. Information about the resolution can also be obtained.

0162RTSP M3 response message 1630 can include an auxiliary stream format. Auxiliary stream formats can include auxiliary stream codec bitmaps, reference overlay resolution bitmaps. Auxiliary stream codec bitmaps can contain information about auxiliary stream formats that WFD sinks can support, and reference overlay resolution bitmaps can contain information about the maximum resolution of auxiliary streams that WFD sinks can support.

0163Auxiliary stream codec bitmaps can be defined as shown in Table 6 below.

0164<tables num="6"><img id="000008" he="49" wi="166" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0165With reference to Table 6, the auxiliary stream codec bitmap can contain information for the codecs of the supported auxiliary content. In the auxiliary stream codec bitmap, the first bit indicates information about whether to support PNG format auxiliary content, and the second bit indicates information about whether to support JPEG format auxiliary content. be able to.

0166The reference overlay resolution bitmap can be defined as shown in Table 7 below.

0167<tables num="7"><img id="000009" he="141" wi="158" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0168With reference to Table 7, the reference overlay resolution bitmap can contain bits to indicate the resolutions supported to generate the auxiliary stream.

0169For supported resolutions / refresh rates for auxiliary streams, refer to CEA (Consumer Electronics Association), VESA (Video Electronics Standards Association), and HH (handheld) format resolutions / refresh rates.

0170Reference Overlay Resolution All supported in the selected graphic format via 1: 0 bit information based on 7: 2 bit information, including information for the graphic format referenced via the bitmap's 1: 0 bit information. Information about the resolution can be indicated.

0171Table 8 illustrates the index for CEA resolution / refresh rate.

0172<tables num="8"><img id="000010" he="163" wi="162" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0173Alternatively, in the embodiments of the present invention, when the user interface and subtitles are transmitted based on overlay multiplexing, the following methods are used for WFD sync display-optimized UI and subtitle settings and output. Can be

0174The resolution of the auxiliary content (or auxiliary stream) output by the WFD sink is the minimum of 1) the maximum resolution for the auxiliary stream of the WFD source, 2) the native resolution of the WFD sink and the maximum resolution of the auxiliary stream of the WFD sink. On the other hand, it is the minimum value of 1) and 2).

0175As a formula, Resolution for auxiliary streams = Minimum {Source device (WFD source)'s maximum resolution of auxiliary streams, Minimum [ At the native resolution of the sink device (WFD sink)'s native resolution, at the maximum resolution of auxiliary streams of the sink device (WFD sink)'s maximum resolution of auxiliary streams}. is there.

0176For example, if the WFD source has a maximum supportable resolution of 2k for the auxiliary stream, the WFD sink has a maximum supportable resolution of 2k for the auxiliary stream, and the WFD sink has a native resolution of 4k, then the WFD The resolution of the auxiliary content output by the sync is 2k.

0177FIG. 17 is a conceptual diagram showing a method of determining the resolution for the auxiliary stream according to the embodiment of the present invention.

0178With reference to Figure 17, the WFD source can support WFD sync with information on the native resolution of the WFD sync, information on the max video resolution of the WFD sync, and WFD sync based on the WFD service discovery procedure and RTSP capability negotiation procedure. Information on the format of the auxiliary stream and information on the maximum resolution of the auxiliary stream that can be supported can be obtained.

0179For example, the WFD source can send the WFD2 service search request frame 1700 to the WFD sink. The WFD sink can send the WFD2 service search response frame 1710 to the WFD source as a response to the WFD2 service search request frame 1700. The WFD2 service search response frame 1710 of the WFD sink can include a video format subelement and an auxiliary stream formats subelement.

0180A video format subelement can include information about the native resolution of the WFD sync and information about the maximum video resolution of the WFD sync.

0181Auxiliary stream format sub-elements can include auxiliary stream codec bitmaps, reference overlay resolution bitmaps. Auxiliary stream codec bitmaps can contain information about auxiliary stream formats that WFD sinks can support, and reference overlay resolution bitmaps can contain information about the maximum resolution of auxiliary streams that WFD sinks can support.

0182Also, based on the RTSP M3 Request Message (M3 RTSP GET PARAMETER Request) 1720 / RTSP M3 Response Message (M3 RTSP GET PARAMETER Response) 1730, information on the native resolution of WFD Sync, information on the maximum video resolution of WFD Sync, in WFD Sync. Information on the formats of supportable auxiliary streams and information on the maximum resolution of supportable auxiliary streams can be obtained.

0183RTSP M3 response message 1730 can include video format and auxiliary stream format.

0184The video format can include information about the native resolution of the WFD sink and information about the maximum video resolution of the WFD sink.

0185Auxiliary stream formats can include auxiliary stream codec bitmaps, reference overlay resolution bitmaps. Auxiliary stream codec bitmaps can contain information about auxiliary stream formats that WFD sinks can support, and reference overlay resolution bitmaps can contain information about the maximum resolution of auxiliary streams that WFD sinks can support.

0186The WFD source can determine the resolution of auxiliary content to send to the WFD sink based on the information contained in the WFD2 service discovery response frame 1710 / RTSP M3 response message 1730.

0187Table 9 below describes a bitmap that provides information about the native resolution of the WFD sync contained in the video format subelement.

0188<tables num="9"><img id="000011" he="126" wi="160" file="JP2018508129A_D0001.tif" img-format="tif" img-content="drawing" /></tables>

0189Information about the native resolution of the WFD sink can be indicated based on the index for the resolution supported by the selected graphic format. Specifically, the video format sub-element can indicate information about the native resolution of the WFD sync via an index for one resolution / refresh rate of CEA, VESA, and HH.

0190Information about the format of the auxiliary stream that the WFD sink can support and information about the maximum resolution of the auxiliary stream that the WFD sink can support can be transmitted based on the bitmaps shown in Tables 6 and 7 above.

0191Alternatively, in the embodiments of the present invention, when the user interface and subtitles are transmitted based on overlay multiplexing, the following methods are used for WFD sync display-optimized UI and subtitle settings and output. Can be

0192The resolutions of the auxiliary content (or auxiliary stream) output by the WFD sink are 1) the maximum resolution of the WFD source, 2) the native resolution of the WFD sink and the minimum resolution of the maximum resolution of the WFD sink 1), 2). Is the minimum value of.

0193As a formula, Resolution for auxiliary streams in WFD sink = Minimum {Source device (WFD source)'s maximum resolution, Minimum [ The native resolution of the sink device (WFD sink)'s native resolution, the maximum resolution of the sink device (WFD sink)'s maximum resolution]}.

0194The maximum resolution of a WFD source is the maximum resolution that can be supported for auxiliary streams supported by a particular AUX (auxiliary) codec selected by the source.

0195The maximum resolution of the WFD sink can be indicated based on the maximum resolution information contained in the RTSP M3 response message. Information about the maximum resolution of the WFD sync can be contained in parameters such as wfd-video-codecs, wfd-3d-video-formats, wfd-preferred-display-mode in RTSP M3 response messages.

0196The native resolution of the WFD sink can be indicated based on the native resolution information contained in the RTSP M3 response message. Native resolution information can be included in parameters such as wfd-video-codecs, wfd-3d-video-formats, wfd-preferred-display-mode in RTSP M3 response messages.

0197For example, if the maximum resolution of the WFD source is 2k, the maximum resolution of the WFD sync is 2k, and the native resolution of the WFD sync is 4k, the resolution of the auxiliary content output by the WFD sync is 2k.

0198FIG. 18 is a conceptual diagram showing a method of determining the resolution for the auxiliary stream according to the embodiment of the present invention.

0199With reference to FIG. 18, the WFD source can obtain information on the native resolution of the WFD sink and information on the maximum resolution of the WFD sink based on the WFD service search procedure and the RTSP capacity negotiation procedure.

0200For example, the WFD source can send the WFD2 service search request frame 1800 to the WFD sink. The WFD sink can send the WFD2 service search response frame 1810 to the WFD source as a response to the WFD2 service search request frame 1800. The WFD2 service search response frame 1810 of the WFD sink can include a video format sub-element.

0201The video format sub-element can include information about the native resolution of the WFD sink and information about the maximum resolution of the WFD sink.

0202Also, based on the RTSP M3 request message (M3 RTSP GET PARAMETER Request) 1820 / RTSP M3 response message (M3 RTSP GET PARAMETER Response) 1830, it is possible to obtain information on the native resolution of the WFD sink and information on the maximum resolution of the WFD sink. it can.

0203RTSP M3 response message 1830 can include a video format.

0204The video format can include information about the native resolution of the WFD sink and information about the maximum resolution of the WFD sink.

0205The WFD source can determine the resolution of auxiliary content to send to the WFD sink based on the information contained in the WFD2 service discovery response frame 1810 / RTSP M3 response message 1830.

0206FIG. 19 is a block diagram showing a wireless device to which an embodiment of the present invention can be applied.

0207Referring to FIG. 19, the radio device is a WFD source 1900 and a WFD sink 1950 that can embody the embodiments described above.

0208The WFD source 1900 includes a processor 1910, memory 1920 and an RF unit (radio frequency unit) 1930.

0209The RF unit 1930 can transmit / receive radio signals in connection with the processor 1910.

0210Processor 1910 can embody the functions, processes and / or methods proposed in the present invention. For example, the processor 1910 can be embodied to perform the operation of the WFD source 1900 according to the embodiment of the present invention described above. The processor can perform the operation of the WFD source 1900 disclosed in the examples of FIGS. 1-18.

0211For example, processor 1910 sends a WFD service search request frame to the WFD sink to search for service capability, receives a WFD service search response frame from the WFD sink as a response to the WFD service search request frame, and receives a WFD. WFD sinks and WFD connections can be configured based on service discovery response frames.

0212In addition, after the WFD connection, the processor 1910 sends an RTSP parameter request message based on the RTSP (real time streaming protocol) to the WFD sink for the capability negotiation procedure with the WFD sink, and responds to the RTSP parameter request message. It can be embodied to send an RTSP parameter response message to the WFD sink.

0213In addition, after the capacity negotiation procedure, the processor 1910 embodies a transmission stream in which a video stream containing video data of the first resolution and an auxiliary stream containing auxiliary data of the second resolution are overlaid and multiplexed to be transmitted to the WFD sink. The WFD search response frame described above may include a second resolution-related information for determining the second resolution, and the RTSP parameter response frame may include the second resolution-related information.

0214Auxiliary data can include subtitle data and user interface data.

0215As mentioned above, the information associated with the second resolution includes information for the native resolution of the WFD sink, the first resolution being determined based on the native resolution of the WFD source and the second. The resolution can be determined based on the native resolution of the WFD sync.

0216Alternatively, the information associated with the second resolution contains information about the maximum resolution supported for the WFD sync auxiliary data, the first resolution is determined based on the native resolution of the WFD source, and the second The resolution can be determined based on information about the maximum resolution supported for the auxiliary data in the WFD sync.

0217The video data can be rescaled at the WFD sink and output at a resolution different from the first resolution, and the auxiliary data can be output at the second resolution without being rescaled at the WFD sink.

0218In addition, the processor 1910 is embodied to receive user operation information by user operation on the WFD sink via UIBC (user input back channel) from the WFD sink and execute an operation corresponding to the user operation information. Can be done.

0219The user action information includes stream information, and the stream information indicates on which of the multiple overlay images the user action is executed based on the video data and auxiliary data output by the WFD sync. Identification information to be used can be included.

0220The WFD sink 1950 includes a processor 1960, a memory 1970 and an RF unit (radio frequency unit) 1980.

0221The RF unit 1980 can transmit / receive radio signals in connection with the processor 1960.

0222Processor 1960 can embody the functions, processes and / or methods proposed in the present invention. For example, the processor 1960 can be embodied to perform the operation of the WFD sink 1950 according to the embodiment of the present invention described above. The processor can perform the operation of the WFD sink 1950 in the embodiments of FIGS. 1-18.

0223For example, the processor 1960 can be embodied to send information to the WFD source to determine the resolution of the auxiliary data output by the WFD sink. For example, the processor 1960 can be embodied to send information about the native resolution of the WFD sink, the resolution of auxiliary content that can be output by the WFD sink, etc., based on the WFD service search response frame / RTSP-based response message. ..

0224The processor 1960 demultiplexes the transmit stream (MPEG2-TS) received from the WFD source, and the video data contained in the video stream is rescaled and output on the display at different resolutions to the auxiliary stream (subtitle stream, user). The subtitle data and user interface data included in the interface stream) can be output with the same resolution as the received data without rescaling.

0225Processors 1910, 1960 can include application-specific integrated circuits (ASICs), other chipsets, logic circuits, data processors and / or converters that convert between baseband and radio signals. The memories 1920 and 1970 can include ROM (read-only memory), RAM (random access memory), flash memory, memory cards, storage media and / or other storage devices. RF units 1930, 1980 may include one or more antennas that transmit and / or receive radio signals.

0226When the embodiments are embodied in software, the techniques described above can be embodied in modules (processes, functions, etc.) that perform the functions described above. Modules are stored in memory 1920, 1970 and can be executed by processors 1910, 1960. The memories 1920, 1970 are inside or outside the processors 1910, 1960 and can be connected to the processors 1910, 1960 by various well-known means.