Streaming techniques for video display systems
Abstract
This record has no abstract on file.
Term
Projected expiry 14 December 2030.
- Priority
- Filed
- Granted
- Today
- Projected expiry
30 claims: 7 independent, 23 dependent
- 1ビデオディスプレイシステムのホストデバイスがデコンポーズドマルチストリーム(DMS)を生成することと、ここで、前記DMSが、ディスプレイウィンドウの第1のエリア中の第1のコンテンツと、前記ディスプレイウィンドウの第2のエリア中の第2のコンテンツとを定義し、前記第1のコンテンツが前記DMS中の第1のフレームレートを定義し、前記第2のディスプレイコンテンツが前記DMS中の第2のフレームレートを定義し、前記第1のフレームレートが前記第2のフレームレートとは異なる、 前記ホストデバイスが前記ビデオディスプレイシステムのクライアントデバイスに前記DMSを通信することと を備え、 前記DMSを生成することが、アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることを含み、前記ホストデバイスが、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することと、前記ホストデバイスが、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することに応答して、前記ホストデバイスにおいてディスプレイバッファから直接、前記第1のコンテンツおよび前記第2のコンテンツを含むシグナルコンポーズドストリーム(SCS)を生成することとをさらに備える、方法。
- 2前記ホストデバイスが、前記第1のコンテンツと前記第2のコンテンツとに基づいて前記DMS中の前記第1のフレームレートと前記第2のフレームレートとを動的に調整することをさらに備える、請求項1に記載の方法。
- 3前記DMSを生成することが、アプリケーションから前記第1のコンテンツをインターセプトし、前記ホストデバイス中のディスプレイバッファから前記第2のコンテンツをキャプチャすることを含む、請求項1に記載の方法。
- 4前記ホストデバイスが、前記第1のコンテンツと前記第2のコンテンツとに基づいて、前記ディスプレイバッファから前記第2のコンテンツをキャプチャすることに関連するキャプチャレートを動的に低減することをさらに備える、請求項3に記載の方法。
- 5前記DMSを生成することが、前記第2のコンテンツを符号化することを含む、請求項1に記載の方法。
- 6前記第1のエリアおよび前記第2のエリアがそれぞれ重なるエリアを含み、 前記ホストデバイスが、前記第1のコンテンツまたは前記第2のコンテンツのどちらが前記重なるエリア中で上にあるかを識別する情報を前記DMS中に生成すること をさらに含む、請求項1に記載の方法。
- 7前記DMSを生成することが、前記重なるエリア中で前記第1のコンテンツまたは前記第2のコンテンツのいずれかのデータを低減するかまたはなくすことを含み、前記低減されるかまたはなくされるデータが前記重なるエリア中で他のデータの下にある、請求項6に記載の方法。
- 8前記第1のコンテンツがビデオシーケンスを備え、前記第2のコンテンツが非ビデオアプリケーションの出力を備える、請求項1に記載の方法。
- 9前記第1のフレームレートが前記第2のフレームレートよりも大きい、請求項8に記載の方法。
- 10前記第1のコンテンツがビデオシーケンスを備え、前記第2のコンテンツがグラフィカルユーザインターフェース要素を備える、請求項1に記載の方法。
- 11前記ホストデバイスが、ユーザ入力に応答して前記第1のコンテンツまたは前記第2のコンテンツのうちの1つのみを含むように前記DMSを調整することをさらに備える、請求項1に記載の方法。
- 12前記ホストデバイスが、前記ホストデバイスと前記クライアントデバイスとの間で利用可能な帯域幅を判断することと、 前記ホストデバイスが、前記利用可能な帯域幅に基づいて前記第1のフレームレートと前記第2のフレームレートの一方または両方を調整することと をさらに備える、請求項1に記載の方法。
- 13前記ホストデバイスが、前記ホストデバイスと前記クライアントデバイスとの間で利用可能な帯域幅を判断することと、 前記ホストデバイスが、前記利用可能な帯域幅に基づいて前記第1のコンテンツと前記第2のコンテンツの一方または両方を動的に符号化することと をさらに備える、請求項1に記載の方法。
- 14前記ホストデバイスが、前記第1のコンテンツと前記第2のコンテンツの一方または両方を動的に符号化することをさらに備える、請求項1に記載の方法。
- 15ビデオディスプレイシステムのホストデバイスであって、 デコンポーズドマルチストリーム(DMS)を生成するDMS生成器と、ここで、前記DMSが、ディスプレイウィンドウの第1のエリア中の第1のコンテンツと、前記ディスプレイウィンドウの第2のエリア中の第2のコンテンツとを定義し、前記第1のコンテンツが前記DMS中の第1のフレームレートを定義し、前記第2のディスプレイコンテンツが前記DMS中の第2のフレームレートを定義し、前記第1のフレームレートが前記第2のフレームレートとは異なる、 前記ホストデバイスから前記ビデオディスプレイシステムのクライアントデバイスに前記DMSを通信するトランスポートユニットと を備え、 前記DMS生成器が、アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトし、前記DMS生成器が、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することと、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することに応答して、前記ホストデバイスにおいてディスプレイバッファから直接、前記第1のコンテンツおよび前記第2のコンテンツを含むシグナルコンポーズドストリーム(SCS)を生成することとを行う、ホストデバイス。
- 16前記DMS生成器が、前記第1のコンテンツと前記第2のコンテンツとに基づいて前記DMS中の前記第1のフレームレートと前記第2のフレームレートとを動的に調整する、請求項15に記載のホストデバイス。
- 17前記DMS生成器が、アプリケーションから前記第1のコンテンツをインターセプトし、前記ホストデバイス中のディスプレイバッファから前記第2のコンテンツをキャプチャする、請求項15に記載のホストデバイス。
- 18前記DMS生成器が、前記第1のコンテンツと前記第2のコンテンツとに基づいて、前記ディスプレイバッファから前記第2のコンテンツをキャプチャすることに関連するキャプチャレートを動的に低減する、請求項17に記載のホストデバイス。
- 19前記DMS生成器が、前記DMS中で前記第2のコンテンツを符号化するために符号器を使用する、請求項17に記載のホストデバイス。
- 20前記第1のエリアおよび前記第2のエリアがそれぞれ重なるエリアを含み、前記DMS生成器が、 前記第1のコンテンツまたは前記第2のコンテンツのどちらが前記重なるエリア中で上にあるかを識別する情報を前記DMS中に生成する、請求項15に記載のホストデバイス。
- 21前記DMS生成器が、前記重なるエリア中で前記第1のコンテンツまたは前記第2のコンテンツのいずれかのデータを低減するかまたはなくし、前記低減されるかまたはなくされるデータが前記重なるエリア中で他のデータの下にある、請求項20に記載のホストデバイス。
- 22前記第1のコンテンツがビデオシーケンスを備え、前記第2のコンテンツが非ビデオアプリケーションの出力を備える、請求項15に記載のホストデバイス。
- 23前記第1のフレームレートが前記第2のフレームレートよりも大きい、請求項22に記載のホストデバイス。
- 24前記第1のコンテンツがビデオシーケンスを備え、前記第2のコンテンツがグラフィカルユーザインターフェース要素を備える、請求項15に記載のホストデバイス。
- 25前記DMS生成器が、ユーザ入力に応答して前記第1のコンテンツまたは前記第2のコンテンツのうちの1つのみを含むように前記DMSを調整する、請求項15に記載のホストデバイス。
- 26前記DMS生成器が、 前記ホストデバイスと前記クライアントデバイスとの間で利用可能な帯域幅を判断することと、 前記利用可能な帯域幅に基づいて前記第1のフレームレートと前記第2のフレームレートの一方または両方を調整することと を行う、請求項15に記載のホストデバイス。
- 27前記DMS生成器が、 前記ホストデバイスと前記クライアントデバイスとの間で利用可能な帯域幅を判断することと、 前記利用可能な帯域幅に基づいて前記第1のコンテンツと前記第2のコンテンツの一方または両方を動的に符号化することと を行う、請求項15に記載のホストデバイス。
- 28前記DMS生成器が、前記第1のコンテンツと前記第2のコンテンツの一方または両方を動的に符号化する、請求項15に記載のホストデバイス。
- 29ビデオディスプレイシステムのホストデバイスのプロセッサ中での実行時に、前記ホストデバイスに、 デコンポーズドマルチストリーム(DMS)を生成することと、ここで、前記DMSが、ディスプレイウィンドウの第1のエリア中の第1のコンテンツと、前記ディスプレイウィンドウの第2のエリア中の第2のコンテンツとを定義し、前記第1のコンテンツが前記DMS中の第1のフレームレートを定義し、前記第2のディスプレイコンテンツが前記DMS中の第2のフレームレートを定義し、前記第1のフレームレートが前記第2のフレームレートとは異なる、 ホストデバイスから前記ビデオディスプレイシステムのクライアントデバイスに前記DMSを通信することと を行わせる命令を備え、 前記DMSを生成することが、アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることを含み、前記命令はさらに、前記ホストデバイスに、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することと、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することに応答して、前記ホストデバイスにおいてディスプレイバッファから直接、前記第1のコンテンツおよび前記第2のコンテンツを含むシグナルコンポーズドストリーム(SCS)を生成することとを行わせる、コンピュータ可読記憶媒体。
- 30ビデオディスプレイシステムのホストデバイスであって、 デコンポーズドマルチストリーム(DMS)を生成するための手段と、ここで、前記DMSが、ディスプレイウィンドウの第1のエリア中の第1のコンテンツと、前記ディスプレイウィンドウの第2のエリア中の第2のコンテンツとを定義し、前記第1のコンテンツが前記DMS中の第1のフレームレートを定義し、前記第2のディスプレイコンテンツが前記DMS中の第2のフレームレートを定義し、前記第1のフレームレートが前記第2のフレームレートとは異なる、 前記ホストデバイスから前記ビデオディスプレイシステムのクライアントデバイスに前記DMSを通信するための手段と を備え、 前記DMSを生成するための手段が、アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトする手段を含み、前記DMSを生成するための手段が、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することと、前記アプリケーションから前記第1のコンテンツまたは前記第2のコンテンツをインターセプトすることが不可能であると判断することに応答して、前記ホストデバイスにおいてディスプレイバッファから直接、前記第1のコンテンツおよび前記第2のコンテンツを含むシグナルコンポーズドストリーム(SCS)を生成することとを行う、ホストデバイス。
Independent claims30
79 paragraphs, as filed
Priority claim
This application claims the benefit of US Provisional Application No. 61 / 286,287 filed December 14, 2009, the entire contents of which are incorporated herein by reference.
The present disclosure relates to video displays, and more particularly to video display systems for generating, communicating, and rendering decomposed multi-streams (DMS).
Video displays are used in a wide range of devices. Such devices include, but are not limited to, digital televisions, wireless communication devices, personal digital assistants (PDAs), laptop or desktop computers, workstations, digital cameras, camcorders, digital media players, video game devices, portables. Includes movie players, electronic readers, tablet computers, cellular or satellite radiotelephones, smartphones and more.
Video display systems often include host devices (such as computers) that generate streams of video content for client devices (such as displays). In many cases, the host device may communicate the video data to a client device that includes its own display and may still include a display that is larger than the host device. Various techniques and standards have been developed to allow host and client devices to communicate such video content over high-speed wireless links so that both the host and client devices can view the video content. Has been done.
Some video display systems generate a stream of video content called a signal-composed stream (SCS). Although the SCS comprises a single video stream, the video content can still contain different areas that may sometimes look like different streams. In this case, the host device may generate an SCS, and the client device may receive the SCS from the host device and render a signal video stream to display the content. In essence, in the case of SCS, the host device sends a "screenshot" frame (eg, the entire content of the display buffer). Screenshots can optionally contain windows with different content, but the windows with different content in the SCS are only part of the screenshot, not separate streams.
In contrast to systems that generate SCS, other types of video display systems generate streams of video content, called decomposed multistreams (DMS). The DMS comprises two or more different video streams of content that can correspond to separate areas of the visible area or, in some cases, overlapping areas within the visible area. In this case, the host device may generate a DMS with two or more different video streams of content, and the client device receives the DMS from the host device and two or more different video streams of content contained in the DMS. Video containing can be rendered on the display screen. The host device may render different streams of DMS in different areas of the screen, and in some cases different areas of different streams in the DMS may overlap.
This disclosure describes techniques that can improve the generation of decomposed multistream (DMS) by the host device of a video display system and the display of DMS by a client device of a video display system. The technique may apply different frame rates to different streams in the DMS, and the frame rate may depend on the content in the different streams. For example, a stream in a DMS may comprise a sequence of full motion video information that can be rendered at a relatively high frame rate (such as 10 to 120 frames per second commonly used in video playback). However, another stream in the DMS can be associated with a display background, various graphic user interface control windows or elements, or a display window containing non-video content (such as email or documents). The second stream in the DMS can be rendered at a frame rate much slower than the frame rate used for the sequence of full motion video information. In addition, if different streams are associated with overlapping areas within the visible area of the display screen, techniques can be applied to reduce the data in one or both streams in the overlapping areas.
Other techniques that can improve DMS are also described. The host device may include a computer device (such as a laptop computer, smartphone or other computer device), and the client device may include a wireless display used to render the same output as the computer device. DMS can be used to communicate data over high speed wireless links so that both host devices (eg, computer devices) and client devices (eg, wireless displays) can display similar content. As the content itself can be different in different streams or overlap within the visible area of the display, this disclosure describes techniques for improving DMS. To generate DMS on the host device, the host device can access not only the display buffer on the host device, but also the content from the application that produces the content. For example, as a fall-back technique for communicating data from a host device to a client device when DMS is not possible or desirable because the content from the application that produces the content is inaccessible. , Signal Compose Stream (SCS) techniques can be implemented.
As an example, in the present disclosure, the DMS is generated via the host device of the video display system, and the DMS is the first content in the first area of the display window and the second area of the display window. The first content defines the first frame rate in the DMS, the second display content defines the second frame rate in the DMS, and the first frame. A method of providing generation, where the rate is different from the second frame rate, will be described. The method also comprises communicating DMS from the host device to the client device of the video display system.
In another example, in the present disclosure, a client device of a video display system receives a DMS from a host device of the video display system, wherein the DMS is the first content in a first area of the display window. , Defines the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content defines the second frame rate in the DMS. Is defined, and a method of providing reception in which the first frame rate is different from the second frame rate will be described. The method also comprises rendering the first display content and the second display content on the client device.
In another example, in the present disclosure, the host device of a video display system, wherein the host device comprises a DMS generator that produces a DMS, the DMS being the first content in a first area of the display window. , Defines the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content defines the second frame rate in the DMS. Is defined, and a host device in which the first frame rate is different from the second frame rate will be described. The host device also comprises a transport unit that communicates the DMS from the host device to the client device of the video display system.
In another example, in the present disclosure, a client device of a video display system, the client device comprising a transport unit that receives DMS from the host device, the DMS being the first in a first area of the display window. And the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content is the second in the DMS. A client device in which the first frame rate is different from the second frame rate will be described. The client device also includes a display unit and a DMS rendering unit that renders the first display content and the second display content on the display unit.
In another example, in the present disclosure, the host device of a video display system, the host device is a means for generating a DMS, the DMS is the first content in the first area of the display window. And the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content is the second frame in the DMS. A host device that defines a rate and provides means for generating, where the first frame rate is different from the second frame rate, and for communicating the DMS from the host device to the client device of the video display system. Will be described.
In another example, in the present disclosure, a client device of a video display system, the means by which the client device receives DMS from the host device of the video display system, the DMS being the first display window. The first content in the area and the second content in the second area of the display window are defined, the first content defines the first frame rate in the DMS, and the second display content A means for receiving, which defines a second frame rate in the DMS and the first frame rate is different from the second frame rate, and the first display content and the second display content on the client device. Describes a client device that provides a means for rendering.
The techniques described in the present disclosure may, in some cases, be implemented in hardware, at least in part, using software or firmware aspects combined with hardware. When implemented in software or firmware, the software or firmware is one or more hardware processors such as microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or digital signal processors (DSPs). Can be performed in. Software that performs this technique may first be stored on a computer-readable medium, loaded into a processor, and executed.
Accordingly, the present disclosure is to generate a DMS on a host device when the video display system is run in the processor of the host device, with the DMS being the first content in the first area of the display window. , Defines the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content defines the second frame rate in the DMS. A computer-readable memory that defines, and causes the first frame rate to be different from the second frame rate, to generate and to communicate the DMS from the host device to the client device of the video display system. Plan the medium.
In another example, in the present disclosure, upon execution in the processor of a client device of a video display system, the client device receives a DMS from the host device of the video display system, where the DMS is the number of display windows. The first content in one area and the second content in the second area of the display window are defined, the first content defines the first frame rate in the DMS, and the second display. The content defines a second frame rate in the DMS, the first frame rate is different from the second frame rate, receiving and the first display content and the second display on the client device. A computer-readable storage medium with instructions for rendering content and doing so will be described.
Details of one or more examples of the present disclosure are given in the accompanying drawings and in the description below. Other features, objectives, and advantages of the present disclosure will become apparent from the description and drawings, as well as the claims.
<figref num="1">A block diagram illustrating an exemplary system that can be used to implement the techniques of the present disclosure.</figref><figref num="2">A conceptual diagram of a display screen that may contain two separate areas of display content that correspond to different video streams of Decomposed Multistream (DMS).</figref><figref num="3">An exemplary block diagram of a host device that matches an example of the present disclosure.</figref><figref num="4">An exemplary block diagram of a client device that matches an example of the present disclosure.</figref><figref num="5">A flow chart showing techniques that can be implemented by host devices consistent with this disclosure.</figref><figref num="6">A flow chart showing techniques that can be implemented by host devices consistent with this disclosure.</figref><figref num="7">A flow chart showing techniques that can be implemented by host devices consistent with this disclosure.</figref><figref num="8">A flow chart showing techniques that can be implemented by host devices consistent with this disclosure.</figref><figref num="9">A flow chart showing techniques that can be implemented by client devices consistent with this disclosure.</figref><figref num="10">A flow chart showing techniques that can be implemented by client devices consistent with this disclosure.</figref><figref num="11">A flow chart showing techniques that can be implemented by client devices consistent with this disclosure.</figref>
This disclosure describes techniques that can improve the generation of decomposed multistream (DMS) by the host device of the video display system and the display of the DMS by the client device of the video display system. DMS refers to a stream of data for a display that itself contains two or more different streams of content that may correspond to separate areas of the visible area, or possibly overlapping areas within the visible area. The host device may generate a DMS with two or more different streams of content, and the client device receives the DMS from the host device and produces a video containing two or more different video streams of content contained in the DMS. Can be rendered on the display screen. The host device may render different streams of DMS in different areas of the screen, and in some cases different areas of different DMS streams may overlap.
Some video display systems, in contrast to DMS streams, produce streams of video content called signal-composed streams (SCS). The SCS comprises one single video stream that can be considered as a screenshot in the host device. In essence, in the case of SCS, the host device sends the entire contents of its display buffer to the client device. These so-called "screenshots" can in some cases contain windows with different content, but the windows with different content in the SCS are not separate streams, but only part of the screenshot. If the host device displays two separate windows, one for full motion video and another for relatively static content, the SCS will generally be at the frame rate for full motion video. Send a screenshot. The SCS technique can be a very inefficient way of sending relatively static content along with full motion video.
The technique may apply different frame rates to different streams in the DMS, and the frame rate may depend on the content in the different streams. For example, a stream in a DMS may comprise a sequence of video information that can be rendered at a relatively high frame rate (such as 10 to 120 frames per second commonly used in video playback). However, another stream in the DMS can be associated with a display background, various graphic user interface control windows or elements, or a display window containing non-video content (such as email or documents). One or more second streams (eg, streams containing relatively static content) can be rendered at a frame rate much slower than the frame rate used for the sequence of full motion video information. In addition, if different streams are associated with overlapping areas within the visible area of the display screen, techniques can be applied to reduce the data in one or both streams in the overlapping areas. Many other techniques that can improve DMS are also described. As an example, the first frame rate can be 10 to 120 frames per second, and about 30 frames per second can be sufficient for many streaming video applications. In contrast, the second frame rate can be between 1 and 10 frames per second, and about 4 frames per second can be sufficient for applications that generally do not include full motion video.
The host device may include a computer device (such as a laptop computer or smartphone) and the client device may include a wireless display used to render the same output as the computer device. DMS can be used to communicate data over high speed wireless links so that both host devices (eg, computer devices) and client devices (eg, wireless displays) can display similar content. Given that the content itself can differ in different streams or overlap within the visible area of the display, the present disclosure provides techniques for improving DMS. To generate DMS on the host device, the host device can access not only the display buffer on the host device, but also the content from the application that produces the content. For example, the SCS technique can be implemented as a fallback technique for communicating data from a host device to a client device when DMS is not possible or desirable because the content from the application that produces the content is inaccessible. ..
FIG. 1 is a block diagram showing an exemplary video display system that can be used to implement the techniques of the present disclosure. The system of FIG. 1 includes a host device 10 and a client device 20. In an exemplary example, the host device 10 may include a laptop or smartphone, and the client device 20 may include a separate external display device used to render the same content that is rendered on the host device. obtain. The host device 10 and the client device 20 may communicate data via a short-range high-speed wireless protocol. However, this example is just an example. The host device 10 and the client device 20 may optionally include any two computer devices that communicate data via the DMS technique. The communication link between the host device 10 and the client device 20 is typically a short-range wireless link. However, in another example, the link between the host device 10 and the client device 20 may comply with any wired or wireless protocol and may include intermediate devices such as routers, switches or other networking devices.
The host device 10 may access different content (eg, content 1 12A, content 2 12B, and content N 12N). The content 12 may be accessible to the host device through the display buffer, but may be separately accessible from one or more specific applications that render the content. As an example, content 12A may include content from a video playback application and content 12B may include content from an email application or word processing application.
The host device 10 includes a DMS generator that produces a decomposed multistream (DMS), the DMS being the first content in the first area of the display window and the second in the second area of the display window. The second content defines the first content, the first content defines the first frame rate in the DMS, the second display content defines the second frame rate in the DMS, and the first frame rate is the first. It is different from the frame rate of 2. For example, if content 12A comprises content from a video playback application and content 12B comprises content from an email or word processing application, content 12A has a higher frame rate than the frame rate used for content 12B. Can be encoded in a first stream of DMS with.
FIG. 2 is a conceptual diagram of a display screen containing two separate areas of display content corresponding to different video streams of DMS. As shown in FIG. 2, the visible area 202 includes a first rendering area 204 and a second rendering area 206. Further, in this example, the first rendering area 204 and the second rendering area 206 both include an overlapping area 205. The DMS may include two separate streams for communicating the contents of the first render area 204 and the second render area 206. The DMS may also include additional streams, i.e. more than two streams.
The first rendering area 204 may include a full motion video sequence and may be encoded by the DMS at 30 frames per second or the like. The second rendering area 206 may include windows associated with a more static application such as an email application or a word processing application. The second rendering area 206 may, as an alternative or addition, include background data, graphic user interface elements (such as control over the content in the first rendering area 204).
Referring again to FIG. 1, the host device 10 includes a multimedia transport unit 16 that communicates DMS from the host device 10 to the client device 20. Again, the DMS contains two or more different streams associated with different rendering areas (such as the first rendering area 204 and the second rendering area 206). The client device 20 also includes a multimedia transport unit 22 for receiving DMS from the host device 10. When communicating the DMS from the host device 10 to the client device 20, the communication may send a stream at the frame rate defined when forming the DMS. That is, in the DMS, the first content is sent at the first frame rate, the second content is sent at the second frame rate, and the first frame rate is different from the second frame rate. It can be sent from the host device 10 to the client device 20.
The transport units 16 and 22 may include wireless units capable of wirelessly communicating with each other. Transport units 16 and 22 may be communicated over any wireless frequency used for wireless communication, short-range or long-range wireless standards, cellphone standards, wi-fi®, ultra-wideband communication, Any of a wide variety of wireless techniques or standards for such communications may be used, including white space communications and the like. When white space or licensed television TV bands are used for communication, the transport unit 16 may include sensing capabilities (or global positioning) to ensure that frequencies are available for use. Other techniques such as can be used).
In the client device, the DMS rendering unit 24 may be called to render different streams of DMS on the display unit 26. As described in more detail below, the DMS rendering unit 24 may utilize different buffering techniques and different latency rules for different streams of content in the DMS. For example, full-motion video may require more buffering to ensure that the video is displayed uninterrupted, but compared to streams related to other applications such as e-mail, full-motion video More latency or delay can be tolerated before rendering. Email applications or other applications may not require the level of buffering required for full motion video (because they use slower frame rates), but latency or latency in changes to the display screen. The delay is unacceptable. For these or other reasons, the DMS reader unit 24 may buffer the first content separately from the second content, for the display unit 26 to display the first content and to display the second content. Different latency rules may be applied for.
According to the present disclosure, the DMS generator 14 of the host device 10 dynamically adjusts the first frame rate associated with the first rendering area 204 in the DMS to the second rendering area 206 in the DMS. The associated second frame rate can be adjusted dynamically. These dynamic adjustments to one or both of the first frame rate and the second frame rate may be based on the first content and the second content. As described in more detail below, the DMS generator 14 intercepts the first content from the application, but may simply capture the second content from the display buffer in the host device 10. In doing so, the DMS generator 14 may also control (dynamically increase or decrease) the capture rate associated with capturing the second content from the display buffer. The DMS generator 14 may optionally include a encoder for encoding one or more of the streams in the DMS, in which case the DMS rendering unit 24 will include a mutual decoder. ..
As mentioned above, the first rendering area 204 and the second rendering area 206 may include an overlapping area 205 (also referred to as an overlapping area). To enable DMS data cleanup and simplification of decryption, the DMS generator 14 identifies, in some examples, whether the first content or the second content is above in the overlapping area. Information to be generated (for example, syntax information) can be generated in the DMS. This syntax information can allow the DMS rendering unit 24 to see whether the first content or the second content is above in the overlapping area 205 z-ordering information (z-ordering). Information) or coordinates can be provided. In this case, the DMS 14 may reduce or eliminate the data of either the first content or the second content in the overlapping area 205, and the reduced or eliminated data may be in the overlapping area. Below the data.
In some cases, the DMS generator 14 may, for example, directly from the display buffer the SCS containing the first content and the second content if it is impossible or impractical to intercept the content from a particular application. May include the ability to generate. In this case, the DMS generator may determine that it is not possible to intercept the first or second content from the application and is unable to intercept the first or second content from the application. In response to determining that it is possible, the host device 10 may generate an SCS containing the first content and the second content directly from the display buffer. User input may also be used to define the DMS. For example, the DMS generator 14 may be configured to adjust the DMS to include only one of the first content or the second content in response to user input.
In addition, the DMS generator 14 can determine the bandwidth available between the host device 10 and the client device 20. Available bandwidth can define the data transfer rate available at a given time. The DMS generator 14 may adjust one or both of the first frame rate and the second frame rate based on the available bandwidth in some examples. As an addition or alternative, the DMS generator 14 may dynamically encode one or both of the first content and the second content based on the available bandwidth. The DMS generator 14 can also dynamically encode one or both of the first content and the second content based on the type of content being communicated. In many cases, it may be desirable to encode the content captured from the display buffer, as described herein. The content intercepted by the application may already be encoded. In some cases, the content intercepted by the application can be transcoded (ie, decoded and then recoded in a different coding format).
In general, the phrase "signal-compressed stream" (SCS) is used herein end-to-end with the entire display buffer in either compressed or uncompressed form (eg, from host device 10 to client device 20). Used to refer to the technique being transmitted. In the case of SCS, partial display updates can also be transmitted end-to-end in compressed or uncompressed form. In contrast, the phrase "decomposed multi-stream" (DMS) refers to the rest of the display content from the display buffer, as well as the precompressed content (eg, video) directly from the application (eg, media player application). Refers to a technique that passes both. Currently, several standardized specifications are under development for the DMS technique, including "VESA Net2 Display" and "USB-IF Display". Specifications that support the SCS method may include VESA Net2 Display, USB-IF Display, and other commercial VNC systems.
The DMS method can define many advantages, such as preserving the quality of pre-compressed content, as well as saving resources and power on the host platform associated with the host device 10. However, DMS has some drawbacks. These drawbacks may include that the DMS may impose additional requirements on the client device 20 that may increase the cost and complexity of the client device 20. Such additional requirements for client device 20 to support DMS include the ability to decode two or more video streams at the same time and the ability to configure and render the decoded output of two or more streams. May include. The ability to compose and render the decoded output of two or more streams may require the ability to perform chroma key and alpha blending as part of the structure of multiple display surfaces. In addition, chroma key and alpha blending requirements can be important when other content is dynamically overlaid over the area where the full motion video is being rendered. Examples of such dynamic content are drop-down menus generated by user actions, dialog boxes and alerts generated by the operating system on the host device 10. Sometimes, for example, these or other items may be rendered by the host device 10 and the client device 20 directly above the area where the movie is being rendered. The techniques of the present disclosure simplify the DMS in these cases and require chroma key and / or alpha blending simply by using syntax information to identify which rendering area is above in the overlapping area. Can be avoided.
Many host device implementations have a high level application program interface (API) (eg, a graphical driver interface) to periodically capture the background surface, even if it may be unnecessary capture and encoding of the background display surface. It can rely on GDI)). For example, it is partially (or completely) hidden by overlay full-motion video to reduce the resource usage of the host device and the amount of data transmitted between the host device 10 and the client device 20. The area can be removed from the DMS.
Also, for wireless touch screen implementations, capturing touch events on the client device 20 may be desirable across the display screen across the rendering location of all received streams in the DMS. In addition, the DMS may not work when the client device 20 does not support a particular media type of pre-compressed content. If the client device 20 does not support that media type due to many aspects of the media type, such as resolution, the particular type of encoding used, the color space used, the orientation, the aspect ratio, or other factors. DMS may not work.
The DMS may also not work when the host device 10 is unable to intercept pre-compressed content, for example from a media player application. This can happen if the media player is implemented in a monolithic form and does not use a well-known media framework that aids in interception. Also, DMS does not work when the combined throughput requirements (for separate streams) exceed the transmit link capacity (ie, available bandwidth) available between the host device 10 and the client device 20. Sometimes. Channel conditions can occur dynamically at the beginning of a wireless display session or at any point during the course of a wireless display session. The techniques of the present disclosure may provide solutions to these issues and problems of the DMS described above.
Systems that use only the SCS technique may be forced to treat the entire display area as a single unit. Therefore, in this case, the client device 20 can handle display areas that are uniformly different in latency by implementing a common jitter buffer for reproduction. This minimizes latency (and thus minimizes jitter buffering) for more static application productivity and responsiveness, and is full (by increasing jitter buffering). It may present a trade-off between improving the smoothness of motion video playback applications. Also, systems that use only SCS techniques may be forced to apply uniform processing to the entire display area when trading off quality parameters, such as compressed data transmission vs. uncompressed data transmission. is there. In this case, dynamic adaptation can be constrained and may need to be applied uniformly over the entire display area.
Consistent with the present disclosure, various requirements of the DMS client (ie, client device 20) can be relaxed, for example, for low cost client implementations. The ability to decode two or more video streams at the same time may still be required in the client device 20, but the background plane encoded frame rate can be significantly reduced to minimize the total decoded frame rate. Reducing the frame rate of one of the streams in the DMS will be described in more detail below.
The ability to configure and render the decoded output of two or more streams can be simplified in the client device 20 only by requiring a relatively simple z-ordering on the content in the overlapping rendering areas. This avoids blending operations and chroma key applications on the client device 20. This scheme can be supplemented to handle content that needs to be displayed above the area where the full motion video is being rendered, such as: In one example, the host device 10 may detect the presence (or subsequent removal) of content to be displayed on the full motion video overlay surface. This detection is performed by the host device 10 in several ways, for example, through the interception of a display driver update that gives the rectangular coordinates of the update, or in a strategic location with knowledge of the coordinates of the full motion video overlay plane. It can be done via scanning the full Graphic Driver Interface (GDI) capture buffer. In addition, the host device 10 may signal the client device 20 via a control message (or other syntax element) to notify the client device of such z-ordering. Upon receiving the control message from the host device 10, the client device 20 may control (possibly swap) the z-order of the two separate streams. This can temporarily hide full-motion video content while there is a user interface alert to be displayed, which can be an acceptable limitation for low-cost, relatively simple client device implementations.
For host device implementations that rely on high-level APIs (eg GDI) to capture the background surface periodically, the host platform overhead for capturing and encoding the background display surface is minimized in several different ways. It can be suppressed. In one example, the GDI capture rate can be significantly reduced (eg to 4 Hz) while there is an active full motion video overlay surface. The effect of this reduced capture rate can reduce the frame rate, but because the surrounding display area is usually static (or generally changes at a much lower rate compared to the full motion video overlay surface). , Can be imperceptible in most situations. In addition, such benefits of reduced frame rates (and reduced capture rates) are due to the full motion video overlay surface area (as is the case when hardware accelerated decryption is done on the host device). Can be even more pronounced when masked by GDI capture. Masked areas within the overlapping areas can be filled with null data or "black" during GDI capture, which can improve coding efficiency at reduced capture rates. Since no motion is generated in the "black" area where the full motion video is being rendered, "blacking" of the masked overlapping areas can be synergistic with lower GDI capture rates, and in some cases , Any encoder bit budget can be depleted in the surrounding area to improve the video quality of a given bit budget.
In an extreme case, if the host device 10 detects that the full motion video overlay surface (which can occur when the media player application is in "full screen" mode) occupies the entire display area, the GDI capture rate will be It can be degenerated to 0 hertz. This reduction in GDI capture rate to 0 hertz is also optional when the host device media player application is not in full screen mode, for example, if the user chooses not to visualize the entire user interface on the client device's wireless display. Can be simulated. This may be a good choice if the client device's wireless display is a television (TV) screen where only the full motion overlay surface is desired to be visible on the client device.
For wireless touch screen implementations, capture of touch events on the client device 20 can be achieved across the display screen across the rendering location of all received DMS streams. The client device 20 may render an additional virtual window surface that matches the entire display area. This virtual window surface can be a window that is transparent to the user so as not to hide any of the actual display streams that are received and rendered. Touch screen captures (using standard computing system interfaces) can be performed on this virtual surface on the client device 20 and can be scaled appropriately to the host device display screen size.
When implementing the techniques described in the present disclosure, the host device 10 may fall back to SCS mode when the client device 20 does not support the media type of precompressed content. Aspects of the media type may include resolution, coding used, color space, orientation, aspect ratio, or other factors. Also, when implementing the techniques described in the present disclosure, when host device 10 is unable to intercept precompressed content from an application (such as a media player application) used to render the content. Can fall back to SCS mode. This can happen, for example, if the application (such as a media player application) does not use a media framework known to the host device 10, or in some cases the media does not help intercept by the host device 10. ..
Host device 10 detects that the combined throughput requirements (for all DMS streaming) exceed the transmit link capacity (eg, available bandwidth) available between host device 10 and client device 20. At that time, the host device 10 can perform various operations. In one such example, the host device 10 is a background surface by adapting one or more parameters such as quantization parameters, frame rate, bit rate, or other parameters used in coding. It can reduce the throughput budget for. If this is still insufficient to fit both streams within the transmit link capacity available between the host device 10 and the client device 20, the client device 20 will have quantization parameters, frame rates, bit rates or other. Can be returned to SCS mode by properly degrading the elements of. Insufficient channel capacity conditions can occur dynamically at the beginning of a wireless display session or during the course of a wireless display session.
The client device 20 may handle each of the streams received differently for latency. For example, the client device 20 may implement a smaller jitter buffer for the background stream compared to the jitter buffer used for full motion video. Client devices may also apply different latency rules to prevent latency from being introduced into the background stream. Latency can be more acceptable for full-motion video, and it may initially be desirable to help avoid interruptions during video playback later in the video sequence. In contrast to the smaller jitter buffer for the background surface stream, the client device 20 may implement a relatively large jitter buffer for the full motion video overlay surface to improve the smoothness of the video playback application. ..
A particular delay implementation for the DMS can also be controlled by the host device 10, for example, via a presentation timestamp added to each stream in the DMS, which can be uniquely compensated by the host device 10. .. The host device 10 also transmits in compressed mode and uncompressed mode for the background surface, depending on the DMS state (ie, whether there is currently an active full motion video overlay surface) and the availability of transmit link capacity. You can choose between. For example, the host device 10 may choose a lower frame rate uncompressed mode whenever there is an active full motion video overlay plane. In this case, the uncompressed mode transmission of the background surface can be either a full screen shot or a partial update that modifies the previous screenshot already received by the client device 20. In either case, the screenshot or update can be lightly compressed using, for example, chroma-sub-sampling, run-length coding or other techniques.
FIG. 3 is a block diagram of a host device 300 that can correspond to the host device 10 of FIG. In this example, the host device 300 includes a host application unit 302 that may include a host operating system running on a processor. The host device 300 also includes a host display system 304 that may include a display and software drivers for display on the host device 300. The display buffer 306 can also form part of the host display system 304, but is shown separately as it can be accessed by the DMS generator 308 as part of the DMS generation. The DMS generator 308 may correspond to the DMS generator 12 of FIG. The host device 300 also includes a multimedia transport unit 310 that may correspond to the multimedia transport unit 14 of FIG. The transport interface 316 may include one or more antennas on the host device 300, as well as an air interface used for wireless communication. The host device 300 may include a media player unit 312, and the DMS generator 308 may be able to obtain content directly from the media player unit 312 (via interrupts or the like) for inclusion in the DMS. The host device 300 may optionally include an encoder 314 for encoding one or more of the streams contained in the DMS. The encoder 314 is ITU-H. 263, ITU-H. 264, ITU-H. It may include any video encoder, such as an encoder that conforms to 265, or any other public or proprietary video coding standard or technique. In many cases, it may be desirable to call the encoder 314 to encode the content captured from the display buffer 306. The content intercepted from the media player unit 312 must already be encoded. is there. In some cases, the content intercepted from the media player unit 312 can also be transcoded by the encoder 314 (ie, decoded and then re-encoded in a different coding format).
The DMS generator 308 generates a DMS that defines a first content in the first area of the display window and a second content in the second area of the display window, and the first content is in the DMS. The first frame rate is defined, the second display content defines the second frame rate in the DMS, and the first frame rate is different from the second frame rate. The first content associated with the first frame rate may include video information received directly from the media player unit 312. In this case, the DMS generator 308 may issue an interrupt to the media player unit 312 to instruct the media player unit 312 to feed its video stream directly to the DMS generator. The media player unit 312 may also provide the host display system 304 with its video stream for display by the display device associated with the host device 300.
In addition to this first stream, which is received directly from the media player unit 312, the DMS generator 308 may also receive a second stream, which is essentially SCS, from the display buffer 306. The DMS generator 308 may generate a DMS containing both a first stream and a second stream, and may generate different streams of DMS with different frame rates. For example, a first stream received directly from the media player unit 312 may define a relatively high frame rate (such as 10-30 frames per second) common to video playback. The second stream received from the display buffer 306 may define a relatively slow frame rate (such as 1-10 frames per second). If the first content area and the second content area overlap, the DMS generator 308 uses the techniques of the present disclosure to avoid sending data for both streams in the overlapping area. Can be done. Alternatively, the DMS generator 308 may generate DMS information to identify which stream is on top and may remove data from the underlying stream (especially in overlapping areas). In this case, generating the DMS may include reducing or eliminating the data of either the first content or the second content in the overlapping area, the data being reduced or eliminated. Below other data in the overlapping area.
In some cases, the DMS generator 308 may dynamically reduce or increase its capture rate associated with capturing the second content from the display buffer 306, these adjustments. Can be based on the first content and / or the second content. Encoder 314 may be called to encode the first content of the DMS, the second content of the DMS, or both the first and second content. The encoder 314 is ITU-H. 263, ITU-H. 264, ITU-H. Data may be encoded according to 265, or other public or proprietary video coding standards or techniques.
In some cases, the DMS generator 308 may determine that it is not possible to intercept the first content from the media player unit 312, in which case it intercepts the first content or the second content from the application. In response to determining that it is not possible, the SCS containing the first content and the second content may be generated directly from the display buffer 306. In this case, the SCS can be considered a fallback to the DMS if the content is not available to the application and is only available from the display buffer 306.
In a different example, the first content comprises a video sequence and the second content comprises the output of a non-video application. In some cases, the second content may include graphical user interface elements that form a shell around the video sequence of the first content. In other cases, the second content may include a graphical user interface control element, or a separate graphical user interface window that may overlap the first content. In yet other cases, the second content may include separate windows associated with different applications such as email applications or word processing applications. If desired, the DMS can be adjusted to include only one of the first or second content in response to user input.
In yet another example, the DMS generator 308 may consider other factors (such as available bandwidth) in generating the DMS. For example, the DMS generator 308 may determine the bandwidth available between the host device 300 and the client device, and based on the available bandwidth, either the first frame rate or the second frame rate. Or both can be adjusted. If the encoder 314 is called by the DMS generator 308 as part of the DMS generation process to create coded content in the DMS, the DMS generator 308 also adjusts the coding parameters as part of the DMS generation process. Can be. As an example, the DMS generator 308 may determine the bandwidth available between the host device 300 and the client device, and the encoder 314 has a first content and a second content based on the available bandwidth. One or both of the contents can be dynamically encoded.
The DMS can be transferred to the multimedia transport unit 310 of the host device 300 for communication to a client device (not shown) via the transport interface 316. The multimedia transport unit 310 and transport interface 316 provide a wide variety of wireless for such communications, including short-range or long-range wireless standards, cell phone standards, wi-fi, ultra-wideband communication, white space communication, and more. Either technique or standard can be used. When white space or licensed television TV bands are used for communication, the multimedia transport unit 310 may include sensing capabilities to ensure that frequencies are available on the transport interface 316. (Or other techniques such as global positioning may be used).
FIG. 4 is a block diagram of a client device 400 that can correspond to the client device 20 of FIG. In this example, the client device 400 includes a client application unit 402 that may include a client operating system running on a processor. The client device 400 also includes a client display system 404 that may include a display and software drivers for display on the client device 400. The DMS rendering unit 406 may correspond to the DMS rendering unit 24 of FIG. The client device 400 also includes a multimedia transport unit 408 that may correspond to the multimedia transport unit 22 of FIG. The transport interface 412 may include one or more antennas on the client device 400, as well as an air interface used for wireless communication from a host device (not shown in FIG. 4). The client device 400 may optionally include a decoder 410 for decoding one or more of the streams contained in the DMS. The decoder 410 is ITU-H. 263, ITU-H. 264, ITU-H. It may include a decoder that conforms to 265, or other public or proprietary video coding standards or techniques.
The client device 400 receives a DMS stream from the host device via the transport interface 412 and the multimedia transport unit 408. As described in the present disclosure, the DMS defines a first content in the first area of the display window and a second content in the second area of the display window, the first content being the DMS. The first frame rate in is defined, the second display content defines the second frame rate in the DMS, and the first frame rate is different from the second frame rate. The DMS rendering unit 406 renders the first display content and the second display content on the client display system 404 of the client device 400. Further, in one example, the first content comprises a video sequence and the second content comprises the output of a non-video application. In another example, the first content comprises a video sequence and the second content comprises a graphical user interface element. In these examples, the first frame rate can be higher than the second frame rate.
In some cases, it may be desirable for the DMS rendering unit 406 and / or the client display system 404 to implement different types of buffering and latency rules for different content in the DMS. For example, the client display system 404 may include a display buffer and buffer the first content separately from the second content. Further, the client display system 404 may apply different latency rules for displaying the first content and for displaying the second content. For example, full-motion video may require more buffering to ensure that the video is displayed uninterrupted, but compared to streams related to other applications such as e-mail, full-motion video More latency or delay can be tolerated before rendering. Email applications or other applications may not require the level of buffering required for full motion video (because they use slower frame rates), but latency or latency in changes to the display screen. The delay is unacceptable. For these or other reasons, the DMS reader unit 406 and / or the client display system 404 may buffer the first content separately from the second content so that the client display system 404 can display the first content. Different latency rules may be applied for the purpose and for the display of the second content.
FIG. 5 is a flow chart showing a technique for generating DMS. Similar techniques may be applied by other devices, but FIG. 5 will be described from the perspective of the host device 300. As shown in FIG. 5, the DMS generator 308 of the host device 300 accesses the first content from the media player unit 312 (501) and the second content from the display buffer 306 (502). The DMS generator 308 generates a DMS containing the first content and the second content at different frame rates (503) and sends the DMS to the client device via the multimedia transport unit 310 and the transport interface 316 and the like. Communicate (504). In some cases, additional content (ie, additional streams) may also be included in the DMS.
FIG. 6 is another flow chart showing a technique for generating DMS. Similar techniques may be applied by other devices, but FIG. 6 will also be described from the perspective of the host device 300. As shown in FIG. 6, the DMS generator 308 of the host device 300 accesses the first content from the media player unit 312 (601) and the second content from the display buffer 306 (602). The DMS generator 308 generates a DMS containing the first content and the second content at different frame rates (603). Further, the DMS generator 308 generates DMS information which may include syntax information (eg, metadata) instructing the client device how to reconstruct the first content and the second content. (604). For example, when the first content and the second content overlap, the DMS information may include z-coordinate information of the overlapping area. In this case, the streams in the DMS, which are below the other streams in the overlapping area, are coded as null data or black data to reduce the amount of data in that area where something is overlaid by the other streams. Can be converted. The host device 300 then communicates DMS and DMS information to the client device via the multimedia transport unit 310, the transport interface 316, and the like (605).
FIG. 7 is another flow chart showing a technique for generating DMS. Similar techniques may be applied by other devices, but FIG. 7 will also be described from the perspective of the host device 300. As shown in FIG. 7, the DMS generator 308 of the host device 300 accesses the first content from the media player unit 312 (701) and the second content from the display buffer 306 (702). The DMS generator 308 dynamically defines the frame rate for the first content and the second content based on the content itself (703). For example, the DMS generator 308 may allocate a higher frame rate for video information and a slower frame rate for a stream that is more static. In some cases, the frame rate can be determined by the location of access. For example, the video information from the media player unit 312 may be assigned a higher frame rate than the data from the display buffer 306. Data from other applications, such as from an email application or word processing application, may also be assigned a slower frame rate than video data from the media player unit 312 or another source. In each case, the DMS generator 308 generates a DMS containing the first content and the second content at different dynamically defined frame rates (704). Different rates can be dynamic in the sense that the rates can change as the content changes. Such changes may be based on the source of the content or the actual type of content in any given instance.
FIG. 8 is a flow chart showing techniques for generating DMS or SCS in different scenarios. Similar techniques may be applied by other devices, but FIG. 8 will also be described from the perspective of the host device 300. As shown in FIG. 8, the DMS generator 308 of the host device 300 determines whether the video content is available to the application associated with the media player unit 312 (801). If not available ("No" 801), the DMS generator may default to the SCS technique. In this case ("No" 801), the DMS generator accesses the content from the display buffer 306 (806) and generates an SCS containing the entire content of the display buffer 306 at one frame rate (807). The host device 300 then communicates the SCS with the client device via the multimedia transport unit 310, the transport interface 316, and the like (808).
If the DMS generator 308 of the host device 300 determines that the video content is available from the application associated with the media player unit 312 ("Yes" 801), the DMS generator may implement the DMS technique. In this case ("Yes" 801), the DMS generator 308 accesses the video content from the application associated with the media player unit 312 (802) and the non-video content from the display buffer 306 separately (803). .. The DMS generator 308 generates a DMS containing different contents at different frame rates (804) and communicates the DMS to the client device via the multimedia transport unit 310, the transport interface 316, and the like (805).
FIG. 9 is a flow chart showing a technique for processing DMS in a client device. Similar techniques may be applied by other devices, but FIG. 9 will be described from the perspective of the client device 400. As shown in FIG. 9, the DMS rendering unit 406 receives the DMS (901). For example, the DMS can be communicated from the host device and received on the client device 400 via the transport interface 412 and the multimedia transport unit 408. The multimedia transport unit 408 transfers the DMS to the DMS rendering unit 406. The DMS rendering unit 406 generates the first content based on the DMS (902) and the second content based on the DMS (903). The DMS rendering unit 406 causes the client display system 404 to display the first content and the second content at different frame rates that can be defined by the DMS (904). This display of two or more streams may optionally include any of the other techniques described herein, such as techniques for handling overlays of different streams in overlapping areas.
FIG. 10 is a flow chart showing a technique for processing DMS in a client device. Similar techniques may be applied by other devices, but FIG. 10 will also be described from the perspective of the client device 400. As shown in FIG. 10, the DMS rendering unit 406 receives DMS (1001) and also receives DMS information that may include metadata or other side or syntax information associated with DMS (1002).
The DMS and DMS information can be communicated from the host device and received on the client device 400 via the transport interface 412 and the multimedia transport unit 408. The multimedia transport unit 408 transfers DMS and DMS information to the DMS rendering unit 406. The DMS rendering unit 406 generates a first content based on the DMS (1003) and a second content based on the DMS (1004). The DMS rendering unit 406 causes the client display system 404 to display the first content and the second content based on the DMS information at different frame rates (1005). Different frame rates can be defined by the DMS itself. The DMS information may include information that defines the dialogue between the first stream and the second stream. For example, the DMS information may include Z-coordinate (ie, depth) information that defines whether the first or second content in the DMS stream is overlaid on the other content. The Z-coordinate (ie, depth) information provides the relative depth of different streams to define which content is on top and which content is underneath for any overlapping area. Can be defined. Content that is below other content in the overlapping area can be nulled or blackened in the DMS stream to improve throughput. Since the data is under other data, it is blocked from view anyway, so nullifying or blackening such data in the DMS stream will not be observed by the user on the client device 400. ..
FIG. 11 is a flow chart showing a technique for processing DMS in a client device. Similar techniques may be applied by other devices, but FIG. 11 will also be described from the perspective of the client device 400. As shown in FIG. 11, the DMS rendering unit 406 receives the DMS including the first content and the second content (1101). For example, the DMS can be communicated from the host device and received on the client device 400 via the transport interface 412 and the multimedia transport unit 408. The multimedia transport unit 408 transfers the DMS to the DMS rendering unit 406. The DMS rendering unit 406 causes the client display system 404 to buffer the first content from the DMS (1102) and the second content from the DMS (1103). The buffer used by the client display system 404 may depend on the content. For example, a larger buffer may be used for real-time video data, but for other types of data (possibly) such as graphical data or data associated with other types of applications such as e-mail applications or word processing applications. Defines a smaller latency than a larger buffer) Smaller buffers may be used. Therefore, the client display system 404 displays the first content and the second content at different frame rates according to buffering and latency rules applied to the different content (1104). These buffering and latency rules allow longer latency and may require more buffering to ensure the smoothness of the output video compared to graphical elements or data from other types of applications. Rendering full motion video information Can be different for such different types of content to improve. For example, a stream of graphical elements or data from other types of applications may require shorter latency (ie, more responsiveness), but a wider level of buffering, which is desirable for full motion video sequences. May not be.
In yet another example, the frame rate associated with the background data (eg, the data in the second stream) can be clearly reduced when the real-time video is included in the first stream of the DMS. This technique can improve the use of limited resources in client devices and can improve video rendering when such resources are limited. To implement such a technique, the GDI capture rate associated with capturing the data in the display buffer for the second stream of DMS on the host device is such that the real-time video is included in the first stream of DMS. Sometimes it can be reduced.
In yet another example, the data may be included in the DMS as yet another stream (eg, a third stream) to provide a touch screen overlay over the first and second contents in the DMS stream. .. In this case, the touch screen overlay may include a transparent window for giving touch screen captures or feedback on the client device. Other techniques may also be used to enable touch screen captures in the host and client devices of the display system.
Note that the above description focuses on DMS containing two streams. However, the techniques of the present disclosure can be extended to DMS containing additional streams, i.e. more than two streams. The DMS may include a first stream, a second stream, a third stream, a fourth stream, and the like. The frame rate of some or all of the different streams in the DSM can be defined in a dynamic way as described herein. Various other techniques may also be used to address the problem of overlapping or other features for three or more streams.
The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the techniques described may include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated circuits. It can be implemented within one or more processors, including individual logic circuits, as well as any combination of such components. The term "processor" or "processing circuit" generally refers to any of the above logic circuits, either alone or in combination with other logic circuits or other equivalent circuits such as individual hardware that performs processing. Sometimes.
Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various behaviors and features described in this disclosure. In addition, any of the units, modules, or components described may be implemented together or separately as separate but interoperable logical devices. Illustrations of various features as modules or units emphasize different functional aspects, implying that such modules or units must necessarily be implemented by separate hardware or software components. do not. Instead, functions related to one or more modules or units are performed by separate hardware, firmware, or software components, or are incorporated within common or separate hardware or software components. Sometimes.
Also, the techniques described herein can be stored, practiced or encoded in a computer-readable medium, such as a computer-readable storage medium that stores instructions. An instruction embedded or encoded in a computer-readable medium may, for example, perform the techniques described herein on one or more processors when the instruction is executed by one or more processors. I can let you. Computer-readable storage media include random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EPROM), It may include flash memory, hard disk, CD-ROM, floppy® disc, cassette, magnetic medium, optical medium, or other tangible computer readable storage medium.
The computer-readable medium may include a computer-readable storage medium corresponding to a tangible storage medium such as the tangible storage medium described above. A computer-readable medium may also include, for example, a communication medium, including any medium that allows the transfer of a computer program from one location to another according to a communication protocol. In this way, the phrase "computer-readable medium" generally corresponds to (1) a tangible computer-readable storage medium that is non-transitory, and (2) a non-tangible computer-readable communication medium such as a temporary signal or carrier. Can be.
Various aspects and examples have been described. However, changes may be made to the structures or techniques of the present disclosure without departing from the claims below. The inventions described in the original claims of the present invention are described below. [C1] To generate a decomposed multistream (DMS) via a host device of a video display system, the DMS is the first content in the first area of the display window and the display window. The first content defines the first frame rate in the DMS, and the second display content defines the second frame in the DMS. A method of defining a rate, generating the first frame rate different from the second frame rate, and communicating the DMS from the host device to a client device of the video display system. .. [C2] The method according to C1, further comprising dynamically adjusting the first frame rate and the second frame rate in the DMS based on the first content and the second content. [C3] The method of C1, wherein generating the DMS comprises intercepting the first content from an application and capturing the second content from a display buffer in the host device. [C4] C3 further comprises dynamically reducing the capture rate associated with capturing the second content from the display buffer based on the first content and the second content. The method described. [C5] The method of C1, wherein producing the DMS comprises encoding the second content. [C6] The first area and the second area each include an overlapping area, and the method identifies whether the first content or the second content is on the overlapping area. The method of C1, further comprising generating information in the DMS. [C7] Generating the DMS comprises reducing or eliminating the data of either the first content or the second content in the overlapping area, the data being reduced or eliminated. The method of C6, which is below the other data in the overlapping area. [C8] The method of C1, wherein generating the DMS comprises intercepting the first content or the second content from an application. [C9] Determining that it is impossible to intercept the first content or the second content from the application, and intercepting the first content or the second content from the application. Further generating a signal-composed stream (SCS) containing the first content and the second content directly from the display buffer in the host device in response to determining that is not possible. The method according to C8. [C10] The method of C1, wherein the first content comprises a video sequence and the second content comprises the output of a non-video application. [C11] The method according to C10, wherein the first frame rate is larger than the second frame rate. [C12] The method of C1, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element. [C13] The method of C1, further comprising adjusting the DMS to include only one of the first content or the second content in response to user input. [C14] Determining the available bandwidth between the host device and the client device, and one or one of the first frame rate and the second frame rate based on the available bandwidth. The method according to C1, further comprising adjusting both. [C15] Determining the available bandwidth between the host device and the client device, and The method of C1, further comprising dynamically encoding one or both of the first content and the second content based on the available bandwidth. [C16] The method according to C1, further comprising dynamically encoding one or both of the first content and the second content. [C17] In a client device of a video display system, receiving a decomposed multistream (DMS) from a host device of the video display system, wherein the DMS is the first in a first area of a display window. The content and the second content in the second area of the display window are defined, the first content defines the first frame rate in the DMS, and the second display content is said. A second frame rate in the DMS is defined, the first frame rate is different from the second frame rate, receiving, and the first display content and the second on the client device. A method that comprises rendering display content. [C18] The method of C17, wherein the first content comprises a video sequence and the second content comprises the output of a non-video application. [C19] The method according to C18, wherein the first frame rate is higher than the second frame rate. [C20] The method of C17, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element. [C21] The method according to C17, further comprising buffering the first content in the client device separately from the second content. [C22] The method according to C21, further comprising applying different latency rules for the display of the first content by the client device and for the display of the second content by the client device. [C23] A host device for a video display system. A DMS generator that generates a decomposed multistream (DMS), wherein the DMS is a first content in a first area of a display window and a second in a second area of the display window. The content is defined, the first content defines the first frame rate in the DMS, the second display content defines the second frame rate in the DMS, and the first frame. A host device comprising a DMS generator whose rate is different from the second frame rate and a transport unit that communicates the DMS from the host device to a client device of the video display system. [C24] The DMS generator dynamically adjusts the first frame rate and the second frame rate in the DMS based on the first content and the second content, C23. The host device described in. [C25] The host device according to C23, wherein the DMS generator intercepts the first content from an application and captures the second content from a display buffer in the host device. [C26] The DMS generator dynamically reduces the capture rate associated with capturing the second content from the display buffer based on the first content and the second content. The host device according to C25. [C27] The host device according to C25, wherein the DMS generator uses the encoder to encode the second content in the DMS. [C28] The first area and the second area each include an overlapping area, and the DMS generator determines whether the first content or the second content is on the overlapping area. The host device according to C23, which generates the identifying information in the DMS. [C29] The DMS generator reduces or eliminates data for either the first content or the second content in the overlapping area, and the reduced or eliminated data is in the overlapping area. Host device according to C28, under other data. [C30] The host device according to C23, wherein the DMS generator intercepts the first content or the second content from an application. [C31] The DMS generator determines that it is impossible to intercept the first content or the second content from the application, and the first content or the second content from the application. In response to determining that it is not possible to intercept the content of, a signal-composed stream (SCS) containing the first content and the second content is delivered directly from the display buffer in the host device. The host device according to C30, which is to generate and perform. [C32] 23. The host device according to C23, wherein the first content comprises a video sequence and the second content comprises the output of a non-video application. [C33] The host device according to C32, wherein the first frame rate is higher than the second frame rate. [C34] The host device according to C23, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element. [C35] The host device according to C23, wherein the DMS generator adjusts the DMS to include only one of the first content or the second content in response to user input. [C36] The DMS generator determines the available bandwidth between the host device and the client device, and the first frame rate and the second frame rate based on the available bandwidth. The host device according to C23, which adjusts one or both of the frame rates of. [C37] The DMS generator determines the available bandwidth between the host device and the client device. The host device according to C23, which dynamically encodes one or both of the first content and the second content based on the available bandwidth. [C38] The host device according to C23, wherein the DMS generator dynamically encodes one or both of the first content and the second content. [C39] A client device of a video display system, a transport unit that receives a decomposed multistream (DMS) from a host device, wherein the DMS is the first in a first area of a display window. The content defines the content and the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display content defines the DMS. A transport unit, a display unit, and the like, which define a second frame rate in which the first frame rate is different from the second frame rate. A client device comprising a DMS rendering unit that renders the first display content and the second display content on the display unit. [C40] The client device according to C39, wherein the first content comprises a video sequence and the second content comprises the output of a non-video application. [C41] The client device according to C40, wherein the first frame rate is larger than the second frame rate. [C42] The client device according to C39, wherein the first content comprises a video sequence and the second content comprises a graphical user interface element. [C43] The client device according to C39, wherein the display unit includes a buffer configured to buffer the first content separately from the second content. [C44] 39. C39, wherein the rendering unit is configured to apply different latency rules for the display unit to display the first content and for the display unit to display the second content. Client device. [C45] When the video display system is executed in the processor of the host device, the DMS is to generate a decomposed multistream (DMS) in the host device, and the DMS is in the first area of the display window. Defines the first content of the display window and the second content in the second area of the display window, the first content defines the first frame rate in the DMS, and the second display. The content defines a second frame rate in the DMS, the first frame rate is different from the second frame rate, the generation and the DMS from the host device to the client device of the video display system. A computer-readable storage medium that comprises instructions for communicating and performing. [C46] Upon execution in the processor of the client device of the video display system, the client device receives a decomposed multistream (DMS) from the host device of the video display system, wherein the DMS is a display window. The first content in the first area and the second content in the second area of the display window are defined, and the first content defines the first frame rate in the DMS. Receiving that the second display content defines a second frame rate in the DMS and the first frame rate is different from the second frame rate. said on the client device. A computer-readable storage medium comprising instructions for rendering a first display content and the second display content. [C47] A host device for a video display system. A means for generating a decomposed multistream (DMS), wherein the DMS is a first content in a first area of a display window and a second in a second area of the display window. The content is defined, the first content defines the first frame rate in the DMS, the second display content defines the second frame rate in the DMS, and the first frame. A host device comprising means for generating, the rate of which is different from the second frame rate, and means for communicating the DMS from the host device to a client device of the video display system. [C48] A client device of a video display system. A means for receiving a decomposed multistream (DMS) from a host device of the video display system, wherein the DMS is a first content in a first area of the display window and a second of the display window. The first content defines the first frame rate in the DMS, and the second display content defines the second frame rate in the DMS. And render the first display content and the second display content on the client device as a means for receiving, the first frame rate being different from the second frame rate. A client device that has the means to do so.
12 members in 7 offices
Priority claims9
| Document | Office | Kind | Date |
|---|---|---|---|
| 28628709 | United States of America | P | |
| 96649510 | United States of America | A | |
| 2010060249 | United States of America | W | |
| 12966495 | – | – | – |
| 61286287 | – | – | – |
| US20090286287P | – | – | – |
| US2010060249 | – | – | – |
| US20100966495 | – | – | – |
| WO2010US60249 | – | – | – |
Members12
| Document | Office | Kind | |
|---|---|---|---|
| US2011145879A1 | United States of America | A1 | |
| WO2011075468A1 | World Intellectual Property Organization (WIPO) | A1 | |
| TW201142701A | Taiwan Province of China | A | |
| CN102656551A | China | A | |
| KR20120099773A | Republic of Korea | A | |
| EP2513773A1 | European Patent Office (EPO) | A1 | |
| JP2013514044A | Japan | A | |
| JP2014179979A | Japan | A | |
| JP5632015B2This record | Japan | B2 | |
| KR101523133B1 | Republic of Korea | B1 | |
| CN102656551B | China | B | |
| US9582238B2 | United States of America | B2 |
14 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written permission of extension of timeJAPANESE INTERMEDIATE CODE: A602A602 | A602 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Written amendmentJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written permission of extension of timeJAPANESE INTERMEDIATE CODE: A602A602 | A602 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 |
Numbers
- Publication
- 5632015
- Publication, DOCDB
- 5632015
- Publication, EPODOC
- JP5632015B
- Application
- 2012544705
- Application, DOCDB
- 2012544705
- Application, EPODOC
- JP20120544705
Titles2
- English
- Decomposed Multistream (DMS) Techniques for Video Display Systems
- Japanese
- ビデオディスプレイシステムのためのデコンポーズドマルチストリーム(DMS)技法
Classification
- CPC, 13
- G06F3/1454
- G09G5/14
- G06F3/14
- G09G2310/04
- G09G2320/0613
- G09G2320/10
- G09G2340/02
- G09G2350/00
- G09G2352/00
- G09G2370/16
- H04L65/607
- H04L65/70
- H04N7/01
- IPC, 4
- H04N21 2343
- H04N5 45
- H04N21 236
- H04N21 2665