Systems and methods for the reuse of encoding information in encoding alternative streams of video data
17 claims: 12 independent, 5 dependent
- 1適応ストリーミングシステムにおいて使用される、マルチメディアコンテンツをエンコーディングするための方法であって、前記方法は、 メディアサーバを使用して、複数のエンコーディングレベルから第1のエンコーディングレベルを選択することと、 前記第1のエンコーディングレベルおよび前記メディアサーバを使用して、ビデオデータの第1のストリームのためのエンコーディング情報を判定すること であって、エンコーディング情報を判定することは、ビデオデータの1つ以上のストリームの解像度における動き推定を判定するために階層的な探索を実行することを含む、こと と、 ビデオデータの前記第1のストリームの解像度を使用して判定された前記動き推定 および前記メディアサーバを使用して、ビデオデータの前記第1のストリームをエンコーディングすることであって、ビデオデータの前記第1のストリームは、第1の解像度および第1のビットレートを含む、ことと、 前記メディアサーバを使用して、前記複数のエンコーディングレベルから第2のエンコーディングレベルを選択することと 、 ビデオデータの第2のストリームの解像度を使用して判定された動き推定の中間結果 および前記メディアサーバを 使用して、前記階層的な探索の完了と並行して ビデオデータの前記第2のストリームをエンコーディングすることであって、ビデオデータの前記第2のストリームは、 前記第1の解像度より低い 第2の解像度および 前記第1のビットレートより低い 第2のビットレートを含む、ことと を含 む、 方法。
- 2エンコーディング情報を判定することは、動き推定結果を参照すること、加重された予測加重を参照すること、場面変化を判定することから成る群から選択されるエンコーディング情報を判定することを含む、請求項1に記載の方法。
- 3前記第1のエンコーディングレベルおよび前記メディアサーバを使用して、ビデオデータの前記第1のストリームのための動き推定を判定することと、 前記第2のエンコーディングレベルおよび前記メディアサーバを使用して、ビデオデータの前記第2のストリームのための前記判定された動き推定の結果をスケーリングすることと をさらに含む、請求項1に記載の方法。
- 4前記メディアサーバを使用して、ビデオデータの前記第1のストリームおよびビデオデータの前記第2のストリームをコンテナファイル内に記憶することをさらに含む、請求項1に記載の方法。
- 5前記コンテナファイルは、Matroskaコンテナファイルである、請求項 4 に記載の方法。
- 6前記メディアサーバを使用して、ビデオデータの前記第1のストリームおよびビデオデータの前記第2のストリームを別個のコンテナファイル内に記憶することをさらに含む、請求項1に記載の方法。
- 7ビデオデータの前記第1のストリームをエンコーディングすることおよびビデオデータの前記第2のストリームをエンコーディングすることは、スケーラブルビデオコーディングを利用する、請求項1に記載の方法。
- 8メディアサーバであって、前記メディアサーバは、 マルチメディアコンテンツを記憶するように構成されたメモリであって、前記マルチメディアコンテンツは、ソースビデオを含む、メモリと、 プロセッサと を備え、 前記プロセッサは、メディアエンコーダアプリケーションによって、 複数のエンコーディングレベルから第1のエンコーディングレベルを選択することと、 前記第1のエンコーディングレベルを使用して、ビデオデータの第1のストリームのためのエンコーディング情報を判定すること であって、エンコーディング情報を判定することは、ビデオデータの1つ以上のストリームの解像度における動き推定を判定するために階層的な探索を実行することを含む、こと と、 ビデオデータの前記第1のストリームの解像度を使用して判定された前記動き推定を使用して、 ビデオデータの前記第1のストリームをエンコーディングすることであって、ビデオデータの前記第1のストリームは、第1の解像度および第1のビットレートを含む、ことと、 前記複数のエンコーディングレベルから第2のエンコーディングレベルを選択することと 、 ビデオデータの第2のストリームの解像度を使用して判定された動き推定の中間結果を使用して、前記階層的な探索の完了と並行して ビデオデータの前記第2のストリームをエンコーディングすることであって、ビデオデータの前記第2のストリームは、 前記第1の解像度より低い 第2の解像度および 前記第1のビットレートより低い 第2のビットレートを含む、ことと を行うように構成されて いる、 メディアサーバ。
- 9前記判定されたエンコーディング情報は、動き推定結果、加重された予測加重、場面変化から成る群から選択される、請求項 8 に記載のメディアサーバ。
- 10前記プロセッサは、 前記第1のエンコーディングレベルを使用して、ビデオデータの前記第1のストリームのための動き推定を判定することと、 前記第2のエンコーディングレベルを使用して、ビデオデータの前記第2のストリームのための前記判定された動き推定の結果をスケーリングすることと を行うようにさらに構成されている、請求項 8 に記載のメディアサーバ。
- 11前記プロセッサは、ビデオデータの前記第1のストリームおよびビデオデータの前記第2のストリームをコンテナファイル内に記憶するようにさらに構成されている、請求項 8 に記載のメディアサーバ。
- 12前記コンテナファイルは、Matroskaコンテナファイルである、請求項 11 に記載のメディアサーバ。
- 13前記プロセッサは、ビデオデータの前記第1のストリームおよびビデオデータの前記第2のストリームを別個のコンテナファイル内に記憶するようにさらに構成されている、請求項 8 に記載のメディアサーバ。
- 14ビデオデータの前記第1のストリームおよびビデオデータの前記第2のストリームは、スケーラブルビデオコーディングを使用してエンコーディングされる、請求項 8 に記載のメディアサーバ。
- 15プロセッサ命令を含む機械可読記憶媒体であって、プロセッサによる前記命令の実行は、 複数のエンコーディングレベルから第1のエンコーディングレベルを選択することと、 前記第1のエンコーディングレベルを使用して、ビデオデータの第1のストリームのためのエンコーディング情報を判定すること であって、エンコーディング情報を判定することは、ビデオデータの1つ以上のストリームの解像度における動き推定を判定するために階層的な探索を実行することを含む、こと と、 ビデオデータの前記第1のストリームの解像度を使用して判定された前記動き推定 を使用して、ビデオデータの前記第1のストリームをエンコーディングすることであって、ビデオデータの前記第1のストリームは、第1の解像度および第1のビットレートを含む、ことと、 前記複数のエンコーディングレベルから第2のエンコーディングレベルを選択することと 、 ビデオデータの第2のストリームの解像度を使用して判定された動き推定の中間結果を使用して、前記階層的な探索の完了と並行して、 ビデオデータの前記第2のストリームをエンコーディングすることであって、ビデオデータの 前記 第2のストリームは、 前記第1の解像度より低い 第2の解像度および 前記第1のビットレートより低い 第2のビットレートを含む、ことと を含むプロセスを前記プロセッサに行わせ る、 機械可読記憶媒体。
- 16ビデオデータの前記第2のストリームの解像度を使用して判定された前記動き推定および前記メディアサーバを使用して、ビデオデータの前記第1のストリームをエンコーディングすることをさらに含む、請求項1に記載の方法。
- 17前記プロセッサは、ビデオデータの前記第2のストリームの解像度を使用して判定された前記動き推定を使用してビデオデータの前記第1のストリームをエンコーディングするようにさらに構成されている、請求項8に記載のメディアサーバ。
Independent claims17
29 paragraphs, as filed
The present invention relates generally to systems and methods for adaptive streaming systems, and more specifically to systems and methods for reusing encoding information in encoding alternative streams of video data.
(Background) The term media streaming describes the playback of media on a playback device, where the media is stored on the server and continuously sent to the playback device over the network during playback. Will be sent. Typically, the playback device is optional during playback to prevent interruptions in playback due to the playback device completing playback of all buffered media prior to receiving the next piece of media. Store a sufficient amount of media in the buffer at a given time. Adaptive bit rate streaming or adaptive streaming involves detecting the streaming state (eg, the user's network bandwidth and CPU capacity) in real time and adjusting the amount of media to be streamed as appropriate. Streaming video over the Internet is becoming a modern phenomenon. YouTube® and ESPN (Bristol,) services of Google, Inc. (Mountain View, California) Many popular websites, such as WatchESPN, a service of Connecticut), utilize streaming video to provide video and television programming to consumers over the Internet.
Scalable Video Coding (SVC) is the International Telecommunication Union Telecommunication Standardization Sector (Geneva,) It is an extension of the H.264 / MPEG-4 AVC video compression standard specified by the ITU-T H.264 standard by Switzerland). The SVC also allows encoding of video bitstreams, including one or more subbitstreams. A subbitstream derives from a videobitstream by dropping packets of data from the videobitstream, resulting in a lower quality and lower bandwidth subbitstream than the original videobitstream. SVC supports three forms of scaling of video bitstreams to subbitstreams: temporal scaling, spatial scaling, and quality scaling. Each of these scaling techniques can be used individually or combined, depending on the particular video system.
In adaptive streaming systems, the source media is typically stored on the media server as a high-level index file that points to some alternative stream that contains the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different indexes and media containers. The Matroska container is a media container developed as an open standard project by the Matroska non-profit organization (Aussonne, France). The Matroska container is based on the Extensible Binary Meta Language (EBML), a binary derivative of the Extensible Markup Language (XML). Decoding of Matroska containers is supported by many consumer electronics (CE) devices. DivX, LLC (San Diego, The DivX Plus file format developed by California) makes use of the Matroska container format extension, which contains elements not specified within the Matroska format.
<p> (Summary of invention) Disclosed are systems and methods for reusing encoding information in encoding alternative streams of video data according to embodiments of the present invention. In one embodiment of the invention, encoding multimedia content for use in an adaptive streaming system involves using a media server to select a first encoding level from a plurality of encoding levels. One encoding level and media server are used to determine the encoding information for the first stream of video data, and the media server is used to encode the first stream of video data. The first stream of video data contains the first resolution and the first bit rate, and the media server is used to select the second encoding level from multiple encoding levels. , Encoding a second stream of video data using encoding information and a media server, that the second stream of video data contains a second resolution and a second bit rate. including.</p><p> In another embodiment of the invention, the first resolution and the second resolution are the same.</p><p> In an additional embodiment of the invention, the first bit rate is lower than the second bit rate.</p><p> In yet another additional embodiment of the invention, the first bit rate exceeds the second bit rate.</p><p> In yet another additional embodiment of the invention, the first resolution is lower than the second resolution.</p><p> In yet another additional embodiment of the invention, the first resolution exceeds the second resolution.</p><p> In yet another embodiment of the invention, determining the encoding information is selected from the group consisting of estimating motion estimation results, selecting weighted predictive weights, and determining scene changes. , Includes determining encoding information.</p><p> In yet another embodiment of the invention, encoding multimedia content further uses a first encoding level and a media server to determine motion estimation for a first stream of video data. That includes scaling the results of the determined motion estimation for a second stream of video data using a second encoding level and media server.</p><p> In yet another additional embodiment of the invention, determining motion estimation involves hierarchical motion estimation using a media server.</p><p> In yet another additional embodiment of the invention, encoding multimedia content further uses a media server to put a first stream of video data and a second stream of video data into a container file. Including remembering.</p><p> In yet another additional embodiment of the invention, the container file is a Matroska container file.</p><p> In yet another embodiment of the invention, the encoded multimedia content further uses a media server to store a first stream of video data and a second stream of video data in separate container files. Including.</p><p> In yet another embodiment of the invention, encoding a first stream of video data and encoding a second stream of video data utilizes scalable video coding.</p><p> Yet another embodiment of the invention includes a media server, the media server being a memory configured to store multimedia content, wherein the multimedia content includes a memory and a processor, including source video. The processor uses the media encoder application to select the first encoding level from multiple encoding levels and uses the first encoding level to encode for the first stream of video data. Determining information and encoding a first stream of video data, the first stream of video data containing a first resolution and a first bit rate, and multiple encodings. From the level, selecting the second encoding level and using the encoding information to encode the second stream of video data, the second stream of video data has the second resolution and It is configured to do things, including a second bit rate.</p><p> In yet another additional embodiment of the invention, the first resolution and the second resolution are the same.</p><p> In yet another additional embodiment of the invention, the first bit rate is lower than the second bit rate.</p><p> In yet another additional embodiment of the invention, the first bit rate exceeds the second bit rate.</p><p> In yet another embodiment of the invention, the first resolution is lower than the second resolution.</p><p> In yet another embodiment of the invention, the first resolution exceeds the second resolution.</p><p> In yet another embodiment of the invention, the determined encoding information is<u style="single">Movement</u>It is selected from the group consisting of estimation results, weighted predictive weights, and scene changes.</p><p> In yet another additional embodiment of the invention, the processor further uses a first encoding level to determine motion estimation for a first stream of video data and uses a second encoding level. It is configured to scale the result of the determined motion estimation for a second stream of video data.</p><p> In yet another additional embodiment of the invention, the determined motion estimation includes a hierarchical motion estimation.</p><p> In yet another additional embodiment of the invention, the processor is further configured to store a first stream of video data and a second stream of video data in a container file.</p><p> In yet another embodiment of the invention, the container file is a Matroska container file.</p><p> In yet another embodiment of the invention, the processor is further configured to store a first stream of video data and a second stream of video data in separate container files.</p><p> In yet another embodiment of the invention, the first stream of video data and the second stream of video data are encoded using scalable video coding.</p><p> Yet another embodiment of the present invention includes a machine-readable medium containing processor instructions, in which the execution of instructions by the processor involves the processor selecting a first encoding level from a plurality of encoding levels. The encoding level of 1 is used to determine the encoding information for the first stream of video data and to encode the first stream of video data, which is the first stream of video data. Encoding a second stream of video data using encoding information, including the first resolution and the first bit rate, and selecting the second encoding level from multiple encoding levels. In other words, the second stream of video data includes a second resolution and a second bit rate, and causes a process including that.<u style="single">For example, the present invention provides the following items.</u><u style="single">(Item 1)</u><u style="single"> A method for encoding multimedia content for use in adaptive streaming systems, said method.</u><u style="single"> Using the media server to select the first encoding level from multiple encoding levels,</u><u style="single"> Using the first encoding level and the media server to determine the encoding information for the first stream of video data.</u><u style="single"> The media server is used to encode the first stream of video data, wherein the first stream of video data includes a first resolution and a first bit rate.</u><u style="single"> Using the media server to select a second encoding level from the plurality of encoding levels,</u><u style="single"> Encoding a second stream of video data using the encoding information and the media server, wherein the second stream of video data includes a second resolution and a second bit rate. And that</u><u style="single"> Including methods.</u><u style="single">(Item 2)</u><u style="single"> The method according to item 1, wherein the first resolution and the second resolution are the same.</u><u style="single">(Item 3)</u><u style="single"> The method according to item 2, wherein the first bit rate is lower than the second bit rate.</u><u style="single">(Item 4)</u><u style="single"> The method according to item 2, wherein the first bit rate exceeds the second bit rate.</u><u style="single">(Item 5)</u><u style="single"> The method according to item 1, wherein the first resolution is lower than the second resolution.</u><u style="single">(Item 6)</u><u style="single"> The method according to item 1, wherein the first resolution exceeds the second resolution.</u><u style="single">(Item 7)</u><u style="single"> Determining encoding information includes determining the encoding information selected from the group consisting of estimating motion estimation results, selecting weighted predictive weights, and determining scene changes. The method described in 1.</u><u style="single">(Item 8)</u><u style="single"> Using the first encoding level and the media server to determine motion estimation for the first stream of video data.</u><u style="single"> Using the second encoding level and the media server to scale the result of the determined motion estimation for the second stream of video data.</u><u style="single"> The method according to item 1, further comprising.</u><u style="single">(Item 9)</u><u style="single"> 8. The method of item 8, wherein determining motion estimation includes hierarchical motion estimation using the media server.</u><u style="single">(Item 10)</u><u style="single"> The method of item 1, further comprising storing the first stream of video data and the second stream of video data in a container file using the media server.</u><u style="single">(Item 11)</u><u style="single"> The method according to item 10, wherein the container file is a Matroska container file.</u><u style="single">(Item 12)</u><u style="single"> The method of item 1, further comprising storing the first stream of video data and the second stream of video data in separate container files using the media server.</u><u style="single">(Item 13)</u><u style="single"> The method of item 1, wherein encoding the first stream of video data and encoding the second stream of video data utilizes scalable video coding.</u><u style="single">(Item 14)</u><u style="single"> A media server, the media server</u><u style="single"> A memory configured to store multimedia content, wherein the multimedia content includes a memory, including a source video.</u><u style="single"> With processor</u><u style="single"> With</u><u style="single"> The processor depends on the media encoder application.</u><u style="single"> Choosing the first encoding level from multiple encoding levels,</u><u style="single"> Using the first encoding level to determine the encoding information for the first stream of video data,</u><u style="single"> Encoding the first stream of video data, wherein the first stream of video data includes a first resolution and a first bit rate.</u><u style="single"> Selecting a second encoding level from the multiple encoding levels</u><u style="single"> Encoding a second stream of video data using the encoding information, wherein the second stream of video data includes a second resolution and a second bit rate.</u><u style="single"> A media server that is configured to do.</u><u style="single">(Item 15)</u><u style="single"> The media server according to item 14, wherein the first resolution and the second resolution are the same.</u><u style="single">(Item 16)</u><u style="single"> The media server according to item 15, wherein the first bit rate is lower than the second bit rate.</u><u style="single">(Item 17)</u><u style="single"> The media server according to item 15, wherein the first bit rate exceeds the second bit rate.</u><u style="single">(Item 18)</u><u style="single"> The media server according to item 14, wherein the first resolution is lower than the second resolution.</u><u style="single">(Item 19)</u><u style="single"> The media server according to item 14, wherein the first resolution exceeds the second resolution.</u><u style="single">(Item 20)</u><u style="single"> The media server according to item 14, wherein the determined encoding information is selected from a group consisting of an emotion estimation result, a weighted predictive weight, and a scene change.</u><u style="single">(Item 21)</u><u style="single"> The processor further</u><u style="single"> Using the first encoding level to determine motion estimation for the first stream of video data,</u><u style="single"> Using the second encoding level to scale the result of the determined motion estimation for the second stream of video data.</u><u style="single"> The media server according to item 14, which is configured to do so.</u><u style="single">(Item 22)</u><u style="single"> The media server according to item 21, wherein the determined motion estimation includes a hierarchical motion estimation.</u><u style="single">(Item 23)</u><u style="single"> The media server of item 14, wherein the processor is further configured to store said first stream of video data and said second stream of video data in a container file.</u><u style="single">(Item 24)</u><u style="single"> The media server according to item 23, wherein the container file is a Matroska container file.</u><u style="single">(Item 25)</u><u style="single"> 14. The media server of item 14, wherein the processor is further configured to store the first stream of video data and the second stream of video data in separate container files.</u><u style="single">(Item 26)</u><u style="single"> The media server of item 14, wherein the first stream of video data and the second stream of video data are encoded using scalable video coding.</u><u style="single">(Item 27)</u><u style="single"> A machine-readable medium containing a processor instruction, the execution of the instruction by the processor is performed by the processor.</u><u style="single"> Choosing the first encoding level from multiple encoding levels,</u><u style="single"> Using the first encoding level to determine the encoding information for the first stream of video data,</u><u style="single"> Encoding the first stream of video data, wherein the first stream of video data includes a first resolution and a first bit rate.</u><u style="single"> Selecting a second encoding level from the multiple encoding levels</u><u style="single"> Encoding a second stream of video data using the encoding information, wherein the second stream of video data includes a second resolution and a second bit rate.</u><u style="single"> A machine-readable medium that allows the process to take place.</u></p>
<figref num="1">FIG. 1 is a system schematic of an adaptive streaming system according to an embodiment of the present invention.</figref><figref num="2">FIG. 2 is a block diagram of a media server configured to encode a stream of video data for use in an adaptive streaming system according to an embodiment of the present invention.</figref><figref num="3">FIG. 3 conceptually illustrates a media server configured to encode a stream of video data for use in an adaptive streaming system according to an embodiment of the invention.</figref><figref num="4">FIG. 4 is a flow diagram illustrating a process for reusing encoding information in encoding an alternative stream of video data according to an embodiment of the present invention.</figref><figref num="5">FIG. 5 is a flow diagram illustrating a process for reusing motion estimation between encodings of alternative streams of video data having matching macroblocks according to an embodiment of the present invention.</figref><figref num="6">FIG. 6 is a flow chart illustrating a process for estimating the hierarchical movement of macroblocks within a frame of video data according to an embodiment of the present invention.</figref>
(Detailed Description) Here, referring to the drawings, a system and a method for reusing encoding information in encoding an alternative stream of video data according to an embodiment of the present invention are disclosed. The present invention will be described below with respect to adaptive streaming systems and block-based video encoding techniques, but the systems and methods described are based on network client connection quality and non-block based video encoding techniques for different streams of video data. It is equally applicable to conventional streaming systems of choice.
In adaptive streaming systems, multimedia content is encoded as an alternative stream of a set of video data. Since each alternative stream of video data is encoded using the same source multimedia content, similar encoding information is determined in the encoding of each alternative stream of video data. Encoding information includes, but is not limited to, motion estimation results, weighted predictive weight selection, and scene change determination. The systems and methods according to many embodiments of the present invention reuse the encoding information determined in the encoding of one alternative stream of video data in the encoding of at least one other alternative stream of video data. By reusing the encoding information in the encoding of some alternative streams of video data, significant improvements in the encoding of alternative streams of video data can be achieved, especially significant time savings in the book. It can be realized according to an embodiment of the invention.
Adaptive streaming systems are configured to stream multimedia content encoded at different maximum bit rates and resolutions over networks such as the Internet. Adaptive streaming systems stream the highest quality multimedia content that can be supported based on current streaming conditions. Multimedia content typically includes video and audio data, subtitles, and other relevant metadata. In order to provide the highest quality video experience independent of network data rates, adaptive streaming systems are used throughout the delivery of video data according to a variety of factors, including, but not limited to, available network data rates and video decoder performance. It is configured to switch between the available sources of video data through. As streaming conditions worsen, adaptive streaming systems typically attempt to switch to multimedia streams encoded at lower maximum bitrates. If the available network data rate cannot support streaming of the stream encoded at the lowest maximum bit rate, then the playback is often buffered with a sufficient amount of content to resume playback. It will be interrupted until it can be done. Systems and methods for switching between playback and video streams that can be used in adaptive streaming systems according to embodiments of the present invention are described in US Patent Application No. 13/221, et al., Filed August 30, 2011.
To create multiple sources of video data used in adaptive streaming systems, the source encoder is configured to encode alternative streams of multiple video data from the source video contained within a set of multimedia content. Can be done. Systems and Methods for Encoding Source Media in Matroska Container, US Patent Application No. 13 / 221,794 by Braness et al., Filed August 30, 2011, for methods and systems for encoding source video for use in adaptive streaming systems Disclosure in Files for Adaptive Bitrate Streaming Using Hypertext Transfer Protocol (incorporated by reference as a whole). According to embodiments of the present invention, the source encoder may be implemented using a media source and / or a media server.
As mentioned above, alternative streams of video data based on the same source video contain similar content. Therefore, the encoding information determined in the encoding of one alternative stream of video data for the source video may be reused in one or more of the other alternative streams of video data. According to an embodiment of the present invention, based on the same source video, alternative stream a set of video data is the same resolution, different bi in Bit Rate, it may contain video data. In many embodiments of the invention, the motion estimation results calculated for the encoding of a particular alternative stream of video data can be reused from among all alternative streams of video data. As discussed below, the various encoding information determined in the encoding of the video data alternative stream may be reused from within the video data alternative stream. Systems and methods for reusing encoding information in encoding alternative streams of video data according to embodiments of the present invention are further discussed below.
(System overview) An adaptive streaming system according to an embodiment of the present invention is configured to adapt to video transmitted to a network client based on streaming conditions. An adaptive streaming system according to an embodiment of the present invention is illustrated in FIG. The adaptive streaming system 10 illustrated includes a media source 100 containing multimedia content. In some embodiments of the invention, the multimedia content stored within the media source 100 is a plurality of pre-encoded streams of video data encoded at multiple bit rates and resolutions for adaptive streaming. The encoding of at least one of the streams of video data in the multiple streams of video data reuses the encoding information determined in the encoding of the second stream of video data. In some embodiments of the invention, the media source 100 encodes a plurality of streams of video data in real time, and at least one of the streams of video data within the streams of video data is the video data. Reuse the encoding information determined in the encoding of the second stream of. In many embodiments of the invention, the multimedia content stored within the media source 100 contains video data encoded using scalable video coding (SVC). In some embodiments, the media source 100 contains multiple streams of video data with equal timelines as alternative streams of video data. The media source 100 is connected to the network renderer 102. In many embodiments, the media source 100 and the network renderer 102 are implemented using a media server. The network renderer 102 contacts a plurality of network clients 104 via the network 110. Will be continued. The network renderer 102 is configured to stream video data to one or more network clients 104. Each network client 104 contains a media decoder 106 configured to decode media streamed from the media source 100 using the network renderer 102.
In many embodiments of the invention, the media source 100 and / or the network renderer 102 is implemented using a single machine. In some embodiments of the invention, the media source 100 and / or the network renderer 102 is implemented using a plurality of machines. In many embodiments of the invention, the media source 100 and network renderer 102 are implemented using a media server. In many embodiments, the network 110 is the Internet. In some embodiments, the network 110 is any IP network. In some embodiments, the network 110 is a cellular data network.
The specific architecture of the adaptive streaming system is shown in FIG. 1, but other implementations suitable for the specific application may also be utilized according to embodiments of the present invention. Systems and methods for reusing encoding information in encoding alternative streams of video data according to embodiments of the present invention are further discussed below.
(Media server capable of encoding media for adaptive streaming) According to embodiments of the present invention, the media server encodes alternative streams of multiple video data for source video contained within multimedia content. It is configured as follows. FIG. 2 is a block diagram of a media server according to an embodiment of the present invention. The media server 200 includes a processor 210 and a video encoder 212. The processor is configured to encode multiple streams of video data, given the source video. In some embodiments, the video encoder 212 is implemented in hardware. In some embodiments, the video encoder 212 is implemented utilizing a graphics processing unit. In many embodiments, the video encoder 212 is a software application that configures the processor 210 to encode an alternate stream of video data.
In some embodiments, the media server 200 includes a storage device 220 connected to a processor 210 that is configured to store alternative streams of multimedia content and / or video data. In many embodiments of the invention, Process 400 uses scalable video coding to encode multimedia content. In some embodiments, alternative streams of video data utilize the Matroska container format and are stored within the container. According to embodiments of the present invention, alternative streams of video data have adaptive groups of photographic structures.
In some embodiments, the media server 200 includes a display device 224 attached to the processor 210, which displays information related to the encoding of the alternate stream of video data and / or the alternate stream of video data. May be used for. According to embodiments of the present invention, the media server 200 is implemented using one or more servers configured with suitable software applications for encoding and / or streaming multimedia content. The media server 200 optionally includes a network device 222 connected to the processor 210. In many embodiments, the network device 222 is configured to establish a network connection and the processor 210 is configured to stream multimedia content using the network connection for use in adaptive streaming systems. To.
The basic architecture of a media server according to an embodiment of the present invention is illustrated in FIG. The media server 300 includes a processor 310 that communicates with the non-volatile memory 330 and the volatile memory 320. In the illustrated embodiment, the non-volatile memory comprises a video encoder 332 that configures the processor to encode multiple alternative streams of video data for the source video, and at least one of the alternative streams of video data. One encoding reuses the encoding information determined in the encoding of the second alternate stream of video data. In some embodiments, the video encoder 332 is implemented using processor 310. The network client 300 also includes a network interface 340 that is configured to send and receive data over a network connection. In some embodiments, the network interface 340 communicates with the processor 310, the non-volatile memory 330, and the volatile memory 320.
In many embodiments of the invention, the media server 300 includes a plurality of processors. In some embodiments, the video encoder 332 is implemented using dedicated hardware. The specific architecture for the media server is illustrated in Figure 3, but the video encoder 332 is located on disk or some other form of storage and is loaded into volatile memory 320 at runtime. Any of a variety of architectures, including architectures, may be utilized to implement a media server for use in adaptive streaming systems according to embodiments of the present invention.
A specific implementation of the media server is shown in FIG. 2, but other implementations suitable for the specific application may also be utilized according to embodiments of the present invention. Systems and methods for reusing encoding information in encoding alternative streams of video data according to embodiments of the present invention are further discussed below.
(Reuse of encoding information in encoding alternative streams of video data) To provide a stream of video data used in adaptive streaming of multimedia content, the media server can encode the multimedia content as an alternative stream of video data. The media server reuses the encoding information determined in the encoding of a stream of video data in the encoding of one or more of the remaining streams of video data, thereby increasing the speed of encoding the alternative stream of video data and Performance can be improved. The process for reusing encoding information in encoding an alternative stream of video data for source video is illustrated in Figure 4. Process 400 includes the step (410) of receiving multimedia content containing the source video. In some embodiments, the encoding level is determined for the source video (412). The encoding level is selected (414). Encoding information is determined for the selected encoding level (416). Encoding information is reused over one or more of the unselected encoding levels (418).
In some embodiments, the encoding level is based on factors including, but not limited to, the expected network data rate, the expected screen resolution of the network client, and / or the resolution and / or bit rate of the source video. Judged (412). In some embodiments, the encoding level is determined by reading the stored value (412). According to embodiments of the present invention, alternative streams of video data with a certain resolution and bit rate are encoded for each encoding level. In many embodiments, the alternative stream of video data has the same resolution and variable bit rate. In some embodiments, the selected (414) encoding level corresponds to an alternate stream of video data with the lowest bit rate. In some embodiments, the selected (414) encoding level corresponds to an alternate stream of video data with the highest bit rate. In some embodiments, the various determined (416) encoding information is of the same resolution and variable bit rate, including, but not limited to, motion estimation results, weighted predictive weight selection, and scene change determination. May be reused across encoding levels with video data in (418). According to embodiments of the present invention, encoding information can be reused across encoding levels (418), resulting in significant savings in computational complexity.
In some embodiments, the alternative stream of video data encoded based on the determined (412) encoding level has variable resolution and bit rate. In many embodiments, the selected (414) encoding level corresponds to an alternative stream of video data with the lowest resolution. In some embodiments, the selected (414) encoding level corresponds to an alternative stream of video data with the highest resolution. In some embodiments, the determined (416) encoding information is reused across streams of video data with different resolutions (418). According to many embodiments of the invention, the reused encoding information (418) may vary across encoding levels. For example, if the reused (418) encoding information is a motion vector, the motion vector can be scaled to different resolutions of alternative streams of video data. Other encoding information, such as the information described above, may likewise be scaled between encoding levels corresponding to alternative streams of video data with different resolutions.
A specific process for reusing encoding information in encoding an alternative stream of video data according to an embodiment of the present invention is a step of reusing encoding information that has been described above but is not specifically listed. And after the initial determination of the encoding information, various processes may be utilized according to embodiments of the present invention, including the step of reusing the encoding information in an alternative stream of video data created later. The process for reusing motion estimation used in the encoding of alternative streams of video data according to certain embodiments of the present invention is discussed below.
(Reuse of motion estimation in encoding alternative streams of video data) Various video compression standards such as H.264 may be used to encode alternative streams of video data for use in adaptive streaming systems. .. Many video compression standards utilize video frames that contain macroblocks, and the encoding of a stream of video data involves the calculation of macroblock motion estimates between video frames. According to an embodiment of the present invention, the bit rate of a stream of video data can be varied by allocating the same number of bits for motion estimation and increasing or decreasing the number of bits assigned to the macroblock description.
A process for reusing a motion estimate of macroblock motion between alternative streams of video data derived from the same source video contained within a set of multimedia content according to an embodiment of the invention is shown in FIG. Illustrated in. Process 500 includes downscaling the source video and determining an alternative stream of video data (510). An alternative stream of video data is selected (512). Macroblocks are determined for selected streams of video data (514). A search (516) is performed to match macroblocks in the second alternate stream of video data. If the second alternative stream of video data contains one or more matching macroblocks, the motion estimation results associated with the matching macroblock will be reused in the encoding of the second alternative stream of video data (518). ).
There are several methods for calculating estimates of macroblock motion within a frame of video data, including, but not limited to, full search, heuristic search, and hierarchical estimation. Hierarchical estimation is commonly used to provide accurate results in a fast and efficient fashion. A process for estimating the hierarchical motion of macroblocks within a frame of video data according to an embodiment of the present invention is illustrated in FIG. Process 600 includes selecting a frame of video data (610), known as zero resolution used in the search. A resolution is created for the selected frame (612). According to embodiments of the present invention, two or more resolutions are created (612), and the resolutions created are lower than the previous resolutions. Resolutions may be created using a variety of image sampling techniques, including, but not limited to, average intensity and subsampling (612). The highest resolution is searched for the macroblock that corresponds to the macroblock at zero resolution (614). Motion estimation is calculated for the corresponding macroblock at the highest resolution (616). When a zero resolution frame is reached (618), the calculated (616) motion estimate is used as the motion estimate for the frame of the video data and the process is complete. If the zero resolution frame is not reached (618), the next resolution is searched (614) and the motion estimate is previously calculated (616) and the motion estimate and the corresponding macroblock are between the previous resolution and the current resolution. It is calculated using the distance traveled in (616). In some embodiments, macroblock motion is calculated by a variety of methods, including mean absolute deviation, mean square deviation, and pixel difference classification.
Returning to FIG. 5, in some embodiments, the search (516) for matching macroblocks utilizes hierarchical motion estimation. In some embodiments, hierarchical motion estimation results, including intermediate results, may be reused between alternative streams of video data with different resolutions (518). In many embodiments, the selected (512) stream of video data is an alternative stream of video data with the lowest resolution. In some embodiments, the selected (512) stream of video data has the highest resolution. In some embodiments, the reused (518) motion estimation results are scaled between alternative streams of video data. The specific process for reusing motion estimation in the encoding of alternative streams of video data has been described above. However, various processes for reusing motion estimation may be utilized according to embodiments of the present invention.
Although the present invention has been described in certain specific aspects, many additional modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that the present invention can be practiced differently from those specifically described without departing from the scope and spirit of the present invention. Therefore, embodiments of the present invention should be viewed in all respects as illustration, not limitation. Therefore, the scope of the present invention should be determined not by the illustrated embodiment but by the appended claims and their equivalents.
6 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6
Every citation, both waysCites: the store holds 8 of 9
| Document | Relation | Office |
|---|---|---|
| JP200633393A | Cites | Japan |
| JP2009502099A | Cites | Japan |
| JP2002142218A | Cites | Japan |
| JP8195956A | Cites | Japan |
| WO2012009485A1 | Cites | World Intellectual Property Organization (WIPO) |
| WO2010114092A1 | Cites | World Intellectual Property Organization (WIPO) |
| JP2005229593A | Cites | Japan |
| JP2001503225A | Cites | Japan |
| 仙田裕三、「シームレスビデオサービスとビデオトランスコーダ」、NEC技報、Vol.51、No.8、株式会社NECクリエイティブ、1998年8月25日、第46~49頁 | Non-patent | – |
| 三村麻梨乃(外3名)、「ストリーミング動画配信機構におけるアプリケーション層レベルのフィードバック型転送レート制御の解析的検討」、電子情報通信学会技術研究報告〔ネットワークシステム〕、Vol.111、No.468、社団法人電子情報通信学会、2012年3月1日、第203~208頁 | Non-patent | – |
18 members in 7 offices
Priority claims9
| Document | Office | Kind | Date |
|---|---|---|---|
| 13485609 | United States of America | – | |
| 201213485609 | United States of America | A | |
| 201213485609 | United States of America | A | |
| 2013041874 | United States of America | W | |
| 2013041874 | United States of America | W | |
| 13485609 | – | – | – |
| US201213485609 | – | – | – |
| US2013041874 | – | – | – |
| WO2013US41874 | – | – | – |
Members18
| Document | Office | Kind | |
|---|---|---|---|
| US2013322517A1 | United States of America | A1 | |
| WO2013181013A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2013181013A8 | World Intellectual Property Organization (WIPO) | A8 | |
| AU2013267738A1 | Australia | A1 | |
| KR20150023323A | Republic of Korea | A | |
| EP2856751A1 | European Patent Office (EPO) | A1 | |
| JP2015523789A | Japan | A | |
| US9532080B2 | United States of America | B2 | |
| EP2856751A4 | European Patent Office (EPO) | A4 | |
| JP2017063495A | Japan | A | |
| US2017104989A1 | United States of America | A1 | |
| AU2017203764A1 | Australia | A1 | |
| BR112014029794A2 | Brazil | A2 | |
| AU2017203764B2 | Australia | B2 | |
| JP6463264B2This record | Japan | B2 | |
| JP2019030022A | Japan | A | |
| KR102144557B1 | Republic of Korea | B1 | |
| US11025902B2 | United States of America | B2 |
16 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Cancellation because of no payment of annual feesLAPS | LAPS | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Request for change of ownership or part of ownershipJAPANESE INTERMEDIATE CODE: R313113S111 | S111 | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Written request for registration of change of nameJAPANESE INTERMEDIATE CODE: R313533S533 | S533 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Written request for extension of timeJAPANESE INTERMEDIATE CODE: A601A601 | A601 | |
| Re-examination (zenchi) completed and case transferred to appeal boardAppealJAPANESE INTERMEDIATE CODE: A912A912 | A912 | |
| Transfer to examiner for re-examination before appeal (zenchi)AppealJAPANESE INTERMEDIATE CODE: A911A911 | A911 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Decision of refusalJAPANESE INTERMEDIATE CODE: A02A02 | A02 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 |
Numbers
- Publication
- 6463264
- Publication, DOCDB
- 6463264
- Publication, EPODOC
- JP6463264B
- Application
- 2015515043
- Application, DOCDB
- 2015515043
- Application, EPODOC
- JP20150515043
Titles2
- Japanese
- ビデオデータの代替ストリームをエンコーディングすることにおけるエンコーディング情報の再使用のためのシステムおよび方法
- English
- Systems and methods for reusing encoding information in encoding alternative streams of video data
Classification
- CPC, 10
- H04N19/56
- H04N19/103
- H04N19/61
- H04N19/33
- H04N19/53
- H04N19/39
- H04N19/87
- H04N19/146
- H04N19/184
- H04N19/30
- IPC, 11
- H04N19 33
- H04N19 115
- H04N19 139
- H04N19 146
- H04N19 157
- H04N19 179
- H04N19 187
- H04N19 513
- H04N19 53
- H04N21 2662
- H04N21 6437
