Adaptive bitrate streaming of media stored in matroska container files using hypertext transfer protocol
61 claims: 9 independent, 52 dependent
- 1拡張可能バイナリマークアップ言語(EBML)コンテナファイル内にパッケージ化されたエンコードされたビデオの複数の代替ストリームを使用して、適応型ビットレートストリーミングを行うように構成された再生デバイスであって、該複数の代替ストリームの各々は、EBMLコンテナファイルの要素中へパックされたエンコードされたビデオの各部分がクローズドピクチャ群(GOP)を始めるイントラフレームで開始し、かつ、該EBMLコンテナファイルの各々が該EBMLコンテナファイル内に含有されるストリームのエンコーディングパラメータを指定する少なくとも1つの要素を含むように、異なるエンコーディングパラメータを使用してエンコードされた同一のソースビデオであり、 該再生デバイスは、 クライアントアプリケーションを介して、ハイパーテキスト転送プロトコル(HTTP)バイト範囲要求を介して遠隔サーバからファイルの部分を要求するように構成されたプロセッサを備え、 該クライアントアプリケーションは、 トップレベルインデックスデータを読み出すことであって、該トップレベルインデックスデータは、複数のEBMLコンテナファイルを識別し、かつ、該複数のEBMLコンテナファイル内に含有される該複数の代替ストリームの最大のビットレートを少なくとも記述する、ことと、 該トップレベルインデックスデータを解析することにより、該複数のEBMLコンテナファイルを識別する情報を取得し、該複数のEBMLコンテナファイルのどの部分が該複数のEBMLコンテナファイル内の該エンコードされたメディアに対するインデックスを含有するかに関する情報を取得することと、 該EBMLコンテナファイル内に含有される該ストリームの該エンコーディングパラメータを指定する該少なくとも1つの要素を含有する該複数のEBMLコンテナファイルのうちの少なくとも1つの部分を要求することであって、該EBMLコンテナファイル内に含有される該ストリームの該エンコーディングパラメータを指定する該少なくとも1つの要素は、該EBMLコンテナファイル内に含有される該代替ストリームのフレーム高さ、フレーム幅、サンプルアスペクト比を指定する、ことと、 該トップレベルインデックスデータを解析することに基づいて、複数のcueポイントを含むインデックスを読み出すことであって、該複数のcueポイントの各々は、対応する1つの要素の開始を参照し、該要素は、該複数のEBMLコンテナファイルのうちの少なくとも1つにおけるエンコードされたビデオの部分を含有する、ことと、 該要素の開始への参照を使用して該要素のサイズを推測することと、 該推測されたサイズを利用してEBMLコンテナファイルの要素に対するHTTPバイト範囲要求を作成することと、 該要求された要素を受信およびバッファリングすることであって、該バッファリングされた要素は、エンコードされたビデオの部分を含有する、ことと、 該バッファリングされた要素内の該エンコードされたビデオのデコーディングの開始に先立って、該複数のEBMLコンテナファイルのうちの該少なくとも1つの該要求された部分内の該少なくとも1つの要素によって指定された該フレーム高さ、該フレーム幅、該サンプルアスペクト比を設定することと、 該エンコーディングパラメータを利用して、該バッファリングされた要素内に含有された該エンコードされたビデオをデコードすることと、 現在のストリーミング状態を測定することと、 該複数のEBMLコンテナファイルのうちの異なるEBMLコンテナファイルを選択することであって、該選択は、デコーディングのために、該選択されたEBMLコンテナファイルからエンコードされたビデオの部分を含有する要素を読み出すことを目的とし、該選択は、該測定されたストリーミング状態および該トップレベルインデックスデータ内に含有される該代替ストリームのビットレートの記述に基づいており、該代替ストリームに対する該EBMLコンテナファイルの対応する要素は、同一のタイムコード要素を含む、ことと を行うように該プロセッサをさらに構成する、再生デバイス。
- 2前記複数のEBMLコンテナファイルのうちの前記少なくとも1つにおけるエンコードされたビデオの部分を含有する前記 要素 は、C l uster要素であり、 該Cluster要素の各々の中の該エンコードされたビデオの部分は、イントラフレームで開始し、少なくとも1つのクローズドピクチャ群を含む、請求項1に記載の再生デバイス。
- 3前記Cluster要素の各々の中の前記エンコードされたビデオの部分は、同一の持続時間を有する、請求項2に記載の再生デバイス。
- 4前記Cluster要素の各々の中の前記エンコードされたビデオの部分は、2秒の持続時間を有する、請求項2に記載の再生デバイス。
- 5各Cluster要素は、タイムコードを含有し、該Cluster要素内に含有される前記エンコードされたビデオの部分の各エンコードされたフレームは、別個のBlockGroup要素内に含有される、請求項2に記載の再生デバイス。
- 6前記Cluster要素中の第1のBlockGroup要素は、前記イントラフレームを含有する、請求項5に記載の再生デバイス。
- 7前記第1のBlockGroup要素は、前記Cluster要素のタイムコードに対して、前記イントラフレームのタイムコード属性を指定するBlock要素を含有する、請求項6に記載の再生デバイス。
- 8前記エンコードされたビデオの代替ストリームの各々は、異なる最大ビットレートでエンコードされる、請求項1に記載の再生デバイス。
- 9前記エンコードされたビデオの代替ストリームのうちの少なくとも2つは、同一の表示アスペクト比でエンコードされるが、異なるサンプルアスペクト比を使用する異なる解像度でエンコードされる、請求項8に記載の再生デバイス。
- 10前記エンコードされたビデオの代替ストリームのうちの少なくとも2つは、異なるフレームレートでエンコードされる、請求項8に記載の再生デバイス。
- 11前記EBMLコンテナファイル内に含有されるストリームの前記エンコーディングパラメータを指定する前記複数のEBMLコンテナファイルの各々の中の少なくとも1つの要素は、該EBMLコンテナファイル内に含有される代替ストリームのフレームレートを指定する、請求項10に記載の再生デバイス。
- 12前記クライアントアプリケーションは、前記複数のEBMLコンテナファイルのうちの1つからバッファリングされた要素内に含有される前記エンコードされたビデオのデコーディングの開始に先立って、前記フレームレートを設定するように前記プロセッサを構成する、請求項11に記載の再生デバイス。
- 13前記クライアントアプリケーションは、ハイパーテキスト転送プロトコル(HTTP)バイト範囲要求を介して、遠隔サーバから、ファイルの部分を要求するように前記再生デバイスを構成する、請求項9に記載の再生デバイス。
- 14前記トップレベルインデックスデータは、SMILデータであり、前記複数のEBMLコンテナファイルを識別する情報は、該複数のEBMLコンテナファイルの各々の場所を識別する複数の汎用リソースインジケータ(URI)である、請求項1に記載の再生デバイス。
- 15前記SMILデータは、前記URIの各々によって識別される前記EBMLコンテナファイル内に含有されるエンコードされたストリームの最大ビットレートを指定する、請求項14に記載の再生デバイス。
- 16前記EBMLコンテナファイル内の前記エンコードされたストリームの部分を含有する要素の各々を参照するインデックスは、該EBMLコンテナファイルの少なくとも1つの要素内に配置されており、 前記SMILデータは、前記エンコーディングパラメータを含有する要素を含有する該EBMLコンテナファイルの部分を指定し、 前記クライアントアプリケーションは、 該SMILデータを解析することにより、該エンコーディングパラメータを含有する要素を含有する該EBMLコンテナファイルの部分を識別することと、 該エンコーディングパラメータを含有する要素を含有する該EBMLコンテナファイルの部分を要求することと を行うように前記プロセッサをさらに構成する、請求項15に記載の再生デバイス。
- 17前記クライアントアプリケーションは、EBMLコンテナファイル内のエンコードされたビデオの部分を含有する各要素を参照する前記複数の cueポイント を含む前記インデックスを読み出すように前記プロセッサを構成し、該インデックスの読み出しは、該インデックスを含有する少なくとも1つの要素を含む該EBMLコンテナファイルの部分を要求することによって行われる、請求項1に記載の再生デバイス。
- 18前記インデックスは、前記EBMLコンテナファイル内のCue要素内に配置されている、請求項17に記載の再生デバイス。
- 19前記Cue要素は、複数のCuePoint要素を含み、該複数のCuePoint要素は、時間属性を含み、かつ、前記EBMLコンテナファイル内のエンコードされたビデオの部分を含有する各要素の場所を指し示す、請求項18に記載の再生デバイス。
- 20前記クライアントアプリケーションは、 前記インデックス内の前記CuePoint要素の時間属性を使用して、前記EBMLコンテナファイル内のエンコードされたビデオの特定の部分を含有する要素の場所を検索することと、 該CuePoint要素を使用して識別された該要素を少なくとも含む該EBMLコンテナファイルの部分を要求することと を行うように前記プロセッサを構成する、請求項19に記載の再生デバイス。
- 21前記エンコードされたビデオをデコードするために利用される前記エンコーディングパラメータは、フレームレート、フレーム高さ、フレーム幅、サンプルアスペクト比、最大ビットレート、最小バッファサイズから成る群から選択される少なくとも1つのエンコーディングパラメータを含む、請求項1に記載の再生デバイス。
- 22前記クライアントアプリケーションは、前記要素が要求された時間から、要求された要素を受信するためにかかった時間を測定することによって、現在のストリーミング状態を測定するように前記プロセッサを構成する、請求項1に記載の再生デバイス。
- 23前記クライアントアプリケーションは、第1のEBMLコンテナファイル内に含有されるストリームの前記エンコーディングパラメータを指定する少なくとも1つの要素を含有する該第1のEBMLコンテナファイルの部分を最初に要求するように前記プロセッサを構成し、 該クライアントアプリケーションが前記測定されたストリーミング状態および前記トップレベルインデックスデータ内に含有される前記代替ストリームのビットレートの記述に基づいて第2のEBMLコンテナファイルからデコードするために、エンコードされたビデオの部分を含有する要素を読み出すことを選択するとき、該クライアントアプリケーションは、該第2のEBMLコンテナファイル内に含有されるストリームのエンコーディングパラメータを指定する少なくとも1つの要素を含有する該第2のEBMLコンテナファイルの部分を要求するように前記プロセッサをさらに構成する、請求項1に記載の再生デバイス。
- 24前記再生デバイスは、トリックプレイトラックを含有するEBMLコンテナファイルを使用して、適応型ビットレートストリーミングを行うようにさらに構成される、請求項1に記載の再生デバイス。
- 25前記クライアントアプリケーションは、ユーザ命令に応答してエンコードされたビデオの部分を含有する要素を読み出すべき前記EBMLコンテナファイルとして、前記トリックプレイトラックを含有する該EBMLコンテナファイルを選択するように前記プロセッサを構成する、請求項24に記載の再生デバイス。
- 26前記再生デバイスは、エンコードされたオーディオのトラックを含有する少なくとも1つのEBMLコンテナファイルを使用して、適応型ストリーミングを行うようにさらに構成される、請求項1に記載の再生デバイス。
- 27前記クライアントアプリケーションは、オーディオトラックを含有する前記少なくとも1つのEBMLコンテナファイルのうちの1つからエンコードされたオーディオを読み出しデコードすることと、該デコードされたオーディオを前記デコードされたビデオと同期させることとを行うように前記プロセッサを構成する、請求項26に記載の再生デバイス。
- 28前記複数のEBMLコンテナファイルの各々は、前記エンコードされたビデオで多重化されたエンコードされたオーディオの少なくとも1つのトラックを含有する要素を含む、請求項1に記載の再生デバイス。
- 29前記クライアントアプリケーションは、 前記エンコードされたオーディオのトラックのうちの1つの部分を含む要素を含有する前記EBMLコンテナファイルの部分を読み出すことと、 該読み出されたエンコードされたオーディオをデコードすることと、 該デコードされたオーディオと前記デコードされたビデオとを同期させることと を行うように構成される、請求項26に記載の再生デバイス。
- 30前記再生デバイスは、字幕トラックを含有する少なくとも1つのEBMLコンテナファイルを使用して、適応型ストリーミングを行うようにさらに構成される、請求項1に記載の再生デバイス。
- 31前記クライアントアプリケーションは、 字幕トラックを含有する前記少なくとも1つのEBMLコンテナファイルのうちの1つから字幕を読み出すことと、 該字幕を前記デコードされたビデオと同期させることと、 該字幕を該デコードされたビデオ上にオーバレイすることと を行うように前記プロセッサを構成する、請求項30に記載の再生デバイス。
- 32前記複数のEBMLコンテナファイルの各々は、前記エンコードされたビデオで多重化された少なくとも1つの字幕トラックを含有する要素を含む、請求項1に記載の再生デバイス。
- 33前記クライアントアプリケーションは、 前記字幕トラックのうちの1つの部分を含む要素を含有する前記EBMLコンテナファイルの部分を読み出すことと、 字幕 を前記デコードされたビデオと同期させることと、 該字幕を該デコードされたビデオ上にオーバレイすることと を行うように構成される、請求項32に記載の再生デバイス。
- 34拡張可能バイナリマークアップ言語(EBML)コンテナファイル内にパッケージ化されたエンコードされたビデオの複数の代替ストリームを使用して、再生デバイス上でメディアの適応型ビットレートストリーミングを行う方法であって、該ビデオの複数の代替ストリームの各々は、EBMLコンテナファイルの要素中へパックされたエンコードされたビデオの各部分がクローズドピクチャ群(GOP)を始めるイントラフレームで開始し、かつ、該EBMLコンテナファイルの各々が該EBMLコンテナファイル内に含有されるストリームのエンコーディングパラメータを指定する少なくとも1つの要素を含むように、異なるエンコーディングパラメータを使用してエンコードされた同一のソースビデオであり、 該方法は、 ハイパーテキスト転送プロトコル(HTTP)バイト範囲要求を介して遠隔サーバからファイルの部分を要求することと、 トップレベルインデックスデータを読み出すことであって、該トップレベルインデックスデータは、複数のEBMLコンテナファイルの各々を識別し、かつ、該複数のEBMLコンテナファイル内に含有される該複数の代替ストリームの各々の最大ビットレートを少なくとも記述する、ことと、 該トップレベルインデックスデータを解析することにより、該複数のEBMLコンテナファイルを識別する情報を取得し、該複数のEBMLコンテナファイルのどの部分が該複数のEBMLコンテナファイル内の該エンコードされたメディアに対するインデックスを含有するかに関する情報を取得することと、 該EBMLコンテナファイル内に含有される該ストリームの該エンコーディングパラメータを指定する少なくとも1つの要素を含有する該複数のEBMLコンテナファイルのうちの少なくとも1つの部分を要求することであって、該EBMLコンテナファイル内に含有される該ストリームの該エンコーディングパラメータを指定する該少なくとも1つの要素は、該EBMLコンテナファイル内に含有されるビデオストリームのフレーム高さ、フレーム幅、サンプルアスペクト比を指定する、ことと、 該トップレベルインデックスデータを解析することに基づいて、複数のcueポイントを含むインデックスを読み出すことであって、該複数のcueポイントの各々は、対応する1つの要素の開始を参照し、該要素は、該複数のEBMLコンテナファイルのうちの少なくとも1つにおけるエンコードされたビデオの部分を含有する、ことと、 該要素の開始への参照を使用して該要素のサイズを推測することと、 該推測されたサイズを利用してEBMLコンテナファイルの要素に対するHTTPバイト範囲要求を作成することと、 該要求された要素を受信およびバッファリングすることであって、該バッファリングされた要素は、エンコードされたビデオの部分を含有する、ことと、 該バッファリングされた要素内の該エンコードされたビデオのデコーディングの開始に先立って、該複数のEBMLコンテナファイルのうちの該少なくとも1つの該要求された部分内の該少なくとも1つの要素によって指定された該フレーム高さ、該フレーム幅、該サンプルアスペクト比を設定することと、 該エンコーディングパラメータを使用して、該バッファリングされた要素内に含有される該エンコードされたビデオをデコードすることと、 現在のストリーミング状態を測定することと、 該複数のEBMLコンテナファイルのうちの異なるEBMLコンテナファイルを選択することであって、該選択は、デコーディングのために、該選択されたEBMLコンテナファイルからエンコードされたビデオの部分を含有する要素を読み出すことを目的とし、該選択は、該測定されたストリーミング状態および該トップレベルインデックスデータ内に含有される該代替ストリームのビットレートの記述に基づいており、該代替ストリームに対する該EBMLコンテナファイルの対応する要素は、同一のタイムコード要素を含む、ことと 含む、方法。
- 35複数のコンテナファイル内にパックされる複数の代替ビデオストリームとしてソースビデオをエンコードするように構成されたソースエンコーダであって、該複数のコンテナファイルは、拡張可能バイナリマークアップ言語(EBML)コンテナファイルであり、 該ソースエンコーダは、 ソースエンコーディングアプリケーションを介して、ソースビデオを含有する少なくとも1つのマルチメディアファイルを取り込むように構成されたプロセッサを備え、 該ソースエンコーディングアプリケーションは、 該ソースビデオの部分を選択することと、 該ソースビデオの該選択された部分をエンコードされたビデオの複数の代替部分にトランスコードすることであって、各代替部分は、異なる一組のエンコーディングパラメータを使用して異なる最大ビットレートでエンコードされ、クローズドピクチャ群(GOP)を始めるイントラフレームで開始する、ことと、 該エンコードされたビデオの該複数の代替部分の各々を異なるEBMLコンテナファイルのCluster要素に書き込むことであって、各EBMLコンテナファイルは、該エンコードされたビデオの代替部分をエンコードするために使用される該エンコーディングパラメータを示す別の要素を含み、該ソースビデオの同一の選択された部分からのエンコードされたビデオの代替部分を含有するCluster要素は、同一のタイムコードを共有する、ことと、 所与のEBMLコンテナファイルに書き込まれた各Cluster要素に対して、エントリを該所与のEBMLコンテナファイル内のCue要素に追加することであって、該Cue要素は、該エンコードされたビデオの該複数の代替部分のうちの1つを含有する次のCluster要素の第1のバイトをインデックスし、該追加されたエントリは、再生デバイスによって、所与のCluster要素のサイズの推測を可能にする、ことと、 トップレベルインデックスファイルを作成することであって、該トップレベルインデックスファイルは、複数のEBMLコンテナファイルを識別し、かつ、該複数のEBMLコンテナファイル内に含有される該複数の代替ビデオストリームの各代替部分のそれぞれの最大ビットレートを少なくとも記述し、該トップレベルインデックスファイルは、該複数のEBMLコンテナファイルとは別個である、ことと を行うように該プロセッサを構成する、ソースエンコーダ。
- 36前記ソースビデオの選択された部分をトランスコードすることは、該選択された部分を少なくとも1つのクローズドピクチャ群にトランスコードすることをさらに含む、請求項35に記載のソースエンコーダ。
- 37前記ソースビデオの部分は、前記ソースビデオの選択された部分の持続時間に基づいて選択される、請求項35に記載のソースエンコーダ。
- 38前記ソースエンコーディングアプリケーションは、2秒の持続時間を有する前記ソースビデオの部分を選択するように前記プロセッサを構成する、請求項37に記載のソースエンコーダ。
- 39前記エンコードされたビデオの代替部分のうちの少なくとも2つは、異なる解像度でエンコードされる、請求項35に記載のソースエンコーダ。
- 40前記エンコードされたビデオの代替部分のうちの少なくとも2つは、異なるフレームレートでエンコードされる、請求項35に記載のソースエンコーダ。
- 41前記エンコードされたビデオの部分は、前記Cluster要素内のBlockGroup要素内に含有される、請求項35に記載のソースエンコーダ。
- 42前記Cluster要素内に含有される前記エンコードされたビデオの代替部分の各エンコードされたフレームは、別個のBlockGroup要素内に含有される、請求項41に記載のソースエンコーダ。
- 43前記Cluster要素内の第1のBlockGroup要素は、IDRフレームを含有する、請求項42に記載のソースエンコーダ。
- 44前記第1のBlockGroup要素は、前記Cluster要素のタイムコードに対して、前記IDRフレームのタイムコード属性を指定するBlock要素を含有する、請求項43に記載のソースエンコーダ。
- 45前記エンコードされたビデオの代替部分の各々が書き込まれる各Cluster要素は、同一のタイムコードが割り当てられる、請求項35に記載のソースエンコーダ。
- 46前記ソースエンコーディングアプリケーションは、前記複数のEBMLコンテナファイルの各々の中の前記エンコードされたビデオの代替部分のうちの1つを含有する要素の場所を、該EBMLコンテナファイルのためのインデックスに追加するように前記プロセッサをさらに構成する、請求項35に記載のソースエンコーダ。
- 47前記ソースエンコーディングアプリケーションは、各EBMLコンテナファイルのための前記インデックスを該EBMLコンテナファイル中へパックするように前記プロセッサをさらに構成する、請求項35に記載のソースエンコーダ。
- 48Cue要素は、HTTPを介して特定のCluster要素内に含有されるメディアの読み出しを容易にする、請求項47に記載のソースエンコーダ。
- 49各Cue要素は、前記EBMLコンテナファイル内の前記エンコードされたビデオの代替部分のうちの1つを含有するCluster要素の場所を指し示すCuePoint要素を含む、請求項48に記載のソースエンコーダ。
- 50前記取り込まれたマルチメディアファイルはまた、ソースオーディオを含む、請求項35に記載のソースエンコーダ。
- 51前記ソースエンコーディングアプリケーションは、前記複数のEBMLコンテナファイルの各々の中に前記ソースオーディオを多重化するように前記プロセッサを構成する、請求項50に記載のソースエンコーダ。
- 52前記ソースエンコーディングアプリケーションは、別個のEBMLコンテナファイルに前記ソースオーディオを書き込むように前記プロセッサを構成する、請求項50に記載のソースエンコーダ。
- 53前記ソースエンコーディングアプリケーションは、前記少なくとも1つのオーディオトラックのうちの少なくとも1つをトランスコードするように前記プロセッサをさらに構成する、請求項50に記載のソースエンコーダ。
- 54前記取り込まれたマルチメディアファイルは、字幕をさらに備える、請求項50に記載のソースエンコーダ。
- 55前記ソースエンコーディングアプリケーションは、前記複数のEBMLコンテナファイルの各々の中に前記字幕を多重化するように前記プロセッサを構成する、請求項54に記載のソースエンコーダ。
- 56前記ソースエンコーディングアプリケーションは、別個のEBMLコンテナファイルに前記字幕を書き込むように前記プロセッサを構成する、請求項54に記載のソースエンコーダ。
- 57前記ソースエンコーディングアプリケーションは、前記ソースビデオをトランスコードすることにより、より低いフレームレートのトリックプレイトラックを作成することと、別個のEBMLコンテナファイルに該より低いフレームレートのトリックプレイトラックを書き込むこととを行うように前記プロセッサをさらに構成する、請求項35に記載のソースエンコーダ。
- 58前記より低いフレームレートのトリックプレイトラックはまた、前記ソースビデオよりも低い解像度である、請求項57に記載のソースエンコーダ。
- 59前記ソースエンコーディングアプリケーションは、一組のエンコーディングパラメータを含有する要素を前記複数のEBMLコンテナファイルの各々の中に書き込むように前記プロセッサをさらに構成する、請求項35に記載のソースエンコーダ。
- 60前記一組のエンコーディングパラメータは、フレームレート、フレーム高さ、フレーム幅、サンプルアスペクト比、最大ビットレート、最小バッファサイズから成る群から選択される少なくとも1つのパラメータを含む、請求項59に記載のソースエンコーダ。
- 61ソースエンコーダを使用して、拡張可能バイナリマークアップ言語(EBML)コンテナファイル中にパックされる複数の代替ストリームとして、ソースビデオをエンコードする方法であって、 該方法は、 該ソースエンコーダを使用して、該ソースビデオの部分を繰り返し選択することと、 該ソースエンコーダを使用して、該ソースビデオの該選択された部分をエンコードされたビデオの複数の代替部分にトランスコードすることであって、各代替部分は、異なる一組のエンコーディングパラメータを使用して異なる最大ビットレートでエンコードされ、クローズドピクチャ群(GOP)を始めるイントラフレームで開始する、ことと、 該ソースエンコーダを使用して、該エンコードされたビデオの該複数の代替部分の各々を異なるEBMLコンテナファイルのCluster要素に書き込むことであって、各EBMLコンテナファイルは、該エンコードされたビデオの部分をエンコードするために使用される該一組のエンコーディングパラメータに対応する一組のエンコーディングパラメータを含有する別の要素を含み、該ソースビデオの同一の選択された部分からのエンコードされたビデオの代替部分を含有するCluster要素は、同一のタイムコードを共有する、ことと、 所与のEBMLコンテナファイルに書き込まれた各Cluster要素に対して、エントリを該所与のEBMLコンテナファイル内のCue要素に追加することであって、該Cue要素は、該エンコードされたビデオの該複数の代替部分のうちの1つを含有する次のCluster要素の第1のバイトをインデックスし、該追加されたエントリは、再生デバイスによって、所与のCluster要素のサイズの推測を可能にする、ことと、 トップレベルインデックスファイルを作成することであって、該トップレベルインデックスファイルは、複数のEBMLコンテナファイルを識別し、かつ、該複数のEBMLコンテナファイル内に含有される該複数の代替ストリームの各代替部分のそれぞれの最大ビットレートを少なくとも記述し、該トップレベルインデックスファイルは、該複数のEBMLコンテナファイルとは別個である、ことと を含む、方法。
Independent claims61
78 paragraphs, as filed
The present invention relates generally to adaptive streaming, and more specifically to adaptive bit rate streaming of encoded media contained within a Matroska container file using a hypertext transfer protocol.
The term media streaming describes the playback of media on a playback device, which is stored on a server and continuously transmitted to the playback device over the network during playback. In general, the playback device completes playback of all buffered media prior to receiving the next piece of media, so that the playback device is sufficient for any given time during playback. A large amount of media is stored in the buffer to prevent interruption of playback. Adaptive bit rate streaming or adaptive streaming involves detecting this streaming state (eg, the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media as appropriate. In general, the source media is encoded at multiple bit rates, and the playback device or client switches between streaming different encodings, depending on the resources available.
Adaptive streaming solutions are commonly referred to as RFC2616, the Hypertext Transfer Protocol (HTTP) published by the Internet Engineering Task Force and the World Wide Web Consortium, or RFC2326, the Internet Engineering Task. Stream media between the server and the playback device using one of the Real Time Streaming Protocol (RTSP) published by Force. HTTP is a stateless protocol that allows playback devices to request a range of bytes in a file. HTTP is described as stateless because the server is not required to record information about the state of the playback device requesting information or the byte range requested by the playback device in order to respond to requests received from the playback device. Will be done. RTSP is a network control protocol used to control streaming media servers. The playback device issues control commands such as "play" and "pause" to the server that streams the media to control the playback of the media file. When RTSP is utilized, the media server records the state of each client device and determines the media to stream based on the instructions received from the client device and the state of the client.
In adaptive streaming systems, the source media is typically stored on the media server as a top-level index file that points to several alternative streams that contain the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions typically utilize different indexes and media containers. Synchronized Multimedia Integration Language (SMIL) developed by the World Wide Web Consortium is IIS Smooth Streaming developed by Microsoft Corporation (Redmond, Washington) and Flash Dynamic Streaming developed by Adobe Systems Incorporated (San Jose, California). Used to index in several adaptive streaming solutions, including. Apple Computer Developed by Incorporated (Cupertino, California), HTTP adaptive bitrate streaming generally uses extended M3U playback files (.M3U8), which are text files containing a list of URIs that identify media container files. Use to implement the index file. The most commonly used media container formats are the MP4 container format specified in MPEG-4 Part 14 (ie ISO / IEC 14496-14) and MPEG-2 Part 1 (ie ISO / IEC standard 13818-1). The MPEG Transport Stream (TS) container specified in. The MP4 container format is used in IIS smooth streaming and flash dynamic streaming. TS containers are used in HTTP adaptive bitrate streaming.
The Matroska container is a media container developed by the Matroska non-profit organization (Aussonne, France) as an open standard project. The Matroska container is based on the Extendable Binary Metalanguage (EBML), which is a binary derivative of the Extendable Markup Language (XML). Decoding of Matroska containers is supported by many consumer electronics (CE) devices. The DivX Plus file format developed by DivX, LLC (San Diego, California) makes use of extensions to the Matroska container format (ie, which is based on the Matroska container format but contains elements not specified within the Matroska format).
<p num="0006"> Disclosed are systems and methods for adaptive bit rate streaming of media stored within Matroska container files utilizing Hypertext Transfer Protocol (HTTP) according to embodiments of the present invention. In one embodiment, the processor is configured to request a portion of a file from a remote server via a client application. In addition, the client application further identifies multiple EBML container files, reads the top-level index data and parses the top-level index data, describing at least the maximum bit rate of the alternate stream contained within the EBML container file. And request information that identifies multiple EBML container files and request at least one part of the EBML container file that contains at least one element that specifies the encoding parameters of the stream contained within the EBML container file. And refer to each element that contains the encoded video part in at least one of the EBML container files, read the index, and use the index to contain the element that contains the encoded video part. Requests a portion of the first EBML container file, receives and buffers the requested element, and uses encoding parameters to decode the encoded video contained within the buffered element and now Measures the streaming state of the file and reads the element containing the encoded video portion for decoding based on the measured streaming state and the description of the bit rate of the alternate stream contained in the top-level data. Configure the processor to select another one of the EBML container files that should be.</p><p num="0007"> Further embodiments include a step of identifying each of the plurality of EBML container files and reading top-level index data that describes at least the maximum bit rate of each of the alternate streams contained within the EBML container file. At least one of the EBML container files that contains at least one element that specifies the encoding parameters of the stream contained within the EBML container file, with the steps to parse and obtain information that identifies multiple EBML container files. Encoded using the index, the step of requesting one part, the step of receiving an index, which references the element containing each encoded video part in at least one of the EBML container files. A step that requests a part of the first EBML container file, including an element that contains a part of the video, a step that receives and buffers the requested element, and an element that is buffered using encoding parameters. Separate the steps of decoding the encoded video contained within, the step of measuring the current streaming state, and the EBML container file that should receive the element containing the part of the encoded video for decoding. A step of selecting one of the above, the selection of which is based on a description of the measured streaming state and the bit rate of the alternate stream contained in the top-level index data.</p><p num="0008"> Another embodiment of the invention includes a processor configured to capture at least one multimedia file containing source video via a source encoding application. In addition, the source encoding application further selects parts of the source video and transcodes the selected parts of the source video into multiple alternative parts of the encoded video, each alternative with a different set of encoding parameters. Encoded using, starting with a closed picture group (GOP), starting with an intraframe, writing each alternative part of the encoded video to an element in a different EBML container file, each element is an alternative to the encoded video. Located within the EBML container file, which indicates the encoding parameters used to encode the parts, and also contains another element, an entry in each of the alternative parts of the encoded video within each of the EBML container files. Configure the processor to add to at least one index that identifies the location of the element that contains one.</p><p num="0009"> Another further embodiment uses the source encoder to repeatedly select a portion of the source video and uses the source encoder to convert the selected portion of the source video into multiple alternative parts of the encoded video. The steps to transcode, each alternative part is encoded using a different set of encoding parameters and starts with an intraframe that begins a closed picture group (GOP), using a step and a source encoder. , The step of writing each of the alternate parts of the encoded video to different elements of the EBML container file, where each element contains a set of encoding parameters that correspond to the encoding parameters used to encode the parts of the video. A step and an entry located within an EBML container file that also contains another element that contains one of the alternative parts of the encoded video within each of the EBML container files. Includes steps to identify, or at least add to one index.<u style="single"> The present invention provides, for example,:</u><u style="single">(Item 1)</u><u style="single"> A playback device configured for adaptive bitrate streaming using multiple alternative streams of encoded video packaged within an extensible binary markup language (EBML) container file. Each of the alternative streams starts with an intraframe where each part of the encoded video packed into the elements of the EBML container file begins a closed picture group (GOP), and each of the EBML container files is said to be The same source video encoded using different encoding parameters so that it contains at least one element that specifies the encoding parameters of the stream contained within the EBML container file.</u><u style="single"> The playback device is</u><u style="single"> It has a processor configured to request parts of a file from a remote server via a client application.</u><u style="single"> The client application</u><u style="single"> Identifying multiple EBML container files and reading top-level index data that describes at least the maximum bit rate of the alternate stream contained within the EBML container file.</u><u style="single"> Analyzing the top-level index data to obtain information that identifies the plurality of EBML container files,</u><u style="single"> Requesting at least one part of the EBML container file containing the at least one element that specifies the encoding parameter of the stream contained within the EBML container file.</u><u style="single"> Reading an index that references each element containing the encoded video portion of at least one of the EBML container files.</u><u style="single"> The index can be used to request a portion of the first EBML container file that contains an element containing an encoded video portion.</u><u style="single"> Receiving and buffering the requested element,</u><u style="single"> Using the encoding parameters to decode the encoded video contained within the buffered element,</u><u style="single"> Measuring the current streaming status and</u><u style="single"> For decoding, selecting another one of the EBML container files to read out the element containing the encoded video portion, the selection of which is the measured streaming state and the top level. Based on the description of the bit rate of the alternative stream contained in the data</u><u style="single"> A playback device that further configures the processor to do so.</u><u style="single">(Item 2)</u><u style="single"> Each of the EBML container files comprises a plurality of Cluster elements, each of which contains a portion of the encoded video.</u><u style="single"> The playback device of item 1, wherein the encoded video portion within each of the Cluster elements begins at an intraframe and comprises at least one closed picture group.</u><u style="single">(Item 3)</u><u style="single"> The playback device of item 2, wherein the encoded video portion within each of the Cluster elements has the same duration.</u><u style="single">(Item 4)</u><u style="single"> The playback device of item 2, wherein the encoded video portion within each of the Cluster elements has a duration of 2 seconds.</u><u style="single">(Item 5)</u><u style="single"> The reproduction according to item 2, wherein each Cluster element contains a timecode, and each encoded frame of the encoded video portion contained within the Cluster element is contained within a separate BlockGroup element. device.</u><u style="single">(Item 6)</u><u style="single"> The reproduction device according to item 5, wherein the first BlockGroup element in the Cluster element contains the intraframe.</u><u style="single">(Item 7)</u><u style="single"> The playback device according to item 6, wherein the first BlockGroup element includes a Block element that specifies a timecode attribute of the intraframe with respect to the timecode of the Cluster element.</u><u style="single">(Item 8)</u><u style="single"> The playback device according to item 1, wherein each of the alternative streams of the encoded video is encoded at a different maximum bit rate.</u><u style="single">(Item 9)</u><u style="single"> The playback device of item 8, wherein at least two of the alternative streams of the encoded video are in the same display aspect ratio, but are encoded using different resolutions using different sample aspect ratios.</u><u style="single">(Item 10)</u><u style="single"> At least one element in each of the EBML container files that specifies the encoding parameters of the stream contained within the EBML container file is the frame height, frame of the video stream contained within the EBML container file. The playback device according to item 9, which specifies the width and the sample aspect ratio.</u><u style="single">(Item 11)</u><u style="single"> The client application performs the frame height, frame width, and sample aspect ratio prior to starting decoding of the encoded video contained within the buffered element from one of the EBML container files. The playback device according to item 10, wherein the processor is configured to set.</u><u style="single">(Item 12)</u><u style="single"> The playback device according to item 8, wherein at least two of the alternative streams of the encoded video are encoded at different frame rates.</u><u style="single">(Item 13)</u><u style="single"> At least one element in each of the EBML container files that specifies the encoding parameters of the stream contained within the EBML container file specifies the frame rate of the video stream contained within the EBML container file. The playback device according to item 12.</u><u style="single">(Item 14)</u><u style="single"> The client application causes the processor to set the frame rate prior to the start of decoding of the encoded video contained within a buffered element from one of the EBML container files. The playback device according to item 13, which is configured.</u><u style="single">(Item 15)</u><u style="single"> The playback device according to item 11, wherein the client application configures the playback device to request a portion of a file from a remote server via a hypertext transfer protocol (HTTP) byte range request.</u><u style="single">(Item 16)</u><u style="single"> The top-level index data is SMIL data, and the information for identifying the plurality of EBML container files is a plurality of general-purpose resource indicators (URIs) for identifying the respective locations of the EBML container files, according to item 1. Playback device.</u><u style="single">(Item 17)</u><u style="single"> The playback device of item 1, wherein the SMIL data specifies the maximum bit rate of the encoded stream contained within the EBML container file identified by each of the URIs.</u><u style="single">(Item 18)</u><u style="single"> An index that references each of the elements in the EBML container file that contains the portion of the encoded stream is located within at least one element of the EBML container file.</u><u style="single"> The SMIL file specifies a portion of the EBML container file that contains elements that contain the encoding parameters.</u><u style="single"> The client application</u><u style="single"> Analyzing the SMIL file to identify parts of the EBML container file that contain elements that contain the encoding parameters.</u><u style="single"> Requesting a portion of the EBML container file that contains an element containing the encoding parameter</u><u style="single"> The playback device according to item 17, wherein the processor is configured to perform the above.</u><u style="single">(Item 19)</u><u style="single"> Each EBML container file contains at least one element containing an index that references each element containing the encoded video portion within the EBML container file.</u><u style="single"> The client application requests an portion of the EBML container file that contains at least one element that contains the index, thereby providing an index that references each element that contains an encoded video portion within the EBML container file. The playback device according to item 1, wherein the processor is configured to read.</u><u style="single">(Item 20)</u><u style="single"> The playback device of item 19, wherein the index is located within a Cue element in the EBML container file.</u><u style="single">(Item 21)</u><u style="single"> 20. The playback device of item 20, wherein the Cue element comprises a time attribute and includes a plurality of CuePoint elements pointing to the location of each element containing a portion of the encoded video in the EBML file.</u><u style="single">(Item 22)</u><u style="single"> The client application</u><u style="single"> Using the time attribute of the CuePoint element in the index to find the location of the element containing a particular part of the encoded video in the EBML file.</u><u style="single"> To request at least a portion of the EBML file that contains the element identified using the CuePoint element.</u><u style="single"> 21. The playback device according to item 21, wherein the processor is configured to perform the above.</u><u style="single">(Item 23)</u><u style="single"> The encoding parameter used to decode the encoded video is at least one selected from the group consisting of frame rate, frame height, frame width, sample aspect ratio, maximum bit rate, and minimum buffer size. The playback device according to item 1, which includes encoding parameters.</u><u style="single">(Item 24)</u><u style="single"> In item 1, the client application configures the processor to measure the current streaming state by measuring the time taken for the element to receive the requested element from the requested time. The playback device described.</u><u style="single">(Item 25)</u><u style="single"> The client application first requests a portion of the first EBML container file that contains at least one element that specifies the encoding parameter of the stream contained within the first EBML container file. Configure and</u><u style="single"> The client application is encoded so that the client application decodes from the second EBML container file based on the measured streaming state and the bit rate description of the alternate stream contained within the top level index. When choosing to read an element that contains a portion of a video, the second EBML container file that contains at least one element that specifies the encoding parameters of the stream contained within the second EBML container file. The playback device of item 1, further configuring the processor to require a portion.</u><u style="single">(Item 26)</u><u style="single"> The playback device according to item 1, wherein the playback device is further configured to perform adaptive bit rate streaming using an EBML container file containing a trick play track.</u><u style="single">(Item 27)</u><u style="single"> The client application configures the processor to select the EBML container file containing the trick play track as the EBML container file that should read the element containing the encoded video portion in response to a user instruction. The playback device according to item 26.</u><u style="single">(Item 28)</u><u style="single"> The playback device according to item 1, wherein the playback device is further configured to perform adaptive streaming using at least one EBML container file containing a track of encoded audio.</u><u style="single">(Item 29)</u><u style="single"> The client application reads the encoded audio from one of the at least one EBML container file containing the audio track, decodes it, and synchronizes the decoded audio with the decoded video. 28. The playback device that constitutes the processor.</u><u style="single">(Item 30)</u><u style="single"> The playback device according to item 1, wherein each of the EBML container files contains an element containing at least one encoded audio track multiplexed with the encoded video.</u><u style="single">(Item 31)</u><u style="single"> The client application</u><u style="single"> Reading and reading the portion of the EBML file that contains an element containing one portion of the encoded audio track.</u><u style="single"> Decoding the read and encoded audio,</u><u style="single"> Synchronizing the decoded audio with the decoded video</u><u style="single"> 28. The playback device according to item 28, which is configured to perform.</u><u style="single">(Item 32)</u><u style="single"> The playback device according to item 1, wherein the playback device is further configured to perform adaptive streaming using at least one EBML container file containing a subtitle track.</u><u style="single">(Item 33)</u><u style="single"> The client application</u><u style="single"> Reading subtitles from one of the at least one EBML container file containing the subtitle track,</u><u style="single"> Synchronizing the subtitles with the decoded video</u><u style="single"> Overlaying the subtitles on the decoded video</u><u style="single"> 32. The playback device according to item 32, wherein the processor is configured to perform the above.</u><u style="single">(Item 34)</u><u style="single">The playback device according to item 1, wherein each of the EBML container files contains an element containing at least one subtitle track multiplexed with the encoded video.</u><u style="single">(Item 35)</u><u style="single"> The client application</u><u style="single"> Reading a portion of the EBML file that contains an element containing one portion of the subtitle track</u><u style="single"> Synchronizing the subtitles with the decoded video</u><u style="single"> Overlaying the subtitles on the decoded video</u><u style="single"> 34. The playback device according to item 34, which is configured to perform.</u><u style="single">(Item 36)</u><u style="single"> A method of performing adaptive bitrate streaming of media on a playback device using multiple alternative streams of encoded video packaged in an extensible binary markup language (EBML) container file. Each of the alternative streams of video starts with an intraframe where each part of the encoded video packed into the elements of the EBML container file begins a closed picture group (GOP), and each of the EBML container files is said to be The same source video encoded using different encoding parameters so that it contains at least one element that specifies the encoding parameters of the stream contained within the EBML container file.</u><u style="single"> The method is</u><u style="single"> To identify each of the plurality of EBML container files and read the top-level index data describing at least the maximum bit rate of each of the alternative streams contained in the EBML container file.</u><u style="single"> Analyzing the top-level index data to obtain information that identifies the plurality of EBML container files,</u><u style="single"> Requesting at least one part of the EBML container file that contains at least one element that specifies the encoding parameter of the stream contained within the EBML container file.</u><u style="single"> Reading an index that references each element containing the encoded video portion of at least one of the EBML container files.</u><u style="single"> The index can be used to request a portion of the first EBML container file that contains an element containing an encoded video portion.</u><u style="single"> Receiving and buffering the requested element,</u><u style="single"> Using the encoding parameters to decode the encoded video contained within the buffered element,</u><u style="single"> Measuring the current streaming status and</u><u style="single"> For decoding, selecting another one of the EBML container files to read the element containing the encoded video portion, the selection is the measured streaming state and the top level. It is based on the description of the bit rate of the alternative stream contained in the index data.</u><u style="single"> Including, method.</u><u style="single">(Item 37)</u><u style="single"> A source encoder configured to encode source video as multiple alternative video streams packed within a container file, the container file being an extensible binary markup language (EBML) file.</u><u style="single"> The source encoder</u><u style="single"> It has a processor configured to capture at least one multimedia file containing the source video via a source encoding application.</u><u style="single"> The source encoding application</u><u style="single"> By selecting the part of the source video,</u><u style="single"> Transcoding the selected portion of the source video into multiple alternative portions of the encoded video, each alternative portion being encoded using a different set of encoding parameters and a group of closed pictures. GOP) Start with an intraframe, and</u><u style="single"> Each of the alternative parts of the encoded video is written to an element of a different EBML container file, each element indicating the encoding parameter used to encode the alternative part of the encoded video. It is located in the EBML container file, which also contains the elements of</u><u style="single"> Adding an entry to at least one index that identifies the location of an element in each of the EBML container files that contains one of the alternative parts of the encoded video.</u><u style="single"> A source encoder that configures the processor to do so.</u><u style="single">(Item 38)</u><u style="single"> 37. The source encoder according to item 37, wherein transcoding a selected portion of the source video further comprises transcoding the selected portion into at least one closed picture group.</u><u style="single">(Item 39)</u><u style="single"> 37. The source encoder according to item 37, wherein the source video portion is selected based on the duration of the selected portion of the source video.</u><u style="single">(Item 40)</u><u style="single"> 39. The source encoder according to item 39, wherein the source encoding application configures the processor to select a portion of the source video having a duration of 2 seconds.</u><u style="single">(Item 41)</u><u style="single"> 37. The source encoder according to item 37, wherein each of the alternative parts of the encoded video is encoded at a different maximum bit rate.</u><u style="single">(Item 42)</u><u style="single"> The source encoder according to item 41, wherein at least two of the alternative parts of the encoded video are encoded at different resolutions.</u><u style="single">(Item 43)</u><u style="single"> The source encoder according to item 41, wherein at least two of the alternative parts of the encoded video are encoded at different frame rates.</u><u style="single">(Item 44)</u><u style="single"> The element of the EBML container file into which each alternative part of the encoded video is written is a Cluster element containing the timecode, and the part of the encoded video is contained within the BlockGroup element within the Cluster element. , The source encoder according to item 37.</u><u style="single">(Item 45)</u><u style="single"> 44. The source encoder of item 44, wherein each encoded frame of the alternative portion of the encoded video contained within the Cluster element is contained within a separate BlockGroup element.</u><u style="single">(Item 46)</u><u style="single"> The source encoder according to item 45, wherein the first BlockGroup element in the Cluster element contains an IDR frame.</u><u style="single">(Item 47)</u><u style="single"> The source encoder according to item 46, wherein the first BlockGroup element includes a Block element that specifies a timecode attribute of the IDR frame with respect to the timecode of the Cluster element.</u><u style="single">(Item 48)</u><u style="single"> 37. The source encoder according to item 37, wherein each element to which each of the alternative parts of the encoded video is written is assigned the same time code.</u><u style="single">(Item 49)</u><u style="single"> 37. The source encoder according to item 37, wherein the source encoding application further configures the processor to create an index for each of the EBML container files.</u><u style="single">(Item 50)</u><u style="single"> The source encoding application said to add the location of the element containing one of the alternative parts of the encoded video in each of the EBML container files to the index for the EBML container file. The source encoder according to item 49, which further constitutes the processor.</u><u style="single">(Item 51)</u><u style="single"> The source encoder according to item 49, wherein the source encoding application further configures the processor to pack the index for each EBML container file into the EBML container file.</u><u style="single">(Item 52)</u><u style="single"> The source encoder according to item 51, wherein each index has a Cues element.</u><u style="single">(Item 53)</u><u style="single"> 52. The source encoder of item 52, wherein each Cue element comprises a CuePoint element that points to the location of the element that contains one of the alternative parts of the encoded video in the EBML file.</u><u style="single">(Item 54)</u><u style="single"> 37. The source encoder according to item 37, wherein the source encoding application further configures the processor to create a top-level index file that identifies each of the EBML container files.</u><u style="single">(Item 55)</u><u style="single"> The source encoder according to item 37, wherein the captured multimedia file also includes source audio.</u><u style="single">(Item 56)</u><u style="single"> 55. The source encoder according to item 55, wherein the source encoding application configures the processor to multiplex audio into each of the EBML container files.</u><u style="single">(Item 57)</u><u style="single"> 55. The source encoder according to item 55, wherein the source encoding application configures the processor to write the audio to a separate EBML container file.</u><u style="single">(Item 58)</u><u style="single"> 55. The source encoder according to item 55, wherein the source encoding application further configures the processor to transcode at least one of the at least one audio track.</u><u style="single">(Item 59)</u><u style="single"> The source encoder according to item 55, wherein the captured multimedia file further comprises subtitles.</u><u style="single">(Item 60)</u><u style="single"> The source encoder according to item 59, wherein the source encoding application configures the processor to multiplex the subtitles into each of the EBML container files.</u><u style="single">(Item 61)</u><u style="single"> The source encoder according to item 59, wherein the source encoding application configures the processor to write the subtitles to a separate EBML container file.</u><u style="single">(Item 62)</u><u style="single"> The source encoding application further configures the processor to transcode the source video to create a lower frame rate trick play track and write the trick play track to a separate EBML container file, item 37. Source encoder described in.</u><u style="single">(Item 63)</u><u style="single"> The source encoder according to item 62, wherein the trick play track also has a lower resolution than the source video.</u><u style="single">(Item 64)</u><u style="single"> 37. The source encoder according to item 37, wherein the source encoding application further configures the processor to write elements containing a set of encoding parameters into each of the EBML container files.</u><u style="single">(Item 65)</u><u style="single"> The source according to item 64, wherein the set of encoding parameters comprises at least one parameter selected from the group consisting of frame rate, frame height, frame width, sample aspect ratio, maximum bit rate, and minimum buffer size. Encoder.</u><u style="single">(Item 66)</u><u style="single"> A method of using a source encoder to encode a source video as multiple alternative streams packed into an extensible binary markup language (EBML) file.</u><u style="single"> The method is</u><u style="single"> Using the source encoder to repeatedly select parts of the source video,</u><u style="single"> The source encoder is used to transcode the selected portion of the source video into multiple alternative portions of the encoded video, each alternative portion using a different set of encoding parameters. Encoded and start with an intraframe that starts a group of closed pictures (GOP),</u><u style="single"> Using the source encoder, each of the alternative parts of the encoded video is written to an element of a different EBML container file, where each element is the encoding used to encode the part of the video. It is located in an EBML container file that also contains another element that contains a set of encoding parameters that correspond to the parameters.</u><u style="single"> Adding an entry to at least one index that identifies the location of an element in each of the EBML container files that contains one of the alternative parts of the encoded video.</u><u style="single"> Including methods.</u></p>
<figref num="1">FIG. 1 is a network diagram of an adaptive bit rate streaming system according to an embodiment of the present invention.</figref><figref num="2">FIG. 2 conceptually illustrates a top-level index file and a Matroska container file generated by the encoding of the source media according to an embodiment of the present invention.</figref><figref num="3">FIG. 3 conceptually illustrates a special Matroska container file that incorporates a modified Cue element according to an embodiment of the invention.</figref><figref num="4a">Figure 4a-4c conceptually illustrates the insertion of different types of media into a cluster element of a Matroska container file, subject to various constraints that facilitate adaptive bitrate streaming, according to embodiments of the present invention. ..</figref><figref num="4b">Figure 4a-4c conceptually illustrates the insertion of different types of media into a cluster element of a Matroska container file, subject to various constraints that facilitate adaptive bitrate streaming, according to embodiments of the present invention. ..</figref><figref num="4c">Figure 4a-4c conceptually illustrates the insertion of different types of media into a cluster element of a Matroska container file, subject to various constraints that facilitate adaptive bitrate streaming, according to embodiments of the present invention. ..</figref><figref num="4d">FIG. 4d conceptually illustrates the multiplexing of different types of media into a cluster element of a Matroska container file, subject to various constraints that facilitate adaptive bitrate streaming, according to an embodiment of the invention. ..</figref><figref num="4e">FIG. 4e conceptually illustrates the inclusion of trick play tracks in the Cluster element of a Matroska container file, subject to various constraints that facilitate adaptive bitrate streaming, according to certain embodiments of the present invention.</figref><figref num="5">FIG. 5 conceptually illustrates a modified Cue element of a special Matroska container file according to an embodiment of the invention, which allows the Cluster element to be read using an HTTP byte range request. Contains information to be.</figref><figref num="5a">FIG. 5a conceptually illustrates a modified Cue element of a special Matroska container file according to an embodiment of the invention, which is similar to the Cue element shown in FIG. 5, but with adaptive bits. Attributes that are not used during rate streaming are removed.</figref><figref num="6">FIG. 6 conceptually illustrates the indexing of Cluster elements in a special Matroska container file that utilizes modified CuePoint elements in a container file according to an embodiment of the present invention.</figref><figref num="7">FIG. 7 is a flow diagram illustrating the process for encoding source media for adaptive bit rate streaming according to an embodiment of the present invention.</figref><figref num="8">FIG. 8 shows between a playback device and an HTTP server associated with the initiation of streaming of encoded media contained within a Matroska container file indexed by a top-level index file, according to an embodiment of the invention. Communication is illustrated conceptually.</figref><figref num="9a">9a and 9b show the switching between streams in response to the streaming conditions experienced by the playback device and in response to the index information available to the playback device prior to the stream switching decision according to the embodiment of the present invention. The communication between the playback device and the HTTP server associated with is conceptually illustrated.</figref><figref num="9b">9a and 9b show the switching between streams in response to the streaming conditions experienced by the playback device and in response to the index information available to the playback device prior to the stream switching decision according to the embodiment of the present invention. The communication between the playback device and the HTTP server associated with is conceptually illustrated.</figref>
Next, with reference to the drawings, a system and method for adaptive bit rate streaming of media stored in a Matroska container file utilizing Hypertext Transfer Protocol (HTTP) according to an embodiment of the present invention is illustrated. There is. In some embodiments, the source media is encoded as some alternative stream. Each stream is stored in a Matroska (MKV) container file. In many embodiments, the Matroska container file is a specialized Matroska container file in that the media in each stream is encoded and the format stored in the container is constrained to improve streaming performance. In some embodiments, the Matroska container file has additional index elements (ie, elements not specified as part of the Matroska container format) to facilitate reading of the desired media during adaptive bitrate streaming. It is even more special in that it can be contained within a file. In some embodiments, each stream (ie, audio, video, or subtitles) is stored in a separate Matroska container file. In other embodiments, the encoded video stream is multiplexed with one or more encoded audio and / or subtitle streams within each Matroska container file. Top-level index files, which contain indexes to the streams contained within each of the container files, are also generated to allow adaptive bitrate streaming of encoded media. In many embodiments, the top-level index file is a Synchronized Multimedia Integration Language (SMIL) file that contains a URI for each of the Matroska container files. In other embodiments, all of the various file formats are used to generate top-level index files.
The performance of the adaptive bit rate streaming system according to embodiments of the present invention is that the video portion starts as a single (or at least one) closed picture group (GOP) starting at an Instant Decoding Update (IDR) frame. It can be significantly improved by encoding each part of the source video at each bit rate so that it is encoded within each stream. The GOP for each stream can then be stored as a Cluster element in the Matroska container file for the stream. In this way, the playback device can be switched between streams when playback is complete, and regardless of the stream from which the cluster is acquired, the first frame in the cluster is the IDR frame, which is in the Cluster element. It can be decoded without reference to any encoded media other than the contained encoded media. In many embodiments, all sections of the source video encoded as a GOP have the same duration. In some embodiments, each 2-second sequence of source video is encoded as a GOP.
During adaptive streaming, reading media using HTTP can be improved by adding additional index information to the Matroska container file used to contain each of the encoded streams. In some embodiments, the index is a simplified index in that the index points only to the IDR at the start of each cluster. In many embodiments, the index of the Matroska container file determines the size of each cluster so that the playback device can read the Cluster element from the Matroska container file over HTTP using a byte range request. Contains additional non-standard attributes that you specify (ie, attributes that do not form part of the Matroska container file format specification).
Adaptive streaming of the source media encoded as described above can be tuned by the playback device according to embodiments of the present invention. The playback device obtains information about each of the available streams from the top-level index file, selects one or more streams, and uses them for media playback. The playback device can then retrieve the header information from one or more bitstreams or a Matroska container file containing the stream, the header providing information about the decoding of the stream. The playback device can also request indexing information that indexes the encoded media stored within the associated Matroska container file. Index information can be stored in the Matroska container file, or separately from the Matroska container file, in the top-level index or in a separate index file. The index information allows the playback device to request a byte range from the server via HTTP, corresponding to the Cluster element in the Matroska container file that contains a particular piece of encoded media. As the playback device receives the Cluster element from the HTTP server, the playback device can evaluate the current streaming state and decide whether to increase or decrease the bit rate of the streamed media. If the playback device determines that a change within the bit rate is required, the playback device can obtain header and index information for the container file containing the desired stream (the playback device obtains this information). Assuming you haven't already obtained it). The index information can then be used to identify the byte range of the Cluster element that contains the next part of the source media encoded at the desired bit rate, and the identified Clus The ter element can be read from the server via HTTP. The next part of the requested source media is generally identified based on the Cluster element already requested by the playback device and the Cluster element buffered by the playback device. The next part of the source media requested by the alternate stream is that the playback device's buffer underflows (ie, media for playback) prior to the playback device receiving the Cluster element containing the next part of the source media. Is required to minimize the possibility of causing (depletion). In this way, the playback device uses top-level indexes and index information that describe the Cluster elements in each of the Matroska container files to sequentially read the Cluster elements from the various streams, depending on their suitability for the streaming state. Allows adaptive bitrate streaming to be achieved.
In some embodiments, bit rate variation between different streams can be achieved by modifying the encoding parameters for each stream, including, but not limited to, bit rate, frame rate, and resolution. .. When different streams contain different resolutions, the display aspect ratios of each stream are the same and the sample aspect ratios are modified to ensure a smooth transition from one resolution to another. The encoding of the source video for use in adaptive bitrate streaming and the playback of the encoded source video using the HTTP request to achieve adaptive bitrate streaming according to embodiments of the present invention are further described below. Be discussed.
(Adaptive streaming system architecture) An adaptive streaming system according to an embodiment of the present invention is illustrated in FIG. The adaptive streaming system 10 includes, as some alternative streams, a source encoder 12 configured to encode the source media. In the illustrated embodiment, the source encoder is a server. In other embodiments, the source encoder is any processing that includes a processor and sufficient resources to transcode the source media (including, but not limited to, video, audio, and / or subtitles). Can be a device. As further discussed below, the source encoding server 12 generates top-level indexes on a plurality of container files containing streams, at least a plurality of which are alternate streams. Alternate streams are streams that encode the same media content in different ways. In many cases, alternative streams encode media content (such as, but not limited to, video) at different bit rates. In some embodiments, the alternate stream is encoded at different resolutions and / or different frame rates. The top-level index file and container file are uploaded to HTTP server 14. The various playback devices can then request parts of the top-level index file and container file over a network 16 such as the Internet, using HTTP or another suitable stateless protocol.
In many embodiments, the top-level index file is an SMIL file and the media is stored within a Matroska container file. As further discussed below, media can be stored within Matroska container files to facilitate adaptive bitrate streaming of media. In many embodiments, the Matroska container file does not form a portion of the Matroska file format specification that facilitates the reading of certain parts of the media over HTTP during adaptive bitrate streaming of the media. A special Matroska container file that contains the element).
In the illustrated embodiment, the playback device includes a personal computer 18 and a mobile phone 20. In other embodiments, the playback device is to a server via a consumer electronics device, such as a DVD player, Blu-ray® player, television, set-top box, video game console, tablet, and HTTP. It can include other devices that can connect and play encoded media. The specific architecture is shown in FIG. 1, but any of any of the various architectures allows the playback device to request parts of the top-level index file and container file according to embodiments of the present invention. Can be used for.
(File structure) A file generated by a source encoder and / or stored on an HTTP server for streaming to a playback device according to an embodiment of the invention is illustrated in FIG. The files utilized within the adaptive bitrate streaming of the source media include a top-level index 30 and a plurality of container files 32, each containing at least one stream. The top-level index file describes the contents of each container file. As further discussed below, top-level index files can take various forms, including SMIL files, and container files can take various forms, including special Matroska container files.
In many embodiments, each Matroska container file contains a single stream. For example, a stream is one of several alternative video streams, an audio stream, one of several alternative audio streams, a subtitle stream, one of several alternative subtitle streams, a trick play stream. , Or can be one of several alternative trick playstreams. In some embodiments, the Matroska container file contains multiple multiplexed streams. For example, a Matroska container can contain a video stream and one or more audio streams, one or more subtitle streams, and / or one or more trick play streams. In many embodiments, the Matroska container file is a special file, as further discussed below. The encoding of the media and the way the media is stored in the Cluster element in the Matroska container file can be subject to constraints designed to improve the performance of adaptive bitrate streaming systems. In addition, Matroska container files can include index elements that facilitate the identification and download of Cluster elements from various Matroska container files during adaptive streaming of media. Top-level index files and Matroska container files that can be used in adaptive bitrate streaming systems according to embodiments of the present invention are discussed below.
(Top level index file) According to many embodiments of the invention, the playback device utilizes a top-level index file to identify a container file that contains a stream available to the playback device for use in adaptive bitrate streaming. In many embodiments, each top-level index file can include a reference to a container file that contains an alternate stream of encoded media. The playback device can utilize the information in the top-level index file to read the encoded media from each of the container files according to the streaming conditions experienced by the playback device.
In some embodiments, the top-level index file allows the playback device to read information about the encoding of the media within each of the container files and the index into the encoded media within each of the container files. , Provide information. In some embodiments, each container file contains information about the encoded media contained within the container file and an index to the encoded media within the container file, and the top-level index file contains this information. The part of each container file contained is shown. Therefore, the playback device reads the top-level index file and uses the top-level index file to include information about the encoded media contained within the container file and an index to the encoded media within the container file. , You can request one or more parts of a container file. Various top-level index files that can be used in adaptive bitrate streaming systems according to embodiments of the present invention are further discussed below.
(Top level index SMIL file) In some embodiments, the top-level index file utilized in adaptive bitrate streaming of media is an XML file, which contains a URI that describes each of the streams and a list of container files that contain the streams. Contains XML files. The URI can include information, such as the "system-bit rate" of the stream contained within the stream and the location of a particular piece of data in the container file.
The basic structure of an SMIL file involves providing an XML declaration and SMIL elements. The SMIL element defines the streams available for use in adaptive bitrate streaming, with only the HEAD element, which is generally left empty, and the PAR (parallel) element, in general. Contains, including BODY elements. The PAR element describes a stream (ie, including media that can be presented at the same time) that can be played at the same time.
The SMIL specification defines several child elements for the PAR element that can be used to specify the streams available for use in adaptive bitrate streaming. The VIDEO, AUDIO, and TEXTSTREAM elements can be used to define a particular video, audio, or subtitle stream. The VIDEO, AUDIO, and TEXTSTREAM elements can be collectively referred to as media objects. The basic attributes of a media object are the SRC attribute, which specifies the full path or URI to the container file containing the associated stream, and the XML: LANG attribute, which contains a three-letter language code. Additional information about the media object can be specified using the PARAM element. PARAM elements are the standard method within the SMIL format for providing common name / value pairs. In some embodiments of the invention, a particular PARAM element is defined to be utilized during adaptive bitrate streaming.
In many embodiments, the "header-request" PARAM element is defined to specify the size of the header section of the container file that contains the stream. The value of the "header-request" PARAM element generally specifies the number of bytes between the start of the file and the start of the encoded media in the file. In many embodiments, the header contains information about the format in which the media is encoded, and the playback device is used to play the encoded media in order to be able to configure a decoder for playing the encoded media. Read the header first. An example of a "header-request" PARAM element is
<maths num="1"><img id="000002" he="25" wi="55" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
In some embodiments, the "mime" PARAM element is defined to specify the MIME type of the stream. A "mime" PARAM element that identifies a stream as an H.264 stream (ie, a stream encoded according to the MPEG-4 Advanced Video Codec Standard)
<maths num="2"><img id="000003" he="27" wi="94" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
The MIME type of a stream can be specified using the "mime" PARAM element, depending on the suitability for the encoding of a particular stream (eg, AAC audio or UTF-8 text stream).
When the media object is a VIDEO element, the additional attributes are the systemBitrate attribute, which specifies the bit rate of the stream in the container file identified by the VIDEO element, and the width and height, which specifies the dimensions of the encoded video in pixels. Defined in the SMIL file format specification that contains the attribute. Additional attributes can also be defined using the PARAM element. In some embodiments, the "vbv" PARAM element is defined to specify the VBV buffer size of the video stream in bytes. A video buffering verifier (VBV) is a theoretical MPEG video buffer model used to ensure that an encoded video stream is accurately buffered and can be played on a decoder device. An example of a "vbv" PARAM element that specifies a VBV size of 1000 bytes is
<maths num="3"><img id="000004" he="25" wi="47" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
Examples of VIDEO elements, including the attributes described above, are:
<maths num="4"><img id="000005" he="57" wi="94" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
An adaptive bitrate streaming system according to an embodiment of the present invention can accommodate trick play streams, which provides a smooth visual search of source content encoded for adaptive bitrate streaming. Can be used for. The trick play stream can be encoded as if visual search is accelerated through the source media during playback, but in reality, the trick play stream is simply a separate encoding of the source media at a lower frame rate. It's a truck. In many embodiments of the system, the reference to the trick play track is indicated by the systemProfile attribute of the VIDEO element. In other embodiments, any of the various techniques can be used to indicate within the top-level index file that a particular stream is a trick play stream. An embodiment of a trick playstream VIDEO element according to an embodiment of the present invention
<maths num="5"><img id="000006" he="43" wi="135" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
In some embodiments of the invention, a "reserve dBandwidth" PARAM element can be defined for the AUDIO element. The "reserve dBandwidth" PARAM element specifies the bit rate of the audio stream in Kbps. Examples of AUDIO elements specified according to certain embodiments of the present invention are
<maths num="6"><img id="000007" he="42" wi="97" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
In some embodiments, the "reserve dBandwidth" PARAM element is also defined for the TEXTSTREAM element. An example of a TEXTSTREAM element, including a "reserve dBandwidth" PARAM element, according to an embodiment of the present invention.
<maths num="7"><img id="000008" he="42" wi="87" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
In other embodiments, any of the various mechanisms can be utilized to specify information about the VIDEO, AUDIO, and SUBTITLE elements, depending on their suitability for a particular application.
The SWITCH element is a mechanism defined within the SMIL file format specification that can be used to define adaptive or alternate streams. An example of a format in which the SWITCH element can be used to specify an alternative video stream at a different bit rate is
<maths num="8"><img id="000009" he="29" wi="114" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
The SWTICH element specifies the URLs of the three alternate video streams. The filename indicates a different bit rate for each of the streams. As further discussed below, the SMIL file format specification is utilized in accordance with embodiments of the invention to specify additional information about the stream and the container file in which it is contained within the top-level index SMIL file. Can provide a mechanism.
In many embodiments of the invention, the EXCL (exclusion) element is used to define alternative tracks that do not adapt during playback with streaming conditions. For example, the EXCL element can be used to define an alternative audio track or an alternative subtitle track. Examples of formats in which the EXCL element can be used to specify alternative English and French audio streams are:
<maths num="9"><img id="000010" he="41" wi="87" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
An example of a top-level index SMIL file that defines two alternative video levels, audio stream and subtitle stream attributes and parameters according to an embodiment of the invention.
<maths num="10-1"><img id="000011" he="231" wi="148" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>
<maths num="10-2"><img id="000012" he="68" wi="141" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Is.
Top-level index SMIL files can be generated when the source media is encoded for playback via adaptive bitrate streaming. Alternatively, a top-level index SMIL file can be generated when the playback device requests the start of playback of the encoded media. When the playback device receives the top-level index SMIL file, the playback device can parse the SMIL file and identify the available streams. The playback device can then select the stream and use it to play the content, use the SMIL file to identify parts of the container file, get information about the encoding of a particular stream, and / Or can be downloaded to get an index to the encoded media in the container file.
The top-level index SMIL file is described above, but any of the various top-level index file formats is used to create a top-level index file, depending on its suitability for a particular application, according to certain embodiments of the invention. Can be used for. The use of top-level index files to enable playback of encoded media using adaptive bitrate streaming according to embodiments of the present invention is further discussed below.
(Store media in MATROSKA file for adaptive bitrate streaming) A Matroska container file used to store encoded video according to an embodiment of the invention is illustrated in FIG. Container file 32 is an extensible binary markup language (EBML) file that is an extension of the Matroska container file format. The special Matroska container file 32 contains standard EBML elements 34 and standard Seek. Includes a Head element 40, a standard Segment information element 42, and a standard Segment element 36 that includes a standard Tracks element 44. These standard elements describe the media contained within the Matroska container file. Segment element 36 also includes standard Clusters element 46. As described below, the manner in which the encoded media is inserted into the individual Cluster elements 48 within the Clusters element 46 is constrained to improve media playback within the adaptive streaming system. In many embodiments, the constraints placed on the encoded video are consistent with the Matroska container file format specification, encoding the video so that each cluster contains at least one closed GOP starting at an IDR frame. Accompanied by that. In addition to the standard elements mentioned above, Segment element 36 also contains a modified version of standard Cues element 52. As further discussed below, the Cues element contains a special CuePoint element (ie, a non-standard CuePoint element) that facilitates the reading of media contained within a particular Cluster element over HTTP.
Constraints imposed on media encoding for adaptive bitrate streaming and the format of the encoded media within the Clusters element of the Matroska container file, as well as additional indexes inserted within the container file, according to embodiments of the present invention. The information is further discussed below.
(Media encoding for insertion inside the Cluster element) The adaptive bitrate streaming system provides the playback device with the option of making selections between different streams of encoded media during playback, according to the streaming conditions experienced by the playback device. In many embodiments, switching between streams separates and pre-encodes discrete parts of the source media according to the encoding parameters of each stream, and then each separately encoded part is its own within the stream's container file. Facilitated by including it within the Cluster element of. In addition, the media contained within each cluster is encoded so that the media is playable without reference to media contained within any other cluster in the stream. Thus, each stream contains a Cluster element that corresponds to the same discrete portion of the source media, and from time to time, the playback device selects the Cluster element from the stream that is most suitable for the streaming state experienced by the playback device. And can start playing the media contained in the Cluster element. Therefore, the playback device can select clusters from different streams as the streaming state experienced by the playback device changes over time. In some embodiments, the Cluster elements are further constrained so that each Cluster element contains a portion of media encoded from source media with the same duration. In some embodiments, each Cluster element comprises 2 seconds of encoded media. The specific constraints that apply to the media encoded within each Cluster element, depending on the type of media (ie, video, audio, or subtitles) are discussed below.
The Clusters element of a Matroska container file containing a video stream according to an embodiment of the invention is illustrated in Figure 4a. Each Clusters element 46 contains a plurality of Cluster elements 48 containing discrete parts of the encoded video. In the illustrated embodiment, each Cluster element 48 comprises 2 seconds of encoded video. In other embodiments, the Cluster element comprises an encoded video having more than or less than 2 seconds. The smaller the Cluster element (ie, the shorter the duration of the encoded media within each Cluster element), the higher the overhead associated with requesting each Cluster element. Therefore, the responsiveness of the playback device to changes in streaming state and the effective data rate of the adaptive streaming system for a given set of streaming states (ie, actually utilized to transmit encoded media). There is a trade-off with (the portion of available bandwidth). In some embodiments, the encoded video sequences within the Cluster element have different durations. Each Cluster element 48 includes a Cluster element and a Timecode element 60 that indicates the start time of the encoded video within the multiple BlockGroup elements. As mentioned above, the encoded video stored in the cluster is constrained so that it can be played without reference to the encoded video contained within any of the other Cluster elements in the container file. Will be done. In many embodiments, the encoding of the video contained within the Cluster element, as a GOP, where the first frame is the IDR frame, imposes constraints. In the illustrated embodiment, the first BlockGroup element 62 contains an IDR frame. Therefore, the first BlockGroup element 62 does not include the ReferenceBlock element. The first BlockGroup element 62 Includes Block element 64, which specifies the Timecode attribute of the frame encoded in Block element 64 for the Timecode of Cluster element 48. Subsequent BlockGroup elements 66 are the frames they can contain.The type is not restricted (except that you cannot refer to frames that are not contained within the Cluster element). Thus, the subsequent BlockGroup element 66 can include a ReferenceBlock element 68, or can contain an IDR frame, which references other BlockGroup elements used in decoding the frames contained within the BlockGroup. Similar to BlockGroup element 62 in 1. As mentioned earlier, the way the encoded video is inserted into the Cluster element of the Matroska file conforms to the Matroska file format specification.
Insertion of encoded audio and subtitle information into Cluster element 46 of a Matroska container file according to an embodiment of the invention is illustrated in Figures 4b and 4c. In the illustrated embodiment, the encoded media is inserted into the Cluster element 48 subject to the same constraints that apply to the encoded video described above in connection with FIG. 4a. In addition, the duration of the encoded audio and subtitle information in each Cluster element corresponds to the duration of the encoded video in the corresponding Cluster element of the Matroska container file containing the encoded video. In other embodiments, the Cluster element in the container file containing the audio and / or subtitle stream does not need to correspond to the start time and duration of the Cluster element in the container file containing the alternate video stream.
(Stream multiplexing in a single MKV container file) The cluster elements shown in Figure 4a-4c assume that a single stream is contained within each Matroska container file. In some embodiments, media from multiple streams is multiplexed within a single Matroska container file. Thus, a single container file can contain a video stream multiplexed with one or more corresponding audio streams and / or one or more corresponding subtitle streams. Reserving the stream in this way can result in duplication of audio and subtitle streams across multiple alternative video streams. However, the search time for reading encoded media from the video stream and associated audio and / or subtitle stream can be reduced due to the adjacent storage of data on the server. A cluster element 46 of a Matroska container file containing multiplexed video, audio, and subtitle data according to an embodiment of the invention is illustrated in FIG. 4d. In the illustrated embodiment, each Cluster element 48 includes an additional BlockGroup element for each of the multiplexed streams. The first Cluster element contains an encoded video frame and contains a Block element 64v that indicates the frame's Timecode attribute (ie, Timecode attribute 60) with respect to the start time of the Cluster element. Contains the BlockGroup element 62v of. The second BlockGroup element 62a contains an encoded audio sequence, the third BlockGroup element 62s contains a Block element 64a indicating the timecode of the encoded audio relative to the start time of the Cluster element, and the third BlockGroup element 62s contains encoded subtitles. It also contains Block elements 64s that indicate the Timecode of the encoded subtitles for the start time of the Cluster element. Implementations not shown, but shown In form, each Cluster element 48 is likely to contain an additional BlockGroup element that contains additional encoded video, audio, or subtitles. The same restrictions apply to encoded media, regardless of the multiplexing of encoded video, audio, and / or subtitle streams.
(Incorporating trick play tracks into MKV container files for use in adaptive bitrate streaming systems) Incorporation of trick play tracks into Matroska container files by DivX, LLC, filed October 29, 2008, U.S. Patent Application No. 12 / 260,404, "Application Enhancement." Proposed in Tracks, the disclosure of which is incorporated herein by reference in its entirety. A trick play track similar to the trick play track described in U.S. Patent Application No. 12 / 260,404 provides a trick play stream to an adaptive bit rate streaming system according to an embodiment of the present invention. Can be used to provide a smooth visual search through encoded source content for. A separate trick play track can be encoded through the source media as if the visual search was accelerated during playback, but in reality, the trick play track simply puts the source media at a lower frame rate. A separate track to encode. In some embodiments, Trick Playstream is a U.S. Patent Application No. 12/260, Created by generating a trick play track and inserting it into a Matroska container file, subject to the above constraints on inserting a video stream into a Matroska container file, as outlined in issue 404. File. In many embodiments, the trick play track is also subject to the additional constraint that each frame in the GOP of each Cluster element in the trick play track is encoded as an IDR frame. Like other video streams, each Cluster element contains the same 2 second source media GOP as the corresponding Cluster element in the other stream. Within the GOP of the trick play track, there are simply fewer frames, and each frame has a longer duration. Thus, the transition to and from the trick play stream is the same as the transition between any of the other encoded streams is processed within the adaptive bitrate streaming system according to embodiments of the present invention. Can be processed in the following way. To achieve visual search acceleration, the playback of frames contained within a trick play track is generally such that the playback device has the desired increase in the rate of search acceleration (eg x2, x4, x6). Etc.), which involves manipulating the timecode assigned to the frame of the encoded video prior to providing the frame to the decoder of the playback device.
The Cluster element containing the media encoded from the trick play track is shown in Figure 4e. In the illustrated embodiment, the encoded trick play is inserted within the Cluster element 48, subject to the same constraints that apply to the encoded video described above in connection with FIG. 4a. However, each Block element contains an IDR. In other embodiments, the Cluster element in the container file containing the trick play track does not have to correspond to the start time and duration of the Cluster element in the container file containing the alternate video stream.
In many embodiments, the source content can be encoded to provide a single trick play track or multiple trick play tracks for use by an adaptive bitrate streaming system. When a single trick play track is provided, the trick play track is generally encoded at a low bit rate. Adaptive rate streaming can also be performed on trick play tracks when multiple alternative trick play tracks are provided. In some embodiments, multiple trick play tracks are provided through the encoded media to accommodate the acceleration of visual search at different rates.
(Incorporation of indexing information into MKV container file) The specification for the Matroska container file format provides an optional Cue element used to index the Block element in the container file. A modified Cue element 52, according to an embodiment of the invention, that can be incorporated into a Matroska container file and can use HTTP to facilitate cluster requests by playback devices is illustrated in FIG. Each modified Cue element 52 contains a plurality of CuePoint elements 70, each containing a CueTime attribute 72. Each CuePoint element contains a CueTrackPosition element 74 that contains the CueTrack76 and CueClusterPosition78 attributes. In many embodiments, the CuePoint element is primarily configured to identify a particular Cluster element as opposed to a particular Block element within the Cluster element. However, some uses require the ability to search for a specific BlockGroup element within the Cluster element, and additional index information is included within the Cue element.
The use of a modified Cues element to index the encoded media within the Clusters element of a Matroska file according to an embodiment of the invention is illustrated in FIG. CuePoint elements are generated for each Cluster element in the Matroska container file. The CueTime attribute 72 of the CuePoint element 70 corresponds to the Timecode attribute 60 of the corresponding Cluster element 48. In addition, the CuePoint element contains a CueTrackPosition element 74 with a CueClusterPosition attribute 78 pointing to the start of the corresponding Cluster element 48. CueTrackPosition element 74 can also include a CueBlockNumber attribute, which is commonly used to indicate a Block element containing the first IDR frame within Cluster element 48.
For easy understanding, the modified Cue element 52 forms an index on each of the Cluster elements 48 in the Matroska container file. In addition, the CueTrackPosition element provides information that can be used by the playback device to request a byte range for a particular Cluster element 48 from a remote server via HTTP or another suitable protocol. The Cue element of a traditional Matroska file does not provide the playback device with information about the number of bytes to request from the start of the Cluster element in order to directly retrieve all of the encoded video contained within the Cluster element. .. The size of the Cluster element can be estimated within the modified Cue element by using the CueClusterPosition attribute of the CueTrackPosition element, which indexes the first byte of the next Cluster element. Alternatively, an additional CueTrackPosition element is added to the modified Cue element according to embodiments of the present invention, which indexes the last byte of the Cluster element (in addition to the CueTrackPosition element that indexes the first byte of the Cluster element). A non-standard CueClusterSize attribute that specifies the size of the Cluster element to obtain and / or pointed to by the CueClusterPosition attribute is contained within each CueTrackPosition element and is identified in the Matroska container file via an HTTP byte range request or similar protocol. Assists in reading the Cluster element of.
Modifications to the Cue element as described above significantly simplify the reading of the Cluster element from the Matroska container file over HTTP or similar protocols during adaptive bitrate streaming. In addition, indexing only the first frame in each cluster will significantly reduce the size of the index. Assuming that the index is generally downloaded prior to playback, a reduction in the size of the Cue element (ie, the index) means that playback can begin more quickly. Using the CueClusterPosition element, the playback device is best suited for the streaming conditions experienced by the playback device, simply by referencing the index of the associated Matroska container file, using the Timecode attribute for the desired Cluster element. Streams can request specific Cluster elements.
In some embodiments, some attributes within the Cue element are not utilized during adaptive bitrate streaming. Therefore, the Cue element can be further modified by removing unused attributes and reducing the overall size of the index for each Matroska container file. A modified Cue element that may be available in a Matroska container file, including a single encoded stream, according to an embodiment of the invention is illustrated in Figure 5a. The Cue element 52'shown in Figure 5a is similar to the Cue element 52 shown in Figure 5, but the CuePoint element 70'does not contain the CueTime attribute (see 72 in Figure 5) and / or the CueTrackPosition element 74'. Does not include the CueTrack attribute (76 in Figure 5). The CueTime attribute is not required when the encoded media pieces within each Cluster element in the Motroska container file have the same duration. The CueTrack attribute is not required when Matroska contains a file containing a single encoded stream. In other embodiments, the Cue element and / or other elements of the Matroska container file are contained within the Matroska container file, assuming that the stream is encoded and inserted into the Matroska container file. Can be modified to remove elements and / or attributes that are not required for adaptive bit rate streaming.
Although various modifications to the Cue element, including information about the size of each Cluster element in the Matroska container file and eliminating unnecessary attributes, are mentioned above, many embodiments of the invention are conventional Matroska. Use a container. In some embodiments, the playback device simply uses the information obtained from the traditional Cues element to size the Cluster element on the fly and / or the size of the Cluster element in the MKV container file. Rely on a separate index file that contains information about and / or location. In some embodiments, additional index information is stored within the top-level index file. In some embodiments, additional index information is stored in a separate file identified within the top-level index file. When the index information used to read the Cluster element from the Matroska container file is stored separately from the container file, the Matroska container file is still generally contained within the Cluster element, as described above. Because of, it is constrained to encode the media. In addition, wherever the index information is located, the index information generally indexes each Cluster element, at least where each Cluster element starts, and in many cases information about its size (but limited to that). Will not be included).
(Source media encoding for adaptive bitrate streaming) A process for encoding source media as a top-level index file and multiple Matroska container files for use in an adaptive bitrate streaming system according to an embodiment of the invention are illustrated in FIG. The encoding process 100 begins with the step of selecting the first part of the source media (102) and the step of encoding the source media with the encoding parameters for each stream (104). When the part of the media is video, the part of the source video is encoded as a single GOP starting with an IDR frame. In many embodiments, the encoding parameters used to create the alternative GOP will vary based on the bit rate, frame rate, encoding parameters, and resolution. In this way, the piece of media is encoded as a set of compatible alternatives, allowing the playback device to select the most appropriate alternative for the streaming conditions experienced by the playback device. When corresponding to different resolutions, the stream encoding is constrained so that each stream has the same display aspect ratio. A constant display aspect ratio can be achieved across different resolution streams by varying the sample aspect ratio with the resolution of the stream. In many cases, reduced resolution can result in higher quality video compared to higher resolution video encoded at the same bit rate. In many embodiments, the source media itself is encoded and the encoding process (104) transcodes or translates the encoded source media according to the respective encoding parameters of the alternate stream supported by the adaptive bitrate streaming system. Including that.
When the source media is encoded as an alternative part of a set of encoded media, each alternative part of the encoded media is in the Cluster element in the Matroska container file that corresponds to the stream to which the encoded media part belongs. Is inserted into (106). In many embodiments, the encoding process also builds an index for each Matroska container file as the media is inserted into the Cluster element inside the container. Therefore, process 100 can also include the step of creating a CuePoint element that points to the Cluster element that will be inserted into the Matroska container file. CuePoint elements can be kept in the buffer until the source media is fully encoded. Although the process described above describes the steps of sequentially encoding each of the alternative parts of the encoded media in a single pass through the source media, many embodiments of the invention are separate through the source media. It involves making a pass and encoding each of the alternate streams.
Looking back at Figure 7, the process continues to select (102) and encode (104) parts of the source media until the entire source media is encoded for adaptive bitrate streaming (108). Then insert the encoded portion of the media into the Matroska container file that corresponds to the appropriate stream (106). At that point, for each stream, the process inserts an index into the Matroska container (110) and creates a top-level index file that indexes each of the encoded streams contained within the Matroska container file (112). )can do. As mentioned earlier, the index is created as encoded media and can be inserted into the Matroska container file so that the CuePoint element indexes each Cluster element in the Matroska container file. Depending on the completion of encoding, each CuePoint element can be contained within a Cue element, and the Cue element can be inserted into the Matroska container file following the Cluster element.
Top-level indexing each of the streams in the Matroska container file, with the Matroska container file containing each of the streams generated during the encoding process, which can include the generation of the trick play stream following the encoding of the source media. After generating the index file, the top-level index file and Matroska container file can be uploaded to the HTTP server for adaptive bit rate streaming to the playback device. Adaptive bitrate streaming of media encoded according to embodiments of the invention using HTTP requirements is further discussed below.
(Adaptive bitrate streaming from MKV container files using HTTP) Media contained within a Matroska container file when the source media is encoded so that there is an alternate stream contained within a separate Matroska container file for at least one of the video, audio, and subtitle content. Adaptive streaming can be achieved using HTTP requests or similar stateless data transfer protocols. In many embodiments, the playback device requests a top-level index file that resides on the server and uses index information to identify the streams available to the playback device. The playback device can then read the index for one or more of the Matroska files and use HTTP requests or similar stateless protocols to read one or more of the streams contained within the Matroska container file. Indexes can be used to request media from. As mentioned above, many embodiments of the invention use modified Cue elements to implement an index for each of the Matroska container files. However, in some embodiments, the encoded media for each stream is contained within a standard Matroska container file, and a separate index file can also be provided for each of the container files. Based on the streaming conditions experienced by the playback device, the playback device can select media from alternative streams encoded at different bit rates. When media from each of the streams is inserted into the Matroska container file, as described above, transitions between streams can occur upon completion of playback of the media within the Cluster element. Therefore, the size of the Cluster element (ie, the duration of the encoded media within the Cluster element) is generally the playback device streaming. It is selected so that it can respond quickly enough to commands from users that involve changing conditions and the use of trick play tracks. The smaller the Cluster element (ie, the shorter the duration of the encoded media within each Cluster element), the higher the overhead associated with each Cluster element's request. Therefore, the responsiveness of the playback device to changes in streaming state and the effective data rate of the adaptive streaming system for a given set of streaming states (ie, actually utilized to transmit encoded media). There is a trade-off with (the portion of the available bandwidth). In many embodiments, the size of the Cluster elements is chosen so that each Cluster element contains 2 seconds of encoded media. In other embodiments, the duration of the encoded media can be greater than or less than 2 seconds, and / or the duration of the encoded media can vary from cluster element to cluster element.
According to certain embodiments of the present invention, communication between a playback device or client and an HTTP server during playback of encoded media in a separate stream contained within a Matroska container file indexed by a top-level index file. , Illustrated in FIG. In the illustrated embodiment, the playback device 200 initiates playback by requesting a top-level index file from server 202 using an HTTP request or a similar protocol to receive the data. Server 202 provides the bytes corresponding to the request. The playback device 200 then parses the top-level index file to identify the respective URI of the Matroska container file that contains the stream of encoded media derived from a particular piece of source media. The playback device can then request a byte range corresponding to one or more headers of the Matroska container file via HTTP or a similar protocol, and the byte range is within the URI for the associated Matroska container file. Determined using the information contained in (see discussion above). The server responds to a request for a byte range containing the header of the Matroska container file,
<maths num="11"><img id="000013" he="50" wi="95" file="JP6038805B2_D0001.tif" img-format="tif" img-content="drawing" /></maths>Returns the information of.
EBML elements are generally processed by the playback device to ensure that they correspond to the file version. The SeekHead element is parsed to find the location of the Matroska index element, and the SegmentInfo element contains two important elements utilized in playback, namely TimecodeScale and Duration. TimecodeScale specifies the timecode scale for all timecodes in the Segment of the Matroska container file, and Duration specifies the duration of the Segment based on the TimecodeScale. The Tracks element contains information used by the playback device to decode the encoded media contained within the Clusters element of the Matroska file. As mentioned above, the adaptive bitrate streaming system according to embodiments of the present invention can accommodate different streams encoded using different encoding parameters, including, but not limited to, frame rate and resolution. .. Thus, the playback device can use the information contained within the header of the Matroska container file to configure the decoder each time a transition is made between the encoded streams.
In many embodiments, the playback device does not receive headers for all of the Matroska container files indexed within the top-level index file. Instead, the playback device first determines the stream that will be used to initiate playback and requests a header from the corresponding Matroska container file. Depending on the structure of the URI contained within the top-level index file, the playback device uses either the information from the URI or the information from the header of the Matroska container file to at least the index from the associated Matroska container file. A byte range can be requested from the server containing the portion. The byte range can cover the entire index. The server provides the playback device with a related byte range that contains the index information, and the playback device uses the index information to request the byte range of the Cluster element that contains the media encoded using this information. be able to. When the Cluster element is received, the playback device can extract the encoded media from the Block element within the Cluster element and decode and play the media within the Block element according to its associated Timecode attribute. it can.
In the illustrated embodiment, the playback device 200 is sufficiently indexed prior to the start of playback so that the playback device can use the index information to stream the entire of each of the selected streams. Request information from the HTTP server. In another embodiment, the playback device continuously reads index information as the media is played. In some embodiments, all index information for the lowest bitrate stream is played so that it is available to the playing device if the streaming state suddenly deteriorates during playback. Required prior to.
(Switching between streams) The communication illustrated in FIG. 8 assumes that the playback device continues to request media from the same stream (ie, the Matroska container file) throughout the playback of the media. In fact, the streaming state experienced by the playback device is likely to change during playback of the streaming media, and the playback device requests the media from an alternate stream (ie, a different Matroska container file) and is experienced by the playback device. It is possible to provide the best picture quality for the streaming state to be done. In addition, the playback device may switch streams to perform a trick play function that utilizes the trick play track stream.
Communication between the playback device and the server when the playback device switches to a new stream according to an embodiment of the invention is illustrated in FIG. 9a. The communication illustrated in Figure 9a is an old stream while index information for the new stream has not previously been requested by the playback device and information is being retrieved for the Matroska container file containing the new stream. Suppose the download of the Cluster element from is proceeding. When the playback device 200 detects a change in the streaming state and determines that a higher bit rate stream is available in this streaming state, or receives a trick play instruction from the user, the playback device is at the top level. The index file can be used to identify the URI for a more suitable alternative stream for at least one of the video, audio, or subtitle streams that the playback device is currently requesting encoded media. it can. The playback device can store information about the current stream and use the corresponding URI parameters to request a byte range of headers for the Matroska container file containing the new stream. Caching information in this way can be useful when the playback device attempts to adapt the bit rate of the stream downwards. When the playback device experiences a decrease in available bandwidth, the playback device will ideally switch quickly to a lower bit rate stream. Due to the bandwidth reduction experienced by the playback device, the playback device is unlikely to have additional bandwidth to request header and index information. Ideally, the playback device would use the full available bandwidth to download the higher rate Cluster elements already requested and use the locally cached index information to use the lower bitrate stream. Matroska container containing
A byte range for index information for the Matroska container file containing the new stream can be requested from HTTP server 202, as described above in connection with FIG. At that point, the playback device can stop downloading the Cluster element from the previous stream and use the index information from the Matroska container file to use the index information from the Matroska container file that contains the new stream from the HTTP server. It is possible to initiate a request for a byte range of the appropriate Cluster element and identify the Cluster element that contains the encoded media following the encoded media in the last Cluster element read by the playback device. As mentioned above, a smooth transition from one stream to another is facilitated by encoding each of the alternate streams so that the corresponding Cluster element starts with the same Timecode element and IDR frame.
When the playback device caches the entire header and index for each stream used in playing the media, the process of reverting to the previously used stream can be simplified. The playback device already has header and index information for the Matroska file that contains the previously used stream, and the playback device simply uses this information to use the previously used stream over HTTP. You can initiate a request for a Cluster element from a Matroska container file. Communication between the playback device and the HTTP server when the playback device reverse-switches the header and index information to a cached stream according to an embodiment of the present invention is illustrated in FIG. 9b. The process illustrated in Figure 9b is ideally performed when adjusting the bit rate downwards, as the reduction in available resources can be exacerbated by the need to download index information in addition to the media. .. The potential for playback interruptions is reduced by increasing the speed at which the playback device switches between streams and reducing the amount of overhead data downloaded to achieve the switch.
Although the present invention has been described in certain specific aspects, many additional modifications and variations will be apparent to those skilled in the art. Thus, the present invention, for example, in various implementations utilizing encoders and decoders, without departing from the scope and spirit of the invention, corresponding to features other than those specified within the particular standard to which they conform. Please understand that it may be practiced differently, including the changes in, other than being explained concretely. Therefore, embodiments of the present invention should be considered in all respects as an example, not a limitation.
27 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27
Every citation, both ways
| Document | Relation | Office | Cited during |
|---|---|---|---|
| US10462537B2 | Cited by | United States of America | Applicant |
| USRE49990E | Cited by | United States of America | Applicant |
| US10225299B2 | Cited by | United States of America | Applicant |
| US11438394B2 | Cited by | United States of America | Applicant |
| US11343300B2 | Cited by | United States of America | Applicant |
| US12470781B2 | Cited by | United States of America | Applicant |
| US10715806B2 | Cited by | United States of America | Applicant |
| US12407906B2 | Cited by | United States of America | Applicant |
| US11102553B2 | Cited by | United States of America | Applicant |
| US10437896B2 | Cited by | United States of America | Applicant |
| US10856020B2 | Cited by | United States of America | Applicant |
| US10498795B2 | Cited by | United States of America | Applicant |
| USRE48761E | Cited by | United States of America | Applicant |
| US12262051B2 | Cited by | United States of America | Applicant |
| US10321168B2 | Cited by | United States of America | Applicant |
| US10687095B2 | Cited by | United States of America | Applicant |
| US12250404B2 | Cited by | United States of America | Applicant |
| US11785066B2 | Cited by | United States of America | Applicant |
| US10397292B2 | Cited by | United States of America | Applicant |
| US10382785B2 | Cited by | United States of America | Applicant |
| US10341698B2 | Cited by | United States of America | Applicant |
| US10225588B2 | Cited by | United States of America | Applicant |
| US11457054B2 | Cited by | United States of America | Applicant |
| US11785066B2 | Cited by | United States of America | Applicant |
| US10212486B2 | Cited by | United States of America | Applicant |
| US12184943B2 | Cited by | United States of America | Applicant |
| US11886545B2 | Cited by | United States of America | Applicant |
| US12244878B2 | Cited by | United States of America | Applicant |
| US11711552B2 | Cited by | United States of America | Applicant |
| US11638033B2 | Cited by | United States of America | Applicant |
| US10368096B2 | Cited by | United States of America | Applicant |
| US11102553B2 | Cited by | United States of America | Applicant |
| US12177281B2 | Cited by | United States of America | Applicant |
| US11683542B2 | Cited by | United States of America | Applicant |
| US10878065B2 | Cited by | United States of America | Applicant |
| US10244272B2 | Cited by | United States of America | Applicant |
| US10484749B2 | Cited by | United States of America | Applicant |
| US10805368B2 | Cited by | United States of America | Applicant |
| JP2007535881A | Cites | Japan | – |
| JP2007036666A | Cites | Japan | – |
| JP2004172830A | Cites | Japan | – |
| JP2004013823A | Cites | Japan | – |
| JP2000201343A | Cites | Japan | – |
| US20060026294A1 | Cites | United States of America | – |
| US20030061369A1 | Cites | United States of America | – |
| WO2010147878A1 | Cites | World Intellectual Property Organization (WIPO) | – |
| US20090150557A1 | Cites | United States of America | – |
90 members in 7 offices
Priority claims19
| Document | Office | Kind | Date |
|---|---|---|---|
| 201161430110 | United States of America | P | |
| 201161430110 | United States of America | P | |
| 61430110 | United States of America | – | |
| 13221682 | United States of America | – | |
| 13221794 | United States of America | – | |
| 201113221682 | United States of America | A | |
| 201113221682 | United States of America | A | |
| 201113221794 | United States of America | A | |
| 201113221794 | United States of America | A | |
| 2011066927 | United States of America | W | |
| 2011066927 | United States of America | W | |
| 13221682 | – | – | – |
| 13221794 | – | – | – |
| 61430110 | – | – | – |
| US2011066927 | – | – | – |
| US201113221682 | – | – | – |
| US201113221794 | – | – | – |
| US201161430110P | – | – | – |
| WO2011US66927 | – | – | – |
Members90
| Document | Office | Kind | |
|---|---|---|---|
| US2012170642A1 | United States of America | A1 | |
| US2012170643A1 | United States of America | A1 | |
| US2012170906A1 | United States of America | A1 | |
| US2012170915A1 | United States of America | A1 | |
| US2012173751A1 | United States of America | A1 | |
| CA2823829A1 | Canada | A1 | |
| WO2012094171A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2012094181A2 | World Intellectual Property Organization (WIPO) | A2 | |
| WO2012094189A1 | World Intellectual Property Organization (WIPO) | A1 | |
| US2013044821A1 | United States of America | A1 | |
| EP2661696A1 | European Patent Office (EPO) | A1 | |
| EP2661875A1 | European Patent Office (EPO) | A1 | |
| EP2661895A2 | European Patent Office (EPO) | A2 | |
| KR20130133266A | Republic of Korea | A | |
| KR20130133830A | Republic of Korea | A | |
| US8649669B2 | United States of America | B2 | |
| JP2014506430A | Japan | A | |
| KR20140035881A | Republic of Korea | A | |
| WO2012094181A3 | World Intellectual Property Organization (WIPO) | A3 | |
| EP2661696A4 | European Patent Office (EPO) | A4 | |
| EP2661875A4 | European Patent Office (EPO) | A4 | |
| US2014250473A1 | United States of America | A1 | |
| US8914534B2 | United States of America | B2 | |
| US9025659B2 | United States of America | B2 | |
| US9210481B2 | United States of America | B2 | |
| US9247312B2 | United States of America | B2 | |
| US2016219303A1 | United States of America | A1 | |
| JP6038805B2This record | Japan | B2 | |
| JP2017063453A | Japan | A | |
| US9883204B2 | United States of America | B2 | |
| KR101874907B1 | Republic of Korea | B1 | |
| KR20180081621A | Republic of Korea | A | |
| US2018220153A1 | United States of America | A1 | |
| JP2018160923A | Japan | A | |
| KR20180123182A | Republic of Korea | A | |
| CA2823829C | Canada | C | |
| JP6453291B2 | Japan | B2 | |
| KR101917763B1 | Republic of Korea | B1 | |
| US2019045219A1 | United States of America | A1 | |
| US2019045220A1 | United States of America | A1 | |
| US10368096B2 | United States of America | B2 | |
| US10382785B2 | United States of America | B2 | |
| KR101988877B1 | Republic of Korea | B1 | |
| US2019356928A1 | United States of America | A1 | |
| EP2661875B1 | European Patent Office (EPO) | B1 | |
| KR102072839B1 | Republic of Korea | B1 | |
| KR20200012048A | Republic of Korea | A | |
| KR20200012049A | Republic of Korea | A | |
| KR20200013095A | Republic of Korea | A | |
| JP6657313B2 | Japan | B2 | |
| EP2661696B1 | European Patent Office (EPO) | B1 | |
| JP2020080551A | Japan | A | |
| ES2767260T3 | Spain | T3 | |
| KR102122189B1 | Republic of Korea | B1 | |
| EP3697096A1 | European Patent Office (EPO) | A1 | |
| EP3700219A1 | European Patent Office (EPO) | A1 | |
| EP3742740A1 | European Patent Office (EPO) | A1 | |
| KR102191317B1 | Republic of Korea | B1 | |
| KR102195414B1 | Republic of Korea | B1 | |
| KR20200144586A | Republic of Korea | A | |
| ES2807884T3 | Spain | T3 | |
| US10992955B2 | United States of America | B2 | |
| KR102274290B1 | Republic of Korea | B1 | |
| KR20210084700A | Republic of Korea | A | |
| US2021250608A1 | United States of America | A1 | |
| JP2021158694A | Japan | A | |
| KR102352043B1 | Republic of Korea | B1 | |
| JP7000475B2 | Japan | B2 | |
| KR20220009503A | Republic of Korea | A | |
| EP3697096B1 | European Patent Office (EPO) | B1 | |
| EP3975574A1 | European Patent Office (EPO) | A1 | |
| ES2911672T3 | Spain | T3 | |
| EP3742740B1 | European Patent Office (EPO) | B1 | |
| KR102408120B1 | Republic of Korea | B1 | |
| KR20220082942A | Republic of Korea | A | |
| ES2917700T3 | Spain | T3 | |
| KR102445689B1 | Republic of Korea | B1 | |
| EP4124048A1 | European Patent Office (EPO) | A1 | |
| US11638033B2 | United States of America | B2 | |
| JP7332655B2 | Japan | B2 | |
| US2023300372A1 | United States of America | A1 | |
| JP2023138806A | Japan | A | |
| EP3700219B1 | European Patent Office (EPO) | B1 | |
| US2024414369A1 | United States of America | A1 | |
| ES2992853T3 | Spain | T3 | |
| US12250404B2 | United States of America | B2 | |
| US12262051B2 | United States of America | B2 | |
| US2025168400A1 | United States of America | A1 | |
| JP7692956B2 | Japan | B2 | |
| JP2025105849A | Japan | A |
19 legal events, as the office reported them to INPADOC
Over the term
Point at a mark for the eventEvents
| Event | Code | |
|---|---|---|
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Receipt of annual feesJAPANESE INTERMEDIATE CODE: R250R250 | R250 | |
| Written notification of registration of transferJAPANESE INTERMEDIATE CODE: R350R350 | R350 | |
| Request for change of ownership or part of ownershipJAPANESE INTERMEDIATE CODE: R313113S111 | S111 | |
| Certificate of patent or registration of utility modelJAPANESE INTERMEDIATE CODE: R150R150 | R150 | |
| First payment of annual fees (during grant procedure)JAPANESE INTERMEDIATE CODE: A61A61 | A61 | |
| Written decision to grant a patent or to grant a registration (utility model)JAPANESE INTERMEDIATE CODE: A01A01 | A01 | |
| Decision of grant or rejection writtenTRDD | TRDD | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Request for written amendment filedJAPANESE INTERMEDIATE CODE: A523A521 | A521 | |
| Notification of reasons for refusalJAPANESE INTERMEDIATE CODE: A131A131 | A131 | |
| Report on retrievalJAPANESE INTERMEDIATE CODE: A971007A977 | A977 | |
| Written request for application examinationJAPANESE INTERMEDIATE CODE: A621A621 | A621 |
Numbers
- Publication
- 6038805
- Publication, DOCDB
- 6038805
- Publication, EPODOC
- JP6038805B
- Application
- 2013548427
- Application, DOCDB
- 2013548427
- Application, EPODOC
- JP20130548427
Titles2
- Japanese
- ハイパーテキスト転送プロトコルを使用してMatroskaコンテナファイル中に記憶されるメディアの適応型ビットレートストリーミング
- English
- Adaptive bitrate streaming of media stored in Matroska container files using the hypertext transfer protocol
Classification
- CPC, 29
- G11B27/005
- H04N21/4621
- H04N19/593
- G11B27/11
- G11B27/322
- H04N21/2387
- H04N21/85406
- H04N21/23439
- H04N21/26258
- H04N21/2662
- H04N21/44209
- H04N21/8455
- H04N21/8456
- H04N21/8543
- H04N21/6587
- H04L65/613
- H04L65/612
- H04L65/70
- H04N21/236
- H04N21/440281
- H04N21/643
- H04N21/42607
- H04N21/435
- H04N21/44004
- H04N21/44008
- H04N19/172
- H04N19/177
- H04N19/40
- H04N21/234345
- IPC, 3
- H04N21 2662
- H04N21 435
- G06F13 00
