Selection of resolutions for seamless resolution switching of multimedia content

Selection of resolutions for seamless resolution switching of multimedia content

FIELD OF THE INVENTION The present invention relates to the delivery of multimedia content, and more particularly, to streaming video content encoded in multiple resolutions optimized for various scaled display resolutions.

The term media streaming refers to the playback of media on a playback device, where the media is stored on a server and continuously transmitted over a network to the playback device during playback. In general, a playback device buffers a sufficient amount of media at any given time during playback to prevent interruption of playback due to completion of playback of all buffered media before the playback device receives the next portion of media. do. Adaptive bit rate streaming or adaptive streaming involves detecting current streaming conditions (eg, a user's network bandwidth and CPU capacity) in real time, and adjusting the quality of the streamed media accordingly.

<p>In adaptive streaming systems, the source media is typically stored on a media server as a top-level index file pointing to multiple alternative streams containing the actual video and audio data. Each stream is typically stored in one or more container files. Different adaptive streaming solutions generally use different indexes and media containers. Matroska Container is a media container developed as an open standard project by the Matroska non-profit organization in Aussonne, France. Matroska containers are based on Extensible Binary Meta Language (EBML), a binary derivative of Extensible Markup Language (XML). Decoding of Matroscar containers is supported by many consumer electronics (CE) devices. The DivX Plus file format, developed by DivX LLC of San Diego, California, uses an extension of the Matroscar container that contains elements not specified within the Matroscar format.</p>

<p>Systems and methods for selecting resolutions for seamless resolution conversion of multimedia content are disclosed in accordance with embodiments of the present invention. In one embodiment of the present invention, the source encoder comprises a processor configured as a source encoder application to receive multimedia content, the multimedia content comprising video data having a main resolution and a main sample aspect ratio and replacing the video data encodes as a set of alternate streams, wherein the plurality of streams in the set of alternative streams have different maximum bitrates and resolutions, each of the plurality of streams comprises a width and a height that are integer pixels, the plurality of streams Each width and height are both a constant fraction of the width and height of the corresponding main resolution.</p><p>In a further embodiment of the invention, the plurality of streams in the set of alternative streams have the same aspect ratio and different maximum bitrates and resolutions.</p><p>In another embodiment of the present invention, the plurality of streams in the set of alternative streams have the same aspect ratio and different maximum bitrates, sample aspect ratios and resolutions.</p><p>In another embodiment of the present invention, the main resolution is 1920x1080, the main aspect ratio is 16:9, and a plurality of streams in the set of alternative streams are encoded with the following resolutions and sample aspect ratios:</p><p>Resolution = 1536×864; Sample Aspect Ratio = 1:1</p><p>resolution = 1152×648; Sample Aspect Ratio = 1:1</p><p>Resolution = 960×544; Sample Aspect Ratio = 1:1</p><p>resolution = 768×432; Sample Aspect Ratio = 1:1</p><p>In another embodiment of the present invention, the main resolution is 1920x1040, the main aspect ratio is 1.85:1, and a plurality of streams in the set of alternative streams are encoded with the following resolutions and sample aspect ratios:</p><p>resolution = 1536×832; Sample Aspect Ratio = 1:1</p><p>resolution = 1152×624; Sample Aspect Ratio = 1:1</p><p>Resolution = 960×520; Sample Aspect Ratio = 1:1</p><p>Resolution = 768×416; Sample Aspect Ratio = 1:1</p><p>In another embodiment of the present invention, the main resolution is 1920x816, the main aspect ratio is 2.35:1, and a plurality of streams in the set of alternative streams are encoded at the following resolutions and sample aspect ratios:</p><p>Resolution = 1536×656; Sample Aspect Ratio = 1:1</p><p>Resolution = 1152×488; Sample Aspect Ratio = 1:1</p><p>Resolution = 960×408; Sample Aspect Ratio = 1:1</p><p>resolution = 768×328; Sample Aspect Ratio = 1:1</p><p>In another embodiment of the present invention, the main resolution is 1920x800, the main aspect ratio is 2.40:1, and a plurality of streams in the set of alternative streams are encoded at the following resolutions and sample aspect ratios:</p><p>resolution = 1536×640; Sample Aspect Ratio = 1:1</p><p>Resolution = 1152×480; Sample Aspect Ratio = 1:1</p><p>resolution = 960×400; Sample Aspect Ratio = 1:1</p><p>Resolution = 768×320; Sample Aspect Ratio = 1:1</p><p>Another embodiment of the present invention comprises a method for encoding multimedia content comprising video data having a main resolution and a main sample aspect ratio as a plurality of alternative streams, the method comprising: using a source encoder, the main resolution and Receiving multimedia content comprising video data having a primary sample aspect ratio, and encoding the video data as a set of alternative streams using a source encoder, wherein the plurality of streams in the set of alternative streams are at different maximums. having bitrates and resolutions, wherein the resolution of each of the plurality of streams includes a width and a height that are integer pixels and both the width and height of each of the plurality of streams are a constant fraction of the width and height of the corresponding main resolution.</p><p>In another embodiment of the present invention, the numerator and denominator of the fraction are each less than 100.</p><p>Another embodiment of the present invention includes a playback device comprising a processor configured to communicate with a memory, the memory comprising a client application; The client application causes the processor to obtain an index file, wherein the index file represents a plurality of alternative video streams having the same aspect ratio and different bitrates and resolutions and wherein the at least two plurality of alternative video streams are different samples. have aspect ratios; configure the video decoder to decode a video having a resolution and a sample aspect ratio of a first one of the plurality of alternative video streams, wherein decoding the video comprises scaling the decoded video to a resolution of a display device, and ; Configure to request part of the first stream.</p><p>In another embodiment of the present invention, there is provided a method of playing back content, the method comprising: using a playback device to obtain an index file, wherein the index file has the same aspect ratio and different bitrates and resolutions a plurality of alternative video streams having different sample aspect ratios, wherein at least two of the plurality of alternative video streams have different sample aspect ratios; configure a video decoder on the playback device to decode a video having a sample aspect ratio and a resolution of a first video stream of the plurality of alternative video streams, wherein the decoding the video scales the decoded video to a resolution of the display device including doing; and requesting a portion of the first stream using the playback device.</p><p>Another embodiment of the present invention comprises a method for determining, as a set of low resolution streams, a set of sub-resolutions for encoding a main video having a main resolution and a main aspect ratio, the method comprising: select a set; where the height and width of each sub-resolution in the set of sub-resolutions is an integer number of pixels and is less than the width and height of the corresponding main resolution; selecting sub-resolutions from a set of sub-resolutions, wherein the width and height of the selected sub-resolutions have an aspect ratio equal to the main aspect ratio.</p><p>In another embodiment of the present invention, the width and height for each sub-resolution in the set of sub-resolutions are multiples of eight.</p><p>In yet a further embodiment of the present invention, the set of sub-resolutions comprises sub-resolutions having a rectangular sample aspect ratio.</p><p>In yet another additional embodiment of the present invention, the set of sub-resolutions is obtained by repeatedly subtracting the width of a column from the width of the main resolution and multiplying the resulting width by the main aspect ratio to determine the corresponding heights, the width and The corresponding height is selected by selecting a sub-resolution that is an integer number of pixels.</p><p>In another embodiment of the present invention, selecting the set of sub-resolutions also includes iteratively subtracting the height of the row from the height of the main resolution and multiplying the resulting height by the main aspect ratio to determine the corresponding widths; and selecting sub-resolutions whose height and corresponding width are an integer number of pixels.</p><p>In another embodiment of the present invention, the width and height for each sub-resolution in the set of sub-resolutions are multiples of eight.</p><p>In another embodiment of the present invention, the set of sub-resolutions includes sub-resolutions having a square sample aspect ratio.</p><p>In another embodiment of the present invention, the set of sub-resolutions comprises sub-resolutions having a square sample aspect ratio, the width and height for each sub-resolution being a multiple of eight.</p><p>In another embodiment of the present invention, sub-resolutions are calculated for the main resolutions of 1920x1080, 1920x1040, 1920x816 and 1920x800.</p><p>In another embodiment of the present invention, the set of sub-resolutions includes sub-resolutions having a common width that is repeated for multiple primary resolutions, the sub-resolutions being any that do not share a common width value. is chosen for the main resolution of</p><p>In another embodiment of the present invention, sub-resolutions selected for main resolutions that do not have a common width value are created by using the width value and selecting a height value having an aspect ratio equal to the aspect ratio of the main resolution. .</p><p>In another embodiment of the present invention, selected height values are rounded to the nearest multiple of 8 pixels.</p><p>In another embodiment of the present invention, the selected height value is rounded down to the nearest multiple of 8.</p><p>In another embodiment of the present invention, the set of sub-resolutions is repeated by subtracting the column height from the height of the main resolution and multiplying the resulting height by the main aspect ratio to determine the corresponding widths, the height and the corresponding width This integer number of pixels is selected by selecting sub-resolutions.</p><p>In another embodiment of the present invention, there is provided a machine readable medium comprising processor instructions, wherein execution of the instructions by the processor causes the processor to produce multimedia content including video data having a main resolution and a main sample aspect ratio. receive and perform processing comprising encoding the video data as a set of alternative streams, wherein a plurality of streams in the set of alternative streams have different maximum bitrates and resolutions, and a resolution of each of the plurality of streams. contains an integer number of pixels width and height, and the width and height of each of the plurality of streams are constant fractions of the width and height of the corresponding main resolution.</p><p>In yet a further embodiment of the present invention, the numerator and denominator of the fraction are each less than 100.</p><p>Another embodiment of the present invention comprises a machine readable medium comprising processor instructions, wherein execution of the instructions by a processor causes the processor to obtain an index file, wherein the index file has the same aspect ratio and different bitrates. and wherein at least two of the plurality of alternative video streams have different sample aspect ratios; configure a video decoder to decode a video having a resolution and a sample aspect ratio of a first video stream of the plurality of alternative video streams, wherein decoding the video includes scaling the decoded video to a resolution of a display device and; A process for requesting a part of the first stream is performed.</p>

1 is a diagram illustrating a network diagram of adaptive bitrate streaming, according to an embodiment of the present invention; Fig. 2 is a flow chart illustrating a process for selecting resolutions in consideration of a main resolution, according to an embodiment of the present invention; 3 is a flow diagram illustrating a process for selecting resolutions taking into account a combination of primary resolutions, according to an embodiment of the present invention; FIG. 4 conceptually illustrates a playback device configured to deliver decoded video data using different video resolutions, in accordance with an embodiment of the present invention; FIG. 5 is a table showing a set of resolutions that are constant fractions of a main resolution calculated using processing according to an embodiment of the present invention; 6 is a table illustrating a set of resolutions calculated for a combination of primary resolutions using processing according to an embodiment of the present invention; 7 is a table showing a final set of resolutions determined for a combination of primary resolutions using processing in accordance with an embodiment of the present invention;

Turning to the figures, systems and methods are illustrated for adaptive bit-trace streaming of streams encoded at different resolutions in accordance with embodiments of the present invention. Adaptive bitrate streaming systems according to embodiments of the present invention are configured to stream encoded multimedia content at different maximum bitrates and resolutions over a network such as the Internet. Multimedia content generally includes video and audio data, subtitles and other related metadata. Video data is generated at a specific resolution and encoded to achieve a target maximum bitrate. Video data of a particular resolution may be encoded at multiple bitrates; The subjective quality of video data at a particular bitrate depends in part on the resolution of the video data. For example, high resolution video data may have subjectively poor quality when encoded at a low bitrate, because information stored in the high resolution video is lost during encoding. Likewise, low resolution video data when encoded at increasingly higher bitrates may not exhibit any improved subjective quality; The subjective quality of low resolution video may be tolerated at low bitrates compared to high resolution encoding. Adaptive bitrate streaming systems according to embodiments of the present invention include multimedia sources containing video data at maximum bitrates of variable video resolutions. To provide the highest quality video experience independent of network data rate, adaptive bitrate streaming systems deliver video data according to various factors including, but not limited to, available network data rate and video decoder performance. configured to switch between available sources of video data throughout. Systems and methods for switching video systems during playback are entitled "Hypertext Delivery" to Braness et al., filed Aug. 30, 2011, the disclosure of which is incorporated herein by reference in its entirety. US Patent Application "Systems and Methods for Adaptive Bitrate Streaming of Media Stored in Matroska Container Files Using Hypertext Transfer Protocol" 13/221,682.

Although streamed multimedia content can be encoded at any of a number of different resolutions, the process of decoding video for display generally involves scaling the decoded video to the resolution of the display device. When the resolution is different from the display device, each pixel of the decoded video is scaled to correspond to one or more pixels of the display. In some cases, it may be desirable for the scaling ratio between the streamed multimedia content and the display resolution to be a fraction of a natural number, such as 1:2, 2:3, 3:4, 7:8, and so on. Adaptive bitrate streaming systems are designed to transition between streams in response to changes in streaming conditions. Transitions between streams encoded at resolutions that are scaled to different ratios for display are often visually interrupted. In many embodiments, the adaptive bitrate streaming system is configured such that the content is encoded in such a way that each stream of video is encoded with an aspect ratio equal to the aspect ratio of the source video. In this way, each pixel of the encoded video scales in the same way as it is decoded and displayed on a display device that receives the video data. Adaptive bitrate streaming systems configured to stream video encoded in multiple resolutions having the same aspect ratio as the source video in accordance with embodiments of the present invention are further discussed.

<u>System overview</u>

An adaptive bitrate streaming system according to an embodiment of the present invention is shown in FIG. 1 . The adaptive bitrate streaming system 100 includes a source encoder 106 configured to encode the source media as multiple alternative streams each having the same aspect ratio. In the illustrated embodiment, the source encoder is a server. In other embodiments, the source encoder can be any processing device including a processor and sufficient resources to perform transcoding of the source media (including but not limited to video, audio and/or subtitles). there is. The source encoding server 106 generally generates a top level index for a plurality of container files comprising streams, at least a plurality of which are alternative streams. Alternative streams are streams that encode the same content in different ways. In many examples, alternative streams encode media content (such as but not limited to video) at different maximum bitrates. In many embodiments, alternative streams are encoded at different resolutions and/or different frame rates. However, alternative video streams are encoded with the same aspect ratio corresponding to the aspect ratio of the source video. The top level index file and container files are uploaded to the HTTP server 104 . Although the source encoding server 106 has been described above as generating a top-level index file, in many embodiments, the top-level index file is generated dynamically in response to a request for a particular portion of content by a playback device.

In the illustrated embodiment, playback devices include personal computers 110 , CE players 108 and mobile phones 112 . In other embodiments, playback devices are DVD players, Blu-ray players, televisions, set-top boxes, video game consoles, tablets and other devices that are connected to a server via HTTP and can play encoded media. consumer electronics devices, such as devices. In the illustrated embodiment, various playback devices use HTTP or another suitable stateless protocol to request portions of the top-level index file and container files over a network 102 , such as the Internet. Before the playback device performs adaptive bitrate streaming using portions of media from alternative streams contained within the container file, a bandwidth probe may be performed by the playback device to determine available bandwidth. When the bandwidth probe is complete, the playback device, as part of the adaptive streaming process, includes (but with data in the top-level index (but not limited to) may be used.

Once starting playback of the content from the initial set of streams, the playback device uses the highest level index to perform adaptive bitrate streaming of the content in response to changes in streaming conditions. In an adaptive bitrate streaming system using streams encoded at different resolutions, the playback device may proceed through a series of operational steps in which the playback device responds differently at each step to changes in streaming conditions. In many embodiments, stability of streaming conditions or improvement of streaming conditions results in a transition to a stage where the playback device assumes stable operating conditions and buffers more content and is less sensitive to variations in streaming conditions. can bring In many embodiments, the worsening of streaming conditions results in the stream switching streaming a set of streams at a lower resolution using lower bandwidth, causing the playback device to assume unstable operating conditions and buffer less content and streaming conditions. transition to a stage that responds quickly to changes in

In the illustrated embodiment, the adaptive bitrate streaming system includes computer systems capable of delivering multimedia content at variable data rates using various video resolutions. In many embodiments, adaptive bitrate streaming systems may be implemented using any device capable of delivering encoded streams of multimedia, wherein the streams are encoded at different maximum bitrates and resolutions. A basic architecture of an adaptive streaming system source encoder according to an embodiment of the present invention is shown in FIG. 4 . The adaptive bitrate streaming system 400 includes a non-volatile memory 430 and a processor 410 in communication with the volatile memory 420 . In the illustrated embodiment, the volatile memory includes a source encoder 422 and alternative streams of video data 424 encoded at different resolutions. Although a particular architecture is shown in FIG. 4 , any of a variety of architectures, including those in which the application is located on disk or some other form of storage and loaded into volatile memory at runtime, may be different in accordance with embodiments of the present invention. It can be used to implement adaptive bitrate streaming systems capable of multimedia delivery of applications at variable data rates using resolutions. Further, any of a variety of system architectures, including, but not limited to, the system architecture shown in FIG. 1 may be used to perform video delivery in accordance with embodiments of the present invention. Systems and methods for encoding streams of video data at different resolutions and delivering the encoded video data using adaptive streaming in accordance with embodiments of the present invention are further discussed below.

<u>Selection of resolutions for the main resolution</u>

As mentioned above, adaptive bitrate streaming systems involve switching between different video streams encoded at different bitrates depending on streaming conditions. In an adaptive streaming system that uses alternative video streams encoded at different resolutions, the resolution of each stream may be selected based on the playback quality of the encoded video at a given bitrate. A general process for selecting resolutions for video streams in accordance with embodiments of the present invention is shown in FIG. Process 200 begins by selecting 210 a main resolution based on a predetermined aspect ratio. For example, for an aspect ratio of 16:9, in many embodiments of the present invention, the main resolution is 1920x1080 pixels. To determine all height and width combinations by width 212 , the width of the main resolution may be reduced one line at a time and the corresponding height is calculated by applying the aspect ratio to the new width. In some embodiments, the width of the image should be a multiple of 8 pixels. In some embodiments, the calculated height may be rounded to the nearest integer value. In many embodiments, determination 212 of height and width combinations is repeated until the width reaches a predetermined minimum width, such as 320 pixels. To determine 214 width and height combinations of the main resolution, the same process can be used as above for height. As above, in some embodiments, the height is twice the height of 8 pixels. However, in other embodiments, the widths and heights of the final image may be defined as any multiple of a number of pixels, such as a multiple of 2 or 4 pixels.

Once the set of resolutions is determined using both width and height, the lists are combined 216 and any overlapping widths and heights are removed. Once the overlapping resolutions are removed, the computed width and height are compared to the main width and height to determine 218 which resolutions are selected (218). In many embodiments of the present invention, computed resolutions are selected 218 where both the height and width can be expressed as natural fractions of the height and width of the primary resolution, where the numerator or denominator is a predetermined value equal to 100. smaller than the number An example of a set 500 of calculated resolutions and ratios according to an embodiment of the present invention using the processing described above is shown in FIG. 5 . For example, one width and height combination is 1280x720, which is a width ratio of 2/3 and a height ratio of 2/3 for the main resolution 1920x1080. In many embodiments, 1280x720 will be selected 218 as a sub-resolution for the main resolution of 1920x1080. However, a resolution such as 1888×1062 will not be selected, because neither 1888 divided by 1920 nor 1062 divided by 1080 can be expressed using fractions of natural numbers, where both the numerator and denominator are greater than 100 respectively. small. In other embodiments, all fractions of a natural number may be less than 10 or 1000, or the numerator may be ten steps smaller and the denominator less than 1000, and vice versa.

Although a specific process for selecting resolutions for use in a multi-resolution adaptive streaming system has been described above, any of a variety of processes involving calculating resolutions based on a main resolution, including those where resolutions higher than the main resolution are computed, have been described above. Processing may be used in accordance with embodiments of the present invention. Systems and methods for selecting resolutions taking into account a combination of primary resolutions are further discussed below.

<u>Selection of resolutions for a combination of primary resolutions</u>

Video data included in multimedia content is often created at a specific resolution and aspect ratio. However, it is desirable to display multimedia content on various display devices; These display devices may not have the same resolution or aspect ratio as the video data. In an adaptive streaming system, video data may be encoded in various primary resolutions with different aspect ratios to maintain a level of consistency in display of the source multimedia content. A general process for selecting resolutions for a video stream having multiple aspect ratios according to embodiments of the present invention is shown in FIG. 3 . Process 300 begins by determining 310 a set of primary resolutions for various aspect ratios. For example, in many embodiments the primary resolutions (with an accompanying aspect ratio) are 1920x1080 (16:9) 1920x1040 (1.85:1), 1920x816 (2.35:1) and 1920x It could be 800 (2.4:1). For each primary resolution, resolutions are calculated (312) and a ratio of the calculated resolution to the primary resolution is determined (314). In many embodiments of the present invention, the processing described above may be used to determine calculated resolutions and ratios for each primary resolution. In some embodiments, determining the ratio 314 represents the ratio, wherein the percentage area the calculated resolution includes proportionally to the primary resolution is rounded to a predetermined number of significant digits, such as one significant digit. It involves converting to a percentage. Common ratios between the calculated resolutions of the primary resolutions are selected (316). In many embodiments, the selected common ratios 316 will be common to all primary resolutions. In some embodiments, the selected common ratios 316 are common for a majority of primary resolutions, such as 3 of 4 primary resolutions, 3 of 5 primary resolutions, 2 of 3 primary resolutions. will be. Once common ratios have been selected, a corresponding width is determined (318). For primary resolutions that do not have a matching ratio, the closest width may be chosen. In many embodiments, the nearest width should be a multiple of 4. In many embodiments, the nearest width should be a multiple of eight. Based on the width, a corresponding height is determined using the aspect ratio (320).

An exemplary set 600 of calculated resolutions according to an embodiment of the present invention is shown in FIG. 6 . Considering the main resolutions and aspect ratios as shown in FIG. 6 , the determined one common ratio 314 is 64%, which occurs in 3 of the 4 main resolutions. The calculated resolutions (for aspect ratio) based on the main resolutions are 1536x864 (16:9), 1536x832 (1.85:1) and 1536x640 (2.4:1). An aspect ratio of 2.35:1 does not have a computed resolution of 1536 pixels wide; The closest height corresponding to a width-to-height ratio equal to the main resolution is 652.8. In some embodiments, this resolution may be rounded to the nearest multiple of 8 or 856. Some other embodiments may use the nearest multiple of 8 or 648 less than the calculated height value. Accordingly, the calculated height is 656, and a resolution of 1536×656 may be used as the calculated resolution. Taking into account the main resolutions of 1920x1080, 1920x1040, 1920x816 and 1920x800 with resolution targets of aspect ratios 16:9, 1.85:1, 2.35:1 and 2.4:1, calculated using the rounding method An example of resolutions is shown in FIG. 7 . Table 700 shows the calculated heights for the common width ratio*height ratio, the number of occurrences of each width ratio*height ratio, the corresponding width, and each aspect ratio that is a multiple of 8, where the calculated heights are underlined. there is. In many embodiments of the present invention, a source encoder receives video data having a resolution of 1920x1080 and encodes the video data into multiple streams of the resolutions shown in FIG.

While the invention has been described in specific specific embodiments, many additional modifications and variations will be apparent to those skilled in the art. Accordingly, it is understood that the present invention may be practiced otherwise than as specifically described, including playback devices in which the set of streaming switching conditions used by the playback device is continuously changed, without departing from the scope and spirit of the invention. will be understood Accordingly, the embodiments of the present invention are to be considered in all respects as illustrative and not non-limiting. Accordingly, the scope of the present invention should be determined not by the illustrated embodiments, but by the appended claims and their equivalents.

100: adaptive bitrate streaming system 102: network 104: HTTP Server 106: Source Encoder 400: adaptive streaming system source encoder 410: processor 420: volatile memory 422: source encoder 424: video data 430: non-volatile memory