A method and system for generating multiple transcoded outputs based on a single unit

{A METHOD AND SYSTEM FOR GENERATING MULTIPLE TRANSCODED OUTPUTS BASED ON A SINGLE UNIT}

Embodiments of the present invention relate to the field of data transcoding. Specifically, embodiments of the present invention relate to methods and systems for generating multiple transcoded outputs based on a single input.

BACKGROUND OF THE INVENTION Portable electronic devices such as cell phones, personal digital assistants (PDAs) and laptop computers are increasingly capable of providing video content to users. Occasionally, video content is generated from a live or broadcast source and transmitted wirelessly to a portable electronic device for presentation. Due to the typical screen size and bit rate format of a typical portable electronic device, the video content is adapted to suit the network properties of the device and the receiving portable electronic device. One way to adapt video content to fit a wide array of networks and client devices is transcoding. Transcoding adapts to viewing media data in different formats by adjusting device and network attributes such as screen size output and bandwidth. In essence, transcoding adjusts the video according to the characteristics of the viewing device.

Due to the wide array of different types of portable electronic devices, it is typically necessary to transcode the video for each type of electronic device to which the video is transmitted. Currently, typical transcoders initiate different transcoding sessions for each type of viewing device. The transcoder transcodes video from the same source, but each transcoding session is performed independently. Different transcoding sessions have different computational loads. For example, one type of device may require a reduction in bit rate and a second type of device may require a reduction in screen resolution that requires a greater computational load. Moreover, a transcoding session can provide very similar video outputs that perform multiple identical video processing operations on the same input video data.

In the described scenario of live video transcoding or broadcast transcoding where one video source is requested by a client with many different devices/connection capabilities, the source needs to be transcoded into multiple types of video output. Current techniques for independently transcoding video data into multiple outputs using separate transcoding sessions are wasting computational power by performing redundant operations in separate transcoding sessions. Moreover, the current technology may not satisfy the scalability requirements for transcoding services.

<u>Summary of the invention</u>

Various embodiments of the present invention are described, methods and systems for generating multiple transcoded outputs based on a single input. A first transcoding session having a first attribute is initiated, wherein the first transcoding session includes a plurality of video processing operations. A second transcoding session having a second attribute is initiated. Intermediate data related to at least one video processing operation of the first transcoding session is generated. A second transcoding session is performed, wherein the second transcoding session is based, at least in part, on intermediate data.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, explain the principles of the invention.

1 shows a block diagram of a multiple output transcoding system, in accordance with an embodiment of the present invention.

2 shows a block diagram of an exemplary decoding and encoding operation of a transcoding process, in accordance with an embodiment of the present invention.

3A shows a two-dimensional graphical representation of two transcoding dimensions, according to an embodiment of the present invention.

3B shows a three-dimensional graphical representation of three transcoding dimensions, according to an embodiment of the present invention.

4 shows a block diagram of an exemplary gradual reuse of discrete cosine transform (DCT) information in a multiple output transcoding process, in accordance with an embodiment of the present invention.

5 depicts a block diagram of an exemplary progressive reuse of rate control information in a multiple output transcoding process, in accordance with an embodiment of the present invention.

6 depicts a block diagram of an exemplary progressive reuse of quantization information in a multiple output transcoding process, in accordance with an embodiment of the present invention.

7 shows a block diagram of an exemplary gradual reuse of error frame information in drift correction in a multiple output transcoding process, in accordance with an embodiment of the present invention.

8 depicts a flow diagram of a process for generating multiple transcoded outputs based on a single input, in accordance with an embodiment of the present invention.

Reference will now be made in detail to various embodiments of the present invention, examples of which are shown in the accompanying drawings. While the present invention will be described in conjunction with these examples, it will be understood that it is not intended to limit the invention to these examples. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims. Moreover, in the following description of the invention, various specific details are set forth in order to provide a thorough understanding of the invention. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present invention.

Aspects of the invention generally include a processor for processing information and instructions, a random access (volatile) memory (RAM) for storing information and instructions, a read-only (non-volatile) memory (ROM) for storing static information and instructions; Data storage devices such as magnetic or optical disks and disk drives that store information and instructions; an optional user input device including alphanumeric characters and function keys (eg, a keyboard) to control the processor, and an optional user input device such as a cursor control device (eg, a mouse) for communicating user input information and command selections to the processor. It may be implemented in a computer system comprising

1 shows a block diagram of a multiple output transcoding system 100, in accordance with an embodiment of the present invention. The multiple output transcoding system 100 efficiently generates multiple transcoded video outputs from a single video input by reusing metadata, also referred to herein as intermediate data, across multiple transcoding sessions. do. The multiple output transcoding system 100 includes a video source 105 , a transcoder 110 , and a memory 115 that produces a first output 120 and a second output 125 . It should be understood that the multiple output transcoding system 100 may generate any number of outputs based on a single video source 105 , eg, a third output 130 . It should also be understood that the multiple output transcoding system 100 may be implemented within a single computer system or within a computer system of a distributed computer network.

The video source 105 provides input video content to the transcoder 110 . In one embodiment, the video source 105 is a live source, eg, a live sporting event or live news conference. In another embodiment, the video source 105 is a broadcast source, eg, a television program or movie. It should be understood that the video source 105 may be any video source that provides video with a set starting point, for example a stored source or a broadcast source.

Transcoder 110 is configured to transcode input video content received from video source 105 according to attributes associated with a particular type of device. The transcoder 110 receives a request for video content from a device having certain attributes. Transcoder 110 performs a plurality of video processing operations 118 based on attributes associated with the request to generate output video. Attributes (also referred to herein as transcoding dimensions) contain information that describes the specific video input requirements of the associated device, including, but not limited to, video format, screen size, frame rate, and bit rate. include It should be understood that the attributes may be based in part on network attributes, eg, network bandwidth.

In one embodiment, the transcoder 110 receives a first request for video content associated with a first device having a first property, and transcodes the input video into a format for viewing on the first device having the first property. Initiate a first transcoding session. The first transcoding session includes a plurality of video processing operations 118 for transcoding the input stream into an output stream suitable for viewing on a first device. Further, the transcoder 110 is operable to initiate a second transcoding session in response to a second request for video content associated with the second device having the second attribute. The second transcoding session is based, at least in part, on intermediate data (eg, metadata) associated with the first transcoding session.

In one embodiment, the multiple output transcoding system 100 also includes a memory 115 for storing intermediate data related to at least one said video processing operation of a first transcoding session. In one embodiment, memory 115 is random access (volatile) memory (RAM) connected to transcoder 110 . It should be understood that memory 115 may be any type of computer memory (eg, flash memory) in which data is stored and rapidly read. It should be understood that a multiple output transcoding system may operate without memory 115 . For example, in a hardware implementation such as a field programmable gate array (FPGA) or digital signal processor (DSP), intermediate data may be generated and reused without explicit storage of the intermediate data.

2 shows a block diagram of an exemplary decoding and encoding operation of a transcoding process 200, in accordance with an embodiment of the present invention. The transcoding process 200 includes a decoding process (eg, blocks 202-212) and an encoding process (eg, blocks 230-244), with a transcoding operation 220 in between. . Each of the blocks represents a video processing operation used in the example transcoding process. It should be understood that the blocks shown are exemplary, and the transcoding process 200 may include different blocks, as well as fewer or more blocks, depending on the video coding standard used by the transcoding process 200 . . In general, useful metadata is identified for motion compensation and transcoding of DCT encoded video streams. Embodiments of the present invention may use the Motion Pictures Experts Group (MPEG) standard (eg, MPEG-1 or MPEG-4), the H.26x standard, or any other standard using motion compensation or DCT encoding.

Each block of transcoding process 200 generates metadata (eg, intermediate data) for an associated video processing operation. The metadata may be stored in a memory (eg, memory 115 in FIG. 1 ). The decoding unit of the transcoding process 200 includes a plurality of video processing operations that generate different metadata that can be stored and reused in other transcoding processes. Metadata that can be generated and stored by the following video processing operation includes the following.

variable length decoding (VLD) - sequence level information such as the screen size of the input video and the bit rate of the input video; picture level information such as picture coding type and number of bits per picture; Macroblock level information such as macroblock coding type, motion vector, coded block pattern (CBP) and quantizer coefficients; and block level such as run-length pairs of quantized DCT coefficients. information.

Run length decoding (RLD) - quantized DCT coefficients in an NxN array, where NxN is the transform block size (eg, N=4 for H.264 format, N=8 otherwise) .

· Inverse quantization (Q<sp>-1</sp>) - DCT coefficients in an NxN array.

· Inverse transform (T<sp>-1</sp>) - pixel (or remainder) values in an NxN array.

· Motion compensation (M<sp>-1</sp>) - YUV color space pixel values in the frame buffer (this block is optional depending on whether the frame is intercoded or not).

· Inverse color conversion (C<sp>-1</sp>) - Red Green Blue (RGB) color space pixel values in the frame buffer.

The encoding unit of the transcoding process 200 also includes a plurality of video processing operations that generate different metadata that can be stored and reused in other transcoding processes. Metadata that can be generated and stored by the following video processing operation includes color transform (C), motion compensation (M), transform (T), run length encoding (RLE), and variable length encoding (VLE). Other examples of metadata that may be generated and stored by the following video processing operations include the following.

Quantization (Q) - quantized DCT coefficients (after operation) and CBP in an NxN array.

Spatial Activity (SA) - Spatial activity value in the macroblock array (eg, given an NxM frame size, the macroblock array is of size 'N/16 by M/16').

Rate Control (RC) - quantization parameter in the macroblock array.

It should be understood that the video processing operations and corresponding metadata described above are exemplary in both the decoding unit and the encoding unit, and may include additional metadata. Moreover, the video processing operations described above store metadata because reuse of the associated metadata is considered particularly useful. However, there may be additional video processing operations as described in FIG. 7 (eg drift correction and error accumulation).

For example, the first transcoding session performs all video processing blocks of the transcoding process 200 to store metadata for each block. The second transcoding session having a different target format selectively uses the metadata generated by the decoding unit of the first transcoding session and supplies it to the encoding unit of the second transcoding session, thereby generating different outputs.

Embodiments of the present invention provide for reuse of intermediate data across multiple transcoding sessions, thereby reducing computational requirements for a transcoder (eg, multiple output transcoding system 100 of FIG. 1 ). Intermediate data generated during the transcoding session is kept in memory. Other transcoding sessions may access and retrieve the intermediate data. In order to efficiently store and reuse the intermediate data, it is desirable to appropriately select whether to store the intermediate data.

3A shows a two-dimensional graphical representation of two transcoding dimensions, according to an embodiment of the present invention. A grid point (eg, a processing point) represents a composed transcoding operation. A first operation a is to reduce the screen size by a factor of two and a bit rate by a factor of two. The second operation b is for reducing the screen size by a factor of 4 and reducing the bit rate by a factor of 4. The second operation may be gradually achieved based on the result of operation a.

The third operation c reduces the screen size by a factor of 8 and the bit rate by a factor of 3. Operation c may reuse the reduced screen size portion of the result from operation b, while reusing the bit rate reduction portion of operation a. Operation c cannot reuse the bit rate reduction portion of operation b because the result from operation b has a lower bit rate. Also, while operation c may utilize the reducing screen size portion of operation a, in one embodiment, operation c may use the reducing screen size portion of operation b, because it reduces the screen size reduction portion of operation a. This is because it creates a smaller computational load than it uses.

Adding another dimension, eg frame rate reduction, transforms the processing space into a three-dimensional processing space. 3B shows a three-dimensional graphical representation of three transcoding dimensions, according to an embodiment of the present invention. A point (eg, a processing point) represents a crafted transcoding operation. The same principle described in Figure 3a applies. In general, actions can be achieved incrementally based on the results of other actions that do not require a larger reduction. In addition, the action selects the action part of the other action whose result is used for transcoding, based on the minimum computational requirements.

It may also be desirable to selectively decide whether to store intermediate data. For example, for bit rate reduction, information about the quantization result (eg, CBP) at the minimum target bit rate level cannot be reused and, therefore, there is no need to store it. For screen size reduction, DCT data at the minimum reduction level cannot be reused by any other transcoding session, nor is it stored. In general, to process points further from the origin, less metadata is stored (eg, at processing point d in FIG. 3A ).

It should be understood that a transcoding session of a multiple output transcoding system (eg, multiple output transcoding system 100 of FIG. 1 ) may progressively reuse metadata from any other transcoding session. In one embodiment, the order in which the requests are received does not matter. For example, a transcoding session initiated in response to an earlier request may reuse intermediate data generated in a transcoding session initiated in response to a later request. When a later request is received, the multiple output transcoding system adjusts the transcoding session so that the later initiated transcoding session stores metadata for use by the earlier initiated transcoding session. These adjustments are performed without interruption to the user.

Figures 4, 5, 6, 7 include examples of gradual reuse of metadata at computing bottlenecks. Examples of such include left-to-right multi-output transcoding processes.

4 shows a block diagram of an exemplary incremental reuse of DCT information in a multiple output transcoding process 400, in accordance with an embodiment of the present invention. Reuse of DCT information provides for gradual screen size reduction. The multiple output transcoding process 400 receives a request 402 to downscal the input video by a factor of two, and a first transcoding session 410 is initiated. A second request 422 to reduce the input video by a factor of 4 is received, and a second transcoding operation 420 is initiated.

The metadata 406 associated with the reduction operation (block 404) of the first transcoding operation may be reused in the second transcoding operation. Block 404 generates intermediate data D2 of video data reduced by a factor of two, which is stored. A second transcoding operation 420 reads the stored metadata from block 404 and performs further reduction by a factor of two. Since there is no need to perform the operation before block 424, the computational load on the transcoder is reduced. Moreover, performing reduction by a factor of 2 is a lower cost operation than reduction by a factor of 4.

Continuing with the example of the multiple output transcoding process 400 , a third request 442 for reduction by a factor of 4 is received and the bit rate is further changed. A third transcoding session initiated in response to the third request 442 reads the metadata 426 associated with block 424 , supplies the metadata 426 to block 444 , and uses different quantization coefficients. to change the bit rate. All operations performed prior to block 444 are saved, reducing the computational load on the transcoder.

5 shows a block diagram of an exemplary incremental reuse of rate control information in a multiple output transcoding process 500, in accordance with an embodiment of the present invention. A rate control (RC) video processing operation uses spatial activity (SA) computed from the original frame to determine the assignment of quantization coefficients. In multiple output transcoding process 500 , a request 502 to adapt a bit rate according to the quantization coefficient Q of block 504 is received, and thus a transcoding session 510 is initiated. A second request 522 to adapt the bit rate according to the other quantization coefficient Q2 of block 524 is received, thus initiating a second transcoding session 520 .

The metadata 506 prior to the quantization operation of block 504 of the transcoding process 510 is reused by the second transcoding process 520 . Metadata 506 is fed directly to block 524 to adapt the bit rate according to Q2. Moreover, the metadata 508 generated in the spatial activity process of block 512 is reused in the second transcoding process 520 and fed directly to block 526 . As shown, in the case of a multi-output transcoder, spatial activity can be reused for other sessions. For example, for bit rate reduction transcoding (e.g., transcoding session 510) computed spatial activity for a different bit rate reduction factor, transcoding (e.g., second transcoding session 520) ) can be reused.

6 shows a block diagram of an exemplary incremental reuse of quantization information in a multiple output transcoding process 600, in accordance with an embodiment of the present invention. Metadata from the macroblock level indicates whether a block in the macroblock is coded. For example, a block may not be coded in a frame with a low bit rate, since the difference between frames may be very small. In MPEG syntax, this coding is referred to as CBP.

The multiple output transcoding process 600 receives a request 602 to reduce the screen size according to the reduction factor D2 of block 604 and to decrease the bit rate according to the quantization factor Q of block 606 . In response to the request 602 , a transcoding session 610 is initiated. A second request 622 is received to reduce the screen size by the same reduction factor D2 of the request 602 and decrease the bit rate by the quantization factor Q2 in block 624 . In response to the request 622 , a second transcoding session 620 is initiated.

The second transcoding session 620 reuses the metadata 608 generated at block 606 . Metadata 608 includes CBP information as well as a reduced bit rate frame. Metadata 608 may be fed to block 624 for further quantization. However, if a block is not coded in one bit rate reduction transcoding (e.g., all coefficients of the block are zero), then, without any action, more stringent bit rate reduction transcoding (resulting in a more coarse quantization) can be achieved. Also, strict quantization is not necessary, since the quantization result will be zero anyway. This indicates that the quantization coefficients do not need to be changed, which results in bit budget savings for the output stream as well as computational savings of the quantization coefficients. Thus, if CBP is equal to zero, metadata 608 may be fed directly to block 628, since processing of blocks 624 and 626 will produce a result of zero. The computational load of the second transcoding session 620 is further reduced by not performing unnecessary operations.

7 shows a block diagram of an exemplary gradual reuse of error frame information in drift correction in multiple output transcoding process 700, in accordance with an embodiment of the present invention. Typically, drift correction requires reconstruction of pixel region information so that error frames can be created that accumulate errors generated by transcoding each frame. A request 702 to decrease the bit rate is received, and a second request 722 to further decrease the bit rate is received.

A second transcoding session 720 reuses the metadata 708 generated at block 704 and supplies the metadata 708 to block 724 . Moreover, FIG. 7 shows that the error frame from the error accumulation (EA) operation of block 706 for the first transcoding session 710 shows that the second transcoding session 720 is more stringent rate reduction transcoding, If the CBP for the corresponding block is zero, it can be reused by the second transcoding session 720 in the drift correction (DC) operation of block 724 .

This is the motion compensation (M) in drift correction.<sp>-1</sp>) can be extended to other types of transcoding required. Since motion compensation is one of the most computationally intensive tasks in a transcoding session, the computational savings by reusing error frames is even more important. The joint multiple output transcoding system stores the reconstructed pixel frame buffer in YUV format, allowing the buffer to be reused by other transcoding sessions that also require drift correction. Typically, rate reduction and screen size reduction transcoding require drift correction.

8 shows a flow diagram of a process 800 for generating multiple transcoded outputs based on a single input, in accordance with an embodiment of the present invention. In one embodiment, the process 800 is performed by a processor and electrical components (eg, a computer system) under the control of computer-readable and computer-executable instructions, such as the multiple output transcoding system 100 of FIG. 1 . do. Although specific steps are disclosed in process 800, such steps are exemplary. That is, embodiments of the present invention are suitable for carrying out various other steps or variations of the steps recited in FIG. 8 .

At step 810 of process 800, a first transcoding session associated with a first device having a first attribute is initiated, wherein the first transcoding session includes a plurality of video processing operations. In step 820, a second transcoding session associated with a second device having a second attribute is initiated. In one embodiment, at least one of the second attributes is a gradual decrease in the corresponding first attribute. In one embodiment, the first attribute includes a first screen size and a first bit rate, and the second attribute includes a second screen size and a second bit rate.

In step 830, it is determined which intermediate data of the first transcoding session to store. In one embodiment, intermediate data relating to the gradual decrease of the first attribute with respect to the second attribute is stored. In step 840, at least one intermediate data related to at least one video processing operation of the first transcoding session is generated. In step 850, at least one intermediate data is stored. It should be understood that step 850 is optional. In step 860, a second transcoding session is performed, wherein the second transcoding session is based, at least in part, on intermediate data.

In one embodiment, the first attribute relates to a reduction in screen size of a first reduction factor, and a corresponding second property relates to a reduction in screen size in a second reduction factor, wherein the second reduction factor is greater than the first reduction factor. Provides screen size reduction. In one embodiment, the intermediate data includes the result of a screen size reduction operation based on the first reduction factor.

In another embodiment, the first property relates to a bit rate reduction in the first bit rate reduction factor, and a corresponding second property relates to a bit rate reduction in the second bit rate reduction factor, wherein the second bit rate reduction factor is and provides a bit rate reduction greater than the first bit rate reduction factor. In one embodiment, the intermediate data includes the result of a bit rate reduction operation based on the first bit rate reduction coefficient. In one embodiment, the intermediate data further comprises a coded block pattern. In an embodiment, performing the second transcoding session also determines whether the coded block pattern is substantially equal to zero, and if the coded block pattern is substantially equal to zero, performs drift correction and This includes not performing error accumulation.

In another embodiment, the first property relates to a screen size reduction of the first reduction factor and a bit rate reduction of the first bit rate reduction factor, and a corresponding second property is the screen size reduction and the second bit of the first reduction factor. relates to a bit rate reduction in a rate reduction factor, wherein the second bit rate reduction factor provides a greater bit rate reduction than the first bit rate reduction factor. In one embodiment, the intermediate data includes the result of the bit rate reduction operation based on the first bit rate reduction coefficient and the coded block pattern. In an embodiment, performing the second transcoding session also determines whether the coded block pattern is substantially equal to zero, and if the coded block pattern is substantially equal to zero, performing quantization on the intermediate data including not doing.

The various embodiments of the invention described provide a combined video transcoding method and system in which, given a request for a single input and multiple outputs, multiple outputs can be efficiently generated. Multiple outputs can be generated in multiple formats, multiple frame rates, multiple bit rates, and multiple screen sizes. Moreover, multiple outputs can be generated in an optimized form that requires a minimal amount of computational resources.

Accordingly, embodiments of the present invention, methods and systems for generating multiple transcoded outputs based on a single input, are described. While the present invention has been described in terms of specific embodiments, it is to be understood that the present invention is not to be construed as being limited by such embodiments, but instead, the present invention is construed in accordance with the following claims.