Reducing latency in video encoding and decoding

1/39

Descriptive Report of the Patent of Invention for METHOD IN A COMPUTER SYSTEM IMPLEMENTING A VIDEO DECODER, METHOD IN A COMPUTER SYSTEM, COMPUTER-READable MEDIUM AND COMPUTER SYSTEM.

BACKGROUND

[0001] Engineers use compression (also called font compilation or font encoding) to reduce the bit rate of digital video. Compression lowers the cost of storing and transmitting video information by converting the information to a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec is an encoder/decoder system.

[0002] Over the past two decades, several codec standards have been adopted, including the H.261, H.262 (MPEG-2 or

ISO/IEC 13818-2), H.263 and H.264 (AVC or ISO/IEC 14496-10) and MPEG-1 (ISO/IEC 11172-2), MPEG-4 Visual (ISO/IEC 144962) standards and SMPTE 421M. More recently, the HEVC standard is under development. A video codec standard typically defines options for the syntax of an encoded video bitstream, detailing parameters in the bitstream when particular aspects are used in encoding and decoding. In many cases, a video codec standard also provides details about decoding operations that a decoder must perform to obtain correct decoding results.

[0003] A basic goal of compression is to provide good rate distortion performance. So, for a particular bitrate, an encoder tries to provide the highest quality video. Or, for a particular level of quality/fidelity to the original video, a

Petition 870200084889, of 07/08/2020, page 9/59

2/39 encoder tries to provide the video encoder with the lowest bitrate. In practice, depending on the usage scenario, considerations such as encoding time, encoding complexity, encoding capabilities, decoding time, decoding complexity, decoding capabilities, overall delay, and/or playback uniformity also affect decisions made during encoding and decoding.

[0004] For example, consider usage scenarios such as playing video from storage, playing video from encoded data continuously transmitted over a network connection, and transcoding video (from one bit rate to another bit rate). bits, or from one pattern to another pattern). On the encoder side, such applications may allow for offline encoding that is by no means time sensitive. Therefore, an encoder can increase encoding time and increase resources used during encoding to find the most efficient way to compress video, and thereby improve rate distortion performance. If a small amount of delay is also acceptable on the decoder side, the encoder can additionally improve rate distortion performance, for example, by exploiting dependencies between images from images farther ahead in a sequence.

[0005] On the other hand, consider usage scenarios such as remote desktop conferencing, video surveillance, video telephony, and other real-time communication scenarios. Such applications are time sensitive. Low latency between recording input images and playing back output images is a key performance factor. When encoding/decoding tools adapted for non-real-time communication are applied in real-time communication scenarios, latency ge

Petition 870200084889, of 07/08/2020, page 10/59

3/39 ral is often unacceptably high. The delays these tools introduce during encoding and decoding can improve performance for normal video playback, but they get in the way of real-time communication.

SUMMARY

[0006] In the Table of Contents, the detailed description presents techniques and tools to reduce latency in video encoding and decoding. Techniques and tools can reduce latency to improve responsiveness in real-time communication. For example, the techniques and tools reduce overall latency by constraining latency due to reordering of video frames, and by indicating constraint regarding frame reordering latency with one or more syntax elements accompanying encoded data to the video frames.

[0007] In accordance with one aspect of the techniques and tools described in this document, a tool such as a video encoder, real-time communication tool with a video encoder, or other tool, establishes one or more syntax elements that indicate a constraint on latency (for example, a constraint on frame reordering latency consistent with interframe dependencies between multiple frames of a video sequence). The tool outputs the syntax element(s), thereby facilitating a simpler and faster determination of when reconstructed frames are ready for output in terms of frame output order.

[0008] In accordance with another aspect of the techniques and tools described in this document, a tool such as a video decoder, real-time communication tool with a video decoder, or other tool, receives and parses one or more syntax elements. that indicate a restriction in relation to latency

Petition 870200084889, of 07/08/2020, page 11/59

4/39 (for example, a constraint regarding frame reorder latency). The tool also receives data encoded for multiple frames of a video sequence. At least some of the encoded data is decoded to reconstruct one of the frames. The tool can determine the latency constraint based on the syntax element(s), then use the latency constraint to determine when a reconstructed frame is ready for emission (in terms of output order). The tool outputs the reconstructed frame.

[0009] The objects, aspects and advantages of the above and other inventions will become more apparent from the following detailed description, which continues with reference to the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[00010] Figure 1 is a diagram of an illustrative computing system in which some described embodiments can be implemented.

[00011] Figures 2a and 2b are diagrams of illustrative network environments in which some described embodiments can be implemented.

[00012] Figure 3 is a diagram of an illustrative encoder system in conjunction with which some described embodiments can be implemented.

[00013] Figure 4 is a diagram of an illustrative decoder system together with which some described embodiments can be implemented.

[00014] Figures 5a through 5e are diagrams showing the coded order and output order for frames in various illustrative series.

[00015] Figure 6 is a flowchart presenting an illustrative technique for establishing and issuing one or more syntax elements that

Petition 870200084889, of 07/08/2020, page 12/59

5/39 indicates a restriction regarding latency.

[00016] Figure 7 is a flowchart showing an illustrative technique for decoding with reduced latency.

DETAILED DESCRIPTION

[00017] The detailed description presents techniques and tools to reduce latency in video encoding and decoding. Techniques and tools can help reduce latency to improve responsiveness in real-time communication.

[00018] In video encoding/decoding scenarios, some delay is unavoidable between when an input video frame is received and when the frame is played back. The frame is encoded by an encoder, delivered to a decoder, and decoded by the decoder, and some amount of latency is caused by practical limitations in terms of encoding capabilities, decoding capabilities, and/or network bandwidth. However, other latency is avoidable. For example, latency can be introduced by an encoder and decoder to improve rate warping performance (e.g. to exploit dependencies between frames from further forward images in a sequence). Such latency can be reduced, although there may be a performance penalty in terms of rate skew, processor utilization, or playback uniformity. [00019] With the techniques and tools described in this document, latency is lowered by constraining latency (thus limiting the temporal extent of the dependency between frames) and indicating the constraint regarding latency for a decoder. Alternatively, the constraint on latency is a constraint in terms of seconds, milliseconds, or some other measure of time. The decoder can then determine the constraint with respect to latency and use the constraint when determining which frames are ready.

Petition 870200084889, of 07/08/2020, page 13/59

6/39 for issue. In this way, the delay can be reduced for remote desktop conferencing, video telephony, video surveillance, web camera video and other real-time communication applications.

[00020] Some of the innovations described in this document are illustrated with reference to syntax elements and operations specific to the H.264 and/or HEVC standard. Such innovations can also be implemented for other standards or formats.

[00021] More generally, several alternatives to the examples described in this document are possible. Some techniques described with reference to flowchart diagrams can be altered by changing the order of the stages presented in the flowcharts, by separating, repeating or omitting some stages, etc. The various aspects of video encoding and decoding latency reduction can be used in combination or separately. Different embodiments use one or more of the techniques and tools described. Some of the techniques and tools described in this document address one or more of the issues cited in the background. Typically, a given technique/tool ​​does not solve all problems.

1. Illustrative Computing System

[00022] Figure 1 illustrates a generalized example of a suitable computing system (100) in which several of the techniques and tools described can be implemented. The computing system (100) is not intended to suggest any limitations as to the scope of use or functionality, as the techniques and tools may be implemented in various general-purpose or special-purpose computing systems.

[00023] With reference to Figure 1, the computing system (100) includes one or more processing units (110, 115) and the memory

Petition 870200084889, of 07/08/2020, page 14/59

7/39 ria (120, 125). In Figure 1, this most basic configuration (130) is enclosed within a dashed line. Processing units (110, 115) execute computer-executable instructions. A processing unit can be a general-purpose central processing unit (CPU), a processor on an application-specific integrated circuit (ASIC), or any other type of processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 1 shows a central processing unit (110) as well as a graphics processing unit or co-processing unit (115). Tangible memory (120, 125) can be volatile memory (e.g. registers, cache, RAM), non-volatile memory (e.g. ROM, EEPROM, flash memory, etc.), or some combination of the two accessible by the unit (s) of processing. The memory (120, 125) stores software (180) implementing one or more innovations to reduce latency in video encoding and decoding, in the form of computer executable instructions suitable for execution by the processing unit(s).

[00024] A computing system may have additional aspects. For example, the computing system (100) includes storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown) such as a bus, controller, or network, interconnects the components of the computing system (100). Typically, operating system software (not shown) provides an operating environment for other software running on the computing system (100), and coordinates the activities of the computing system components.

Petition 870200084889, of 07/08/2020, page 15/59

8/39 (100).

[00025] Tangible storage (140) may be removable or non-removable, and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-temporary manner and that can be accessed within the computing system (100). Storage (140) stores instructions for software (180) implementing one or more innovations for reducing latency in video encoding and decoding.

[00026] The input device(s) (150) may be a touch input device, such as a keyboard, mouse, pen, or trackball, a voice input device, a digitizing device, or other device that provide input to the computing system (100). For video encoding, the input device(s) (150) may be a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM or CD-RW that reads video samples to the computing system (100). The input device(s) (160) may be a video, printer, speaker, CD recorder, or other device that provides output from the computing system (100).

[00027] The communication connection(s) (170) allow communication over a communication medium to another computing entity. The communication medium carries information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in a way that encodes information in the signal. By way of example, and not limitation, the means of communication may use an electrical, optical, RF or other carrier per Petition 870200084889, of 07/08/2020, p. 16/59

9/39 ador.

[00028] The techniques and tools can be described in the general context of computer-readable media. Computer readable media are any available tangible media that can be accessed within a computing environment. By way of example and not limitation, with the computer system (100), the computer readable medium includes memory (120, 125), storage (140), and combinations of any of the above.

[00029] Techniques and tools can be described in the general context of computer-executable instructions, such as those included in program modules, being executed in a computing system on a real or virtual target processor. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform particular tasks or implement particular types of abstract data. The functionality of the program modules may be combined or separated among the program modules as desired in the various embodiments. Computer-executable instructions for program modules can be executed within a local or distributed computing system.

[00030] The terms “systems and “device are used interchangeably throughout this document. Unless the context clearly indicates otherwise, neither term implies any limitation as to a type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and/or general-purpose hardware with software implementing the functionality described in this document.

[00031] For the purpose of presentation, the detailed description

Petition 870200084889, of 07/08/2020, page 17/59

10/39 uses terms such as “determines and “uses to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer, and should not be confused with acts performed by a human being. Actual computer operations corresponding to these terms vary depending on implementation.

II. Illustrative Network Environments

[00032] Figures 2a and 2b show illustrative network environments 201, 202 that include video encoders 220 and video decoders 270. Encoders 220 and decoders 270 are connected through a network 250 using an appropriate communication protocol. Network 250 may include the Internet or other computer network.

[00033] In the network environment 201 shown in Figure 2a, each real-time communication ("RTC) tool 210 includes both an encoder 220 and a decoder 270 for bidirectional communication. A given encoder 220 can output according to the SMPTE 421M standard, the ISO-IEC 14496-10 standard (also known as H.264 or AVC), HEVC standard, another standard, or an appropriate format, with a corresponding decoder 270 accepting encrypted data from encoder 220. Two-way communication can be part of a video conference, video phone call, or other two-party communication scenario. While the network environment 201 in Figure 2a includes two real-time communication tools 210, the network environment 201 may instead include three or more real-time communication tools 210 that participate in multiparty communication.

[00034] A real-time communication tool 210 manages encoding by an encoder 220. Figure 3 shows a

Petition 870200084889, of 07/08/2020, page 18/59

11/39 illustrative encoder system 300 which may be included in real-time communication tool 210. Alternatively, real-time communication tool 210 uses another encoder system. A real-time communication tool 210 also manages decoding by a decoder 270. Figure 4 shows an illustrative decoder system 400, which may be included in the real-time communication tool 210. Alternatively, real-time communication tool 210 uses another decoder system.

[00035] In the network environment 202 shown in Figure 2b, an encoding tool 212 includes an encoder 220 that encodes video for distribution to multiple playback tools 214, which include decoders 270. One-way communication may be provided for a video surveillance system, web camera monitoring system, remote desktop conference presentation, or other scenario in which video is encoded and sent from one location to one or more other locations. . While the network environment 202 in Figure 2B includes two playback tools 214, the network environment 202 may include more or fewer playback tools 214. In general, a playback tool 214 communicates with the encoding tool 212 to determine a video stream for the playback tool 214 to receive. Playback tool 214 receives the stream, buffers the encoded data received during an appropriate period, and begins decoding and playback.

[00036] Figure 3 shows an illustrative encoder system 300 that may be included in the coding tool 212. Alternatively, the coding tool 212 uses another encoder system. Encoding tool 212 may also include logic from the

Petition 870200084889, of 07/08/2020, page 19/59

12/39 server-side controller for managing connections to one or more playback tools 214. Figure 4 shows an illustrative decoder system 400 which may be included in playback tool 214. Alternatively, playback tool 214 uses another decoder system. A playback tool 214 may also include client-side controller logic to manage connections to the encoding tool 212.

[00037] In some cases, the use of a syntax element to indicate latency (eg frame reorder latency) is specific to a particular pattern or format. For example, encoded data may contain one or more syntax elements that indicate a restriction with respect to latency as part of the syntax of an elementary encoded video bitstream defined according to the standard or format, or as defined media metadata if relating to the encoded data. In such cases, the real-time communication tool 210, the encoding tool 212, and/or the playback tool 214 with reduced latency may be codec dependent, in that the decisions it makes may depend on the syntax of the stream. bits to a particular pattern or format.

[00038] In other cases, the use of a syntax element to indicate a restriction regarding latency (eg frame reorder latency) is outside of a particular standard or format. For example, syntax element(s) that indicate a constraint regarding latency may be signaled as part of the syntax of a streaming media stream, media storage file, or, more generally, a multiplexing protocol or streaming protocol. transport of the media system. Or, the syntax element(s) that indicates latency may be negotiated between real-time communication tools 210, encoding tools 212, and/or tools

Petition 870200084889, of 07/08/2020, page 20/59

13/39 playback minds 214 in accordance with a media ownership negotiation protocol. In such cases, the real-time communication tool 210, the encoding tool 212, and the playback tool 214 with reduced latency can be codec-independent in that they can work with any available video encoder and decoder, assuming a level of control over the dependency between frames established during encoding.

III. Illustrative Encoding Systems

[00039] Figure 3 is a block diagram of an illustrative encoder system 300 in conjunction with which some described embodiments may be implemented. Encoder system 300 may be a general-purpose encoding tool capable of operation in any of multiple encoding modes, such as a low-latency encoding mode for real-time communication, transcoding mode, and normal encoding mode for media playback from a file or stream, or it may be a special-purpose encoding tool adapted for such encoding mode. Encoder system 300 can be implemented as an operating system module, as part of an application library, or as a standalone application. Overall, the encoder system 300 receives a sequence of source video frames 311 from a video source 310 and produces encoded data as output to a channel 390. The encoded data output to the channel may include one or more syntax elements that indicate a restriction with respect to latency (eg, frame reorder latency) to facilitate decoding with reduced latency.

[00040] Video source 310 can be a camera, tuner card, storage medium, or other digital video source. THE

Petition 870200084889, of 07/08/2020, page 21/59

14/39 video source 310 produces a sequence of video frames at a frame rate of, for example, 30 frames per second. As used in this document, the term “frame” generally refers to source, encoded, or reconstructed image data. For progressive video, a frame is a frame of progressive video. For interlaced video, in the illustrative embodiments, an interlaced video frame is de-interlaced prior to encoding. Alternatively, two complementary interlaced video fields are encoded as an interlaced video frame or separate fields. In addition to indicating a progressive video frame, the term “frame can indicate a single unpaired video field, a pair of complementary video fields, a video object plane representing a video object at a given time, or a region of interest in a larger image. The video object plane or region can be part of a larger image that includes multiple objects or regions of a scene.

[00041] An incoming source frame 311 is stored in a source frame buffer storage area 320 that includes multiple frame buffer storage areas 321, 322, ..., 32n. A frame buffer 321, 322, etc. maintains a source frame in the source frame storage area 320. After one or more of the source frames 311 have been stored in frame buffers 321, 322, etc., a frame selector 330 periodically selects an individual source frame from the source frame storage area 320. The order in which frames are selected by frame selector 330 for input to encoder 340 may be different from the order in which frames are produced by video source 310, for example, one frame may be ahead in order to facilitate the prediction temporarily backwards. Before the 340 encoder, the encoder system

Petition 870200084889, of 07/08/2020, page 22/59

15/39 (300) may include a preprocessor (not shown) that performs preprocessing (e.g., filtering) of the selected frame 331 prior to encoding.

[00042] Encoder 340 encodes selected frame 331 to produce an encoded frame 341 and also produces memory management control signals 342. If the current frame is not the first frame that was encoded, when performing its encoding process, the encoder 340 may utilize one or more previously encoded/decoded frames 369 that have been stored in a decoded frame storage area in buffer 360. Such stored decoded frames 369 are used as reference frames for prediction between frames of current source frame content 331. Generally, encoder 340 includes multiple encoding modules that perform encoding tasks such as motion estimation and compensation, frequency transformations. , entropy encoding quantization. The exact operations performed by the 340 encoder may vary depending on the compression format. The format of the output encoded data can be a Windows Media Video format, VC-1 format, MPEG-x format (for example, MPEG-1, MPEG-2, or MPEG4), H.26x format (for example, H.261, H.262, H.263, H.264), format

HEVC or other format.

[00043] Encoded frames 341 and memory management control signals 342 are processed by a decoding process emulator 350. The decoding process emulator (350) implements some of the functionality of a decoder, for example, decoding tasks. decoding to reconstruct reference frames that are used by encoder 340 in motion estimation and compensation. The 350 decoding process emulator uses the 342 memory management control signals

Petition 870200084889, of 07/08/2020, page 23/59

16/39 to determine whether a given encoded frame 341 needs to be reconstructed and stored for use as a reference frame in inter-frame prediction of subsequent frames to be encoded. If the control signals 342 indicate that an encoded frame 431 needs to be stored, the decoding process emulator 350 models the decoding process that would be conducted by a decoder that receives the coded frame 341 and produces a corresponding decoded frame 351. In doing so, when the encoder 340 has used the decoded frame(s) 369 that was stored in the decoded frame storage area 360, the decoding process emulator 350 also uses the decoded frame(s) 369 from the decoded area. 360 storage as part of the decoding process.

[00044] Decoded frame storage area in buffer 360 includes multiple frame storage areas in buffer 361, 362, ..., 36n. Decoding process emulator 350 uses memory management control signals 342 to manage the contents of storage area 360 so as to identify any frame buffers 361, 362, etc. with frames that are no longer needed by the encoder 340 for use as reference frames. After modeling the decoding process, the decoding process emulator 350 stores a most recently decoded frame 351 in a frame buffer (361, 362, etc. which has been identified in this manner.

[00045] The coded frames 341 and the memory management control signals 342 are also buffered in a temporary coded data area 370. The coded data that is aggregated in the coded data area 370 may contain, as part of the syntax of a bitstream elementPetition 870200084889, of 07/08/2020, p. 24/59

17/39 tar of encoded video, one or more syntax elements that indicate a restriction with respect to latency. Or, the encoded data that is aggregated in the encoded data area 370 may include syntax elements that indicate a restriction regarding latency as part of media metadata relating to encoded video data (e.g., as one or more parameters in one or more supplemental enhancement information (“SEI) messages or video usability information (“VUI”) messages.

[00046] The aggregated data 371 from the encoded data temporary area 370 is processed by a channel encoder 380. The channel encoder 380 may package the aggregated data for transmission as a media stream, in which case the encoder channel 380 may add, as part of the streaming media stream syntax, the syntax element(s) that indicates a constraint regarding latency. Or, channel encoder 380 may organize the aggregated data for storage as a file, in which case channel encoder 380 may add, as part of the storage media file syntax, the syntax element(s) indicating a constraint regarding latency. Or, more generally, channel encoder 380 may implement one or more multiplexing protocols or media system transport protocols, in which case channel encoder 380 may add, as part of the syntax of the protocol(s), the element Syntax(s) that indicate a restriction regarding latency. Channel encoder 380 provides output to a channel 390, which represents storage, a communication connection, or other channel for the output.

IV. Illustrative Decoder Systems

[00047] Figure 4 is a block diagram of an illustrative decoder system 400 in conjunction with which some described embodiments may be implemented. The 400 po decoder system

Petition 870200084889, of 07/08/2020, page 25/59

18/39 of being a general purpose decoding tool capable of operating in any of multiple decoding modes such as a low latency decoding mode for real-time communication and normal decoding mode for playing media from a file or stream, or it may be a special-purpose decoding tool adapted for such a decoding mode. The decoder system 400 can be implemented as an operating system module, as part of an application library, or as a standalone application. Overall, the decoder system 400 receives encoded data from a channel 410 and produces reconstructed frames as output to an output destination 490. The encoded data may include one or more syntax elements that indicate a restriction with respect to latency (eg, frame reordering latency) to facilitate decoding with reduced latency.

[00048] Decoder system 400 includes a channel 410 which represents storage, a communication connection, or other channel for encoded data as input. Channel 410 produces encoded data that has been encoded per channel. A channel decoder 420 can process the encoded data. For example, channel encoder 420 de-packets data that has been aggregated for transmission as a media stream, in which case channel decoder 420 may parse, as part of the streaming media stream syntax, the element Syntax(s) that indicate a restriction regarding latency. Or, channel decoder 420 separates encoded video data that has been aggregated for storage as a file, in which case channel decoder 420 may parse, as part of the storage media file syntax, the syntax element(s) which indicates a constraint regarding latency. Or, more generally, the channel decoder

Petition 870200084889, of 07/08/2020, page 26/59

19/39

420 may implement one or more demultiplexing protocols or media system transport protocols, in which case the channel decoder 420 may parse, as part of the protocol(s) syntax, the syntax element(s) that indicates a restriction on regarding latency.

[00049] The scrambled data 421 that is output from the channel decoder 420 is stored in a temporary scrambled data area 430 until a sufficient amount of such data has been received. Encoded data 421 includes encoded frames 431 and memory management control signals 432. The encoded data 421 in the encoded data area 430 may contain, as part of the syntax of an encoded video elementary bit stream, one or more syntax elements that indicate a restriction with respect to latency. Or, the encoded data 421 in the encoded data area 430 may include syntax element(s) that indicate a restriction with respect to latency as part of the media metadata relating to the encoded video data (e.g., as one or more parameters in one or more SEI messages or VUI messages). In general, scrambled data area 430 temporarily stores scrambled data 421 until such scrambled data 421 is used by decoder (450). At this point, encoded data for an encoded frame 431 and memory management control signals 432 are transferred from the encoded data area 430 to the decoder 450. As decoding continues, new encoded data is added to the encoded data area. 430 and the oldest scrambled data remaining in scrambled data area 430 is transferred to decoder 450.

[00050] Decoder 450 periodically decodes an encoded frame 431 to produce a corresponding decoded frame

Petition 870200084889, of 07/08/2020, page 27/59

20/39

451. As appropriate, when performing its decoding process, decoder 450 may use one or more previously decoded frames 469 as reference frames for interframe prediction. Decoder 450 reads such previously decoded frames 469 from a decoded frame storage area in buffer 460. Generally, decoder 450 includes multiple decoding modules that perform decoding tasks such as entropy decoding, inverse quantization, inverse frequency transformations, and motion compensation. The exact operations performed by the 450 decoder may vary depending on the compression format. [00051] Decoded frame buffer storage area 460 includes multiple frame buffer storage areas 461, 462, ..., 46n. Decoded frame storage area 460 is an example of a decoded image buffer. Decoder 450 uses memory management control signals 432 to identify a frame buffer 461, 462, etc. in which it can store a 451 decoded frame. Decoder 450 stores decoded frame 451 in this frame buffer. [00052] An output sequencer 480 uses memory management control signals 432 to identify when the next frame to be produced in the output order is available in decoded frame storage area 460. To reduce the latency of the encoding-decoding system, the output sequencer 480 uses syntax elements that indicate restrictions on latency to expedite identification of frames to be produced in the output order. When the next frame 481 to be produced in the output order is available in decoded frame storage area 460, it is read by output sequencer 480 and output

Petition 870200084889, of 07/08/2020, page 28/59

21/39 to output destination 490 (e.g. video). In general, the order in which frames are output from decoded frame storage area 460 by output sequencer 480 may differ from the order in which frames are decoded by decoder 450.

V. Elements of Syntax that Facilitate Encoding and Decoding with Low Latency

[00053] In most video codec systems, the encoded order (also called decoding order or bitstream order) is the order in which video frames are represented in encoded data in a bitstream, and consequently , processed during decoding. The encoded order may be different from the order in which frames are captured by a camera before encoding and different from the order in which decoded frames are displayed, stored, or otherwise output after decoding (output order or display order) . Reordering frames relative to output order has benefits (mainly in terms of compressibility), but it increases the end-to-end latency of the encoding and decoding processes. [00054] Techniques and tools described in this document reduce latency due to the reordering of video frames and, by providing information on restrictions regarding the reordering latency for the decoding systems, it also facilitates the reduction of latency by the decoding systems. Such latency reduction is useful for several purposes. For example, it can be used to reduce the response time that occurs in interactive video communication using a video conferencing system, so that the flow of conversation and interactivity of communication between remote participants will be faster and more natural.

A. Approaches to Output Synchronization and Output Ordering

Petition 870200084889, of 07/08/2020, page 29/59

22/39

[00055] According to the H.264 standard, a decoder can use two approaches to determine when a decoded frame is ready to be output. A decoder may use sync information in the form of decoding timestamps and output timestamps (e.g., as signaled in SEI picture sync messages). Or, the decoder may use the buffering capacity limits signaled with various syntax elements to determine when a decoded frame is ready to be output.

[00056] Synchronization information can be associated with each decoded frame. The decoder can use the synchronization information to determine when a decoder frame can be output. However, in practice, such synchronization information may be unavailable to a decoder. Furthermore, even when sync information is available, some decoders do not actually use this information (for example, because a decoder has been designed to function independently of whether sync information is available).

[00057] Buffering capacity limits are indicated with various syntax elements according to the H.264 standard (and earlier versions of the HEVC standard), including the max_dec_frame_buffering syntax element, the num_reorder_frames syntax element, information of relative ordering (referred to as “picture order count information) and other memory management control information signaled in the bitstream. The max_dec_frame_buffering syntax element (or the derived variable specified as MaxDpbFrames) specifies the required size of a decoded image buffer (“DPB) in frame buffer units. Thus, the max_dec_frame_buffering syntax element expresses an ability to

Petition 870200084889, of 07/08/2020, page 30/59

23/39 high-level memory used for an encoded video stream to allow a decoder to output images in the correct order. The syntax element num_reorder_frames (or max_num_reorder_frames) indicates the maximum number of frames (or pairs of complementary fields, or unpaired fields) that can precede any frame (or pair of complementary fields, or unpaired fields) in the coded order and follow the same in outgoing order. In other words, num_reorder_frames specifies a constraint on the memory capacity needed for image reordering. The max_num_ref_frames syntax element specifies the maximum number of short-term and long-term reference frames (or pairs of reference fields, or complementary unpaired reference fields) that can be used by the decoding process for inter prediction of any image. in sequence. The max_num_ref_frames syntax element also determines the size of the sliding window for marking the decoded reference image. Like num_reorder_frames, max_num_ref_frames specifies a restriction on the required memory capacity.

[00058] A decoder uses the max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements to determine when a buffering capacity limit has been exceeded. This happens, for example, when a new decoded frame needs to be stored in the DPB, but there is no available space remaining in the DPB. In this situation, the decoder uses the picture order count information to identify, among the pictures that have been decoded, which is the oldest in the output order. The image that is oldest in the output order is then output. Such processing is sometimes called “collision due to an image being “pushed out of the BPD”

Petition 870200084889, of 07/08/2020, page 31/59

24/39 for the arrival of a new image that needs to be stored.

[00059] The information indicated with the max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements is sufficient to determine the memory capacity required in a decoder. When used to control the “collision to image output” process, however, the use of such information can introduce unnecessarily latency. As defined in the H.264 standard, the max_dec_frame_buffering and num_reorder_frames syntax elements do not set a limit on the amount of reordering that can be applied to any particular image, and therefore do not set a limit on an end-to-end latency. tip. Regardless of the values ​​of these syntax elements, a particular image can be held in the DPB for an arbitrary long time before it is output, which corresponds to the substantial latency added by pre-bushing the source images by an encoder.

B. Elements of Syntax Indicating Restrictions Regarding Frame Reorder Latency

[00060] Techniques and tools described in this document reduce latency in a video communication system. An encoding tool, real-time communication tool, or other tool sets a limit on the extent of reordering that can be applied to any frame in an encoded video sequence. For example, the threshold is expressed as a number of frames that can precede any given frame in a video stream encoded in the output order and follow it in the encoded order. The threshold restricts the reordering latency allowed for any particular frame in the sequence. In other words, the limit restricts the temporal extent of reordering (in terms of frames) between encoded order and output order that

Petition 870200084889, of 07/08/2020, page 32/59

25/39 can be applied to any particular frame. Limiting the reorder extent helps reduce end-to-end delay. Also, setting such a limit can be useful in real-time system negotiation protocols or in application specifications for usage scenarios where reducing latency is important.

[00061] One or more syntax elements indicate restriction regarding frame reorder latency. Flagging a constraint on frame reorder latency facilitates system-level negotiation for interactive real-time communication and other usage scenarios. This provides a way to directly express restrictions regarding frame reordering latency and characterize properties of a stream or media session.

[00062] A video decoder may use a stated restriction on frame reordering latency to allow output of decoded video frames with reduced latency. In particular, compared to “collision” processes, signaling a constraint on frame reordering latency allows a decoder to more simply and more quickly identify frames in a DPB that are ready to be emitted. For example, a decoder can determine the latency condition of a frame in the DPB by calculating the difference between encoded order for the frame and output order. By comparing the frame latency condition to the constraint against frame reordering latency, the decoder can determine when the constraint versus frame reordering latency has been reached. The decoder can immediately output any frame that has reached this limit. This can help the decoder more quickly identify frames that are ready for emission, compared to “collision” processes that use multiple syntax elements and trace structures. In this way, the decoder can quickly (and more

Petition 870200084889, of 07/08/2020, page 33/59

26/39) determine when a decoded frame can be output. The faster (and sooner) the decoder can identify when frames can be output, the faster (and sooner) the decoder can output video to a video or to subsequent stages of processing.

[00063] Thus, using the restriction regarding reordering latency, a decoder can start emitting frames from the decoded frame storage area before the decoded frame storage area is full, but still provide adequate decoding (i.e. i.e. decoding all frames so that the frames are exact bit combinations of frames decoded using another conventional scheme). This significantly reduces the delay when the delay (in frames) indicated by the latency syntax element is much smaller than the size (in frames) of the decoded frame storage area.

[00064] Figures 5a to 5e illustrate series (501 to 505) of frames having different dependencies between frames. The series are characterized by different values ​​for (1) the constraint on the memory capacity required for image reordering (i.e., the number of frame buffers used to store reference frames for reordering purposes, for example, as indicated with a num_reorder_frames syntax element, and (2) a constraint on frame reorder latency, for example, as specified by a MaxLatencyFrames variable. In Figures 5a to 5e, for a given frame the subscript^J indicates frame position in output order and superscript indicates frame position in encoded order. Frames are displayed in order of output - the subscript value of the order of output increases from left to right. The arrows illustrate

Petition 870200084889, of 07/08/2020, page 34/59

27/39 the dependencies between frames for motion compensation, whereby preceding frames in the encoded order are used for prediction of subsequent frames in the encoded order. For simplicity, Figures 5a through 5e present dependencies between frames at the frame level (and not at the level of macroblocks, blocks, etc., in which reference frames can change), and Figures 5a through 5e present a maximum of two frames as reference frames for a given frame. In practice, in some implementations, different macroblocks, blocks, etc., in a given frame, may use different reference frames, and more than two reference frames may be used for the given frame.

[00065] In Figure 5a, the series (501) includes nine frames. The last frame F8¹ in output order uses the first frame F0⁰ as a frame of reference. The other frames in the series (501) use both the last F8 frame¹ as the first frame F0⁰ as a frame of reference. This means that the F0 frame⁰ is decoded first, followed by frame F8¹, followed by frame F1², etc.. In the series (501) presented in Figure 5a, the value of num_reoder_frames is 1. At any point in the processing of the decoder system, among the frames presented in Figure 5a, there is only one frame (F8¹) stored in the decoded frame storage area for reordering purposes. (The first frame F0⁰ it is also used as a frame of reference and is stored, but not stored for reordering purposes. Due to the output order for the first frame F0⁰ be less than the output order of the intermediate frames, the first frame F0⁰ is not counted for the purpose of num_reoder_frames). Regardless of the low value for num_reorder_frames, the series (501) has a relatively high latency - the MaxLatencyFrames value is 7. After encoding the first frame F0⁰, the encoder waits until it has put in me

Petition 870200084889, of 07/08/2020, page 35/59

28/39 temporary memory eight more source frames before encoding the next F1 frame² in output order, due to the next F1 frame² depend on the last frame F8¹ in the series (501). The MaxLatencyFrames value is effectively the maximum difference allowed between the subscript value and the superscript value for any particular encoded frame.

[00066] In Figure 5b, the series (502) includes nine frames, as in the series (501) of Figure 5a, but the dependencies between frames are different. The temporal reordering of frames takes place over short spans. As a result, the (502) series has much less latency - the value of MaxLatencyFrames is 1. The value of num_reorder_frames is still 1.

[00067] In Figure 5c, the series (503) includes ten frames. The longer interframe dependency is shorter (in temporal span) than the interframe dependency in Figure 5a, but longer than the longer interframe dependency in Figure 5b. The series (5 03) has the same low value of 1 for num_reorder_frames, and it has a relatively low value of 2 for MaxLatencyFrames. Therefore, the (503) array allows lower end-to-end latency than the (501) array of Figure 5a, although not as low as the allowable latency of the (502) array of Figure 5b.

[00068] In Figure 5d, the series (504) includes frames organized in a temporal hierarchy with three temporal layers according to the dependencies between frames. The lowest temporal resolution layer includes the first F0 frame⁰ and the last frame F8¹. The next temporal resolution layer adds the F4 frame², which depends on the first frame F0⁰ and from the last frame F8¹. The highest temporal resolution layer added the remaining frames. The series (504) shown in Figure 5d has a relatively low value of 2 for num_reorder_frames, but a relatively high value of 7 for

Petition 870200084889, of 07/08/2020, page 36/59

29/39

MaxLatencyFrames, at least for the highest temporal resolution layer, due to the difference between the encoded order and the Aida order for the last Fe frame¹. If only the intermediate temporal resolution layer or the lowest temporal resolution layer is decoded, the constraint on frame reordering delay can be reduced to 1 (for the middle layer) or 0 (for the lowest layer). To facilitate decoding with reduced latency at various temporal resolutions, syntax elements can indicate restrictions regarding frame reordering latency for different layers in a temporal hierarchy.

[00069] In Figure 5e, the series (505) includes frames organized in a temporal hierarchy with three temporal layers according to different dependencies between frames. The lowest temporal resolution layer includes the first F0 frame⁰, the middle frame F4¹ and the last frame Fe⁵. The next temporal resolution layer adds the F2 frames² (which depends on the first frame F0⁰ and from the middle frame F4¹) and F6⁶ (which depends on the middle frame F4¹ and from the last frame Fe⁵). The highest temporal resolution layer adds the remaining frames. Compared to the (504) series of Figure 5d, the (505) series of Figure 5e still has a relatively low value of 2 for num_reorder_frames, but has a lower value of 3 for MaxLatencyFrames, at least for the highest temporal resolution layer. , due to the difference between the encoded order and the output order for the middle frame F4¹ and for the last frame Fe⁵. If only the intermediate temporal resolution layer or the lowest temporal resolution layer is decoded, the constraint on frame reordering delay can be reduced to 1 (for the middle layer) or 0 (for the lowest layer).

[00070] In the examples presented in Figures 5a to 5e, if the MaxLatencyFrames value is known, a decoder can

Petition 870200084889, of 07/08/2020, page 37/59

30/39 identify some frames as being ready for immediate emission upon receiving the preceding frame in the output order. For a given frame, the frame output order value minus the frame encoded order value can equal the value of MaxLatencyFrames. In this case, the given frame is ready for transmission as soon as the frame preceding it in the output order is received. (In contrast, such frames could not be identified as broadcast-ready using num_reorder_frames alone until additional frames were received or the end of the sequence was reached). In particular, a decoder can use the MaxLatencyFrames value to allow early output of the following frames:

• In the series (501) of Figure 5a, the frame F8¹.

• In the series (502) of Figure 5b, the frames F2¹, F4³, F5⁵, and F87.

• In the series (503) of Figure 5c, the frames F3¹, F6⁴, and F9⁷.

• In the series (504) of Figure 5d, the frame F8¹.

• In the series (505) of Figure 5e, the F4 frames¹ and F8⁵.

[00071] In addition, declaring or negotiating the system-level MaxLatencyFrames value can provide a summarized expression of bitstream or session latency characteristics in a way that is not permitted by measuring the reordering and storage capacity. indication of such capability using num_reorder_frames.

C. Illustrative Implementations

[00072] Syntax elements that indicate a constraint on frame reorder latency can be signaled in various ways depending on the implementation. Syntax elements can be signaled as part of a string parameter set (“SPS), picture parameter set (“PPS), or another bitstream element, signaled as part of messages.

Petition 870200084889, of 07/08/2020, page 38/59

31/39

KNOW. VUI messages or other metadata, or flagged in some other way. In either implementation, a syntax element indicating a constraint value can be encoded using unsigned exponential Golomb encoding, some other form of entropy encoding, or fixed-length encoding, and then signed. A decoder performs the corresponding decoding after receiving the syntax element.

[00073] In a first implementation, a max_latency_limitation_flag flag is flagged. If the indicator has a first binary value (eg 0), no restrictions on frame reorder latency are imposed. In this case, the value of a max_latency_frames syntax element is not flagged or ignored. Otherwise (the indicator having a second binary value such as 1), the value of a max_latency_frames syntax element is flagged to indicate the constraint regarding the reorder latency of when. For example, in this case, the signaled value for the max_latency_frames syntax element can be any non-negative integer value.

[00074] In a second implementation, a max_latency_frames_plus1 syntax element is flagged to indicate the constraint regarding frame reorder latency. If max_latency_frames_plus1 has a first value (eg 0), no restrictions on frame reorder latency are imposed. For other values ​​(e.g. non-zero values), the constraint value regarding frame reorder latency is set to max_latency_frames_plus1 - 1. For example, the value of max_latency_frames_plus1 is in the range of 0 to 2³² - 2 inclusive.

[00075] Similarly, in a third implementation, a max_latency_frames syntax element is flagged to indicate the res

Petition 870200084889, of 07/08/2020, page 39/59

32/39 in relation to the reorder latency of when. If max_latency_frames has a first value (eg a maximum value), no restrictions on frame reorder latency are imposed. For other values ​​(for example, values ​​less than the maximum value), the constraint value regarding frame reorder latency is set to max_latency_frames.

[00076] In a fourth implementation, the constraint on frame reordering latency is stated in relation to a maximum frame memory size. For example, the latency constraint is flagged as an increase relative to the num_reorder_frames syntax element. Typically, the constraint on frame reorder latency (in terms of frames) is greater than or equal to num_reorder_frames. To save bits in signaling the latency constraint, the difference between the latency constraint and num_reorder_frames is encoded (eg, using unsigned exponential Golomb encoding, some other form of entropy encoding) and then signaled. A max_latency_increase_plus1 syntax element is flagged to indicate the constraint regarding frame reorder latency. If max_latency_increase_plus1 has a first value (eg 0), no restrictions on frame reorder latency are imposed. For other values ​​(eg non-zero values), the constraint value regarding frame sort latency is set to num_reorder_frames + max_latency_increase_plus1 - 1. For example, the value of max_latency_increase_plus1 is in the range of 0 to 2³² - 2 inclusive.

[00077] Alternatively, one or more syntax elements that indicate a constraint on frame reorder latency are signaled in some other way.

D. Other Ways to Indicate Restrictions Regarding Latency.

Petition 870200084889, of 07/08/2020, page 40/59

33/39

[00078] In the previous several examples, the constraint on latency is a constraint on frame reorder latency that is expressed in terms of a frame count. More generally, the constraint on latency is a constraint on delay that can be expressed in terms of a frame count or expressed in terms of seconds, milliseconds, or another measure of time. For example, the constraint on latency can be expressed as an absolute time measure such as 1 second or 0.5 second. An encoder can convert such a time measurement to a frame count (considering the video frame rate), then encode the video so that the dependencies between frames between multiple frames of a video sequence are consistent with the count. of frames. Or, independent of frame reordering and dependencies between frames, the encoder can use time measurement to limit the extent to which delay is used to smooth out short-term fluctuations in the bitrate of the encoded video, in the complexity of the encryption, network bandwidth, etc. A decoder can use this time measurement to determine when a frame can be output from a decoded image buffer.

[00079] The constraint on latency can be negotiated between a transmitter side and receiver side in order to negotiate speed of response (no delay) with the ability to smooth out short-term fluctuations in video bitrate coded, ability to smooth out short-term fluctuations in encoding complexity, ability to smooth out short-term fluctuations in network bandwidth, and/or another factor that benefits from increased delay. In such negotiations, it may be useful to establish and characterize the constraint with respect to latency in a way that is independent

Petition 870200084889, of 07/08/2020, page 41/59

34/39 frame rate. Then, the restriction can be applied during encoding and decoding, considering the frame rate of the video. Or, the restriction can be applied during encoding and decoding regardless of the video frame rate.

E. Generalized Techniques for Establishing and Emitting Elements of Syntax.

[00080] Figure 6 presents an illustrative technique (600) to establish and output syntax elements that facilitate decoding with reduced latency. For example, a real-time communication tool or encoding tool described with reference to Figure 2a and Figure 2b performs technique (600). Alternatively, another tool performs the technique (600).

[00081] To begin with, the tool establishes (610) one or more syntax elements that indicate a constraint with respect to latency (e.g. frame reorder latency, latency in terms of a measure of time) consistent with dependency between the frames between multiple frames of a video sequence. When the tool includes a video encoder, the same tool can also receive the frames, encode the frames to produce encoded data (using dependencies between frames that are consistent with the constraint on frame reorder latency), and output encoded data for storage or transmission.

[00082] Typically, a constraint on frame reordering latency is an allowable reordering latency for any frame in the video sequence. However, the restriction can be expressed in several ways and have several other meanings. For example, the constraint can be expressed in terms of a maximum count of frames that can precede a given frame in the output order, but follow the given frame in the encoded order. Or the

Petition 870200084889, of 07/08/2020, page 42/59

This constraint can be expressed as a maximum difference between the encoded order and the output order for any frame in the video sequence. Or, focusing on an individual frame, the constraint can be expressed as the reordering latency associated with a given specific frame in the video sequence. Or, focusing on a group of frames, the constraint can be expressed as the reordering latency associated with the group of frames in the video sequence. Or, the restriction may be expressed in some other way.

[00083] Next, the tool outputs (620) the syntax element(s). This makes it easier to determine when the reconstructed frames are ready for output in terms of the output order of the multiple frames. Syntax element(s) can be output as part of a string parameter set or image parameter set in an encoded video bitstream, as part of syntax for media storage file or streaming media stream which also includes encoded data for the frames as part of a media properties negotiation protocol (e.g. when exchanging stream or session parameter values ​​in system-level negotiation), as part of media system information multiplexed with data encoded for frames, or as part of media metadata relating to data encoded for frames (e.g. in SEI messages or VUI messages). Different syntax elements can be issued to indicate memory capacity requirements. For example, a buffer size syntax element (such as max_dec_frame_buffering) can indicate the maximum size of a DPB, and a frame memory syntax element (such as num_reorder_frames) can indicate the maximum frame memory size for reordering.

[00084] The value of the constraint regarding latency can be represented

Petition 870200084889, of 07/08/2020, page 43/59

36/39 sitting in various ways, as described in the VC section. For example, the tool emits an indicator that indicates the presence or absence of the syntax element(s). If the indicator indicates that syntax elements are missing, then the latency constraint is either undefined or has a pre-set value. Otherwise, the syntax element(s) follows and indicates the constraint regarding latency. Or, a value of the syntax element(s) that indicates the constraint on latency is undefined or has a pre-set value, and other possible values ​​of the syntax element(s) indicate an integer count for the constraint on the latency. latency. Or, for cases where the constraint on latency is a constraint on the reordering latency of when, a given value of the syntax element(s) indicates an integer count for the constraint on the reordering latency of frame against a maximum frame memory size for reordering, which is indicated with a different syntax element such as num_reorder_frames. Alternatively, the constraint on latency is represented in some other way.

[00085] In some implementations, the video sequence frames are organized according to a temporal hierarchy. In this case, different syntax elements may indicate different restrictions regarding frame reordering latencies for different temporal layers of the temporal hierarchy.

F. Generalized Techniques for Receiving and Using Elements of Syntax.

[00086] Figure 7 presents an illustrative technique (700) to receive and use syntax elements that facilitate decoding with reduced latency. For example, a real-time communication tool or playback tool describes with reference to Figures 2a and 2b performs technique (700). Alternatively, another tool Petition 870200084889, of 07/08/2020, p. 44/59

37/39 mint performs the technique (700).

[00087] To start, the tool receives and parses (710) one or more syntax elements that indicate a constraint with respect to latency (eg frame reorder latency, latency in terms of a measure of time). For example, parsing includes reading the one or more syntax elements that indicate the constraint regarding latency from the bit stream. The tool also receives (720) data encoded for multiple frames of a video stream. The tool can parse the syntax element(s) and, based on the syntax element(s), determine the constraint regarding latency. Typically, a constraint on frame reordering latency is an allowable reordering latency for any frame in the video sequence. The restriction can be expressed in several ways, however, and have several other meanings, as described in the previous section. Syntax element(s) may be signaled as part of a sequence parameter set or image parameter set in an encoded video elementary bitstream, as part of syntax for media storage file or streaming video stream. media, as part of a media properties negotiation protocol, as part of the media system information multiplexed with encoded data, or as part of media metadata relating to the encoded data. The tool can receive and parse different syntax elements that indicate memory capacity requirements, for example, a buffer size syntax element such as max_dec_frame_buffering and a frame memory syntax element such as num_reorder_frames.

[00088] The value of the constraint in relation to latency can be represented in several ways, as described in the VC section For example, the tool receives an indicator that indicates the presence or absence of the

Petition 870200084889, of 07/08/2020, page 45/59

38/39 syntax element(s). If the indicator indicates that the syntax element(s) is missing, then the latency constraint is either undefined or has a pre-set value. Otherwise, the syntax element(s) follows and indicates the constraint regarding latency. Or, a value of syntax element(s) that indicates constraint on latency is undefined or has a pre-set value, and other possible values ​​of syntax element(s) indicate an integer count for constraint on latency . Or, for cases where the constraint on latency is a constraint on frame reordering latency, a given value of the syntax element(s) indicates an integer count for the constraint on frame reordering latency. frame against a maximum frame memory size for reordering, which is indicated with a different syntax element such as num_reorder_frames. Alternatively, the constraint on latency is signaled in some other way.

[00089] Returning to Figure 7, the tool decodes (730) at least some of the encoded data to reconstruct one of the frames. The tool outputs (740) the reconstructed frame. In doing so, the tool can use the constraint on latency to determine when the reconstructed frame is ready to be output, for example, in terms of the output order of frames in the video sequence.

[00090] In some implementations, the video sequence frames are organized according to a temporal hierarchy. In this case, different syntax elements may indicate different restrictions regarding frame reordering latencies for different temporal layers of the temporal hierarchy. The tool can select one of different restrictions regarding frame reordering latencies depending on the temporal resolution of the output.

Petition 870200084889, of 07/08/2020, page 46/59

39/39

[00091] In view of the various possible embodiments to which the principles of the disclosed invention can be applied, it should be recognized that the illustrated embodiments are only preferred examples of the invention and should not be considered as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. I therefore claim as my invention all that is within the scope and spirit of these claims.

Petition 870200084889, of 07/08/2020, page 47/59