Selection of motion vector precision.
Abstract
Approaches are presented for selecting the accuracy of the motion vector ("MV") during video encoding. These approaches can facilitate compression that is effective in terms of rate distortion performance and / or computational efficiency. For example, a video encoder determines the MV precision for a video unit among multiple MV accuracies, which include one or more MV accuracies of fractional sample and the MV precision of the entire sample. The video encoder can identify a group of MV values that have the MV precision of fractional sample, then select the MV precision for the unit based at least partly on the prevalence of MV values (within the group) that has a fractional part of zero. Otherwise, the video encoder can perform a rate distortion analysis, wherein the rate distortion analysis is driven towards the whole sample MV precision. Otherwise, the video encoder can collect information about the video and select the MV precision for the unit based at least in part on the information collected.

Term
8.2 yearsleft in the term
Expires 19 December 2034.
- Priority
- Filed
- Granted
- Today
- Expires
16 claims: 2 independent, 14 dependent
- 1- Un medio legible por computadora siendo seleccionado del grupo que consiste en memoria no volátil, disco magnético, CD-ROM, y DVD, que provoca que un dispositivo de cómputo realice un método que comprende:codificar un video, que incluye determinar una precisión de vector de movimiento (“MV”) para una unidad del video, en donde los valores de MV para los bloques dentro de la unidad del video tienen la precisión de MV para la unidad, y en donde la determinación de la precisión de MV para la unidad incluye: identificar un grupo de valores de MV que tienen una precisión de MV de muestra fraccionada;medir la prevalencia, dentro del grupo de valores de MV, de valores de MV que tienen una parte fraccionada de cero;comparar la medida de prevalencia a un umbral;y seleccionar la precisión de MV para la unidad con base por lo menos en parte en los resultados de la comparación;y emitir el video codificado.
- 2- El medio legible por computadora de conformidad con la reivindicación 1, en donde:la medida de prevalencia es la fracción del grupo de valores de MV que tienen una parte fraccionada de cero;o una región o grupo de regiones usan el grupo de valores de MV, y la medida de prevalencia es la fracción de la región o el grupo de IMPI INSTITUTO mexicano regiones que usan uno de los valores de MV'^J^^eiW^nKr'parte fraccionada de cero. -
- 3- El medio legible por computadora de conformidad con la reivindicación 2, en donde, si la medida de prevalencia excede el umbral, la precisión de MV seleccionada para la unidad es una precisión de MV de muestra entera, y en donde, si la medida de prevalencia no excede el umbral, la precisión de MV seleccionada para la unidad es la precisión de MV de muestra fraccionada.
- 4- El medio legible por computadora de conformidad con la reivindicación 1, en donde la precisión de MV seleccionada para la unidad es una precisión de MV de muestra entera, en donde la precisión de MV seleccionada para la unidad también se usa para las unidades posteriores del video hasta que un evento provoca un cambio de regreso a la precisión de MV de muestra fraccionada, y en donde el evento es:codificación de un número definido de unidades;un cambio de escena;o una determinación, con base en las observaciones durante la codificación, que el cambio de regreso a la precisión de MV de muestra fraccionada será benéfico.
- 5- El medio legible por computadora de conformidad con la reivindicación 1, en donde el grupo de valores MV es:permitido para incluir MVs de valor cero y MVs de valor no cero;restringido para incluir solamente MVs de valor no cero;o restringido para incluir solamente MVs de valor no cero de los bloques de un cierto tamaño de bloque o más grandes. con la IMPIO INSTITUTO MEXICANO DE LA ΡΕΟίΊΕΟΑΟ \ *«3.- 6.- El medio legible por computadora dé L J Se)ttformn reivindicación 1, en donde seleccionar la prooioión de MV palé raHiTidad también se basa por lo menos en parte en una medida de prevalencia de MVs de valor no cero, de modo que se permite el cambio a la precisión de MV de muestra entera si hay una cantidad umbral de MVs de valor no cero, en donde la medición de prevalencia de los MVs de valor no cero es (a) la fracción de valores de MV que son MVs de valor no cero;(b) la cuenta de bloques que usan MVs de valor no cero, o (c) la fracción de una región o grupo de regiones que usan MVs de valor no cero, y en donde el grupo de valores de MV que tienen una precisión de MV de muestra fraccionada se identifica de entre los MVs de valor no cero de la región o del grupo de regiones. 7, - El medio legible por computadora de conformidad con la reivindicación 1, en donde la unidad del video es una unidad actual del video, y en donde: la codificación es una codificación de un solo pase, y la precisión de MV seleccionada para la unidad actual depende por lo menos en parte en una o más unidades previas del video;o la codificación es una codificación de múltiples pases, y la precisión de MV seleccionada para la unidad actual depende al menos en parte en la unidad actual del video.
- 68. - El medio legible por computadora de conformidad con la reivindicación 1, en donde las operaciones además comprenden:ajustar la cantidad de impulso hacia o contra la precisión de MV de muestra entera con base por lo menos en parte en (a) un grado de •««.te*. IMPJ4 INSTITUTO MEXICANO 7 D€ ΙΛ FHOHtUAtS confianza de que la precisión de MV de muestra enterarás apfó'ÉHada y/o (b) la capacidad computacional de codificac’on“y/o la déc63TFTca t cíom’ 9.- El medio legible por computadora de conformidad con la reivindicación 1, en donde la precisión de MV seleccionada para la unidad es para los componentes de MV horizontales y/o para los componentes de MV verticales de los valores MV para los bloques dentro de la unidad del video.
- 710. - El medio legible por computadora de conformidad con la reivindicación 1, en donde la unidad se selecciona del grupo que consiste en una secuencia, serie de imágenes entre cambios de escena, grupo de imágenes, imagen, mosaico, unidad de árbol de codificación de rebanada, y unidad de codificación, y en donde los bloques son bloques de predicción, unidades de predicción, macrobloques, o divisiones de sub-macrobloques.
- 811. - Un dispositivo de cómputo, que comprende:una o más unidades de procesamiento;memoria volátil;y memoria no volátil y/o almacenamiento, la memoria no volátil y/o almacenamiento que provocan que el dispositivo de cómputo realice un método que comprende: codificar un video, con un codificador de video, que incluye determinar una precisión de vector de movimiento (“MV”) para una unidad del video de entre múltiples precisiones de MV, las múltiples precisiones de MV incluyen una o más precisiones de MV de muestra fraccionada y una precisión de MV de muestra entera, en donde los IMP INSTITUTO MLXICaNC DE LA PROMEDAT INDUSTRIA!. valores de MV para los bloques dentro de la unidad del video tienen la precisión de MV para la unidad, y en donde la determinación incluye: recolectar información acerca del video, en donde la información recolectada incluye un grupo de valores de MV en una de la una o más precisiones de MV de muestra fraccionada;medir la prevalencia, dentro del grupo de valores de MV, de valores de MV que tienen una parte fraccionada de cero;comparar la medida de prevalencia a un umbral;y seleccionar la precisión de MV para la unidad con base por lo menos en parte en resultados de la comparación;y almacenar, en una memoria, el video codificado para su salida.
- 912. - El dispositivo de cómputo de conformidad con la reivindicación 11, en donde los valores de MV recolectados son organizados de acuerdo al valor de sus partes fraccionadas.
- 1013. - El dispositivo de cómputo de conformidad con la reivindicación 11, en donde la unidad del video es una unidad actual del video, y en donde:la codificación es una codificación de un solo pase, y la precisión de MV seleccionada para la unidad actual depende por lo menos en parte en una o más unidades previas del video;o la codificación es una codificación de múltiples pases, y la precisión de MV seleccionada para la unidad actual depende al menos en parte en la unidad actual del video.
- 1114. - El dispositivo de cómputo de conformidad con la reivindicación 11, en donde la codificación comprende además;IMPI INSTITUTO MEXICANO INSTITUTO MEXICANO v ajustar la cantidad de impulso hacia o de muestra entera con base por lo menos en parte en (a) un grado de confianza de que la precisión de MV de muestra entera es apropiada y/o (b) la complejidad computacional de codificación y/o la decodificación.
- 1215. - El dispositivo de cómputo de conformidad con la reivindicación 11, en donde la precisión de MV seleccionada para la unidad es para los componentes de MV horizontales y/o para los componentes de MV verticales de los valores MV para los bloques dentro de la unidad del video.
- 1316. - El dispositivo de cómputo de conformidad con la reivindicación 11, en donde la unidad se selecciona del grupo que consiste de una secuencia, serie de imágenes entre cambios de escena, grupo de imágenes, imagen, mosaico, unidad de árbol de codificación de rebanada, y unidad de codificación, y en donde los bloques son bloques de predicción, unidades de predicción, macrobloques, o divisiones de sub-macrobloques.
- 1417. - En un dispositivo de cómputo, un método que comprende:codificar un video, que incluye determinar una precisión de vector de movimiento (“MV”) para una unidad del video, en donde los valores de MV para los bloques dentro de la unidad del video tienen la precisión de MV para la unidad, y en donde la determinación de la precisión de MV para la unidad incluye: identificar un grupo de valores de MV que tienen una precisión de MV de muestra fraccionada: medir la prevalencia, dentro del grupo de valores de MV, de IMPI INSTITUTO MEXICANA D€ LA PROPIEDAD Ό©τvalores de MV que tienen una parte fraccionada de tfSW* IAL comparar la medida de prevalencia a un-emi'bral;y— seleccionar la precisión de MV para la unidad con base por lo menos en parte en los resultados de la comparación;y emitir el video codificado. 18.- El método de conformidad con la reivindicación 17, en donde: la medida de prevalencia es la fracción del grupo de valores de MV que tienen una parte fraccionada de cero;o una región o grupo de regiones que usan el grupo de valores de MV, y la medida de prevalencia es la fracción de la región o el grupo de regiones que usan uno de los valores MV que tienen una parte fraccionada de cero.
- 1519. - El método de conformidad con la reivindicación 17, en donde si la medida de prevalencia excede el umbral, la precisión de MV seleccionada para la unidad es una precisión de MV de muestra entera, y en donde, si la medida de prevalencia no excede el umbral, la precisión de MV seleccionada para la unidad es la precisión de MV de muestra fraccionada.
- 1620. - El método de conformidad con la reivindicación 17, en donde la precisión de MV seleccionada para la unidad es una precisión de MV de muestra entera, en donde la precisión de MV seleccionada para la unidad también se Usa para las unidades posteriores del video hasta que un evento provoca un cambio de regreso a la precisión de MV de muestra fraccionada, y en donde el evento es:codificación de un número definido de unidades;
Independent claims16
377 paragraphs in 45 sections, as filed
Engineers use compression (also called source encoding or source encoding) to reduce the bit rate of digital video. Compression lowers the cost of storing and transmitting video information by converting the information into a lower bitrate form. Decompression (also called decoding) reconstructs the version of the original information from the compressed form. A "codec" is an encoder / decoder system.
Background of the invention
Over the past two decades, various video encoding standards have been adopted, including the ITU-T H.261, H.262 (MPEG-2 or ISO / IEC 13818-2), H.263 AND H.264 (MPEG-4 AVC or ISO / IEC 14496-10), MPEG-1 (ISO / IEC 11172-2) and MPEG-4 Visual (ISO / IEC 14496-2) and SMPTE 421m (VC-1). In recent years, the HEVC standard (ITU-T H.265 or ISO / IEC 23008-2) has been approved. Extensions for the \ HEVC standard (for example, scalable video encoding / decoding to encode / decode video with higher fidelity in terms of sample bit depth or color sampling rate, or for multi-view encoding / decoding ) is currently under development. A norm
ΙΜ.ΡΙ
INSTITUTO MEXICANO JA for a typical video encoder / decoder W ^ j ^^ fify ^ ü ^^ Options for the syntax of an encoded video bitstream, which details on the bitstream when particular characteristics are used for encoding and decoding. In many cases, a video encoder / decoder standard also provides details about the decoding operations that a decoder must update to achieve decoding results. In addition to the encoder / decoder standards, several unique codec formats define other options for the syntax of an encoded video bitstream and the corresponding decoding operations.
In general, video compression techniques include “intra-picture” compression and “inter-picture compression. Intra-image compression techniques compress individual images and inter-image compression techniques compress images with reference to the next image or the previous image (often called a reference or anchor image) or images.
Inter-image compression techniques often use motion estimation and motion compensation to reduce the bit rate by exploiting temporal redundancy in the video stream. Motion estimation is a process for calculating motion between images. In a common technique, an encoder using motion estimation attempts to match a current block of sample values in a current image to a candidate block of the same size in a search area in another image, the reference image. When the encoder finds an exact match or
JL Ivi. k JL
INSTITUTO MEXICANO DS LA PROPERTY
V · »jKÍM tHDQSTFIAJ“ close enough ”in the search area in the reference image, the encoder parameterizes the change in ¡a * position between the current and candidate blocks as motion data (such as a motion vector (“ MV ”)).
A MV is a conventional, two-dimensional value, which has a horizontal MV component that indicates left or right spatial shift and a vertical MV component that indicates upward or downward spatial shift. In general, motion compensation is a process of reconstructing images from reference images with the use of motion data.
An MV can indicate spatial displacement in terms of an integer number of sample grid positions starting from the position co-located in the energy image for a current block. For example, for a current block at position (32, 16) in a current image, the MV (-3, 1) indicates position (29, 71) in the reference image. Otherwise, the MV may indicate a spatial shift in terms of a fractional number of grid positions displayed from the position co-located in the reference image for a current block. For example, for a current block at position (32, 16) in a current image, the MV (-3.5, 1.25) indicates position (28.5, 17.25) in the reference image. To determine the sample values at fractional offsets in the reference image, the encoder typically interpolates between the sample values at the integer sample positions. Such interpolation can be computationally intensive. During motion compensation, the decoder
<img file="MX359698B_D0001.tif" />
it also performs interpolation as needed to compute sample values at fractional offsets in reference images.
Different video codec standards and formats have used MVs with different MV precisions. For integer sample MV precision, an MV component indicates an integer of the sample grid positions for spatial displacement. For the fractional sample MV position, such as the MV precision of Vi sample or the MV position of% of sample, the MV component can indicate an integer of the sample grid positions or the fractional number of the sample grid positions. sample for spatial displacement. For example, when the MV precision is the MV position of% sample, an MV component can indicate the spatial shift of 0 samples, 0.25 samples, 0.5 samples, 0.75 samples, 1.0 samples, 1.25 samples, and so on. Some codec standards and formats offer support for changing MV precision during encoding. Encoder-side decisions about which MV precision to use are not as effective in certain encoding scenarios.
Brief description of the invention
In summary, the detailed description presents innovations in encoder-side operations for motion vector ("MV") precision selection. For example, when a video encoder encodes the video, the video encoder determines the MV accuracy for one unit of the video.
IMPI
MEXICAN INSTITUTE
<img file="MX359698B_D0002.tif" />
In accordance with one aspect of the innovations described herein, when determining the MV precision for the unit, the video encoder can identify a group of MV values that have the fractional sample MV precision. The video encoder may select the MV precision for the unit based at least in part, on the prevalence, within the group of MV values, of MV values having a fractional part of zero.
Consistent with another aspect of the innovations described here, when determining the MV precision for the unit, the video encoder can update a rate distortion analysis to decide between multiple MV precision, which includes one or more sample MV precision. fractional and whole sample MV precision. Rate distortion analysis is driven toward integer sample MV precision by: (a) scaling the distortion cost; (b) by adding a distortion cost penalty; (c) scaling the cost of the bit rate; (d) add a penalty to the cost of the bit rate and / or (e) adjust the multiplier factor
Langrangian.
Consistent with another aspect of the innovations described here, when determining the MV precision for the unit, the video encoder can collect information about the video and select the MV precision for the unit from multiple MV precisions based on what less in part, in the information collected. Multiple MV precisions include one or more fractional sample MV precision and whole sample MV precision.
WICKED
MEXICAN INSTITUTE
The innovations for the Igfdt ^^^ potB ^ iCTÓ ^ options for the selection of MV precision can be implemented as Darte. , of a method, as part of a computing device adapted to perform the method or as part of a computer-readable medium that stores computer-executable instructions to cause the computing device to perform the method. The different innovations can be used in combination or separately.
The foregoing and other objects, advantages and characteristics of the invention will be apparent from the detailed description, which is specified with reference to the accompanying figures.
Brief description of the drawings
Figure 1 is a diagram of an exemplary computer system where some of the modalities described here can be implemented.
Figures 2a and 2b are exemplary network environment diagrams where the modalities described here can be implemented.
Figure 3 is a diagram of an exemplary encoder system in conjunction with which the embodiments described herein may be implemented.
Figures 4a and 4b are diagrams illustrating an exemplary video encoder in conjunction with which the modalities described herein can be implemented.
Figure 5 is a diagram illustrating a powered desktop environment that can provide input for the
<img file="MX359698B_D0003.tif" />
IMPI screenshot.
Fig. 6 is a diagram illustrating video with content mixed with natural video content and artificial video content.
Figures 7a and 7b are diagrams illustrating motion compensation with MV values having the whole sample spatial shift and fractional sample spatial shift, respectively.
Figure 8 is a flow chart illustrating a generalized technique for tailoring MV precision during encoding.
Figure 9 is a flow chart illustrating an exemplary technique for tailoring MV precision during encoding using a low complexity approach.
Figure 10 is a diagram illustrating different regions of an image in accordance with certain low complexity focus variations.
Detailed description of the invention
The detailed description presents innovations for the selection of the precision of the motion vector ("MV") during encoding. These approaches can facilitate compression that is effective in terms of rate distortion performance and / or computational efficiency. For example, a video encoder determines the MV precision for a video unit between multiple MV precision, which includes one or more fractional sample MV precision and the sample MV precision.
<img file="MX359698B_D0004.tif" />
IMPI
MEXICAN INSTITUTE OF THE ENTIRE PROPERTY. The video encoder can identify?<sup>D</sup>or<sup>T</sup>n '<sup>TO</sup>group<sup>J</sup>^ é'MV values that have the fractional sample MV precision, then the MV precision for the unit is selected based, at least in part, on the prevalence of MV values (within the group) that have a partial fractional of zero. Otherwise, the video encoder can carry out the rate distortion analysis, where the rate distortion analysis targets the whole sample MV precision. Otherwise, the video encoder can collect information about the video and select the MV accuracy for the unit based at least in part on the information collected. Otherwise, the video encoder can determine the MV accuracy for a video unit in some other way.
Although the operations described here are in places described as being carried out by the video encoder, in many cases, the operations can be carried out by another type of media processing tool.
Some of the innovations described here are illustrated with reference to the syntax elements and operations specific to the HEVC standard. The innovations described here can also be implemented for other standards or formats.
More generally, different alternatives are possible for the examples described here. For example, some of the methods described here can be altered by changing the order of actions in the described method, dividing, repeating, or omitting certain actions in the method, and so on. The different aspects of the technology described can be used in combination or separately. The different modalities use a
<img file="MX359698B_D0005.tif" />
or more of the innovations described.
actions
IMPIOUS
MEXICAN INSTITUTE. . OF THE PROPERTY, Some / o ^ of ^ the * described here address one or more problems in the background. Typically, a given technique / tool does not solve all problems.
1. Exemplary computer systems
Figure 1 illustrates a generalized example of a suitable computing system (100), where some of the innovations described here can be implemented. The computer system (100) is not intended to suggest any limitation to the scope of use or functionality, as the innovations can be implemented in various computer systems, including special-purpose computer systems adapted for video encoding.
With reference to Figure 1, the computing system (100) includes one or more processing units (110, 115) and a memory (120, 125). A processing unit can be a central processing unit ("CPU"), a processor in an application specific integrated circuit ("ASIC"), or any other type of processor. In a multiple-processing system, multiple-processing units execute computer-executable instructions to increase processing power. For example, Figure 1 shows a central processing unit (110) as well as a graphics processing unit or co-processing unit (115). The tangible memory (120, 125) can be a volatile memory (for example, registers, cache, RAM), a non-volatile memory (for example, ROM, EEPROM, memory
IMPI flash, etc.), or some combination of the twoTa ^^^ SlSn indüItui / ú.
<img file="MX359698B_D0006.tif" />
through the processing units. The memory (120, 125) stores software (180) that implements one or more innovations for the selection of MV precision during encoding, in the form of appropriate computer-executable instructions to be executed by the processing units.
A computer system may have additional features. For example, the computing system (100) includes a storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown) such as a busbar, controller or network interconnects the components of the computing system (100). Typically, the operating system software (not shown) provides an operating environment for other software that runs on the computing system (100) and coordinates the activities of the components of the computing system (100).
Tangible storage (140) can be removable or non-removable and includes magnetic disks, magnetic tapes or magnetic cartridges, CD-ROM, DVD or any other medium that can be used to store information and is accessed within the system (100 ) of computation. Storage 140 stores instructions for software 180 that implements one or more innovations for MV precision selection during encoding.
The input devices (150) can be a touch input device, such as a keyboard, a mouse, a pen, a ball
IMPI mexican institute
DíuraoMsoAi; · A
INOUSTI ::, ··· .:
follower, a voice input device, a scanning device or · _ J> 4; · another device that provides inputs to the computing system (100).
For video, the input devices (150) can be a camera, a video card, a TV tuner card, a screen capture module or a similar device that accepts video input in analog or digital form or a CD-ROM or CD-RW that reads the video input into the computer system (100). The output devices (160) can be a display, printer, speaker, CD writer, or other device that provides the output from the computer system (100).
The communication connections (170) allow communication over a communication medium to another computing entity. The communication medium conducts information, such as computationally executable instructions, audio or video input or output, or other data into a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics fixed or changed in such a way as to encode the information in the signal. As an example and without any intention of limiting, the communication medium may use an electrical, optical, RF or other carrier.
Innovations can be described in the general context of computer-readable media. Computer-readable media is any tangible medium available that is accessed within the computing environment. As an example and not intended to limit, with the computer system (100), the computer-readable medium includes a memory (120, 125), a storage (140) and combinations of the
<img file="MX359698B_D0007.tif" />
<img file="MX359698B_D0008.tif" />
Previous INSTITUI OR MEXICAN. mlamopiepaü miiiviiwiuq. ΐΝηΐ /, ΓΓκιν
Innovations can be described in the general context of computer-executable instructions, such as those included in program modules, to be executed in a computer system on a current or virtual object processor. In general, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that can carry out tasks or implement particular abstract data types. The functionality of the program modules can be combined or divided between the program modules, as appropriate in various modes. The compute-executable instructions for the program modules can be executed within a local or distributed computer system.
The terms "system" and "device" are used interchangeably. Unless the context clearly indicates otherwise, no term implies any limitation on the type of computing system or computing device. In general, the computer system or computing device can be local or distributed and can include any combination of special-purpose hardware and / or hardware with software that implements the functionality described herein.
The methods described can also be implemented with the use of specialized computing hardware configured to carry out any of the methods described here. For example, the methods described can be implemented by an integrated circuit (eg, an ASIC, such as an ASIO digital signal processor ("DSP"), a graphics processing unit ("GPU"), or a programmable logic device.
ΙΜΡΙΟ ^ (“PLD”), such as a compuePtéTiH ^^ taf ^^^^ re field array (“FPGA”)), specially designed or configured, Qara._jmnJgment any of the methods described.
For clarity of presentation, the detailed description uses terms such as "determine" and "use" to describe the operations of the computer in a computer system. These terms are high-level abstractions for operations carried out by the computer and should not be confused with actions carried out by the user. The actual compute operations for those terms vary depending on the implementation. As used herein, the term "optimize" (including its variations, such as optimization and optimized) refers to a choice between options under a decision scope and does not imply that an optimal option is the "best" option for the extended scope. decision making.
II. Exemplary network environments
Figures 2a and 2b show exemplary network environments (201, 202) that include video encoders (220) and video decoders (270). Encoders (220) and decoders (270) are connected over network (250) using an appropriate communication protocol. The network 250 may include the Internet or other appropriate computer network.
In the network environment (201) shown in Figure 2a, each current-time communication ("RTC") tool (210) includes both an encoder (220) and a decoder (270) for communication.
Bidirectional IMPI. An encoder (220) determined ^ '- f ^^^ ipr ^ WÍE'atrna output compatible with the variation or extension of the HEVC-hamhián standard known as H.265), the SMPTE 421M standard, the ISO / IEC 14496- 10 (also known as H.264 or AVC), another standard or a unique format, with a corresponding decoder (270) that accepts the encoded data from the encoder (220). Two-way communication can be part of a video conference, video phone call, or other two-party communication or multi-party communication scenario. Although the network environment (201) in Figure 2aa includes two current-time communication tools (210), the network environment (201) may instead include three or more current-time communication tools (210). , which participate in multi-party communication.
The current-time communication tool (210) handles the encoding by an encoder (220). Figure 3 shows an exemplary encoder system (300) that may be included in the present-time communication tool (210). Alternatively, the current time communication tool 210 uses another encoder system. The current-time communication tool (210) also handles decoding by the decoder (270).
In the network environment (202) shown in Figure 2b, the encoding tool (212) includes an encoder (220) that encodes the video for delivery to multiple playback tools (214), including set-top boxes (270). One-way communication can be provided for a video surveillance system, a video surveillance system,
<img file="MX359698B_D0009.tif" />
IMPI
INSTITUTO MEXICANO DE LA FROPIEDAD INDUSTRIAL network camera monitoring, a screen capture module, a remote conference presentation or other scenario where video is encoded and sent from one location to one or more other locations. Although the network environment (202) in Figure 2b includes two playback tools (214), the network environment (202) may include more or fewer playback tools (214). In general, the playback tool (214) communicates with the encoding tool (212) to determine that the playback tool (214) receives a video stream. The playback tool (214) receives the current, memorizes the received encoded data for an appropriate period, and begins decoding and playback.
Figure 3 shows an exemplary encoder system (300) that may be included in the encoding tool (212). Alternatively, the encoding tool 212 uses another encoder system. The encoding tool (212) may also include server-side controller logic to handle connections to one or more replay tools (214). A replay tool (214) may also include client-side controller logic to handle connections to the encoding tool (212).
III. Exemplary Encoder Systems
FIG. 3 is a block diagram of an exemplary encoder system 300 in conjunction with which some of the described embodiments may be used. The encoder system (300) may be a
MEXICAN INSTITUTE
Di THE INDUSTRIAL PROPERTY tWjXjW general purpose encoding tool with the ability to operate in any of multiple encoding modes, such as the current low latency encoding mode, a transcoding mode and for time communication a high latency encoding mode for producing media for playback from a file or stream or may be a special purpose encoding tool adapted for such encoding mode. The encoder system 300 can be implemented as an operating system module, as part of an application library, or as a standalone application. In general, the encoder system (300) receives the sequence of source video frames (311) from the video source (310) and produces encoded data as output for one channel (390). The encoded data output for the channel may include the content encoded with the use of a selected MV precision.
The video source 310 can be a camera, tuner card, storage medium, screen capture module, or other digital video source. The video source 310 produces a sequence of video frames at a frame rate of, for example, 30 frames per second. As used herein, the term "frame" generally refers to source, encoded, or reconstructed image data. For progressive scan video, a frame is a progressive scan video frame. For interlaced video, in exemplary embodiments, an interlaced video frame can be deinterlaced prior to encoding. Alternatively, two complementary interlaced video fields are encoded together as a single
<img file="MX359698B_D0010.tif" />
video frame
IMPI / T
MEXICAN INSTITUTE
I HEARD THE INDUSTRIAL CURRENCY is encoded as two separate encoded fields. In addition to indicating a progressive scan video frame or interlaced scan video frame, the term "frame or" image may indicate a single unparalleled video field, a complementary pair of video fields, an object plane of video representing the video object at a point in time or a region of interest in a larger image. The video object plane or region can be part of a larger image that includes multiple objects or regions of a scene.
An incoming source frame (311) is stored in a source frame buffer storage area (320), which includes multiple frame buffer storage areas (321, 322, ..., 32n). A frame buffer (321, 322, etc.) maintains a source frame in the source frame storage area (320). After one or more of the source frames (311) have been stored in the frame memories (321, 322, etc.), a frame selector (330) selects an individual source frame from the storage area (320). source box. The order in which the frames are selected by the frame selector (330) to be input to the encoder (340) may differ from the order in which the frames are produced by the video source (310), for example, the encoding of some frames can be delayed in order, thus allowing subsequent frames to be coded first and thus facilitating temporal back-prediction. Before encoder 340, encoder system 300 may include a preprocessor (not shown) that performs preprocessing (e.g.
<img file="MX359698B_D0011.tif" />
example, the
INSTITUTO MEXICANO Filtration) Box (331) se ^ eoc ^ Rtá
<img file="MX359698B_D0012.tif" />
encoding. Preprocessing can · * μ * ιηϊγ of color within the primary (for example, brightness) and secondary (for example, color differences towards red and towards blue) components and resampling processing (for example, to reduce the spatial resolution of color components) for encoding. Typically, prior to encoding, the video has been converted into a color space, such as YUV, where the sample values of the brightness component (Y) represent the values of brightness or intensity and the sample values of the color (U , V) represent the values of different colors. Color sample values can be subsampled at a lower color sample rate (for example, for a YUV 4: 2: 0 or YUV 4: 2: 2 format) or color sample values can have the Same resolution as the brightness sample values (for example, for a 4: 4: 4 YUV format). In the YUV 4: 2: 0 format, the color components are sampled descending by a factor of two horizontally and by a factor of two vertically. In the YUV 4: 2: 2 format, the color components are sampled descending by a factor of two horizontally. Otherwise, the video can be encoded in another format (for example, a 4: 4: 4 RGB format).
Encoder 340 encodes selected frame 331 to produce encoded frame 341 and also outputs memory management control operation ("MMCO") signals 342 or reference image group information ( "RPS"). When the current frame is not the first frame that has been encoded, when its processing is carried out
<img file="MX359698B_D0013.tif" />
IMPI
INSTITUTO MEXICANO OS LA FRCriFDA ·?
In order to encode, the encoder 340 may utilize one or more previously encoded / decoded frames 369 that have been stored in a decoded frame buffer storage area 360. Such stored decoded frames (369) are used as the reference frames for inter-frame prediction of the content of the current source frame (331). The MMCO / RPS information 342 indicates to a decoder the reconstructed frames that can be used as reference frames and therefore must be stored in a frame storage area.
In general, the encoder 340 includes multiple encoding modules that perform encoding tasks such as tiling, intraprediction estimation and prediction, motion estimation and compensation, frequency transformations, quantization and entropy encoding. The exact operations updated by the encoder 340 may vary depending on the compression format. The output encoded data format can be a variation or extension of the HEVC (H.265) format, Windows Media video format, VC-1 format, MPEG-x format, (for example, MPEG-1, MPEG- 2 or MPEG-4), the H.26x format (for example, H.261, H.252, H.263, H.264), or another format.
Encoder 340 can divide the frame into multiple tiles of the same size or of different sizes. For example, the encoder (340) divides the mosaic along rows of mosaic and columns of rows that, with the limits of the mosaic, define the horizontal and vertical limits of the mosaics within the frame, where each mosaic is a
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PIIOETEDaO
<img file="MX359698B_D0014.tif" />
rectangular region. Tiles are often used to provide options for parallel processing. A square can also be arranged as one or more slices, where one slice can be a whole square or region of the square. A slice can be decoded independently of other slices in a frame, improving error resilience. The content of a slice or mosaic is also divided into blocks or other groups of samples for the purposes of encoding and decoding.
For HEVC-compliant syntax, the encoder divides the content of a frame (or slice or tile) into encoding tree units. A coding tree unit ("CTU") includes the brightness sample values organized as one color coding tree block ("CTB") and corresponding sample values organized as two color CTBs. The size of a CTU (and its CTB) is selected by the encoder. A color CTB can contain, for example, 64x64, 32x32, or 16x16 sample brightness values. A CTU includes one or more coding units. A coding unit ("CU") has a brightness coding coding block ("CB) and two corresponding color CBs. For example, a CTU unit with one 64x64 brightness CTB block and two 64x64 color CTBs (4: 4: 4 YUV format) can be divided into four CUs, each CU includes one 32x32 brightness CB and two color CBs 32x32 and each CU is possibly divided into smaller CUs. The smallest allowable CU size (eg 8x8, 16x16) can be signaled within the bit streams.
In general, a CU has a prediction mode that is inter- or
<img file="MX359698B_D0015.tif" />
INSTITUTO MEXICANO Dt LA PROPERTY
INDUSTRIAL intra-. A CU includes one or more prediction units for the purpose of signaling prediction information (such as prediction mode details, offset values, etc.) and / or prediction processing. A prediction unit (“PU”) has one brightness prediction block (“PB) and two color PBs. For an intrapredicted UC, the PU is the same size as the UC, unless the UC is the smallest size (for example, 8x8). In this case, the CU can be divided into four smaller PUs (for example, every 4x4 when the smallest CU is 8x8) or the PU can have the smallest CU size, as indicated by a syntax element for the CU. A CU also has one or more transform units for residual encoding / decoding purposes, wherein the transform unit ("TU") has the brightness transform block ("TB") and two TBs of color. A PU in an intra-predicted CU can contain a single TU (equal in size to the PU) or multiple TUs. The encoder decides the way to divide the video into CTU, CU, PU, TU, etc. In the context of the H.264 / AVC standard, the term “macroblock” indicates a block-shaped region similar to that of a CTU for the H.265 / HEVC standard, and the term “sub-macroblock division indicates a region with a similar to that of a CU or PU. As used herein, the term "block" can indicate a CB, PB, TB, CTU, CU, PU, TU, macroblock, sub-macroblock split, or other group of sample values, depending on the context.
Referring again to Figure 3, the encoder represents an intracoded block of a source frame (331) in terms of predicting other previously reconstructed sample values in frame (331).
IMPI
MEXICAN USTITUTE
For an intrablock copy prediction<sup>THE</sup>i ^ ért¿A<sup>,</sup>heh,
<img file="MX359698B_D0016.tif" />
r „to other intraimage computes the offset of previously reconstructed sample values. An intra-frame prediction reference region (or intra-prediction region for short) is a region of samples in the frame that are used to generate the BC prediction values for the block. The intra-frame prediction region can be indicated with a block vector value ("BV") (determined in the BV estimate). For the intraspatial prediction for a block, the intraimage estimator estimates the extrapolation of the reconstructed sample values, neighbors within the block. The intra-image estimator can output the prediction information (such as BV values for intra-BC prediction or prediction mode (direction) for intraspatial prediction), which is entropy encoded. An intraframe prediction predictor applies the prediction information to determine the intraprediction values.
Decoder 340 represents a predicted block, interframe encoded from a source frame 331 in terms of prediction from reference frames. The motion estimator calculates the motion of the block with respect to one or more reference frames (369). The motion estimator can select the precision of the motion vector (“MV”) (for example, whole sample MV precision, MV precision of<sup>1</sup>Λ sample or sample% MV precision) as described here, then use the MV precision selected during motion estimation. When using multiple reference frames, the multiple reference frames may be in different
MEXICAN INSTITUTE OF PROPERTY ... industrial temporal directions or in the same temporal direction Tijna motion compensated prediction reference region is a region of the samples in the reference frames that is used to generate the motion compensated prediction values for a block of samples in a current frame. The motion estimator outputs motion information such as MV information, which is entropy encoded. A motion compensator applies the MV values having the MV precision selected to the reference frames 369 to determine the motion compensated prediction values for the interframe prediction.
The encoder can determine the differences (if any) between the prediction values of the blocks (intra or inter) and the corresponding original values. These prediction residuals are also encoded using a frequency transform (when the frequency transform is not skipped), quantization, and entropy encoding. For example, encoder 340 adjusts the values for the quantization parameter ("QP") for an image, mosaic, slice, and / or other portion of video and quantizes the transform coefficients accordingly. The encoder's entropy encoder 340 compresses the qualified transform coefficient values, as well as certain side information (eg, MV information, selected MV precision, BV values, QP values, mode decisions, parameter options). Entropy encoding techniques include Exponential-Golomb encoding, Golomb-Rice encoding, arithmetic encoding, differential encoding,
Prevented
Huffman, length encoding, encoding ¥ i<sup>LJ</sup>Wia & le eFe-WTTgitude to variable length (“V2V”), variable encoding i ^ ngitt<sup>M, t</sup>'”T“<sup>+ rrw</sup>Y<sup>rrirl </sup>Fixed ("V2F"), Lempel-Ziv ("LZ") encoding, Dictionary encoding, Probability Interval Division Entropy ("PIPE") encoding, and combinations of the above. The entropy encoder can use different encoding techniques for different types of information, can apply multiple techniques in combination (for example, applying Golomb-Rice encoding followed by arithmetic encoding), and can select from multiple code tables within a particular coding technique. In some implementations, the frequency transformation can be skipped. In this case, the prediction residuals can be quantized and entropy encoded.
An adaptive unblocking filter is included within the motion compensation loop (ie, "looping" filtering) in encoder 340 to smooth out discontinuities across block boundary rows and / or columns in a decoded frame. Other filtration (such as distinctive filtration, adaptive loop filtration ("ALF"), or sample adaptive displacement filtration ("SAO"), not shown), may be applied, additionally or alternatively as in the operations loop filtration.
The encoded frames (341) and the MMCO / RPS information (342) (or information equivalent to the MMCO / RPS information (342), because the dependencies and order structures for the frames are already known in the encoder (340 )) are processed by a decoding process emulator (350). The emulator (350) of the process
<img file="MX359698B_D0017.tif" />
IMPI <sup>, NST</sup>OÍU? Z<sup>SX1CANG</sup> decoding implements some example functionality, decodes tasks to rebuild ~ ctF «h * ee - de-peia £ aa £ jaiJ In a manner consistent with the (342) MMCO / RPS information, the emulator (350) of the decoding process determines if a The determined encoded frame (341) needs to be reconstructed and stored to be used as a reference frame in inter-frame prediction of the subsequent frames to be encoded. When an encoded frame 341 needs to be stored, the decoding process emulator 350 models the decoding process that will be conducted by the decoder that receives the encoded frame 341 and produces a corresponding decoded frame 351. In doing this, when the encoder (340) has used the decoded frames (369) that have been stored in the decoded frame storage area (360), the decoding process emulator (350) also uses the frames (369) decoded from the storage area (360) as part of the decoding process.
The decoded frame buffer storage area (360) includes multiple frame buffer storage areas (361, 362, ... 36n). In a manner consistent with the MMCO / RPS information (342), the decoding process emulator (350) manages the contents of the storage area (360) in order to identify any buffer memory (361, 362, etc.) of the frame with frames that are no longer needed by encoder 340 to be used as reference frames. After modeling the decoding process, the decoding process emulator 350 stores
<img file="MX359698B_D0018.tif" />
INSTITUTO MEXICANO DE LA PROPERTY a new painting (351) decoded in a méffi '(STTá<sup>L</sup> (3 &? F3b2, etc.) intermediate frame, which has been identifiedUU<sup>1</sup> Of this * roí mu .---- The encoded frames (341) and MMCO / RPS information (342) are memorized in a temporarily encoded data area (370). The coded data that is aggregated in the coded data area 370 contains, as part of the syntax of an elementary coded video bitstream, the coded data for one or more pictures. The encoded data that is aggregated in the encoded data area (370) may also include metadata related to the encoded video data (eg, as one or more parameters in one or more Supplemental Enhancement Information Messages ("SEP") or video usage information ("VUI") messages.
The aggregated data (371) from the time-coded data area (370) is processed by the channel encoder (380). The channel encoder (380) can pack and / or multiplex the aggregated data for transmission or storage as a media stream (e.g., in accordance with a media program stream or a transport stream format, such as ITU -T H.222.0 | ISO / IEC 13818-1 or an Internet current-time transport protocol format, such as IETF RFC 3550), in which case, The channel encoder (380) may add syntax elements as part of the streaming media syntax. Otherwise, the channel encoder (380) may organize the aggregated data for storage as a file (for example, in accordance with a media counter format, such as ISO / IEC 14496-12), in which case ,
<img file="MX359698B_D0019.tif" />
IMPI
IMJTITUTO MIXiCANO 'OI LA FROPILDAO industry<sup>1</sup> The channel encoder 380 may add syntax elements as part of the media storage file syntax. Otherwise, more generally, the channel encoder (380) may implement one or more media system multiplexing protocols or transport protocols, in which case the channel encoder (380) may add syntax elements as part of the syntax of the protocols. Channel encoder 380 provides the output for a channel 390, which represents storage, a communications link, or other channel for the output. The channel encoder (380) or channel (390) may also include other elements (not shown), for example, for forward error correction coding ("FEC") and signal analog modulation.
IV. Exemplary video encoders
Figures 4a and 4b are a block diagram of a generalized video encoder (400) together with which certain modalities described herein can be implemented. The encoder (400) receives a sequence of video images including a current image as an input video signal (405) and produces data encoded in a bit stream (495) of encoded video as output.
Encoder 400 is block-based and uses an implementation-dependent block format. The blocks can also be subdivided into different stages, for example, the prediction, frequency transformation and entropy coding stages. For example, an image can be divided into 64x64 blocks, 32x32 blocks, or
ΙΜΡΙΓ
<img file="MX359698B_D0020.tif" />
INSTITUTO MEXICANO DE EA PROflECAO 16x16 blocks, which in turn, can be divided into tfl ^ small titles of sample values for encoding , (CTB), CU, (CB), PU, (PB) and TU (TB).
The encoder (400) compresses the pictures with the use of intra-picture coding and / or inter-picture coding. Many of the components of the encoder 400 are used for intra-picture coding and for inter-picture coding. The exact operations carried out by these components may vary depending on the type of information to be compressed.
A tile module 410 optionally divides the image into multiple tiles of the same or different sizes. For example, the mosaic module (410) divides the image along rows of mosaic and mosaic columns that, with the limits of the image, define the horizontal and vertical limits of the mosaics within the image, where each mosaic it has a rectangular region.
The overall encoding control (420) receives images for the input video signal (405), as well as feedback (not shown) from various encoder modules (400). In general, general encoding control (420) provides control signals (not shown) to other modules (such as tile module (410), transformer / scaler / quantizer (430), inverted transformer / scaler (435) ), the intra-image estimator (440), the motion estimator (450) and the intra / inter switch), to adjust and change the coding parameters during coding. In particular, together with
IMPI, INSTITUTO MEXICANO motion estimator (450), control (42Φ) * ^ $ & $ ϊίi '& S ^ f & general can determine MV precision during encoding. The general encoding contrQL_ (420) can also evaluate intermediate results during encoding, for example, it performs rate distortion analysis. The overall encoding control 420 produces overall control data 422 indicating the decisions made during encoding, so that the corresponding encoder can make consistent decisions. General control data (422) is provided to the entropy encoder / header formatter (490).
When the current image is predicted using inter-image prediction, the motion estimator (450) calculates the movement of the blocks of the sample values of the current image of the input video signal (405) with respect to one or more. more pictures for reference. The motion estimator 450 may select the motion vector precision (MV ") (eg, whole sample MV precision, 'Λ sample MV precision, or% sample MV precision), as described here, then use the MV precision selected during motion estimation. Decoded image buffer 470 stores one or more encoded images, previously reconstructed for use as reference images. When using multiple reference images, the multiple reference images can be from different temporal directions or from the same temporal direction. The motion estimator (450) produces lateral information motion data (452), such as MV data, blend mode index values, and reference image selection data, as well
IMPI as the lateral information that indicates the precision m ^^ eeléc ^ ÓJñraua. The side information including data "• is provided to the entropy encoder / header formatter (490) as well as to the motion compensator (455).
The motion compensator (455) applies the MV values having the selected MV precision to the reconstructed reference images from the decoded image buffer (470). When the color data for the image has the same resolution as the brightness data (for example, when the format is YUV 4: 4: 4: or RGB 4: 4: 4), the MV value that is applied for the color block can be the same as the MV value applied for the brightness block. On the other hand, when the color data for an image has a reduced resolution relative to the brightness data (for example, when the format is YUV 4: 2: 0 or YUV 4: 2: 2), the value MV that is applied for the color block can be an MV value that has been scaled down and possibly rounded to accommodate the difference in color resolution (for example, for the YUV 4: 2: 0 format, by dividing the vertical and horizontal components of the MV value by two and truncating or rounding them with the precision used for the color movement compensation process, for YUV 4: 2: 2, by dividing the horizontal component of the MV value by two and truncating or round it to the precision used for the color movement compensation process). Motion compensator (455) produces motion compensated predictions for the current image.
On a separate path within encoder 400, a
<img file="MX359698B_D0021.tif" />
IMPI
MEXICAN INSTITUTE,
The intraimage estimator (440) PRpPIFDAO determines the form '^ r'á: abo the intra-image prediction for the blocks of values mü ^^ ra ~ de'trn? Rrrrrrrrr current input video signal (405). The actual image can be fully or partially encoded with the use of intra-image encoding. Using the values of a reconstruction (438) of the current image, for intraspatial prediction, the intraimage estimator (440) determines how the sample values of a current block of the current image are spatially predicted from the Sample values, previously reconstructed, another way, for predicting intra-BC neighbors of the current image. Using the BV values, the intra-image estimator (440) estimates the displacement of the sample values of the current block for different candidate regions within the current image.
The intra-image estimator (440) produces intra-prediction data (442) as side information, such as information indicating whether the intra-prediction uses spatial prediction or intra-BC prediction (for example, one flag value per intra-block), the direction of prediction mode (for intraspatial prediction) and BV values (for intra-BC prediction). The intraprediction data (442) is provided to the entropy encoder / header formatter (490) as well as the intra-picture predictor (445).
In accordance with the intraprediction data (442), the intraimage predictor (445) spatially predicts the sample values of a current block of the current image from the previously reconstructed sample values neighboring the current image. Otherwise, for the
IMPI
INÍTITUTO MEXICANO intra-BC prediction, the (445) intraimagerP predictor<sup>£</sup>| A®S3 »ficék ^ ¿^ HOres sample of the current block with the use of the« ainrac sample prRviamentñ reconstructed from an intraprediction region, which is indicated by the BV value for the current block.
The intra / inter switch selects the motion compensated prediction values or the intra-image prediction to be used as the prediction (458) for a given block. When the residual encoding is not skipped, the difference (if any) between a prediction block (458) and a corresponding portion of the original current picture of the input video signal (405) provides values of the residual (418). During the reconstruction of the current picture, when the residual values have been encoded / flagged, the reconstructed residual values are combined with the prediction (458) to produce a reconstruction (438) of the original content from the video signal (405). In lossy compression, however, some information is lost from the video signal (405).
In the transformer / scaler / quantizer (430), when the frequency transformation is not skipped, the frequency transformer converts the video data from the spatial dimension to data from the frequency dimension (ie, spectral, transformation). For block-based video coding, the frequency transformer applies the discontinuous cosine transform ("DCT"), an integer approximation thereof, or another type of forward block transform (for example, a discontinuous sine transform or an integer approximation thereof) in the blocks of the prediction residual data
IMPI
INSTITUTO MEXICANO Dt LA HOPIRDAD (or the value data shows when the prediction fWSBF<sup>L</sup>it is
<img file="MX359698B_D0022.tif" />
which produces blocks of transformation coefficients of · revaeirvla? ' Encoder 400 may also have the ability to indicate that such a transformation step is skipped. The scaler / quantizer scales and quantizes the transformation coefficients. For example, the quantizer applies a sample zone scaled quantization on the frequency domain data with the size of the quantization stage varying on a frame-by-frame, mosaic-by-mosaic basis, on a slice-by-slice basis, on a block by block basis, on a specific frequency basis or other basis. The quantized transform coefficient data (432) is provided to the entropy encoder / header formatter (490). When the frequency transformation is skipped, the scaler / quantizer can scale and quantize the blocks of the prediction residual data (or the sample value data when the prediction (458) is null), which produces quantized values that are provided by header formatter / entropy encoder (490).
In the inverted scaler / transformer (435), the inverted scaler / quantizer performs invert scaling and invert quantization on the quantized transform coefficients. An inverted frequency transformer performs an inverted frequency transformation, which produces blocks of reconstructed prediction residuals or sample values. When the transform stage has been skipped, the inverted frequency transform is also skipped. In this case, the inverted scaler / quantizer can perform inverted scaling and inverted quantization in blocks.
IMPI
INSTITUTO MEXICANO DE LA FRCUEDAD of the residual prediction data (or the data of 'VálÓt m'
<img file="MX359698B_D0023.tif" />
which produces reconstructed values.
When coded / flagged, the encoder (400) combines the reconstructed residual values with values from the prediction (458) (e.g., the motion compensated prediction values, the intra-image prediction values), to form the reconstruction ( 438). When the residual values have not been encoded / flagged, the encoder (400) uses the values from the prediction (458) as the reconstruction (438).
For intra-image prediction, the reconstruction values (438) can be supplied back to the intra-image estimator (440) and the intra-image predictor (445). Also, the reconstruction values (438) can be used for motion compensated prediction of subsequent images. The reconstruction values (438) can also be filtered. A filtering control (460) determines the way to perform deblocking filtering and SAO filtering on the reconstruction values (438), for a given image of the video signal (405). The filter control (460) produces filter control data (462), which is provided to the entropy header / encoder (490) formatter and to the merger / filter (465).
In the merger / filter (465), the encoder (400) mixes the content of different tiles within a reconstructed version of the image. Encoder 400 selectively performs deblocking filtering and SAO filtering in accordance with filter control data 462 to adaptively smooth discontinuities across the boundaries of the paintings. Other leak (such<sup>1</sup> I like you shape shape (400),
IMPI • βπτυτο MEXICAN M LA MONEDAR the distinctive filtration or ALF, not shown ^ W ^^ ueae ^ '^ pficar in alternative or additional. The mosaic boundaries can be selectively filtered or unfiltered, depending on the encoder settings, and the encoder 400 can provide the syntax within the encoded bitstreams to indicate and whether or not the filtering was applied. Decoded image buffer 470 stores the current reconstructed image for use in post-motion compensated prediction.
Entropy encoder / header formatter (490) formats and / or entropy encodes general control data (422), quantized transform coefficient data (432), intraprediction data (442), data (452) motion and filter control data (462). MV values can be coded predictively. For example, the entropy encoder / header formatter (490) uses Exponential-Golomb encoding for entropy encoding of various syntax elements, such as the syntax elements for differential MV values, after MV prediction.
The header / encoder formatter (490) provides the encoded data in the encoded video bitstream (495). The encoded video bitstream (495) format can be a variation or extension of the HEVC format, the Windows Media format
Video, the VC-1 format, the MPEG-x format (for example, MPEG-1, MPEG-2, or MPEG-4), the H.26x format (for example, H.261, H.262, H.263 , H.264) or other format.
D depending on the implementation and the type of compression
IMPb
MEXICAN INSTITUTE? OWNERSHIP '1 desired, the encoder modules can be added to' Bfh<sup>1</sup> or multiple modules, combined with other nTOOUlUS — yfo can be replaced with similar modules. In alternative embodiments, encoders with different modules and / or other module configurations carry out one or more of the techniques described. Encoder-specific embodiments typically use a variation or a complementary version of the encoder (400). Relationships shown between modules within encoder 400 indicate general information flows in the encoder, other relationships are not shown for clarity.
V. Selecting MV precision during encoding
This section presents various aspects of selecting the precision of the motion vector (“MV”) during encoding. These aspects can facilitate compression that is effective in terms of rate distortion performance and / or the computational efficiency of encoding and decoding.
These aspects described for selecting MV precision can be applied when encoding any type of video. However, in particular, selecting MV precision as described here can improve performance when encoding certain artificially created video content, such as screenshot content.
A. Types of video
In general, the screenshot video (also called as
ΙΜΡΪβΒ ^ INSTITUTO MiXICaN.
.... ... ... MUMOHEDAO video of screen content or content to ^ ea ^ unSiMfe ^ screen) represents the output of a process that renders content for a computer screen or for another screen. This is contrary to natural video, which refers to video images captured from a video or real-world object camera sensor viewfinder that has the same characteristics. Screenshot video typically contains text, computational graphics, animation-generated content, or other similar types of content captured for the output of the production process for a computer screen, opposite (or in addition to) video content captured with one camera only.
Common scenarios for encoding / decoding screen capture content include remote conferencing and encoding / decoding of graphics or text representations into natural video or other “mixed content” video. Several of the innovations described here are adapted to encode screenshot video or other artificially created video. These innovations can also be used for natural video, but they are not as effective. Other innovations described here are effective for encoding natural video or artificially created video.
Figure 5 shows a desktop computer environment 510 with content that can provide input for screen capture. For example, the screenshot video may represent a series of images from the desktop computer (511). Otherwise, the screenshot video may represent a series of images for one of the windows of the computer environment of the
IM PI
INSTITUTO MEXICANO '^' • feggíSia Di LA MOHEDAL V ^ eigKeíí ^ desk, such as a window (513) by aplicatffbWé<sup>1</sup>; qwAircTuye the content of a game, a window (512) of 'fna'g'ayduiún corveonteTiide of web page or a window (514) with a word processor content.
Like artificially created, computer-generated video content, screenshot content tends to have relatively few discontinuous sample values compared to natural video content, which is captured using a video camera. . For example, a region in screenshot content often includes a single, solid color, while a region in natural video content is more likely to include gradually varying colors. Also, screenshot content typically includes different structures (eg graphics, text characters), which repeat exactly from frame to frame, even when the content may be spatially shifted (eg due to scrolling). Screenshot content is typically encoded in a format (for example, YUV 4: 4: 4 or RGB 4: 4: 4) with high color sampling resolution, although it can also be encoded in a format with a lower color sampling resolution (for example, YUV 4: 2: 0, YUV 4: 2: 2).
FIG. 6 shows mixed video content 620 including certain natural video 621 and certain artificially created video content. The artificially created video content includes a graphic (622) in addition to the natural video (621) and a ticker (623) that runs below the natural video (621). Like the content of
<img file="MX359698B_D0024.tif" />
IMP
INFHTUTO MfiX'CANO screenshot shown in Figure 5, the artificially trending d ^ wtffr $ * shown in Figure 6, tends to have relatively few discontinuous sample values. It also tends to have different structures (eg graphics, text characters) that repeat exactly from frame to frame (eg due to unfolding).
Screenshot video or mixed content video can be periodically read from an output buffer for the display device or from one or more buffers that store frames. Otherwise, the screen capture video can be provided from a screen capture module (which can periodically read the values from the output buffer for the display device, Intercept display commands from a display module). operating system or otherwise, it can capture sample values to be displayed). Screenshot video or mixed content video can be from a “live” stream or from a previously recorded stream in storage.
B. Different MV precisions
In many encoding scenarios, when encoding screenshot video or other artificially created video content, most MV values represent whole sample spatial shifts and very few MV values represent fractional sample spatial shifts . This offers other opportunities to reduce MV accuracy in order to improve overall performance.
IMPI f
MEXICAN INSTITUTE \
Figure 7aa shows the compensation
<img file="MX359698B_D0025.tif" />
n MV (720) that has an integer sample _ spatial offset. The MV (720) indicates a spatial shift of four samples to the left and one sample up, relative to the position (710) co-located in a reference image for the current block. For example, for a current 4x4 block at position (64, 96) in a current image, the
MV (720) indicates a 4x4 prediction region (730) whose position is (60, 95) in the reference image. Prediction region 730 includes reconstructed sample values at the entire sample positions in the reference image. An encoder or decoder does not need interpolation to determine the values of the prediction region (730).
Figure 7b shows motion compensation with an MV (721) having a fractional sample spatial shift. The MV (721) indicates a spatial shift of 3.75 samples to the left and 0.5 samples up relative to the position (710) co-located in a reference image for a current block. For example, for a current 4x4 block at position (64, 96) in the current image, the MV 721) indicates a 4x4 prediction region 731) whose position is 60.25, 95.5) in the reference image. Prediction region 731 includes interpolated sample values at fractional sample positions in the reference image. An encoder or decoder performs interpolation to determine the sample values of the prediction region (731). When fractional sample spatial shifts are allowed, there are more candidate prediction regions that can match the current block and thus the quality
<img file="MX359698B_D0026.tif" />
ΙΝΠΊΤυΤΟ MEXICAN
OF THE PROPERTY
INWJSTRIAL of motion compensated prediction is usually improved, at least for certain types of video content (for example, natural video).
When MV precision is integer sample precision for a video unit, all MV values for blocks in the unit indicate integer sample spatial displacements. When the MV precision is a fractional sample precision for a video unit, the MV value for a block in the unit can indicate the fractional sample spatial shift or an entire sample spatial shift. That is, when the MV precision is a fractional sample precision for a video unit, certain MV values for the blocks in that unit can indicate the fractional sample spatial displacements, while other MV values for the blocks in the unit indicate the displacements. integer sample spaces.
When coding a block with the use of motion estimation and motion compensation, an encoder often computes the sample-by-sample differences (also called residual values or error values) between the sample values of the block and its compensated prediction of movement. The residual values can then be encoded. For residual values, the encoding efficiency depends on the complexity of the residual values and the amount of loss or distortion that is introduced as part of the compression process. In general, a good compensated motion prediction is very close to a block, so the values i € ar in
<img file="MX359698B_D0027.tif" />
INSTITUTO MEXICANO ZA ... Dt LA PRCPltDAn ζ S.-2Í?
Residuals are differences of small amplitude that effectively shape. Moreover, a prediction<sup>1</sup> foiti thought d & -mwiwwto is often deficient, it offers residual values that include values with greater amplitude, which are more difficult to encode efficiently. Encoders typically take a large proportion of encoding time to perform motion estimation, trying to find good matches and thus improve rate distortion performance.
When an encoder-decoder uses MV values with integer sample MV precision, the encoder and decoder do not need to perform interpolation operations between the sample values of the reference images for motion compensation, since the MV values indicate integer sample spatial displacements. When an encoder / decoder uses MV values with fractional sample MV precision, the encoder and decoder can perform interpolation operations between the sample values of the reference images for motion compensation (adding computational complexity, thus minus for MV values indicating fractional sample spatial displacements), but motion compensated predictions tend to approximate blocks (leading to residuals with fewer significant values), compared to whole sample MV precision.
C. Representation of MV values
MV values are typically represented with the use of
IMPI *
INSTITUTO MEXICANO D »LA FRORIRDAD 1 INDUSTRIAL“ 'integer values, whose meaning depends on the precision M
<img file="MX359698B_D0028.tif" />
associated.
For integer sample MV precision, for example, a value of 'eñTeTO'iJe 1 indicates a spatial shift of 1 sample, an integer value of 2 indicates a spatial shift of two samples, and so on. For sample Vi MV precision, for example, an integer value of 1 indicates a spatial shift of 0.25 samples. The integer values of 2, 3, 4 and 5 indicate spatial displacements of 0.5,
0.75, 1.0, and 1.25 samples, respectively. Regardless of MV precision, the integer value can indicate the magnitude of the spatial offset and a separate flag value can indicate whether the offset is negative or positive. The horizontal MV component and the vertical MV component of a given MV value can be represented using two integer values. Thus, the meaning of two integer values represents an MV value that depends on the MV precision. for example, for an MV value that has a horizontal offset of 2 samples and no vertical offset, when the MV precision is the MV precision of% of sample, the MV value is represented as (8, 0). When the MV precision is an integer sample MV precision, however, the value
MV is represented as (2, 0).
MV values in encoded video data bitstreams are typically entropy encoded (eg, on an MV component type basis). An MV value can be differentially encoded relative to the predicted MV value (for example, on an MV component basis). In many cases, the MV value equals the predicted MV value, so that the differential MV value is zero, the
IMPI Mexican institute Ά
From INDUSTRIAL MOFICITY which can be encoded efficiently. A differential MV value (or the MV value, when MV prediction is not used) can be entropy encoded with the use of Exponential Golomb encoding, binary arithmetic encoding, context adaptive, or other form of entropy encoding. Although the exact relationship between the MV value (or the differential MV value) and the encoded bits depends on the form of entropy encoding used, in general, smaller values are encoded more efficiently (that is, with the use of less bits) because they are more common, and higher values are encoded less efficiently (that is, with more bits) because they are less common.
D. Adaptive MV Accuracy - Introduction
To summarize the previous three sections, using MV values with whole sample MV precision tends to reduce the bit rate associated with signaling MV values and to reduce the computational complexity of encoding and decoding (by avoiding interpolation of the sample values at the fractional sample positions in the reference images), but may reduce the quality of the motion compensated prediction and therefore increases the width of residual values, at least for certain types of video content. On the other hand, using MV values with fractional sample MV precision tends to increase the bit rate associated with signaling MV values and increases the computational complexity of encoding and decoding (by including interpolation of sample values at sample positions
<img file="MX359698B_D0029.tif" />
<img file="MX359698B_D0030.tif" />
INSTITUTO MEXICANO Di LA PWOHEOAÓ industrial fractionated in the reference images), but it can improve the quality of the motion compensated prediction and reduce the amplitude of the residual values, at least for certain types of video content. In general, the computational complexity, the bit rate for signaling MV values, and the quality of the motion compensated prediction increase as the MV precision increases (for example, from the entire sample to <sup>1</sup>Λ sample or <sup>1</sup>Z sample at% of sample), up to a point of decrease in return. At the same time, although increased MV precision tends to increase the bit rate required to signal MV values, when encoding natural content, the associated improvement in the quality of motion compensated prediction can reduce the bit rate required to send a suitable approximation of the residual values and therefore reduce the total bit rate required to encode the video content with a suitable image quality.
When encoding screenshot video or other artificially created video content, the added costs of fractional sample MV precision (in terms of bit rate and computational complexity) may not be justified. For example, when most MV values represent whole sample spatial shifts and very few MV values represent fractional sample spatial shifts, the added costs of fractional sample MV precision are not guaranteed. The encoder can skip searching at fractional sample positions (and skip interpolation operations to determine sample values at fractional sample positions) during
INSTITUTO MEXICANO DE LA PROPIEDAD \ industrial the estimation of movement. For such content, the bit rate and computational complexity can be reduced, without major penalties in the quality of the motion compensated prediction, with the use of MV values with integer sample MV precision.
Because fractional sample MV precision can be useful for other types of video content (for example, natural video captured with a camera), an encoder and decoder can be adapted to switch between MV precisions. For example, an encoder and decoder may use whole sample MV precision for screenshot video, but will use fractional sample MV precision (such as sample% MV precision) for natural video. The actions that an encoder can take when MV precision is selected are described in the next section. The encoder can signal the selected MV precision to the decoder with the use of one or more syntax elements in the bit stream.
In one approach to signaling MV precision, when adaptive selection of MV precision is enabled, the encoder selects MV precision on a slice-by-slice basis. A flag value in a sequence parameter setting ("SPS"), picture parameter setting ("PPS"), or other syntax structure indicates whether adaptive selection of MV precision is enabled. When so, one or more syntax elements in the slice header for a given slice indicates the selected MV precision for the blocks in that slice. For example, a flag value of 0 indicates the MV precision of
<img file="MX359698B_D0031.tif" />
IMPI
INSTITUTO MEXICANO DE LA PROPIEDAD INDUSTRIAL% sample and a flag value of 1 indicates the whole sample MV precision.
In another measure, to signal MV precision, the encoder selects MV precision on a Picture-by-Picture basis or on a slice-by-slice basis. A syntax element in a PPS indicates one of three MV precision modes: (0) sample% MV precision for the MV values of the slices of an image associated with the PPS; (1) the whole sample MV precision for the MV values of the slices of an image associated with the PPS or (2) the adaptive MV precision per slice depending on the flag value signaled by the slice header, where the value The flag in the slice header of a slice can indicate the MV precision of% of sample or the MV precision of integer sample for the MV values of that slice. For additional details about this measure in an implementation, please see JCTVC-P0277.
In another approach to signaling MV precision, when adaptive selection of MV precision is enabled, the encoder selects MV precision on a CU by CU basis. One or more syntax elements in a structure for a given CU indicate the selected MV precision for the blocks in that CU. For example, a flag value in a CU syntax structure for a CU indicates whether the MV values for all PUs associated with the CU have whole sample MV precision or sample% MV precision. For additional details about this measure in an implementation, please see JCTVC-P0283.
IMPI
MEXICAN INSTITUTE
DS THE PROPERTY
<img file="MX359698B_D0032.tif" />
In either of these approaches, the encoder and decoder
<td>they can use different</td><td>MV precisions for MV components</td>
horizontal and vertical. This can be useful when encoding screenshot video that has been scaled horizontally or vertically (for example, using the whole sample MV precision in an unscaled dimension and using the MV precision of fractional sample in scaled dimension). In some implementations, when rate control cannot be achieved by adjusting QP values alone, the encoder can resize the screenshot video horizontally or vertically to reduce the bit rate, then encode the resized video. On the decoder side, the video is scaled back to its original dimensions after decoding. The encoder can signal the MV precision for horizontal MV components (e.g. with a first flag or syntax element value) and also signals MV precision for vertical MV components (e.g. with a second flag or element value syntax) for the decoder.
More generally, when adaptive selection of MV precision is enabled, the encoder selects MV precision and signals the precision
MV selected in some way. For example, the flag value in an SPS, PPS, or other syntax structure can indicate whether adaptive selection of MV precision is enabled. When adaptive MV precision is enabled, one or more syntax elements in a sequence layer syntax, in a layered image group syntax, (“GOP layer syntax”), the image layer syntax, the syntax from
<img file="MX359698B_D0033.tif" />
IMPI
MEXICAN XSTITUTE
DE LA MIOrilDAL · slice layer, mosaic layer syntax, <sup>lf</sup>the<sup>,$</sup>A block page or other syntax structure can indicate the selected MV pieuisióTr for MV values. Otherwise, one or more syntax elements in the stream layer syntax, the GOP layer syntax, the image layer syntax, the slice header layer syntax, the slice data layer syntax, the Tile layer syntax, block layer syntax, or other syntax structure can indicate MV precisions for different MV components. When there are two MV precisions available, a flag value can indicate the selection between the two MV precisions. When more MV precisions are available, an integer value can indicate the selection between those MV precisions.
In addition to modifications to flag / parse the syntax elements indicating selected MV precision, the decoding can be modified to change the way the pointed MV values are interpreted depending on the selected MV precision. The details of how MV values are encoded and reconstructed may vary depending on the MV precision. For example, when MV precision is whole sample precision, predicted MV values can be rounded to the next nearest integer and differential MV values can indicate whole sample offsets. Otherwise, when the MV precision is the% sample precision, the predicted MV values can be rounded to the next closest% sample offset and the differential MV values can indicate% sample offsets. Otherwise, the MV values are
IMPI * »
MEXICAN INSTITUTE
M LA MOHBDX !, they can point out in some other way. When the'VáWes
<img file="MX359698B_D0034.tif" />
In the whole sample MV precision and the video uáy ^ TIflCrestréO '^ αΐσΤΐ5Τ · 4 ”2 · 2 or
4: 2: 0, color MV values can be derived by scaling, etc., which can result in sample% offsets for color. Otherwise, color MV values can be rounded to integer values.
E. Approaches to Selecting MV Accuracy
When MV precision can be adapted during video encoding, the encoder selects MV prediction for a video unit. The encoder can select the MV accuracies to be used based on tracks from the video source (see approach 1, below). For example, the video source may indicate that the video is from screenshot content or is natural video (captured with a camera). Otherwise, the encoder can select MV precisions based on a thorough evaluation of various MV precisions (see approach 2, below). Furthermore, the encoder may select the MV precisions based on the analysis of statistical data from various units and / or statistical data from the current unit to be encoded (see approaches 3-4 below).
Some of the approaches to selecting MV precision are adapted for screenshot encoding scenarios. Other approaches generally apply when encoding any kind of video content.
In some examples in this section, the encoder selects between using sample% MV precision and sample MV precision
<img file="MX359698B_D0035.tif" />
whole. More generally, tiples
IMPI g
MEXICAN USTITUTE <?>
. . . DS LA FliOHMOA'.í the entire encoder selecowroraiAienT, the MV precision of sample, the precision MV of% of sample and / or other precision MV.
When the encoder selects an MV precision for a video unit, the video unit can be a stream, a GOP, a picture, a slice, a tile, a CU, a PU, another block, or another type of video unit. Depending on the trade-off between complexity and flexibility, selecting an MV precision on a highly local basis (for example, CU by CU basis), a larger region-by-region basis (for example, mosaic basis by mosaic or base slice-by-slice), on a full image basis, or on a more global basis (eg, per encoding session, per sequence, per GOP, or per image series between detected scene changes) may be appropriate.
1. Approaches that use application tracks, operating system, or video source
An encoder can select MV accuracy based on a track pointed to by an application, operating system, or video source. For example, the track may indicate that the video content to be encoded was produced by a particular application, such as a word processor, spreadsheet application, or web browser (without the embedded video region, which may be natural video content). When produced with such an application it will tend to produce only
ΝΓΠΤυΤΟ MEXICAN integer sample spatial displacements EI ^ KS ^ nW ^ oíM Based on such a track, the encoder can gpio ^ innate the whole sample MV precision. For content produced with a word processor, spreadsheet application, web browser or other application usually does not produce natural video content, integer sample MV precision is preferable than fractional sample MV precision.
(But fractional sample MV precision may be preferable when video is resized.)
Otherwise, the track may indicate that the video content was delivered as a screen capture module or other video source that typically delivers the artificially created video content. For such content, the whole sample MV precision is preferable to the fractional sample MV precision, so the encoder selects the whole sample MV precision. (But fractional sample MV precision may be preferable when video is resized.)
On the other hand, when the track indicates that the video content was delivered by a camera, DVD or other disc, tuner card or produced by a video player, the encoder can select fractional sample MV precision. For such content, fractional sample MV precision is preferable than whole sample MV precision.
A track can be applied in a coding session, in a series of frames, in a single video frame, or in part of a video frame (such as an area corresponding to a window associated with an application).
Mexican institute D! INDUSTRIAL PROPERTY
<img file="MX359698B_D0036.tif" />
In some cases, the encoder may have the ability to interpret a clue provided by a video source, the operating system, or the application concerning the nature of the video content. Otherwise, the track may be incorrect or inaccurate (for example, for mixed video content that includes natural video content and artificially created video content, or for video that has been resized). In such a case, the encoder can use another approach to determine the MV accuracies to select.
two. Forced encoding approaches
In another group of approaches to selecting MV precision, the encoder encodes a video unit multiple times using different MV precision (e.g. once with whole sample MV precision, once with MV precision of% of show). The encoder selects the MV precision that provides the best performance and uses the selected MV precision when encoding the unit for its output. The video unit can be a block, PU, CU, slice, tile, picture, GOP, sequence, or other type of video unit. Typically, the encoder performs multiple encoding passes in such approaches.
To evaluate the MV precision that provides the best performance, the encoder can determine the cost of rate distortion when different MV precision is used during unit encoding and
<img file="MX359698B_D0037.tif" />
<img file="MX359698B_D0038.tif" />
select the option with the lowest cost of disTSiíS ^ I ^ Wot
INDUSTRY !, rate distortion has a distortion cost D and a bit rate cost R, with a factor λ (often called a Lagrangian multiplier) that weights the cost of the bit rate relative to the cost of distortion ( D + XR) or vice versa (RX + D). The bit rate cost can be estimated or it can be the actual bit rate cost. In general, the cost of distortion is based on the comparison of the original samples with the reconstructed samples. Distortion cost can be measured as the sum of absolute differences (“SAD”), the sum of absolute Hadamard transformed differences (“SAHD”), mean square error (“MSE”), mean variance, or some other distortion metric. The factor λ may vary during encoding (for example, it increases the relative weight of the bit rate cost when the size in the quantization stage is larger). Rate distortion cost usually provides the most accurate evaluation of the performance of the different MV precision options, but it also has high computational complexity.
The encoder can vary one or more of the terms of the rate distortion cost function to drive the rate distortion analysis toward the integer sample MV precision option. For example, when determining MV precision for a video unit using rate distortion analysis to decide between multiple MV precisions, in rate distortion analysis you go towards whole sample MV precision by scaling the cost distortion, by adding
MEXICAN INSTITUTE
Ot LA FROPIIDAI a distortion cost penalty by scaling the bit rate cost, adding a bit rate cost penalty and / or adjusting the Lagrangian multiplier factor. When evaluating fractional sample MV precision, the encoder can scale the distortion cost (by a factor greater than 1), scale the bit rate cost (by a factor greater than 1), add a distortion penalty, add a distortion rate penalty and / or use a Lagrangian multiplier factor. Otherwise, when evaluating the whole sample MV precision, the encoder can scale down the distortion cost (by a factor less than 1), scale down the bit rate cost (by a factor less than 1) and / or use a lower Lagrangian multiplier factor.
The encoder may vary the limit to drive towards or against the whole sample MV precision during encoding. For example, the encoder may adjust the boost towards whole sample MV precision depending on the degree of confidence that whole sample MV values are more appropriate for encoding video content (for example, increasing the momentum towards sample MV precision whole when the video content is artificially created content). On the other hand, the encoder can adjust the drive towards whole sample MV precision depending on the computational complexity to encode and / or decode (for example, increase the drive towards whole sample MV precision when the computational complexity available is lower ).
Alternatively, the encoder can use another approach to evaluate the MV precision that provides the best performance. For example,
<img file="MX359698B_D0039.tif" />
IMPI
INSTITUTO MEXICANO M LA ΜΟΠΕΡλι: · industrial the encoder measures the MV precision that results in less encoded, for a size of the quantization stage of the rriiñád0. "Otherwise, the encoder evaluates any distortion to encode, which uses different MV precisions. otherwise, the encoder uses a simpler measure, such as the benefit of distortion reduction for fractional sample MV precision compared to whole sample MV precision, which may be simple enough to be determined in a single encoding pass. . For example, the encoder examines the amount of distortion reduction (in terms of SAD, SATD, TSE, MSE, or other distortion metric) when using fractional sample MV precision, compared to when using whole sample MV precision. .
Forced coding approaches can be computationally intensive. Potentially, they may involve additional computations, additional memory storage, and additional memory read and write operations, compared to encoding using fixed MV precision.
3. Approaches that use content analysis
In another group of approaches to selecting MV precision, an encoder selects MV precision for a video unit based on analysis of the input video content and / or encoded video content. The video unit can be a block, PB, PU, CU, CTU, a sub-macroblock division, macroblock, slice, mosaic, picture,
GOP, sequence or other type of video unit.
<img file="MX359698B_D0040.tif" />
tNSTTTUTQ MEXICAN
Figure 8 shows a technique (800) for MV during encoding. The technique (800) can be carried out by the encoder, such as that described with reference to Figure 4 or Figures 4aa and 4b or by another encoder. In accordance with technique 800, during video encoding, the encoder determines one MV precision out of multiple MV precision for the video units. Multiple MV precisions can include one or more fractional sample MV precision, as well as a whole sample MV precision. For example, multiple MV precisions can include whole sample MV precision and% sample MV precision. Otherwise, the multiple MV precisions can include the whole sample MV precision, the sample% MV precision, and the sample% MV precision.
Specifically, when encoding a video unit, the encoder determines (810) whether to change the MV precision. At the start of encoding, the encoder may initially adjust the MV precision according to a preset value or proceed as if the MV precision had changed. For the later video units, the encoder can use the current MV precision (which was used for one or more previously encoded units) or change the MV precision. For example, the encoder may decide to change the MV precision based on the presence of a defined event (for example, after encoding a threshold value number of units, after a scene change, after determining the video that was changed).
To change the MV precision, the encoder collects (820) the information about the video. In general, the information collected
<img file="MX359698B_D0041.tif" />
it can be the characteristics of the input video or the characteristics of the encoded video. The information collected can be related to the current unit to be encoded and / or it can be related to the previously encoded units of the video. (When the information collected is related to one or more previously encoded units of the video, the collection (820) of such information can occur before, during or after the encoding of the previous units. This collection (820) is different than the time shown in Figure 8 and happens without considering the decision (810) to change the precision MV). The encoder then selects (830) the accuracy for that video unit based at least in part on the information collected.
As an example, the encoder can collect sample values for the current unit. The presence of a small number of discontinuous sample values tends to indicate the screenshot content, and therefore suggests that the whole sample MV precision should be selected. On the other hand, the presence of a large number of discontinuous sample values tends to indicate natural video and therefore suggests that fractional sample MV precision should be selected. Sample values can be arranged as a histogram. Sample values can be collected from only the brightness samples (Y) in the YUV color space, from the brightness as well as from the color samples (U. V) in the YUV color space, from the R, G samples and B in the RGB color space or from just the G (or R or B) samples in the RGB color space. For example, when MV precision is selected, the encoder determines the count of the different sample values between the
<img file="MX359698B_D0042.tif" />
IMPI
INSTITUTE MtXICAN! i MU PIUWDAD «WISTRIAI.
collected sample values. The encoder buys the account with a threshold. When the count is less than the threshold, the encoder selects the whole sample MV precision. When the count is higher than the threshold, the encoder selects the fractional sample MV precision. The limiting condition (the count equals the threshold) can be handled with the use of any option, depending on the implementation. Otherwise, the encoder considers the statistics of the collected sample values. For example, the encoder determines whether the x most common sample values count for more than y% of the sample values. When so, the encoder selects the whole sample MV precision, otherwise, the encoder selects the fractional sample MV precision. The values of x and y are implementation dependent. The value of x can be 10 or some other account. The value of y can be 80, 90, or some other percentage less than 100.
As another example, the encoder can collect the distortion measurements for the current unit blocks encoded with the respective MV accuracies. For example, the encoder records the improvement (reduction) in distortion when using fractional sample MV precision, compared to whole sample MV precision. When MV precision is selected, the encoder determines whether the reduction in distortion justifies the increase in MV precision.
As another example, the encoder may collect MV values (which has fractional sample MV precision) for one or more previous units. The collected MV values can be organized according to the value of their fractional parts, for example, for
IMPI
MEXICAN INSTITUTE
<img file="MX359698B_D0043.tif" />
MU PROPERTY MV precision MV values of% of sample, in a '<sup>,</sup>KísVógráhTff * ^ n a deposit for the MV values that have the fractional part of 0.57w ^ TI ^ * tTenrüTTjn deposit for MV values that have the fractional part of 0.25, a deposit for the MV values that have the fractional part of 0.5 and a deposit for the MV values that have the fractional part of 0.75. Low complexity variations of this approach are described in the next section.
As another example, the encoder can collect information about the encoded bit count for the MV data (the differential MV values) for the coded blocks with the use of fractional sample MV precision. A low average number of bits for differential MV values indicates regular (predictable) movement and is more common when integer sample MV precision is appropriate. A high average number of bits used for differential MV values is more common when fractional sample MV precision is appropriate. When MV precision is selected, the encoder measures the average (or mean) number of bits between the encoded bit counts for differential MV values. The encoder compares the measurement with a threshold. When the measurement is lower than the threshold, the encoder selects the whole sample MV precision. When the measurement is higher than the threshold, the encoder selects the fractional sample MV precision. The limiting condition (measurement equals threshold) can be handled with the use of any option, depending on the implementation.
As another example, when a drive is encoded, the encoder
IMPIOS • «UTUTO MiXICAfO
HEARD THE PROPERTY //
INDCÜTRIAI y, evaluates the multiple MV precisions per block (for example, PU) of the unit and collects the information per block indicating the MV precision that offers the best performance for that block. The encoder can determine the rate distortion cost (eg, D + XR) when the block is encoded using the fractional sample MV precision. The encoder determines how many times each of the multiple MV precisions is best for the respective blocks within the unit and selects the MV precision with the highest count. For example, for each of the blocks in an image, the encoder determines the rate distortion cost when the block is encoded using the whole sample MV precision and also determines the rate distortion cost when the block is encoded. encodes with the MV precision of% sample. The encoder counts the number of times the whole sample MV precision will be better and the number of times the sample% MV precision will be better, then selects the higher of the two. Alternatively, the encoder determines a count of how many times the whole sample MV precision is best for the unit blocks, then selects the whole sample MV precision only when the count is higher than a threshold percentage of the number. of blocks in the unit. In some implementations, the encoder considers blocks with any value of MV. In other implementations, the encoder considers blocks with any value of MV. In other implementations, the encoder considers only blocks with non-zero MVs. This block-type evaluation of multiple MV precisions can be carried out for blocks of a given unit with
<img file="MX359698B_D0044.tif" />
KST1TUTO MEXICANO M LA PROHEDAL · 'INDUSTRIAL
<img file="MX359698B_D0045.tif" />
in order to select the MV precision for one or more subsequent units, regardless of the MV precision mode used for the given unit.
Otherwise, block-type evaluation of the multiple MV precisions can be carried out for a given unit in order to select the MV precision for that unit.
Alternatively, the encoder uses another approach to collect information and select the MV precision based at least in part on the selected information.
Referring again to FIG. 8, whether or not the MV precision is changed, the encoder 840 encodes the unit using the selected MV precision. The MV values for the blocks (eg PU, macroblocks, or other blocks) within the video unit have the MV precision selected. The encoder outputs the encoded data for the current unit, for example, in a bit stream. The encoded data can include syntax elements that indicate precision
MV selected.
The encoder decides (850) whether to proceed to the next unit. When so, the encoder 810 decides to change the MV precision for the next unit. Thus, the MV precision can be selected for each unit (e.g. per segment, per GOP, per image, per slice, per CTU, per CU, per PU, per PB, per macroblock, per sub-macroblock division ). Otherwise, to reduce complexity, the MV precision for one unit can be changed from time to time (for example, periodically or after the presence of a defined event), then repeated for one or more subsequent units.
IΑ / ί ΡI ** CTITOTOM £) 8CANV) t
When the encoder uses the same pattern of ^ mo ^ icoá -'- M-rfíiagen in picture, the encoder can repeat the precisions MV | Jl) i -rrrersai © or - picture in picture. Image-in-image colocated tiles can use the same MV precision. Similarly, image-to-image co-located slices can use the same MV precision. For example, the video is supposed to illustrate a desktop computer and part of the desktop has a window that displays natural video content. Fractional sample MV precision can be used within that region of the picture-in-picture desktop, while other areas displaying text or other content are encoded using whole sample MV precision.
In this group of approaches, the encoder can use single pass encoding. For the current unit of video to be encoded, the MV precision selected for the current unit depends, at least in part, on the information collected from one or more previous units of the video (in the encoding order, which is also called the order decoding or bitstream order, not input order, which is also called temporal order, output order, or display order).
Alternatively, in this group of approaches, the encoder may use multi-pass encoding or encode with a short forward window (sometimes called 1-5-pass encoding). For the current unit of video to be encoded, the selected MV precision depends, at least in part, on the information collected from the current unit. The MV precision selected for the current unit is also
<img file="MX359698B_D0046.tif" />
IMPI
INSTITUTO MIXICano DS LA KOPISDAÜ INDUSTRIAL may depend, at least in part, on the information collected from one or more previous units of the video (in coding order, not input order).
In this group of approaches, the encoder may adjust the amount of pulse toward or against the whole sample MV precision based at least in part on the degree of confidence that the whole sample MV precision is appropriate. The encoder can also adjust the amount of Boost towards or against the whole sample MV precision based at least in part on the computational complexity of encoding and / or decoding (which favors the whole sample MV precision to reduce computational complexity when there is less computational complexity available). For example, to favor the selection of the whole sample MV precision, the encoder may adjust thresholds used in comparison operations to cause the whole sample MV precision to be selected.
In this group of approaches, the selected MV precision can be for the horizontal MV components and / or for the vertical MV components of the MV values for the blocks within the video unit, where the horizontal MV components and the Vertical MV components have different MV accuracies. otherwise, the selected MV precision can be for the horizontal MV components and for the vertical MV components of the MV values for the blocks within the video unit, where the horizontal MV components and the vertical MV components have the same precision
MV.
IMPI
INSTITUTE MMWC ΜΙΛ thowkom; tMMmiML
<img file="MX359698B_D0047.tif" />
In this group of approaches, the encoded video (eg, in the bitstream) includes one or more syntax elements indicating the MV precision selected for that unit. Alternatively, the encoded video may lack syntax elements indicating the selected MV precision for the unit (see later in the section on non-normative approaches). For example, even when the bitstream supports signaling MV values with fractional sample MV precision, the encoder can restrict the motion estimation for the video unit to use only MV values with fractional parts of zero. This can reduce the computational complexity of encoding and decoding by avoiding interpolation operations.
Four. Approaches that use low complexity content analysis
To simplify the decision-making process, the encoder can consider a smaller group of data before selecting MV precision or use simpler decision logic when selecting MV precision, avoiding multiple encoding passes.
Figure 9 shows a technique (900) for adapting MV precision during encoding using the low complexity approach. Technique 900 can be carried out by an encoder, such as that described with reference to Figure 3 or Figures 4a and 4b, or by another encoder. Technique (900) details an approach to collect information about video and select MV accuracy based on
<img file="MX359698B_D0048.tif" />
less in
IMPI
Í'F Ι.Λ Μ (·· * »ΕίΜΓ · 'INMMKL starts from the collected information, as described with reference to figure 8.
In accordance with technique 900, during video encoding, the encoder determines the MV accuracy for a video unit.
When determining the MV precision for unity, the encoder identifies (910) a group of MV values that have the fractional sample MV precision. The MV value group can be allowed to include zero-valued MVs and zero-valued MVs. Otherwise, the set of MV values may be restricted to include only non-zero MVs. Otherwise, the group of MV values may also be restricted to include only non-zero value MVs from blocks of a certain block size or larger.
The encoder selects (920) the MV precision for the unit based at least in part on the prevalence, within the group of MV values, of MV values having a fractional part of zero. Prevalence can be measured in terms of a fraction of the group of MV values that have a fractional part of zero. For example, for an image, the encoder can determine the percentage of MV values that have a fractional part of zero. Otherwise, for a region or group of regions that use the set of MV values, prevalence can be measured in terms of the fraction of that region or group of regions that have the fractional part of zero. When the fraction exceeds a threshold, the MV precision selected for that unit is the whole sample MV precision. When the fraction does not exceed the threshold, the MV precision selected for the unit is the sample MV precision
<img file="MX359698B_D0049.tif" />
IMPI
INSTITUTO MEXICANO DE LA rtOMEDAO INDUSTRIAL fractional. The limiting condition (fraction equals threshold) can be handled with the use of any option, depending on the implementation.
The selection (920) of the MV precision for the unit may also be based at least in part on the prevalence of non-zero value MVs, so that the change to the whole sample MV precision is allowed when there is a threshold amount of non-zero MV. The prevalence of non-zero MV can be measured in terms of the fraction of MV values that are non-zero MV, in terms of the count of blocks used by non-zero MV, or in terms of the fraction of a region or a group of regions that use non-zero VMs. In this case, the group of MV values having the fractional sample MV precision can be identified from the non-zero value MVs of the region or group of regions. In this way, the encoder can consider the prevalence of non-zero MVs having a fractional part of zero within the group of MVs that are non-zero MVs. For example, the encoder switches to integer sample MV precision when two conditions are satisfied: (1) a sufficiently large number of non-zero value MVs are detected and (2) within that of non-zero value MVs, there are enough that have a fractional part of zero (or alternatively, sufficiently few that they have a fractional part not zero). The prevalence of non-zero MV values and the prevalence of MV values that have a fractional part of zero can be determined by counting the MV values (without considering the size of their associated block) or by considering the size of the associated block for the
IMPI MV values (for example, because some
MEXICAN INSTITUTE OF MONEY
<img file="MX359698B_D0050.tif" />
apply in larger blocks than others). '
The encoder encodes the unit using the MV precision selected for that unit. The MV values for the blocks (for example, PU, macroblocks, or other blocks) within the video unit have the MV precision selected for the unit. The encoder outputs encoded data for the current unit, for example, in a bit stream. The encoded data can include syntax elements indicating the MV precision selected for that unit.
To reduce the amount of time the encoder spends adjusting the MV precision, after the whole sample MV precision is selected for a unit, the selected MV precision can be used for subsequent units of the video until an event causes the precision to MV change back to fractional sample MV precision. For example, the event may be the encoding of a defined number of units, a scene change, or a determination, based on observations during encoding, that the change back to fractional sample MV precision will be beneficial.
In an exemplary implementation, the encoder encodes the video unit (eg, picture, mosaic, slice, or CU) only once. To begin with, the encoder encodes the unit with the use of a sample% MV position. During encoding, the encoder determines whether the fractional parts of the MV values are zero or not, for example, the encoder measures the fraction of the MV values that have non-zero fractional parts. Otherwise, because
<img file="MX359698B_D0051.tif" />
MV values affect image regions larger than others, the encoder measures the fraction of interimage predicted regions using MV values with non-zero fractional parts (measures area, not count of MV values). When the fraction exceeds a threshold (which is implementation dependent and is for example 75%), the encoder switches to whole sample MV precision for one or more subsequent units of the video.
In this exemplary implementation, after the encoder changes from the whole sample MV precision, the encoder can maintain the whole sample MV precision indefinitely or until a defined event triggers a change back to the fractional sample MV precision, for at least temporarily. The event may be for example the encoding of a particular number of units (eg 100 units. Otherwise the event may be a scene change. Otherwise, the event may be a determination, based on statistics collected during encoding, that a change back to fractional sample MV precision is likely to be beneficial. (Such statistics can be collected during the coding of a certain limited area, to decide if fractional sample MV precision will work better for that area, then the change to MV precision for one or more units is applied.)
Whether the video content is natural video content or artificially created video content, large portions of the video can be static. For example, the static portions could be a stationary background in a natural video or a stationary content.
IMPI
W «~> urm« / -. k.,. '
<img file="MX359698B_D0052.tif" />
INSTITUTO MiXICANO DELA PROPERTY INDUSTRIAL in a screenshot content. The static portions of video have non-zero MVs, which have fractional parts of zero when the MV precision is the fractional sample MV precision. The presence of a significant number of zero-value MVs can confuse the decision logic that considers the fraction of MV values with non-zero fractional parts.
Therefore, the encoder can remove zero value MVs from consideration. Figure 10 shows an image (1000) that includes a non-moving portion (1001) with (most) zero-value MVs and two moving portions (1002, 1003) with (most) non-zero-value VMs. The encoder considers the non-zero value MVs in the moving portions (1002, 1003), but does not consider the MV values of the non-moving portion (1001). The encoder can change from the whole sample MV precision when the fraction of the non-zero value MVs (in the movable portions (1002, 1003) with fractional parts of zero, exceeds a threshold (or when the fraction of the image that uses non-zero MVs with fractional parts of zero (in terms of area) exceed a threshold).
The encoder can also verify that the number of non-zero MVs being evaluated exceeds the threshold quantity, so that no decisions are made based on a smaller number of MV values. This can make for a more robust decision-making process.
In another exemplary implementation, the encoder encodes a particular unit of video (eg, picture, mosaic, slice, or CU) using the precision of% sample. The encoder switches to integer sample MV precision for one or more subsequent units of the
<td></td><td>IMPI MEXICAN INSTITUTE OF THE ηοΗίΛΑο · Λ, * Μ INDUSTRIAL</td>
<td>video</td><td>when (1) more than x% of the unit uses inter-image prediction</td>
with non-zero MV and (2) more than y% of the part of the unit that uses the non-zero MV has integer value MV (fractional parts of zero). The values of x and y are implementation dependent and can be, for example, and 75, respectively.
In a similar exemplary implementation, the encoder encodes a given unit of video (eg, picture, tile, slice, or CU) using the MV position of% sample. Encoder switches to integer sample MV precision for one or more subsequent video units when (1) more than z PUs in the unit have non-zero MVs and (2) more than y% of those PUs have MVs of value integer (fractional parts of zero). The values of z and y are implementation dependent and can be for example 100 and 75, respectively.
MV values for larger regions are more reliable than MV values for smaller regions. The encoder can limit the MV values to be evaluated. For example, the encoder can only evaluate MV values for blocks of a certain block size or larger (eg 16x16 or larger).
In another exemplary implementation, the encoder encodes a given unit of video (eg, picture, mosaic, slice, or CU) using the MV precision of% sample. The encoder switches to integer sample MV precision for one or more subsequent units of the video when (1) more than z unit PUs are wxwo larger and have non-zero MVs and (2) more than y% of those PU have integer MVs (fractional parts of zero). The values of w, z, and y depend on
<img file="MX359698B_D0053.tif" />
Implementation and can be respectively.
75,
5. Non-normative approaches
In most of the above examples, an encoder points to one or more syntax elements that indicate the selected MV precision in encoded data, for example, in the bit stream. A decoder parses the syntax elements indicating the selected MV precision and interprets the MV values according to the selected MV precision.
Alternatively, in a non-normative approach, the encoder does not point to any syntax elements indicating the MV precision selected by the encoder. For example, the encoder selects between whole sample MV precision and fractional sample MV precision, but always encodes MV values in fractional sample MV precision. A decoder reconstructs and applies the MV values to the fractional sample MV precision.
When you select a whole sample MV precision, the encoder simply estimates the motion by avoiding interpolation of sample values at fractional sample offsets and evaluating candidate prediction regions only at whole sample offsets. Also, when the MV prediction produces a fractional value - for example, with the use of the temporal MV prediction - the encoder can consider only those MV differences which will result in integer values when the MV difference is added in the
IMPI MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX359698B_D0054.tif" />
fractional value MV prediction (for example, from temporal MV prediction). During decoding, motion compensation can be simplified by avoiding interpolation of sample values at fractional sample offsets.
Certain approaches described in the preceding sections (for example, using scaled rate distortion cost by scaling the distortion cost and / or bit rate cost or by adding a distortion cost penalty or a penalty in bit rate cost or by adjusting the weighting factor) can also be adapted for a non-normative approach. The encoder can vary the pulse limit toward or against the whole sample MV precision during encoding. Through scaling, penalties and / or weighting factor, the encoder can adjust towards whole sample MV precision depending on the degree of confidence that whole sample MV values are more appropriate for encoding video content or depending on of computational complexity to encode or decode.
6. Alternatives and variations
In some usage scenarios, the image encoding order (also called decoding order or decoding order) differs from time order in camera input / capture and display (also called display order). The encoder can take such rearrangement into account when selecting the MV precision. For example, the encoder may select the MV accuracies based on the temporal order of the images rather than the encoding order of the images.
IMPI
Dt THE INDUSTRIAL HOPIEDAO
<img file="MX359698B_D0055.tif" />
In many of the examples here described, intra-BC and motion compensation are implemented in separate components or processes and BV estimation and motion estimation are implemented in separate components or processes. . Alternatively, intra-BC prediction can be implemented as a special case of motion compensation and BV estimation can be implemented as a special case of motion estimation, for which the current image is used as the reference image. . In such implementations, a BV value can be designated as an MV value but is used for BC intraprediction (within current image) rather than interimage prediction. As the term is used here, "intra-BC prediction" indicates the prediction within the current image, whether the prediction is provided with the use of the intra-image prediction module, the motion compensation module, or some other module. Similarly, the BV value can be represented with the use of an MV value or with the use of a different type of parameter or syntax element and the BV estimation can be provided with the use of the intra-image estimation module, the module of motion estimation or some other module. The approaches described here for selecting MV precision can be applied to determine the precision of MV values that will be used as BV values for intra-BC prediction (that is, with the current image as the reference image).
<img file="MX359698B_D0056.tif" />
<img file="MX359698B_D0057.tif" />
INSTITUTO MEXICANO DE LA FKOi'IEÜAÜ INDUSTRIAL
SAW. Innovative features
In addition to the claims presented below, the innovative features described herein include, but are not limited to, the following.
<td> #</td><td>Characteristic</td>
<td>A1</td><td>A computing device comprises: means for encoding video, including means for determining the precision of the motion vector ("MV") for a video unit, wherein the MV values for the blocks within the video unit have a MV precision for the unit and in where the means of determining the MV precision for the unit include: means for identifying a group of MV values having the fractional sample MV precision; Y means for selecting the MV precision for the unit based at least in part on the prevalence, within the group of MV values, of MV values having a fractional part of zero; Y</td>
<img file="MX359698B_D0058.tif" />
INSTITUTO MEXICANO gave the reortF.0AD INDUSTRIAL
<td></td><td>Means to broadcast the encoded video.</td>
<td>B1</td><td>A computing device comprising: means for encoding video, including means for determining the precision of the motion vector ("MV") for a video unit, wherein the MV values of the blocks within the video unit have the MV precision for the unit, in where the means for determining includes means for carrying out a rate distortion analysis to decide between multiple MV precisions, multiple MV precisions include one or more fractional sample MV precision and whole sample MV precision and where the rate distortion analysis is driven toward whole sample MV precision by: (a) scaling a distortion cost; (b) add a distortion cost penalty; (c) scaling the cost of the bit rate; (d) add a bit rate cost penalty and / or (e) adjust the Lagrangian multiplier factor; Y means to broadcast the encoded video.</td>
<td>C1</td><td>A computing device, which comprises: means for encoding video, including means for determining the precision of the motion vector ("MV") for a video unit from among multiple MV precisions, the multiple MV precisions include one or more fractional sample MV precisions and the MV precision of integer sample, where the MV values for the blocks within the video unit have the MV precision for the unit and where the means to determine include: means of collecting information about the video; Y means for selecting the MV precision for the unit based at least in part on the information collected; Y means to broadcast the encoded video.</td>
In view of the many possible embodiments in which the principles of the invention may be applied, it should be recognized that
<img file="MX359698B_D0059.tif" />
Illustrated embodiments are only preferred exemplary of the invention and should not be construed as limiting the scope of the invention. Rather, the scope of the invention is defined by the following claims. Therefore what is within the scope and spirit of the claims is claimed as our invention.
Contents45
75 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75
76 members in 11 offices
Priority claims7
| Document | Office | Kind | Date |
|---|---|---|---|
| 61925090 | United States of America | – | |
| 201461925090 | United States of America | P | |
| 61934574 | United States of America | – | |
| 201461934574 | United States of America | P | |
| 14513132 | United States of America | – | |
| 201414513132 | United States of America | A | |
| 2014071332 | United States of America | W |
Members76
| Document | Office | Kind | |
|---|---|---|---|
| US2015195525A1 | United States of America | A1 | |
| US2015195557A1 | United States of America | A1 | |
| CA2935340A1 | Canada | A1 | |
| CA2935562A1 | Canada | A1 | |
| CA3118603A1 | Canada | A1 | |
| WO2015105661A1 | World Intellectual Property Organization (WIPO) | A1 | |
| WO2015105662A1 | World Intellectual Property Organization (WIPO) | A1 | |
| AU2014376190A1 | Australia | A1 | |
| AU2014376189A1 | Australia | A1 | |
| CN105900419A | China | A | |
| CN105900420A | China | A | |
| KR20160106155A | Republic of Korea | A | |
| KR20160106703A | Republic of Korea | A | |
| MX2016009023A | Mexico | A | |
| MX2016009025A | Mexico | A | |
| EP3075153A1 | European Patent Office (EPO) | A1 | |
| EP3075154A1 | European Patent Office (EPO) | A1 | |
| JP2017508344A | Japan | A | |
| JP2017508348A | Japan | A | |
| BR112016015243A2 | Brazil | A2 | |
| BR112016015854A2 | Brazil | A2 | |
| US9749642B2 | United States of America | B2 | |
| US2017359587A1 | United States of America | A1 | |
| RU2016127410A | Russian Federation | A | |
| RU2016127312A | Russian Federation | A | |
| US9900603B2 | United States of America | B2 | |
| US9942560B2 | United States of America | B2 | |
| US2018131947A1 | United States of America | A1 | |
| AU2014376189B2 | Australia | B2 | |
| MX359698BThis record | Mexico | B | |
| MX360925B | Mexico | B | |
| AU2014376190B2 | Australia | B2 | |
| RU2679349C1 | Russian Federation | C1 | |
| RU2682859C1 | Russian Federation | C1 | |
| JP6498679B2 | Japan | B2 | |
| CN105900419B | China | B | |
| US10313680B2 | United States of America | B2 | |
| CN105900420B | China | B | |
| CN110099278A | China | A | |
| CN110149513A | China | A | |
| CN110177274A | China | A | |
| US2019281309A1 | United States of America | A1 | |
| JP6576351B2 | Japan | B2 | |
| EP3075154B1 | European Patent Office (EPO) | B1 | |
| MX373555B | Mexico | B | |
| US2020177887A1 | United States of America | A1 | |
| BR112016015854A8 | Brazil | A8 | |
| US10681356B2 | United States of America | B2 | |
| US10735747B2 | United States of America | B2 | |
| US2020329247A1 | United States of America | A1 | |
| EP3075153B1 | European Patent Office (EPO) | B1 | |
| KR20210073608A | Republic of Korea | A | |
| KR102270095B1 | Republic of Korea | B1 | |
| KR102271780B1 | Republic of Korea | B1 | |
| US11095904B2 | United States of America | B2 | |
| CA2935562C | Canada | C | |
| US2021337214A1 | United States of America | A1 | |
| KR102360403B1 | Republic of Korea | B1 | |
| KR20220019845A | Republic of Korea | A | |
| CN110099278B | China | B | |
| CN110149513B | China | B | |
| CN110177274B | China | B | |
| KR102465021B1 | Republic of Korea | B1 | |
| KR20220153111A | Republic of Korea | A | |
| CA2935340C | Canada | C | |
| BR112016015243B1 | Brazil | B1 | |
| BR112016015854B1 | Brazil | B1 | |
| BR122022001631B1 | Brazil | B1 | |
| US11638016B2 | United States of America | B2 | |
| US2023209070A1 | United States of America | A1 | |
| KR102570202B1 | Republic of Korea | B1 | |
| KR20230127361A | Republic of Korea | A | |
| US12108054B2 | United States of America | B2 | |
| KR102725421B1 | Republic of Korea | B1 | |
| KR20240161211A | Republic of Korea | A | |
| US2024414356A1 | United States of America | A1 |
1 legal event, as the office reported them to INPADOC
Events
| Event | Code | |
|---|---|---|
| Grant or registrationFG | FG |
Numbers
- Publication
- 359698
- Application
- 9025
Titles2
- Spanish
- SELECCION DE PRECISION DE VECTOR DE MOVIMIENTO.
- English
- PRECISION SELECTION OF VECTOR OF MOTION.
Classification
- CPC, 17
- H04N19/109
- H04N19/174
- H04N19/523
- H04N19/136
- H04N19/42
- H04N19/139
- H04N19/147
- H04N19/177
- H04N19/17
- H04N19/179
- H04N19/105
- H04N19/70
- H04N19/13
- H04N19/169
- H04N19/52
- H04N19/142
- H04N19/521
- IPC, 13
- H04N19 174
- H04N19 17
- H04N19 105
- H04N19 109
- H04N19 136
- H04N19 139
- H04N19 142
- H04N19 147
- H04N19 177
- H04N19 179
- H04N19 513
- H04N19 52
- H04N19 523