Image encoding method, image decoding method, image encoding device, image decoding device, and image encoding/decoding device.
Abstract
A dependency indication is included at the beginning of a packet, i.e., in the vicinity of an analysis target slice header or parameter set, and is delivered by a signal. This is achieved, for example, by including the dependency indication at the beginning of the slice header, ideally after a syntax element identifying the parameter set, and before the slice address, by using a separate message and providing the dependency indication to an NALU header, or by using a special NALU type used in an NALU holding a dependent slice.

Term
7 yearsleft in the term
Expires 19 September 2033.
- Priority
- Filed
- Granted
- Today
- Expires
13 claims: 4 independent, 9 dependent
- 1REIVINDICACIONES IMPI INSTITUTO MEXICANO K LA FROFIEDAD INDUSTRIAL Habiéndose descrito la invención como antecede, se reclama como propiedad lo contenido en las siguientes reivindicaciones:1. Un método de codificación de imágenes para realizar el procesamiento de codificación al dividir una representación visual en una pluralidad de cortes, caracterizado porque comprende transmitir una corriente de bits la cual incluye: un indicador de habilitación de cortes dependientes que señala si la representación visual incluye o no un corte de dependiente en el cual se realiza el procesamiento de codificación dependiendo de un resultado del procesamiento de codificación en un corte diferente de un corte actual;una dirección de corte que señala una posición de inicio del corte actual;y una indicación de dependencia que señala si el corte actual es o no el corte dependiente, en donde el indicador de habilitación de cortes dependientes se dispone en un conjunto de parámetros común para los cortes, la dirección de corte se dispone en un encabezado de corte del corte actual y la indicación de dependencia se dispone en el encabezado de corte y se dispone antes de la dirección de 152 IMPI INSTITUTO MEXICANO DE LA FROFIEDAD INDUSTRIAL corte y después de un elemento de sintaxis que identifica el conjunto de parámetros.
- 2El método de codificación de imágenes de conformidad con la reivindicación 1, caracterizado porque la indicación de dependencia se incluye en la corriente de bits cuando el indicador de habilitación de cortes dependientes señala la inclusión del corte dependiente.
- 3El método de codificación de imágenes de conformidad con una de las reivindicaciones 1 y 2, caracterizado porque el indicador de habilitación de cortes dependientes se dispone al comienzo del conjunto de parámetros.
- 4El método de codificación de imágenes de conformidad con una de las reivindicaciones 1 a 3, caracterizado porque cada uno de los corte incluye una pluralidad de macrobloques y el procesamiento de codificación en el corte actual se inicia después de que el procesamiento de codificación se realiza en dos de los macrobloques incluidos en un corte actual inmediatamente precedente.
- 5El método de codificación de imágenes de conformidad con una de las reivindicaciones 1 a 4, caracterizado porque la indicación de dependencia no se incluye en un encabezado de corte de un corte el cual es procesado primero para la representación visual, entre los cortes. 153 IMPI INSTITUTO MtXICAN! Dt i.a rroriedad INDSTRMt
- 6Un método de decodificación de imágenes para realizar el procesamiento de decodificación al dividir una representación visual en una pluralidad de cortes, caracterizado porque comprende extraer, de una corriente de bits codificada, un indicador de habilitación de cortes dependientes que señala si la representación visual incluye o no un corte dependiente en el cual se realiza el procesamiento de decodificación dependiendo de un resultado del procesamiento de decodificación en un corte diferente de un corte actual, una dirección de corte que señala una posición de inicio del corte actual y una indicación de dependencia que señala si el corte actual es o no el corte dependiente, en donde el indicador de habilitación de cortes dependientes se dispone en un conjunto de parámetros común para los cortes, la dirección de corte se dispone en un encabezado de corte del corte actual y la indicación de dependencia se dispone en el encabezado de corte y se dispone antes de la dirección de corte y después de un elemento de sintaxis que identifica el conjunto de parámetros.
- 7El método de decodificación de imágenes de conformidad con la reivindicación 6, caracterizado porque la indicación de dependencia se extrae de la corriente de bits 154 IMPI INSTITI/TÜ MEXICANO DELA «OREDAÍ» INDUSTRIAL _ cuando el indicador de habilitación de cortes dependientes señala la inclusión del corte dependiente.
- 8El método de decodificación de imágenes de conformidad con una de las reivindicaciones 6 y 7, caracterizado porque el indicador de habilitación de cortes dependientes se dispone en el comienzo del conjunto de parámetros.
- 9El método de decodificación de imágenes de conformidad con una de las reivindicaciones 6 a 8, caracterizado porque cada uno de los cortes incluye una pluralidad macrobloques y el procesamiento decodificación en el corte actual se inicia después de que el procesamiento de decodificación se realiza en dos de los macrobloques incluidos en un corte actual inmediatamente precedente.
- 10El método de decodificación de imágenes de conformidad con una de las reivindicaciones 6 a 9, caracterizado porque la indicación de dependencia no es incluida en un encabezado de corte de un corte el cual es procesado primero para la representación visual, entre los cortes.
- 11Un aparato de codificación de imágenes el cual realiza el procesamiento de codificación al dividir una representación visual en una pluralidad de cortes, caracterizado porque comprende 155 IMPI INSTITUTO MEXICANO DE LA PROPIEDAD INDI ISTRIAL un codificador el cual transmite una corriente de bits la cual incluye:un indicador de habilitación de cortes dependientes que señala si la representación visual incluye o no un corte dependiente en el cual se realiza el procesamiento de codificación dependiendo de un resultado del procesamiento de codificación en un corte diferente de un corte actual;una dirección de corte que indica una posición de inicio del corte actual;y una indicación de dependencia que señala si el corte actual es o no el corte dependiente, en donde el indicador de habilitación de cortes dependientes se dispone en un conjunto de parámetros común para los cortes, la dirección de corte se dispone en un encabezado de corte del corte actual y la indicación de dependencia se dispone en el encabezado de corte y se dispone antes de la dirección de corte y después de un elemento de sintaxis que identifica el conjunto de parámetros.
- 12Un aparato de decodificación de imágenes el cual realiza el procesamiento de decodificación al dividir una representación visual en una pluralidad de cortes, caracterizado porque comprende un decodificador el cual extrae, de una corriente de bits codificada, un indicador de habilitación de cortes dependientes que señala si la representación visual incluye o 156 IMPI ínsito rro mexicano de LA PROHEOA/·· INDUSTRIAL no un corte dependiente en el cual se realiza el procesamiento de decodificación dependiendo de un resultado del procesamiento de decodificación en un corte diferente de un corte actual, una dirección de corte que indica una 5 posición de inicio del corte actual y una indicación de dependencia que señala si el corte actual es o no el corte dependiente, en donde el indicador de habilitación de cortes dependientes se dispone en un conjunto de parámetros común 10 para los cortes, la dirección de corte se dispone en un encabezado de corte del corte actual y la indicación de dependencia se dispone en el encabezado de corte y se dispone antes de la dirección de 15 corte y después de un elemento de sintaxis que identifica el conjunto de parámetros.
- 13Un aparato de codificación y decodificación de imágenes, caracterizado porque comprende:el aparato de codificación de imágenes de 20 conformidad con la reivindicación ll;y el aparato de decodificación de imágenes de conformidad con la reivindicación 12. 157
Independent claims13
1,076 paragraphs in 120 sections, as filed
(54) Title: IMAGE CODING METHOD, IMAGE DECODING METHOD, IMAGE CODING DEVICE, IMAGE DECODING APPARATUS AND IMAGE CODING AND DECODING APPARATUS.
(54) Title: IMAGE ENCODING METHOD, IMAGE DECODING METHOD, IMAGE ENCODING DEVICE, IMAGE DECODING DEVICE, AND IMAGE ENCODING / DECODING DEVICE.
(57) Summary
A dependency indication is signaled within the beginning of a packet, that is, in the vicinity of a parsing header being parsed or a set of parameters. This is accomplished, for example, by including the dependency indication at the beginning of the string header, preferably after a syntax element identifying the parameter set and before the break address, by including the dependency indication before the break address, by providing the dependency indication to a NALU header using a separate message or by using a special NALU type for NALUs that carry dependent breaks.
(57) Abstract
A dependency indication is included at the beginning of a packet, Le., In the vicinity of an analysis target slice header or parameter set, and is delivered by a signal. This is achieved, for example, by including the dependency indication at the beginning of the slice header, ideally after a syntax element identifying the parameter set, and before the slice address, by using a separate message and providing the dependency indication to an NALU header , or by using a special NALU type used in an NALU holding a dependent slice.
Institute
Mexican Property
Industrial
PATENT TITLE NO. 339463
<img file="MX339463B_D0001.tif" />
Headlines): PANASONIC INTELLECTUAL PROPERTY CORPORATION OF AMERICA
Address: 20000 Mariner Avenue, Suite 200, Torrance, California, 90503, USA
Name: IMAGE CODING METHOD, IMAGE DECODING METHOD, IMAGE CODING DEVICE, IMAGE DECODING APPARATUS AND IMAGE CODING AND DECODING APPARATUS.
Classification: lnt.CI.8: G06K9 / 36; H04N19 / 00; H04N19 / 30; H04N19 / 70
<td>Inventors): SEMIH</td><td colspan="2">ESENLIK; MATTHIAS NARROSCHKE; THOMAS WEDI</td>
<td></td><td>REQUEST</td><td>FF, F 'Itíj; .<sup>!</sup>.. <sub>;</sub></td>
<td>Number:</td><td colspan="2">International filing date:</td>
<td>MX / a / 2015/002889</td><td colspan="2">September 19, 2013</td>
<td> 1</td><td>PRIORITY</td><td></td>
<td>Country:</td><td>Date:</td><td>Number:</td>
<td>US</td><td>September 26, 2012</td><td> 61/705,846</td>
<td>US</td><td>October 10, 2012</td><td> 61/711,892</td>
<td>Validity: Twenty years</td><td></td><td></td>
Expiration Date: September 19, 2033
The reference patent is based on articles 1, 2, section V, 6, section III, and 59 of the Industrial Property Law.
In accordance with article 23 of the Industrial Property Law, this patent has a non-renewable term of twenty years, counted from the filing date of the international application and will be subject to the payment of the fee to keep the rights in force. . ..... '<sup>/:</sup>'
Whoever subscribes to this title does so based on the provisions of articles 6 fractions III and 7 bis 2 of l | Industrial Property Law (DlarloSBficlal deiiia Federación (DOF) 06/27/1991, amended on 08/02/1994, 10/25/1996, 12/26/1997, 05/17/1999, 01/20/2004, 06/16/2005, 01/25/2006, 06/05/2009, 06/01/2010, 06/18/2010, 06/28/2010, 01/27/2012 and 04/09/2012); Articles 1, 3 · section V, subsection a), 4 · and 12 · sections I and III of the Regulations of the Federal Institute of Industrial Property (DOF 14/12/1999, -reformed on 07/01/2002, 07/15 / 2004, 07/28/2004 and 09/07/2007); Articles 1, 3, 4, 5, section V, Section a), 16 sections l and III and 30 of the Organic Statute of the Mexican Institute of Industrial Property (DOF) 12/27/1999, amended on 10/10/2002, 07/29/2004, 08/04/2004 and 09/13/2007); 1, 3 and 5 Clause a) of the Agreement that delegates powers to the Deputy Directors General, Coordinator, Divisional Directors, Holders of the Regional Offices, Divisional Deputy Directors, Departmental Coordinators and other subordinates of the Mexican Institute of Industrial Property. (DOF 12/15/1999, amended on 02/04/2000, 07/29/2004, 08/04/2004 and 09/13/2007).
<img file="MX339463B_D0002.tif" />
IMPI
MEXICAN INSTITUTE OE IX INDUSTRIAL PROPERTY!
IMAGE CODING METHOD, IMAGE DECODING METHOD, IMAGE CODING DEVICE, IMAGE DECODING DEVICE AND CODING DEVICE AND
DECODING IMAGES
Field of the Invention
The present invention relates to an image encoding method for encoding an image and an image decoding method for decoding an image.
Background of the Invention
Most of today's standardized video encoding algorithms are based on hybrid video encoding. In hybrid video encoding methods, several different lossless and lossy compression schemes are used in order to achieve the desired compression gain. Hybrid video encoding is also the basis for ITU-T standards (H.26x standards such as H.261, H.263), as well as ISO / IEC standards (MPEG-X standards such as MPEG-1,
MPEG-2 and MPEG-4).
The latest and most advanced video encoding standard is currently the standard represented as H.264 / MPEG-4 Advanced Video Encoding (AVC). This is the result of efforts of
<img file="MX339463B_D0003.tif" />
Ref. 252685
<img file="MX339463B_D0004.tif" />
standardization by the Joint Video Team (JVT), a joint ITU-T team and ISO / IEC MPEG groups.
Furthermore, a video coding standard called High Efficiency Video Coding (HEVC) is being considered by the Joint Collaborative Video Coding Team (JCT-VC). , with the particular aim of improving efficiency with respect to high-resolution video encoding.
List of References
Bibliography that is not Patent
Non-Patent Bibliography 1: C. Gordon et al., Wavefront Parallel Processing for HEVC Encoding and Decoding, JCTVC-F274-v2, from the Meeting in Torino, July 2011, Internet <URL: http: // phenix. intevry. f r.>
Non-Patent Bibliography 2: A. Fuldseth et al., Tiles, JCTVC-F355-vl, from the Meeting in Torino, July 2011, Internet <URL: http: // phenix. intevry. fr.>
Non-Patent Bibliography 3: JCTVC-J1003_d7, High efficiency video coding (HEVC) text specification draft 8, July 2012, page 73, dependent_slice_flag, Internet <URL: http: // phenix. IT-sudparis. eu / jct />
IMPI
Brief Description of the Invention
ΙΝΤΓΓΤυΤΟ MEXICANO • E LA PROPIEDA »INDUSTRIAL
<img file="MX339463B_D0005.tif" />
Technical problem
However, there is a problem that an image encoding method, an image decoding method, and the like do not have sufficient processing efficiency.
Therefore, the present invention provides an image encoding method and an image decoding method that are capable of increasing processing efficiency.
Solution to the problem
An image encoding method in accordance with an aspect of the present invention is an image encoding method for performing encoding processing by dividing a visual into a plurality of slices, the image encoding method comprises transmitting a stream of bits which includes: a dependent cut enable indicator that indicates whether or not the display includes a dependent cut on which encoding processing is performed depending on a result of encoding processing on a slice different from a current slice; a cut direction that signals a current cut start position; and a dependency indication indicating whether or not the current cut is the dependent cut, where the
<img file="MX339463B_D0006.tif" />
Dependent cut enable indicator is provided in a common set of parameters for cuts, the cut direction is set in the cut heading of the current cut, and the dependency indication is set in the cut header, and is set before the break address and after a syntax element that identifies the parameter set.
An image decoding method in accordance with one aspect of the present invention is an image decoding method for performing decoding processing by dividing a visual into a plurality of slices, the image decoding method comprises extracting, from a encoded bitstream, a dependent cut enable indicator that indicates whether or not the display includes a dependent cut on which decoding processing is performed depending on a result of decoding processing on a cut different from a current cut, a cut direction that indicates a starting position of the current cut and a dependency indication that indicates whether or not the current cut is the dependent cut, where the dependent cut enable indicator is provided in a common set of parameters for cuts, the cut direction is set in a cut heading of the current cut and the dependency indication is set in the cut header and set before the break address and after a syntax element that identifies the parameter set.
The general and specific aspects disclosed above can be implemented using a system, method, integrated circuit, computer program or computer-readable recording medium such as a CDROM, or any combination of systems, methods, integrated circuits, computer programs or computer readable recording media.
Advantageous Effects of the Invention
An image encoding method and an image decoding method according to the present invention are capable of increasing the encoding efficiency.
Brief Description of the Figures
These and other objects, advantages and qualities of the description will become apparent from the following description thereof taken in conjunction with the associated figures illustrating a specific embodiment of the present invention.
FIGURE 1 is a block diagram showing an example of a HEVC compliant encoder.
FIGURE 2 is a block diagram showing an example of a HEVC compliant decoder.
IMPI
MEXICAN INSTITUTE Of. THE INBUSTRIAL FRORDA
FIGURE 3 is a diagram showing an example
<img file="MX339463B_D0007.tif" />
of an image configuration in parallel wavefront processing (WPP).
FIGURE 4 is a diagram showing an example of a relationship between a normal cut and a dependent cut in parallel wavefront processing.
FIGURE 5 is a diagram showing an example of a packet header.
FIGURE 6 is a diagram showing an example of a cut heading of an entropic cut or a dependent cut.
FIGURE 7 is a diagram showing dependencies and signal transmission when normal cutoff is used.
FIGURE 8 is a schematic view showing dependencies and signal transmissions when using dependent cut and entropic cut.
FIGURE 9A is a diagram showing an example to show a syntax implementation of inter-layer dependencies, temporary dependencies and inter-cut dependencies in HM8.0.
FIGURE 9B is a diagram to explain the analysis steps that are performed to analyze the inter-layer dependencies in HM8.0.
FIGURE 9C is a diagram to explain the analysis steps that are performed to analyze dependencies.
ΙΜΡΙ
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0008.tif" />
inter-layers in HM8.0.
FIGURE 10 is a diagram showing an example of the position of the dependent_slice_flag.
FIGURE 11 is a diagram showing an example of the syntax when removing the parsing condition with respect to dependent_slice_enabled_flag in FIGURE 10.
FIGURE 12 is a diagram showing an example of the syntax when the dependent_slice_flag is moved before the first_slice_in_pic_flag.
FIGURE 13 is a diagram showing an example of the syntax when the dependent_slice_flag is moved before the slice_address syntax element.
FIGURE 14 is a diagram showing an example of the syntax when the dependent_slice_flag is moved within the NAL header.
FIGURE 15 is a diagram showing an example of a cut header syntax for a dependent break when a new type is added to the NAL unit types used for dependent breaks.
FIGURE 16 is a diagram showing an example of the syntax of a cut header and a NAL unit header when the dependent_slice_flag is assumed to be set to 1 for certain types of NALUs.
FIGURE 17 shows a complete configuration of a content delivery system to implement
IM ΡI
INSTITUTO MEXICANO DE IA PROPIEDAD industrial content distribution services. _
FIGURE 18 shows a complete configuration of a digital broadcast system.
FIGURE 19 shows a block diagram illustrating an example of a television configuration.
FIGURE 20 shows a block diagram illustrating an example of an information playback / recording unit configuration that reads and writes
<td>information of</td><td>and in</td><td>a recording medium that is</td><td>a</td><td>disk</td>
<td>optical.</td><td></td><td></td><td></td><td></td>
<td>The</td><td>FIGURE</td><td>21 shows an example</td><td>of</td><td>a</td>
<td>setting</td><td>of a</td><td>recording medium which is</td><td>a</td><td>disk</td>
optical.
FIGURE 22A shows an example of a cell phone.
FIGURE 22B is a block diagram showing an example of a cell phone configuration.
FIGURE 23 illustrates a multiplexed data structure.
FIGURE 24 schematically shows how each stream is multiplexed into multiplexed data.
FIGURE 25 shows in more detail how a video stream is stored in a packet stream.
PES.
FIGURE 26 shows a TS packet structure
ΙΜΡΙ
INSTITUTO MEXICANO DF LA PROPIEDAD INDUSTRIAL
<img file="MX339463B_D0009.tif" />
and source packets in the multiplexed data. _
FIGURE 27 shows a data structure of a
PMT.
FIGURE 28 shows an internal structure of multiplexed data information.
FIGURE 29 shows an internal structure of current attribute information.
FIGURE 30 shows steps to identify video data.
FIGURE 31 shows an example of a configuration of an integrated circuit to implement the encoding method of moving visuals and the decoding method of moving visuals according to each of the modalities.
FIGURE 32 shows a configuration for switching between drive frequencies.
FIGURE 33 shows steps for identifying video data and for switching between drive frequencies.
FIGURE 34 shows an example of a lookup table in which video data standards are associated with drive frequencies.
FIGURE 35A is a diagram showing an example of a configuration for sharing a module of a signal processing unit.
<img file="MX339463B_D0010.tif" />
The figure. 35B is an example diagram of a sharing configuration — air uiódulu lié 15 signal processing unit.
Detailed Description of the Invention Fundamental Knowledge that Forms the Basis of
Present Description
With respect to the image encoding method and the image decoding method described in the Background section of the Invention, the inventors have discovered the following problem.
First, an image encoding apparatus and an image decoding apparatus will be described in HEVC.
A video signal input to an image encoding apparatus is a sequence of images called frames (visual representations). Each frame includes a two-dimensional pixel array. All of the aforementioned standards based on hybrid video encoding include dividing each individual video frame into smaller blocks that include a plurality of pixels. The size of the blocks may vary, for example, according to the content of the image. The coding method can typically be varied on a block basis. The largest possible size for this block, for example in the HEVC, is 64 x 64 pixels. Is
ΙΜΡΙ
MEXICAN INSTITUTE Df. LA RROflEOAI '
INDUSTRIAL called the largest coding unit (LCU, for its
<img file="MX339463B_D0011.tif" />
acronym in English). The LCU can be repetitively divided into 4 CUs.
In H.264 / MPEG-4 AVC, a macroblock (usually indicating a 16 x 16 pixel block) was the basic picture element, for which encoding is performed. The macroblock can be further divided into smaller sub-blocks. The encoding steps included in the encoding method and / or the decoding steps included in the decoding method are performed on a sub-block basis.
1-1. Hybrid video encoding
The following simply describes hybrid video encoding.
Typically, the encoding steps of a hybrid video encoding include spatial and / or temporal prediction (space prediction and / or time prediction). Accordingly, each block that is encoded is first predicted using either the blocks in its spatial vicinity or the blocks in its temporal vicinity, ie, previously encoded video frames. A residual block which is a difference between the block that is encoded and its prediction result is then calculated. The residual block is then transformed from the spatial domain (pixel) into a domain of
<img file="MX339463B_D0012.tif" />
INSTITUTO MEXICANC 'OF LA FROFIEDAD INDUSTRIAL frequency. The transformation is aimed at reducing the correlation of the input block.
Additionally, the transformation coefficients obtained from the transformation are quantified. This quantification is lossy compression (irreversible). Usually, the compressed transformation coefficient values are further compressed without loss by means of entropic encoding. Furthermore, the auxiliary information necessary for the reconstruction of the encoded video signal is encoded and provided together with the encoded video signal. This is for example information about spatial prediction, temporal prediction and / or quantization.
1-2. Configuration of the image coding apparatus
FIGURE 1 is an example of a typical H.264 / MPEG-4 AVC and / or HEVC image encoding apparatus (encoder 100).
As shown in FIGURE 1, encoder 100 includes a subtractor 105, transformation unit 110, quantization unit 120, reverse transformation unit 130, adder 140, unlock filter 150, adaptive loop filter 160, frame memory 170 , prediction unit 180 and entropic encoder 190.
Prediction unit 180 drives a signal of
<img file="MX339463B_D0013.tif" />
IMPI
MEXICAN INSTITUTE
OE THE PROPERTY
INDUSTRIAL s2 prediction by means of temporal prediction or spatial prediction. The type of prediction used in the prediction unit 180 can be varied on a per frame basis or on a block basis. Temporal prediction is called inter-prediction and spatial prediction is called intra-prediction. Coding using a s2 prediction signal by means of temporal prediction is called inter-coding and coding using a s2 prediction signal by means of spatial prediction is called intra-coding. In deriving a prediction signal using temporal prediction, encoded images that are stored in memory are used. In deriving a prediction signal using spatial prediction, an encoded or decoded adjacent block boundary pixel value that is stored in memory is used. The number of prediction addresses in intra-prediction depends on the size of the encoding unit (CU). It should be noted that the details of the prediction will be described later.
The subtractor 105 first determines a difference (prediction error signal e) between a current block that is encoded from an input image (= input signal if) and a corresponding prediction block (= prediction signal s2). Difference is used for prediction
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY of the current block that is coded. It should be noted that the prediction error signal e is also called a prediction residual signal.
Transformation unit 110 transforms a prediction error signal e into coefficients. Generally, transformation unit 110 uses an orthogonal transformation such as a two-dimensional discrete cosine transformation (DCT) or an integer version thereof. Orthogonal transformation can reduce the correlation of the input signal if (the video signal before encoding) efficiently.
After transformation, lower frequency components are usually more important to image quality than high frequency components so that more bits can be consumed for encoding low frequency components than high frequency components .
The quantization unit 120 quantizes the coefficients and obtains quantized coefficients.
Entropic encoder 190 performs entropic encoding on quantized coefficients. The quantized coefficients are compressed without loss by means of entropic coding. Additionally, by means of entropic encoding, the volume of data stored in memory and the volume of data (bit stream) that
<img file="MX339463B_D0014.tif" />
IMPI * NSTITUTO MEXICANO LA PWOriSDAP 1ΜΠ »KTF1AL
<img file="MX339463B_D0015.tif" />
is transmitted can be further reduced. Entropic encryption is done by primarily applying the encoding using a variable length password. The length of a password is selected based on the probability of its occurrence.
Entropic encoder 190 transforms the two-dimensional matrix of quantized coefficients into a one-dimensional matrix. Typically, the entropic encoder 190 performs this conversion through a commonly called zigzag scan. Zigzag scanning starts with the DC coefficient in the upper left corner of the two-dimensional matrix and scans the two-dimensional matrix in a predetermined sequence ending with an AC coefficient in the lower right corner. Energy is typically concentrated in the upper left of the two-dimensional coefficient matrix. Generally, when the coefficients are located in the upper left corner, they are coefficients of low frequency components. When the coefficients are located in the lower right corner, they are coefficients of high frequency components. Therefore, zigzag scanning results in a matrix where usually the last values are consecutively a plurality of ones or zeros. This enables efficient encoding using path length codes as part of / before actual entropic encoding.
IMPI
MEXICAN INSTITUTE OF THE P »OPIK) AD INDHSTKIAl
<img file="MX339463B_D0016.tif" />
H.264 / MPEG-4 AVC and HEVC use different types of entropic encoding. Although some syntax elements are encoded with a fixed length, most syntax elements are encoded with variable length codes. In particular, among the syntaxes, Context Adaptive Variable Length Codes (CABACs) are used to encode prediction error signals (prediction residual signals). Generally, various other integer codes other than context-adaptive variable length codes are used for the encoding of other syntax elements. However, context-adaptive binary arithmetic coding can be used.
Variable length codes allow lossless compression of the encoded bitstream. However, since passwords are variable in length, decoding must be done sequentially on passwords. In other words, it is not possible to encrypt or decode passwords before encoding or decoding previous passwords without restarting (initializing) entropic encryption or without separately indicating a password position (starting point) to start decoding.
Arithmetic encryption encodes a sequence of bits into an individual password based on a model of
<img file="MX339463B_D0017.tif" />
INSTITUTO MEXICANC V LA PRi'PlíOAP IN »I ISTRIAL default probability. The predetermined probability model is determined according to the content of the video sequence in the case of CABAC. Arithmetic encoding, and thus also CABAC, are more efficient when the length of the bit stream being encoded is larger. In other words, CABAC applied to bitstreams is efficient for larger blocks. At the beginning of each sequence, CABAC restarts. In other words, at the beginning of each video sequence, your probability model starts with some predefined or predetermined values.
Entropic encoder 109 transmits, on one side of the decoder, a bit stream that includes encoded quantized coefficients (encoded video signals) and encoded auxiliary information.
The H.264 / MPEG-4 and H.264 / MPEG-4 AVC as well as the HEVC include two functional layers, a Video Coding Layer (VCL) and a Network Abstraction Layer (NAL). The VCL provides the encoding functionality as previously described. The NAL encapsulates information elements in standardized units called reporting units.
NAL according to your additional application such as transmission over a channel or storage on a storage device. The information elements
IMPI
MEXICAN INSTITUTE r> £ LA MOREDA! '
INDI ISTPIAL
<img file="MX339463B_D0018.tif" />
encapsulated by the NAL are, for example, (1) -1? norial Hp encoded prediction error (compressed video data) or (2) other information necessary for decoding the video signal such as the type of prediction, quantization parameter, motion vectors, etc. There are VCL NAL units that contain the compressed video data and related information, as well as non-VCL units that encapsulate additional data such as a parameter set that relates to a complete video stream, or a Supplemental Enhancement (SEI) that provides additional information that can be used to improve decoding performance.
Some non-VCL NAL units include, for example, parameter sets. A parameter set is a set of parameters that relate to the encoding and decoding of a certain portion of the video stream. For example, there is a Sequence Parameter Set (SPS) which includes a parameter relevant to encoding and decoding the entire sequence of visuals. In particular, the sequence parameter set is a syntax structure that includes syntax elements. In particular, syntax elements apply to zero or more encoded video streams
ΙΜΡΙ
MEXICAN INSTITUTE Ot IA RROPIEDAI? INDUSTRIAL
<img file="MX339463B_D0019.tif" />
Complete as determined by the content of a seq_parameter_set_id. The seq_parameter_set_id is a syntax element included in the visual representation parameter set (described later) referred to by the pic_parameter_set_id. The pic_parameter_set_id is a syntax element included in each cut header.
The Visual Representation Parameter Set (PPS) is a parameter set which defines parameters applied to the encoding and decoding of a visual representation of a sequence of visual representations (video sequence). In particular, PPS is a syntax structure that includes syntax elements. Syntax elements are applied to zero or more complete encoded visuals as determined by the pic_parameter_set_id which is a syntax element found in each cut header.
Therefore, it is easier to keep track of an SPS than a PPS. This is due to the fact that the PPS changes for each visual representation, while the SPS is constant for the entire video sequence which can be even minutes or hours in length.
Encoder 100 includes a reconstruction unit (commonly called a decoding unit) which gets a reconstructed signal (commonly called a decoded signal) s3. By means of the unit of
<img file="MX339463B_D0020.tif" />
<sup>20</sup> IMPI
INSTITUTO MiXICANf OE LA FXOREDaP inoustuial reconstruction, a reconstructed image that is obtained by means of the reconstruction (decoding) of the encoded image is generated and stored in the frame memory
170.
The rebuild unit includes the reverse transformation unit 130, the adder 140, the unlock filter 150, and the adaptive loop filter 160.
The inverse transformation unit 130, in accordance with the coding steps described above, performs inverse quantization and inverse transformation. It should be noted that the prediction error signal e 'obtained from the inverse transformation unit 130 is different from the prediction error signal e' due to the quantization error, also called quantization noise.
The adder 140 obtains a reconstructed signal s 'by adding a reconstructed prediction error signal e' reconstructed by the inverse transformation unit 130 to a prediction signal s2.
The unlock filter 150 performs the unlock filter processing to reduce the quantization noise which is superimposed on the reconstructed signal s' due to the quantization. In this document, since the coding steps described above are performed on a block basis, there is a case where a block boundary is visible when noise is
IMPL
MEXICAN INSTITUTE OF LA PROREDAD industrial
<img file="MX339463B_D0021.tif" />
overlays (noise blocking characteristics). The overlapping noise is called a blocking noise. In particular, when strong quantization is performed by quantization unit 120, there are more visible block boundaries in the reconstructed image (decoded image). This blocking noise has a negative effect on human visual perception, which means that a person feels that the image quality is impaired. For the purpose of reducing blocking noise, the unlock filter 150 performs unlock filter processing on each reconstructed signal s' (rebuilt block).
For example, in H.264 / MPEG-4 AVC unlock filter processing, for each area, a suitable filter processing is selected for the area. In the case of a high degree of blocking noise, a strong low-pass filter is applied (narrow band), while for a low degree of unlocking noise, a weaker low-pass filter is applied (wide band). The resistance of the low-pass filter is determined by the prediction signal e2 and by the prediction error signal e '. Unlock filter processing generally smooths the block edges. This leads to an improved subjective image quality of the decoded signals. The filtered image is used for the prediction of compensated movement of the following image.
Since filter processing also reduces
IMPI industrial prediction errors, the coding efficiency can be improved.
Adaptive Loop Filter 160 applies Sample Adaptive Compensation Processing (SAO) and / or Adaptive Loop Filter Processing (ALF) to the reconstructed image s after processing of the unblocking filter in the unlocking filter 150, to obtain a reconstructed signal (decoded signal) s3.
The deblocking filter processing in the deblocking filter 150 is aimed at improving subjective quality. Meanwhile, ALF processing and ODS processing in adaptive loop filter 160 are aimed at improving pixel fidelity (objective quality). SAO processing is used to add a compensation value to a pixel value for each pixel using a pixel value from the immediately adjacent pixel. ALF processing is used to compensate for image distortion caused by compression.
Typically, the filter used in ALF processing is a Wiener filter with filter coefficients determined such that the mean square error (MSE) between the reconstructed signal s' and the input signal if minimizes. Filter coefficients of ALF processing are calculated and transmitted
ΙΜΡΙ
IΝST1TUTO MEXICANO HF IA INDUSTRIAL RRORIEDAD
<img file="MX339463B_D0022.tif" />
on a frame-by-frame basis, for example. ALF processing can be applied to the entire frame (image) or to local areas (blocks). Auxiliary information indicating which areas should be filtered can be transmitted on a block-by-block basis, a frame-by-frame basis, or a quaternary tree-by-quaternary tree basis.
Frame memory (frame buffer)
170 stores part of the image encoded and reconstructed (decoded) (reconstructed signal s3). The stored reconstructed image is used to decode an intercoded block.
The prediction unit 180 obtains a prediction signal s2 using the (same) signal that can be used on both the encoder side and the decoder side, in order to maintain compatibility between the encoder side and the decoder side . The signal that can be used on both the encoder and decoder side is a reconstructed signal s3 (video signal after filter processing by adaptive loop filter 160) on the encoder side that is encoded and then reconstructed (decoded), and a reconstructed signal s4 (video signal after filter processing by the adaptive loop filter in FIGURE 2) on the decoder side that is decoded from a bit stream.
IMPI
INSTITUT · MEXICANO OF LA FROFIEDAD INDUSTRIAL
<img file="MX339463B_D0023.tif" />
Prediction unit 180, when generating a prediction signal s2 by intercoding, predicts using motion compensation prediction. A motion estimator of the prediction unit 180 (which is not illustrated) finds a best matching block for the current block of the blocks within the previously encoded and reconstructed video frames. The best matching block then becomes a prediction signal. The relative displacement (motion) between the current block and its best matching block is then signaled as motion data included in the auxiliary information in the form of three-dimensional motion vectors. The signal is transmitted along with the encoded video data. The three-dimensional motion vector includes two spatial dimension motion vectors and a time dimension motion vector. For the purpose of optimizing prediction accuracy, motion vectors can be determined at spatial sub-pixel resolution, eg, half-pixel or quarter-pixel resolution. A motion vector with spatial sub-pixel resolution can signal a spatial position within an already reconstructed frame where an actual pixel value, ie, a sub-pixel position, is not available. Therefore, spatial interpolation of these pixel values is necessary.
<img file="MX339463B_D0024.tif" />
IMPI
INSTITUTO MEXICANO oe the pnoFitoA® iNHUSTKIAt in order to make the prediction of compensated movement. This can be achieved by means of an interpolation filter (integrated within the prediction unit
180 in FIGURE 1).
1-3. Configuration of the image decoding apparatus
A configuration of a decoder (image decoding apparatus) will be described with reference to the
FIGURE 2.
FIGURE 2 is a block diagram showing an example of a decoder 200 in accordance with the H.264 / MPEG-4 AVC or HEVC video encoding standard.
As shown in FIGURE 2, decoder 200 includes an entropic decoder 290, inverse transformation unit 230, adder 240, unlock filter 250, adaptive loop filter 260, frame memory 270, and a prediction unit 280.
A bit stream fed into the decoder
200 (encoded video signal) is first transmitted to entropic decoder 290.
The entropic decoder 290 extracts the quantized coefficients that are encoded from the bit stream and the encoded auxiliary information and decodes the encoded quantized coefficients and the encoded auxiliary information. Auxiliary information,
IMPI
INSTITUTO MEXICANO OE LA PROPIEDAD INDUSTRIAL as described above, information necessary to
<img file="MX339463B_D0025.tif" />
decoding such as motion data (motion vector) and prediction mode (prediction type).
Entropic decoder 290 transforms decoded quantized coefficients in a one-dimensional matrix into those in a two-dimensional matrix by means of reverse scanning. The entropic decoder
290 introduces, to the inverse transformation unit 230, the quantized coefficients after being transformed into those in a two-dimensional matrix.
The inverse transformation unit 230 performs the inverse quantization and the inverse transformation on the quantized coefficients transformed into those in a two-dimensional matrix, to obtain a prediction error signal e '. The prediction error signal e 'corresponds to the differences obtained by subtracting the prediction signal from the input signal to the encoder in case no quantization noise is input and an error does not occur.
Prediction unit 280 drives a prediction signal s2 by means of temporal prediction or spatial prediction. Information such as the type of prediction included in the auxiliary information is used in the case of intra-prediction (spatial prediction). Furthermore, information such as motion data
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY included in the auxiliary information is used in the case of the prediction of compensated movement (inter-prediction, temporal forecast).
The adder 240 adds a prediction error signal e 'obtained from the inverse transformation unit 230 and a prediction signal e2 obtained from the prediction unit 280, to obtain a reconstructed signal s'.
Unlock filter 250 performs unlock filter processing on a reconstructed signal s'. Adaptive loop filter 260 applies ODS processing and ALF processing to the reconstructed signal s to which the unlock filter processing by the unlock filter 250 is applied. A decoded signal S4 obtained from the application of the SAO and ALF processing in adaptive loop filter 260 is stored in frame memory
270. The decoded signal S4 stored in the frame memory 270 is used, in the prediction unit 280, to predict the next current block which is the current image that is decoded.
1-4. Processing efficiency
Generally, parallelization is considered for the purpose of improving encoding processing processing and decoding processing.
<img file="MX339463B_D0026.tif" />
decoded or the processing the efficiency of
IMPI
INSTITUTO MEXICANO PF LA FfcOPIFPAÍ! NfX »« TP.lAl
Compared to H.264 / MPEG-4 AVC, the HEVC has
<img file="MX339463B_D0027.tif" />
a function to support high-level parallel processing (parallelization processing) of encoding and decoding. In HEVC, it is possible to divide a frame into slices, similarly to H.264 / MPEG-4 AVC. In this
<td>document, the</td><td>courts are groups of LCUs</td><td>in</td><td>the order</td><td>of</td>
<td>exploration.</td><td>In H.264 / MPEG-4 AVC,</td><td>the</td><td>cuts</td><td>are</td>
<td>decodable</td><td>independently and not</td><td>I know</td><td>apply</td><td>the</td>
spatial prediction between cuts. Therefore, parallel processing can be performed on a cut-by-cut basis.
However, since the cuts have significantly large headers and there is a lack of dependencies between the cuts, the compression efficiency is reduced. On the other hand, CABAC encoding loses efficiency when applied to small blocks of data.
In order to enable more efficient parallel processing, Wavefront Parallel Processing (WPP) is proposed. The WPP maintains a constant dependency which is different from parallel processing in which each of the breaks is independent.
The following description will be made by referring to the case where a visual representation comprises the LCUs each in visual representations which are
IMPI iNSTmrro muucano »t LA MOMEBaD
<img file="MX339463B_D0028.tif" />
they have in a matrix and each row of LCU comp ¥? Wde tn? cut (refer to FIGURE 3). In<sup>-</sup> WPP, it eliLltí the »'LCUs comprising the current row of LCU 32, as the CABAC probability model to restore the CABAC state of the first LCU (main LCU), the CABAC probability model is used just after Processing is completed on the second LCU of the previous LCU row 31. All inter-block dependencies are maintained. This allows the decoding of the LCU strings to be parallelized. The timing for starting each row processing of LCUs is delayed by two LCUs compared to the previous one. Information about the starting points to initiate LCU row decoding is included in the break header. WPP is described in detail in
Bibliography that is not Patent 1.
Another approach to improve parallelization is called mosaics. Therefore, a plot (visual representation) is divided into tiles. The tiles are rectangular groups of LCUs. The boundaries between the tiles are set in such a way that the complete visual representation is divided into a matrix. The tiles are processed in the gridded scanning order.
All dependencies are broken at the tile boundaries. Entropic coding such as CABAC is also restored at the beginning of each mosaic. Processing only
IMPI
ΙΝδΤΤΤίΓΓΟ MEXICANO OE LA PMOniDAO (NDUSTRIAL
<img file="MX339463B_D0029.tif" />
Unlock filter and adaptive sample compensation processing can be applied over the tile boundaries. In this way, the tiles can be encoded and decoded in parallel. Mosaics are described in detail in the Non-Patent 2 Bibliography and the Non-Patent 3 Bibliography.
On the other hand, in order to improve the concept of cuts and make it suitable for parallelization, preferably that for the resilience of errors which was the original purpose of cuts in H.264 / MPEG-4 AVC, the concept has been proposed of dependent cuts and entropic cuts.
In other words, in HEVC, there are three types of cuts supported: (1) normal cuts; (2) entropic cuts; and (3) dependent cuts.
Normal cuts indicate cuts already known from H.264 / MPEG-4 AVC. Spatial prediction between normal cuts is not allowed. In other words, prediction of cutoff limits is not allowed. This means that a normal cut is encoded without reference to any other cut. In order to enable independent decoding of these cuts, the
CABAC restarts at the beginning of each cut.
When the cut being processed is a normal cut, the CABAC restart includes the final processing
ΙΜΡΙ <ΜCTITI ι * Τ> / - »ι <e ν ι / ->. , «MEXICAN NSTITUTE ΠΕ INDUSTRIAL PROPERTY
INDUSTRIAL
<img file="MX339463B_D0030.tif" />
(termination processing) arithmetic encoding processing or arithmetic decoding processing at the end of the preceding break, and initialization processing of the context table (probability table) to a default value at the beginning of the normal break.
Normal cuts are used at the beginning of each frame. In other words, each frame has to start with a normal cut. A normal cut has a header that includes parameters necessary for decoding the cut data.
The term entropic sections indicates sections in which the spatial prediction between the parent section and the entropy section is allowed. The analysis of the parent cut and the entropic cut is performed independently.
However, the parent cut is, for example, a normal cut just before the entropic cut. The parent slice is required for the reconstruction of the pixel values of the entropy slice. In order to make the independent analysis of the entropic cuts possible, CABAC is also restarted at the beginning of the cut. As the cut header of entropic cuts, it is possible to use a cut header which is shorter than the cut header of the normal cut. The cut heading of the entropic cuts includes a subset of the parameters of
IMPI
INSTITUTO MEXICANO DE LA FROFIEDAD INDUSTRIAL coding with respect to the information transmitted within the heading of a normal break. Missing elements in the entropy cut heading are copied from the parent cut heading.
When the cut being processed is an entropic cut, restarting the CABAC, similar to the normal cut, includes final processing (processing of
<img file="MX339463B_D0031.tif" />
<td>termination)</td><td>in</td><td>The end of</td><td>cut</td><td colspan="2">preceding</td><td>and the</td>
<td>processing</td><td>of</td><td>initialization</td><td>of the</td><td>table</td><td>of</td><td>context</td>
<td colspan="4">(probability table) at a value per</td><td>default</td><td>to the</td><td>Start</td>
of the current cut.
(3) The dependent cut is similar to an entropic cut, but is partially different in the processing in which the CABAC is restarted.
When the break that is processed is a dependent break and the WPP is not effective, the CABAC reset includes the final processing on the preceding break (termination processing) and the initialization processing of the context table to a state value of end of the preceding cut. When the break that is processed is a dependent break and the WPP is not effective, the CABAC reset includes the final processing on the preceding break (termination processing) and the initialization processing of the context table to a state value after of the LCU processing which belongs to the
IMPI
<img file="MX339463B_D0032.tif" />
mexican 'preceding cut and is the second extreme iz- ·· ®' ^ beginning of current cut
As previously described, restarting CABAC always includes termination processing. By contrast, on CABAC restart, the CABAC state is frequently moved.
Dependent cuts cannot be analyzed without a parent cut. Therefore, dependent cuts cannot be coded when the parent cut is not received. The parent cut is usually a preceding cut of the dependent cuts in a coding order and a cut which includes a full cut heading. This is the same for the parent cut of an entropic cut.
As described above, dependent entropic cuts use the cut heading (in particular, the cut heading information which is missing from the dependent cut heading) of the immediately preceding cut according to the coding order of the cuts. This rule is applied recursively. The parent cut of the current dependent cut depends on it being recognized as available for reference. Reference includes the use of spatial prediction between outages, shared CABAC states, and the like. A dependent break uses CABAC context tables that are generated at the end of the immediately preceding break. In this way, a dependent break does not initialize the tables of
IMPI Mexican institute
OF THE INDUSTRIAL RRORIEDAC
<img file="MX339463B_D0033.tif" />
CABAC to default values, but instead maintains the use of already developed context tables. Additional details regarding entropic and dependent cuts can be found in the Bibliography which is not
Patent 3.
HEVC provides several profiles. A profile includes some settings of the image encoding apparatus and the image decoding apparatus suitable for a particular application. For example, the main profile only includes normal and dependent cuts, but not entropic cuts.
As described above, the coded outages are further encapsulated in NAL units, which are further encapsulated, for example, in a Real Time Protocol (RTP) and finally in Internet Protocol (IP) packets. , for its acronym in English). Whether this or other protocol stacks, it makes it possible to transmit the encoded video on packet-oriented networks, such as the internet or some proprietary networks.
Networks typically include at least one or more routers, which employ special hardware that operates very fast. The role of the router is to receive IP packets, parse their IP packet headers, and
IMPI
MEXICAN INSTITUTE OF MONEDAD INWSTWAI.
<img file="MX339463B_D0034.tif" />
Accordingly, forward the IP packets to their respective destinations. Since routers need to manage traffic from many sources, the packet handling logic needs to be as simple as possible. The minimum requirement for the router is to verify the destination address field in the IP header in order to determine which route to take to send them. In order to further provide quality of service (QoS) support, intelligent (media-aware) routers further verify specialized fields in network protocol headers, such as an IP header, an RTP header and even a NALU header.
As can be seen from the above description of video encoding, the different types of cuts defined for the purpose of parallel processing such as dependent cuts and entropic cuts, are of different importance with respect to quality distortion with its damage. In particular, the dependent cuts cannot be parsed and decoded without a parent cut. This is because at its start of the dependent cut, the entropic encoder or decoder cannot be reset. Additionally, the father cut is more important for the reconstruction of the image or video.
At HEVC, dependent and entropic cuts
Ws iwsmvro MEXICANO DE LA noniDAD (NttrSTRIAL
<img file="MX339463B_D0035.tif" />
they introduce an additional dimension of dependency, specifically, inter-court dependency (a dependency within the plot). This kind of dependency is not considered by routers.
The dependencies described above and, in particular, the inter-court dependency are not considered at the network level. However, it would be desirable to take into account the dependency described above at the network level in order to provide better support for quality of service. Therefore, there is a need to improve the flexibility of packet handling at the network level when considering outage dependencies.
Details of the problem
1-5. WPP and dependent cut
Dependent cuts can be used in conjunction with parallel processing tools such as parallel wavefront processing (WPP) and mosaics. In particular the dependent cuts make the wavefront (undercurrent) able to decrease the transmission delay without causing a loss of coding.
On the other hand, dependent outages serve as starting points for CABAC undercurrents since CABACs are not reset on dependent outages. Furthermore, the information indicating that the start points can be transmitted in the bit stream with the
IMPI
MEXICAN INSTITUTE ----, ^ r-DE LA PROPIEDAD INDUSTRIA!
- -INDUSTRIAL purpose of providing starting points for possibly independent analysis. In particular, if more than two CABAC subcurrents are encapsulated in a normal or dependent break, the starting points are explicitly indicated in the form of the number of bytes per subcurrent. In this document, the undercurrent indicates a portion of the current which is independently analyzable thanks to the start points. Additionally, dependent breaks can be used as starting point markers, since each dependent break needs to have a NAL unit header. This means that the starting points can be marked with respect to these markers.
The two approaches, in particular the explicit starting point marking and the marking of the starting points via dependent cuts are used together.
As a rule, the starting point of each NAL unit (beginning of each NAL header) has to be identifiable. There is no requirement about the exact identification operation. For example, the following two methods can be applied.
The first method is a method of putting a start code (for example, 3 bytes long) at the beginning of each NAL header. The second method is a method
ΙΜΡΙ $
MEXICAN INSTITUTE φΣDi LA MOPIEÍMI} C *
INDUSTRIAL to place each NAL unit in a separate package
Due to dependency on cuts, the size of the cut header may be reduced.
With respect to entropic cuts, the method makes possible the parallel analysis of CABAC. This is because CABAC actually restarts at the beginning of the entropy cuts. In the case of CABAC parallel processing, CABAC represents a bottleneck which can be overcome by CABAC parallel analysis followed by sequential pixel reconstruction operations. In particular, the WPP parallelization tool makes it possible to decode each row of LCUs by means of a processing kernel (intellectual property code (IP kernel), a function block). It should be noted that the allocation of the LCU rows to the nuclei may be different. For example, two rows can be assigned to one core and one row can be assigned to two cores.
FIGURE 3 is a diagram showing an example of a configuration of visual 300. In FIGURE 3, visual 300 is subdivided into 31 to 3 m (m is the ordinary number of LCU) rows of encoding units plus large (LCU). Each of the rows of LCU 3 i (I = 1 for m) comprises LCUs 3il to 3in (n is the ordinary number of a column of LUC) that are arranged in a row. The row of LCU 3i corresponds to the Front of
ΙΜΡΙ
MEXICAN INSTITUTE ΟΙ ''. A PROPERTY
INDUSTRIAL
<img file="MX339463B_D0036.tif" />
wave i. Parallel processing can be done for wavefronts. The CABAC status arrow in FIGURE 3 indicates a relationship between LCU that refers to the CABAC status and the reference destination.
Specifically, in FIGURE 3, first of all, among the LCUs included in row LCU 31, the processing (encoding or decoding) starts for the main LCU 311. The processing in the LCUs is done in an order of the LCU 311 to 31n. After processing is performed on the first two LCUs 311 and 312 on row LCU 31, processing begins on row LCU 32. In the processing of the first row of LCU 321 of the LCU 32 column, as shown by the status arrow of
CABAC in FIGURE 3, the CABAC state just after processing in LCU 312 in row LCU 31 in the first row is used as the initial state of CABAC state. In other words, there is a delay of two LCUs between the two parallel processes.
FIGURE 4 is a diagram showing an example of the case where a dependent cut using the WPP is used. LCU rows 41 to 43 correspond to Wave Front 1, Wave Front 2 and Wave Front 3, respectively. LCU rows 41 to 43 are processed by their respective independent cores. In FIGURE 4, the row of LCU 41 is a normal cut and the rows of LCU
ΙΜΡΙ
MEXICAN INSTITUTE ΠΕ LA FROFIE0AD INDUSTRIAL
<img file="MX339463B_D0037.tif" />
at 4 m are dependent cuts.
Dependent outages make WPP capable of reducing delay. Dependent cuts do not have a complete cut heading. On the other hand, the dependent cuts can be decoded independently of other cuts as long as the start points (or the start point of dependent cuts, which is known as a ruler as described above) are known. In particular, dependent outages can make WPP suitable for low lag applications as well without incurring coding loss.
In the usual case of encapsulating the subcurrents (LCU rows) in cuts, it is mandatory to insert explicit start points in the cut header to ensure parallel entropic encoding and entropic decoding. As a result, a cut is ready for transmission only after the last undercurrent of the cut is fully encoded. The cutoff header is completed only after the encoding of all the subcurrents in the cutout is completed. This means that transmission of the start of a break cannot be initiated via packet fragmentation at the RTP / IP layer until the complete break is completed.
However, since the dependent cuts are
IMPI
MEXICO INSTITUTE '
I »E! A MONEDAD INDUSTRIAL can be used as starting point markers, no explicit starting point marking is required. Therefore, it is possible to divide a normal cut into many dependent cuts without loss of coding. Dependent outages can be transmitted as soon as the encapsulated substream encoding is complete (or even earlier in the case of packet fragmentation).
The dependent cuts do not break the spatial prediction dependency. Dependent outages don't break even parsing dependency. This is because analysis of the current dependent cut ordinarily requires the CABAC states of the previous cut.
When dependent outages were not allowed, then each row of LCUs can be configured to be one outage. This setting decreases the transmission delay, but at the same time, leads to a fairly high encoding loss as discussed in the section on
Background of Previous Invention.
Alternatively, the entire plot (visual representation) is encapsulated in a single cut. In this case, the starting points for the subcurrents (LCU rows) need to be marked in the cut header with the
<img file="MX339463B_D0038.tif" />
<td>purpose</td><td colspan="2">to make possible</td><td>its</td><td>parallel analysis.</td><td>How</td>
<td>Outcome,</td><td>exists</td><td>a delay</td><td>of</td><td>level transmission</td><td>of</td>
<td>frames. In</td><td>others</td><td>words,</td><td>the</td><td>header needs</td><td>to be</td>
IMPIféB »
INS UTUTO MEXICANO ^ '«n Xi»
K LA MOPCDAD
INDUSTRIAL ^ ** «modified after the entire frame is encoded. Having a complete visual representation encapsulated in an individual slice by itself does not increase transmission delay. For example, transmission of some parts of the cut may start before the complete encoding is complete. However, if WPP is used, then the break header needs to be modified later in order to write the starting points. Therefore, the complete cut needs to be delayed for transmission.
The use of dependent cuts thus makes it possible to reduce the delay. As shown in
FIGURE 4, a visual 400 is divided into a row of LCU 41 which is a normal cut and the rows of LCU 42 to 4 m which are dependent cuts. When each row of LCUs is a dependent cut, a transmission delay of one row of LCUs can be achieved without any loss of coding. This is caused by the fact that the dependent outages do not break any spatial dependencies and do not restart the CABAC engine.
1-6. Package Configuration
As previously described, network routers have to parse packet headers in order to enable quality of service provision. The quality of service is different according to
ΙΜΡΙ6®> *
MEXICAN INSTITUTE
PROPERTY 'i
INDUSTRIAL · * »τ13Η1 the type of application and / or priority of the service and / or the relevance of the packet for the distortion caused by its packet loss.
FIGURE 5 is a diagram showing an example of encapsulation (packing) of a bit stream.
Generally, the Real Time Protocol (RTP) is used for packaging. RTP is usually used for streaming media in real time. The header lengths of the respective involved protocols are basically fixed. Protocol headers have extension fields. Extension fields can extend the length of the headers by 4 bytes. For example, the IP header can be extended by up to 20 bytes. The syntax elements in the IP, User Datagram Protocol (UDP) and RTP headers are also fixed in length.
<td>The</td><td>FIGURE 5 shows a</td><td colspan="2">head of</td><td>package 500</td><td></td>
<td>included in</td><td>an IP packet.</td><td>The</td><td>header</td><td>package</td><td></td>
<td>shown in</td><td>FIGURE 5 includes</td><td>a</td><td>header</td><td>IP 510,</td><td></td>
<td>head of</td><td>ι UDP 530, header</td><td>of</td><td colspan="2">RTP 54 0, header</td><td></td>
RTP H264 560 payload and NAL 570 header. IP 510 header is a 20-byte-long header with a 4-byte 520 extension field. The payload of the IP packet is a UDP packet. The UDP packet
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0039.tif" />
It includes a UDP 53 0 header with a length of 8 bytes and the UDP payload. The UDP payload is made up of the RTP packet. The RTP packet includes a 540 RTP header with a 12-byte head length and a 4-byte 550 extension field. The RTP packet can be selectively extended by the extension field. The RTP packet payload includes a special 560 H264 RTP payload header with a length of 0 to 3 bytes followed by a HEVC NAL 570 header which is bytes in length. The NALU payload that includes the encrypted video packet follows the packet headers
500 (which are not shown in FIGURE 5).
Routers which are capable of providing improved quality of service are called Media Conscious Network Elements (MANE). The Media Conscious Network Elements verify some of the fields in the packet headers shown in FIGURE 5. For example, a MANE is called temporal_id and is included in the NAL header.
570 or the decoding order number included in the RTP 540 header can be verified for the purpose of detecting losses and the order of presentation of the contents of received packets. Routers (network elements) manage packets as fast as possible in order to enable high performance on the network.
IMPI Mexican Institute OF LA PRORIEDA »INDUSTRIAL
<img file="MX339463B_D0040.tif" />
the fields in the and simple with the of the
Logic is required to access packet headers quickly in order to keep the complexity of network elements low.
The NALU is encapsulated by header 500. The NALU may include break data when a break header is present.
FIGURE 6 is a diagram showing an example of a cut header syntax 600. The dependent_slice_flag syntax element 601 is a syntax element indicating whether or not a cut is a dependent cut. The syntax element can be used to identify the inter-cut dependency. However, the cut heading is the content of a NALU. Parsing syntax elements before dependent_slice_flag 601 requires quite complicated logic. This is a level which cannot be considered efficiently by ordinary routers as will be shown later.
As described above, a NALU includes common information for a plurality of cuts such as parameter sets, or includes cuts directly encoded with information necessary for decoding included in the cut header. The syntax of a cut header used for an entropic or dependent cut is exemplified in FIGURE 6. FIGURE 6
IMPI 46
INSTITUT
MEXICAN INSTITUTE Í> F LA PBOUcr, ΛΠ
DE LA PROFlií / AÓ industrial
<img file="MX339463B_D0041.tif" />
displays a table with a break heading structure.
When the dependent_slice_flag syntax element is set to 1, all cuts to the first normal cut (a cut which is not an entropic cut and is not a dependent cut) preceding the current cut in the decoding order are required. When the cuts are not decoded, in general, the current dependent cut cannot be decoded. In some special cases, for example, the dependent cutoff may be decodable when some other noted or obtained secondary information is available. The dependent_slice_flag 601 syntax element is included in the middle of the cut header. On the other hand, the break header includes the number of CABAC subcurrents within the current break indicated by the num_entry_point_offsets information element 602 and the number of bytes in a substream indicated by the entry_point_offset [i] 603 syntax element. In this document , information element num_entry_point_offsets 602 corresponds to the number of entry points. Additionally, i is an integer and an index indicating the particular entry points (entry point offsets). The number of bytes in a substream indicated by entry_point_offset [i] 603 makes easy navigation within the bitstream possible.
1-7. Dependence on visual representation
IΛ4 ΡI
INSTITUTO MEXICANI) • ε la reoniDAf industrial
As described above, there are several types of dependencies that result from the HEVC encoding approach.
FIGURE 7 is a diagram showing the dependencies and their signaling in the case in which only normal cuts are used, that is, no dependent cuts and entropic cuts are used. FIGURE 7 shows three visuals 710, 720 and 730.
Visual 710 is a base layer visual carried in two VCL NALUs, specifically VCL NAL Unit 1 and VCL NAL Unit 2. POC indicates the order in which the visuals should be converted. VCL NALU includes a syntax element that indicates whether a visual representation belongs to a base layer or an enhancement layer and a temporary_id syntax element. The syntax element indicating whether a visual belongs to a base layer or an enhancement layer is transmitted under a condition of being within the NAL 570 header of the packet header 500 shown in FIGURE 5. The temporal_id syntax element is also transmits under a condition of being within the NAL 570 header. The temporal_id syntax element indicates the degree of dependency on the other visuals. For example, decodes or slices encoded with decodable temporal_id = 0 are
<img file="MX339463B_D0042.tif" />
IMPI
MEXICAN INSTITUTE
OF IA OWNED
INDUSTRIAL regardless of other visuals / cuts which have a higher temporal_id. It should be noted that in the HEVC, the temporal_id is denoted in the NAL header as nuh__temporal_id_plusl (refer to FIGURE 9A). In particular, the following Expression 1 can be applied to the relationship between the temporary_id used in those examples and the nuh__temporal_id_plusl syntax element.
Mathematical equation 1 temporaljd = nuhjemporaljd_plusl -1 (Expression 1)
Cuts with temporal_id = 1 depend on cuts of temporal_id with a lower value. In other words, the value of temporal_id in this case is 0. In particular, the syntax element temporal_id refers to the prediction structure of the visual. In general, cuts with a particular value of temporal_id depend only on cuts with a lower or equal value of temporal_id.
Accordingly, a display 710 in FIGURE 7 can be decoded first.
A visual 720 is an enhancement layer for the base layer of visual 710. Thus, there is a dependency which requires visual 720 to be decoded after decoding visual 710. The representation
IMPIg
MEXICAN INSTITUTE oE LA MONEDA! '' Λ »
INDUSTRIAL visual 720 includes two NALUs, specifically Unit 3 of
<img file="MX339463B_D0043.tif" />
VCL NAL and Unit 4 of VCL NAL. Both 710 and 720 visuals have their POC value of 0. This means that the 710 and 720 visuals belong to the same image that is immediately displayed. The images comprise the base layer and the enhancement layer.
Visual 730 is a base layer which includes two NALUs, specifically VCL Unit 5
<td>NAL and</td><td>Unit 6 of</td><td>VCL</td><td>NAL.</td><td>The</td>
<td>has</td><td>the value of</td><td>POC</td><td>of</td><td> 1.</td>
<td colspan="2">visual representation</td><td colspan="2">(portion)</td><td> 730</td>
This means that of the visual representations 720 and 710. On the other hand, the visual representation 730 has the value of temporal_id = 1. This means that the visual representation 730 depends temporarily on a visual representation with temporal_id = 0. Therefore, with Based on the dependency noted in the NAL header, visual 730 depends on visual 710.
FIGURE 8 is a diagram showing dependencies (degree of dependency) and their signaling in the case where dependent cuts and non-entropic cuts are used. FIGURE 8 shows three visuals 810, 820, and 830. FIGURE 8 differs from FIGURE 7 described above in that the cut dependencies
ΙΜΡΙ
INSTITUTO MEXICANO OE LA RRORIEDAD INDUSTRIAL
<img file="MX339463B_D0044.tif" />
dependents and entropics noted within the cut heading are added.
In FIGURE 7, the inter-layer dependency is shown with the example of the visual representations 710 and 720. On the other hand, the temporal dependency is shown in the example of the visual representations 710 and 730. These dependencies are both indicated in the NAL header.
Inter-cut dependency as shown in FIGURE 8 is inherent in dependent and entropic cuts. In particular, the base layer frame 810 and the enhancement layer frame 820 both have two cuts. Of the two cuts, one is a parent cut (normal cut) and the other is a child cut (dependent cut). In frame 810, the VCL NAL Unit 1 cut is the parent cut of VCL NAL Unit 2. In frame 820, the VCL NAL Unit 3 cut is the parent cut of VCL NAL Unit 4. As described above, the term parent cut of a dependent cut refers to a cut on which the dependent cut depends, that is, the cut on which the cut heading information is used by the dependent cut. This is as a general rule that the first preceding cut is a cut that has a full heading. The cut that has a full heading is a normal cut, not another dependent cut, for example.
The corresponding syntax of the header of the
IMPI
INSTITUT · MEXICANO • E INDUSTRIAL RIORITY as used in HM8.0 is
<img file="MX339463B_D0045.tif" />
NAL unit and cut heading currently in the HEVC and in particular will be described with reference to FIGURE 9A.
FIGURE 9A is a diagram showing the syntax of a NAL 910 unit header and the syntax of cut heading 920. In particular, inter-layer dependencies are planned (in current standardization) to be flagged within the header of NAL unit using the nuh_reserved_zero_6bits syntax element. Temporary dependencies are flagged using the nuh_temporal_id_plusl syntax element. Cutoff header 920 includes a signal showing the inter-cut dependency indicator. The inter-cut dependency flag is signaled by the dependent_slice_flag syntax element. In other words, the inter-cut dependency (eg temporary dependency) is noted within the cut header, somewhere in the cut header.
For the purpose of parsing this syntax element, all syntax elements preceding dependent_slice_flag must be parsed as well as the parameter set syntax elements necessary to parse cut header syntax elements preceding dependent_slice_flag.
1-8. Processing on the router
As described above, in the
IMPI
MEXICAN INSTITUTE OF PROPERTY
<img file="MX339463B_D0046.tif" />
determination of traffic settings is <sup>ST</sup>click.
take into account the dependencies induced by the córces<sup></sup>dependents and entropics, in addition to the dependencies indicated in the NAL heading. For example, a router can be implemented as a media-aware mobile base station. Bandwidth on the downlink is very limited and needs to be handled very carefully. The following exemplary case is assumed. A packet is assumed to be randomly abandoned in the upstream by a normal router. In this case, a media aware network element (MAME) discovers the packet loss by verifying the packet number. After verifying the packet loss, MANE drops all packets which are dependent on the dropped packet and which follow. This is a desirable quality for media conscious network elements. In this way, packets can be more intelligently abandoned. When it is determined that a router is leaving a NAL unit, it would immediately infer that the following dependent outages need to be abandoned as well. In the current syntax introduced in FIGURE 9A, accessing the dependent_slice_flag requires parsing of a considerable amount of information. This is not essential for packet routing operations or traffic configuration on routers. All the information that is
ΙΜΡΙ
INSTITUTO MEXICANO OE LA RROHEDAD INDUSTRIAL
<img file="MX339463B_D0047.tif" />
necessary to discover inter-layer and intertemporal relationships is present in the video parameter set. The video parameter set is the highest set in the parameter set hierarchy.
Accordingly, the information described above is noted within the heading of NAL 570.
However, in the case of the NAL header and the cut header shown in FIGURE 9A, access to cut dependency information requires keeping track of additional parameter sets such as PPS and SPS. This, on the other hand, reuses the capacity of media-aware gates or routers. As seen from FIGURE 9A, the cut header 920 has to be parsed down to the dependent_slice_flag and the parsed parameters are useless for network operation.
In order to enable parsing direction analysis which precedes the dependent_slice_flag, the following syntax elements are required from the syntax elements included in the SPS930 as shown in FIGURE 9B. FIGURE 9B is a diagram showing an example of the syntax included in the SPS.
• pic_width_in_luma_samples (reference sign 931 in FIGURE 9B) • pic_height_in_luma_samples (reference sign
932 in FIGURE 9B)
ΙΜΡΙ
MEXICAN INSTITUTE OE INDUSTRIAL PROPERTY
<img file="MX339463B_D0048.tif" />
• log2 min coding block size minus 3 (sign of .. ...
reference 933 in FIGURE 9B) • log2_diff_max_min_coding_block_size (reference sign 934 in FIGURE 9B).
Those parameters are shown in the right table of FIGURE 9B and are required to obtain the slice_address parameter. The slice_adress syntax element is variable-length encoded (as can be seen when looking at length v in the descriptor, second column, of slice_address and cut header 920 in FIGURE 9A). In order to know the length of this variable length encoded parameter, those SPS syntax elements are required. In fact, in order to make parsing of the dependent_slice_flag possible, the actual value of the slice_address syntax element is not required. Only the length of the syntax element which is variable should be known so that the parsing process can continue.
Therefore, the SPS needs to be parsed to point 935 of the syntax elements within SPS 930 shown in FIGURE 9B. All four syntax elements are required to be stored. These are later used in a formula to calculate the length of the slice_address syntax element.
On the other hand, in order to access the
ΙΜΡΙ
MEXICAN INSTITUTE ΠΕ LA FROWSDAO INDUSTRIAL
<img file="MX339463B_D0049.tif" />
dependent_slice_enabled_flag which also precedes the dependent_slice_flag, the PPS needs to be parsed up to point 945 of the syntax elements within the PPS shown in FIGURE. 9C. FIGURE 9C is a diagram showing an example of syntax included in the PPS. It should be noted that the syntax elements whose parsing methods are described with reference to FIGURES 9A through 9C and which are located within the break header and the SPS and PPS are not required for common router operations. On the other hand, some syntax elements cannot simply be omitted since some of the syntax elements are encoded with variable length codes. Consequently, even if a jump is made to the bitstream by a predefined number of bits, the jump to the dependent_slice_enabled_flag is not possible.
In other words, for the purpose of reading the dependent_slice_flag (dependency indication), the MANE needs to go further into the cut header (refer to cut header 920) whose analysis is quite complicated.
Specifically, the first_slice_in_pic_flag flag has to be parsed. The first_slice_in_pic_flag flag is a flag that indicates whether or not a cut is the first cut within the visual.
IMPI
INSTITVT »MEXICAN · • e raontwAD industrial
<img file="MX339463B_D0050.tif" />
Then no_output_of_prior_pics_flag whose presence is conditional on the NALU type has to be parsed.
Furthermore, the variable length encoded pic_parameter_set_id has to be decoded. The pic_parameter_set_id syntax element is a syntax element that indicates which of the parameter sets is used (a syntax element that identifies the parameter set). By parsing the pic_parameter_set_id, you can identify the parameter set that is used.
Finally, the slice_address syntax element is required. The slice_address syntax element is a syntax element that indicates the start position of the cut. The syntax element further requires parsing of the PPS and SPS as well as additional computation.
As the last step, the value of dependent_slice_enabled_flag (dependent cut-off enabled flag) has to be obtained from the PPS, in order to know whether or not the dependent_slice_f lag is present in the bitstream. When dependent_slice_enabled_flag = 0, it means that the current cut is a normal cut since dependent cuts are not enabled. In order to obtain the value of the dependent_slice_enabled_flag, the PPS is required to be analyzed to about half.
Unfortunately, the syntax elements before
<img file="MX339463B_D0051.tif" />
dependent_slice_flag cannot be omitted and need to be parsed unlike in the case of RTP and NAL header data, in which the position of the data is predefined. This is caused by the fact that the syntax elements in the cut header are variable length encoded. Therefore, the presence and length of the element needs to be calculated for each VCL NAL unit. Also, additional session data needs to be stored because it is needed later (refer to PPS and SPS). Furthermore, the presence of some syntax elements depends on the presence or value of other syntax elements possibly included in other parameter structures (syntax elements are conditionally encoded).
In the current standardization, there is a proposal to point out the dependency structure of the video stream in the Video Parameter Set (VPS) that describes how many layers are contained in the bit stream and the indicators of dependency to indicate the various inter-layer dependencies. The VPS is signaled very early in the video, before the first SPS. Multiple SPSs can refer to an individual VPS. This means that a VPS carries information that is valid for multiple video streams. The main purpose of the VPS is to inform a router or decoder about the
IMPI
<img file="MX339463B_D0052.tif" />
INSTITUTO MEXICANO PE LA WOEIt »AD content of the video that includes information. How many video sequences are included and how they are interrelated. The SPS is valid only within a video sequence while the VPS carries information related to multiple video sequences.
Furthermore, the information feature carried in the VPS is informative especially for routers. For example, the VPS could carry information that is required for establishing an uninterrupted transmission session since the design is not finalized. The router analyzes the information in the VPS. The router, without the need for other parameter sets (only by looking in the NAL headers), can determine which data packets to send to the decoder and which packets to drop.
However, for the purpose of discovering the VPS
<td>at present</td><td>active,</td><td>is</td><td colspan="2">necessary the following steps</td>
<td>ordered it</td><td>perform</td><td></td><td></td><td></td>
<td>analysis of</td><td colspan="2">PPS_id in the</td><td>cutting header;</td><td></td>
<td>analysis of</td><td>SPS_id</td><td>in</td><td>the active PPS determined by</td><td>the</td>
<td>PPS_id; and</td><td></td><td></td><td></td><td></td>
<td>analysis of</td><td>VPS_id</td><td>in</td><td>the active SPS determined by</td><td>the</td>
<td>SPS id.</td><td></td><td></td><td></td><td></td>
In order to solve the problem described above, an image encoding method of
ΙΜΡΪ> ~
MEXICAN INSTITUTE γ. γλ The PkOWEOAD £
<img file="MX339463B_D0053.tif" />
in accordance with an aspect of the present invention<sup>Nn</sup>and<sup>s</sup>to*<sup>,TO</sup>image coding tm to perform and the prüL'SüailliéñLÜ ¿15 'coding by dividing a visual representation into a plurality of cuts, the image coding method comprises transmitting a bit stream which includes: a dependent cut enable indicator that indicates whether or not the display includes a dependent cut in which encoding processing is performed depending on a result of encoding processing in a slice other than a current slice; a cut direction indicating a current cut start position; and a dependency indication (dependent_slice_flag) that indicates whether or not the current cut is the dependent cut, where the dependent cut enable flag is provided in a common set of parameters for cuts, the cut direction is arranged in a cut header of the current cut and the dependency indication is arranged in the cut header and is arranged before the cut direction and after a syntax element (pic_parameter_set_id) that identifies the set of parameters.
In the image encoding method described above, an inter-cut dependency dependency indication is located in a position suitable for analysis by the router. With this,
IMPI
MEXICAN INSTITUTE nt IA INDUSTRIAL PROPERTY
<img file="MX339463B_D0054.tif" />
it is possible to encode the syntax dependency indication independently, in other words, unconditionally, from other syntax elements.
For example, the dependency indication may be included in the bitstream when the dependent break enable flag signals the inclusion of the dependent break.
For example, the dependent break enable indicator may be set at the beginning of the parameter set.
For example, each slice can include a plurality of macroblocks, and encoding processing on the current slice can start after encoding processing is performed on two of the macroblocks included in an immediately preceding current slice.
For example, the dependency indication may not be included in a cut heading of a cut which is processed first for visual representation, between cuts.
For the purpose of solving the problem described above, an image decoding method in accordance with one aspect of the present invention is an image decoding method for performing decoding processing by dividing a visual into a
<img file="MX339463B_D0055.tif" />
<sup>61</sup> IMPI
INSTITUT »MEXICAN» £ LA
INDUSTRIAL plurality of cuts, the image decoding method comprises extracting, from an encoded bitstream, a dependent cut enable indicator indicating whether or not the display includes a dependent cut in which the decoding processing is performed depending of a decoding processing result in a different cut than a current cut, a cut direction indicating a starting position for the current cut and a dependency indication indicating whether or not the current cut is the dependent cut, where the dependent cut enable indicator is arranged in a common set of parameters for the cuts, the cut address is provided in the cut header of the current cut and the dependency indication is provided in the cut header and is arranged before the cut address and after a syntax element that identifies the parameter set.
For example, the dependency indication may be extracted from the bitstream when the dependent break enable flag signals the inclusion of the dependent break.
For example, the dependent break enable indicator may be set at the beginning of the parameter set.
For example, each of the cuts may include
ΙΜΡΙ
<img file="MX339463B_D0056.tif" />
MEXICAN PROPERTY INSTITUT a plurality of macroblocks and decoding processing in the current cut can be started after the decoding processing is done in two of the macro blocks included in an immediately preceding current cut.
For example, the dependency indication may not be included in a cut heading of a cut which is processed first for visual representation, between cuts.
For the purpose of solving the problem, an image coding apparatus in accordance with one aspect of the present invention is an image coding apparatus which performs coding processing by dividing a visual representation into a plurality of cuts, the apparatus Image encoding comprises an encoder which transmits a bit stream which includes: a dependent cut enable indicator that indicates whether or not the display includes a dependent cut in which coding processing is performed depending on the result of coding processing in a cut different from a current cut; and a cut direction indicating a current cut start position; and a dependency indication that indicates whether or not the current cut is the dependent cut, where the dependent cut enable indicator is provided
<img file="MX339463B_D0057.tif" />
<sup>63</sup> IMPI
INSTITUTO MEXICANO DE LA FROFIEDAD industrial in a common set of parameters for cuts, the cut direction is provided in a cut heading of the current cut and the dependency indication is set in the cut header and is placed before the cut direction. cut and after a syntax element that identifies the parameter set.
For the purpose of solving the problem, an image decoding apparatus in accordance with one aspect of the present invention is an image decoding apparatus which performs decoding processing by dividing a visual representation into a plurality of cuts, the apparatus decoding system comprises a decoder which extracts, from an encoded bitstream, a dependent cut enable indicator that indicates whether or not the display includes a dependent cut in which decoding processing is performed depending on a result of decoding processing in a cut other than a current cut, a cut direction that indicates a start position of the current cut and a dependency indication that indicates whether or not the current cut is the dependent cut, where the dependent cut enable indicator is provided in a common set of parameters for cuts, the cut direction is provided in a cut header of the current cut and the indication of
IMPI
<img file="MX339463B_D0058.tif" />
Dependency is arranged in the break header and is arranged before the break address and after a syntax element that identifies the parameter set.
For the purpose of solving the problem described above, an image encoding and decoding apparatus in accordance with an aspect of the present invention includes the image encoding apparatus described above and the image decoding apparatus described above.
In accordance with the image encoding method, the image decoding method and the like that are configured above, an inter-cut dependency indication is located within the bit stream syntax related to a cut independently of other elements. The dependency indication is located, without unnecessarily analyzing other elements, separately from the other elements. In the HEVC examples above, the inter-cut dependency flag dependent_slice_flag is pointed at a location where it is not necessary to parse irrelevant syntax elements for network operation.
Specifically, the present invention provides an apparatus for analyzing a bit stream of an image video stream encoded at least partially with a variable length code and which
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL RROlfcDAD
<img file="MX339463B_D0059.tif" />
they include data units that carry encoded cuts of video stream. The apparatus comprises an analyzer for extracting a dependency indication from the bit stream which is a syntax element that indicates for a cut whether variable length decoding or cut analysis depends or not on other cuts, where the indication dependency is extracted from the bitstream of and without the need to extract other syntax elements beforehand.
This apparatus can be included, for example, within the entropic decoder 290 shown in FIGURE 2.
When reference is made to bitstream extraction, extraction and, where necessary for extraction, an entropic decoding is implied. Entropic encoding is a variable length encoding, eg, arithmetic encoding such as CABAC. In HEVC, this applies to encoding image data. Data units in this document refer, for example, to NAL units or access units. The expression without having to extract other syntax elements refers to a situation in which the dependency indication is preceded only by elements, of which the length is known and of which the presence is known or conditioned on elements already analyzed or not conditionally encoded at all.
The present invention further provides an apparatus
<img file="MX339463B_D0060.tif" />
<sup>IN?</sup>VTUTO MEXICANO OE THE INDUSTRIAL MOHITY to generate a bit stream of a video stream encoded at least partially with a variable length code and including data units carrying encoded cuts of video images. The apparatus comprises a bit stream generator to incorporate a dependency indicator into the bit stream which is a syntax element indicating for a cut whether or not the variable length decoding of the cut is dependent on other cuts, where the dependency flag is incorporated into the bitstream independently of and without the need to incorporate other syntax elements beforehand.
This apparatus can be included, for example, within the entropic encoder 190 shown in FIGURE. one.
In accordance with the image encoding method, the image decoding method and the like which are configured above, the bitstream includes encoded cut data and header data with respect to the cut and the dependency indicator is located within the bit stream at the beginning of the break header.
This means that the break heading begins with the syntax elements that indicate the cut dependency.
It should be noted that the dependency indication does not have to be located at the very beginning of the break header. However, it is advantageous when no other
IMPI
MEXICAN INSTITUTE • E LA FROFIEDAD
INOU5T1IAI conditionally encoded syntax element and / or
<img file="MX339463B_D0061.tif" />
encoded by variable length precedes the dependency indicator within the break header.
For example, the current position of the dependent_slice_flag is changed from the prior art described above so that it is located at the beginning of the cut header. With this change, the amount of syntax elements that need to be analyzed is reduced. Complicated router analysis operations are avoided, such as variable length decoding and information analysis that requires additional calculations and / or storage of additional parameters for future use and / or analysis of other parameter sets. Furthermore, the number of parameter sets that are required to be tracked is reduced.
Hereinafter, the embodiments are specifically described with reference to the figures. Each of the modalities described below shows a general or specific example. The numerical values, configurations, materials, structural elements, the arrangement and connection of the structural elements, steps, the order of processing of the steps, etc. shown in the following modalities are only examples and therefore do not limit the scope of this invention. Thus,
IMPI
MEXICAN INSTITUTE OF THE INDUSTRIAL PMOFIEDAP
<img file="MX339463B_D0062.tif" />
Among the structural elements in the following embodiments, the structural elements not set forth in any of the independent claims are described as arbitrary structural elements.
Mode 1
FIGURE 10 shows an example of the bitstream syntax according to the present embodiment. A NAL 1010 header shown in FIGURE 10 is the same as the NAL 910 header shown in FIGURE 9A. In other words, there is no change.
However, the cut header 1020 syntax structure is different from the cut header 920 syntax structure of FIGURE 9A. In particular, in cut header 1020, the dependent_slice_flag is moved upward within the cut header such that there is no syntax element preceding the dependent_slice_flag. The dependent_slice_flag is conditionally encoded, encoded using a variable length code, or receives parsing that requires additional calculation.
The first_slice_in_pic_flag and dependent_slice_flag syntax elements both actually determine the spatial dependencies. Syntax elements are encoded immediately after the NAL header such that no other syntax element needs to be
IMPI
MEXICAN INSTmrro eg currency. INm'STPIAL
<img file="MX339463B_D0063.tif" />
analyzed. Since the first_slice_in_pic_flag also carries information which is related to inter-court dependencies, it can precede dependent_slice_flag. The first_slice_in_pic_flag syntax element is a flag which is set according to the rule that each frame has to start with a normal break. Therefore, when the first_slice_in_pic_flag flag is set, it means that the cut is a normal cut and thus independent. In this way, the dependent1ice_flag and the first_slice_in_pic_flag can both be seen together as an indicator of inter-cut dependencies.
In other words, the dependency indicator can be defined to include a first cut indication indicating whether or not the cut is a first cut in a visual display and a dependent cut indicator indicating whether the variable length decoding of the cut depends or not of other cuts. The first cut in a visual representation is always a cut for which variable length decoding does not depend on other cuts.
Advantageously, the bit stream includes a dependent break enable indicator that indicates whether or not the dependent breaks can be included within the bit stream. The dependency indication is included in the bit stream only when the
IMPI
INSTTTVTO MtXICANC ΠΕ LA PHOFIÍDAn (NrWfSTWlAl
<img file="MX339463B_D0064.tif" />
Enabling dependent cuts indicates that cuts
.....<sup>.</sup> c - dependents can be included in the bit stream. The dependent break enable indicator is located within the bit stream in a common parameter set for a plurality of breaks and located at the beginning of the parameter set. The parameter set can be, for example, the visualization parameter set which carries parameters for an individual visual representation. Alternatively, the dependent cut enable indicator is located within a set of sequence parameters which carries parameters for the complete image (video) sequence.
However, in the present invention, the dependent_slice_flag (dependency indication) is unconditionally encoded in the dependent_slice_enabled_flag syntax element (dependent break enable flag). In the present embodiment, since the identification of the set of visual representation parameters is located after the dependency indication, it is advantageous to avoid a possible analysis error in the case where the identification of the set of visual representation parameters is indicated within the cutting header.
This change can also be observed as and / or can be interpolated by the change in the position of the
IMPI
MEXICAN INSTITUTE OF LA MOHEDAL ' <sub>Ί</sub> INDUSTRIAL
<img file="MX339463B_D0065.tif" />
other syntax elements required in the box ^ 'unto parameters or headers to reduce the cañtldád de<sup>1 </sup>syntax elements that are required to be analyzed to determine dependencies between breaks.
For example, the dependent slice £ lag syntax element in the cut header of the present HM8.0 syntax is only present when the value of the dependent_slice_enabled_flag syntax element indicates that the use of dependent breaks within the bitstream is enabled . Enabling dependent breaks and thus also the dependent_slice_enabled_flag syntax element is included in the PPS as shown in FIGURE 9C. Accordingly, the dependent_slice_enabled_flag syntax element in the PPS is moved upwards within the PPS syntax in order to simplify its parsing necessary to parse the dependent_slice_flag (eg the beginning of the parameter set). This can also be useful when the dependent_slice_flag is encoded after the pic_parameter_set_id (the syntax element that identifies the parameter set). This is because when doing it this way, the parsing error is avoided even when the dependent break enable indicator is conditioning the presence of the dependency indication.
<img file="MX339463B_D0066.tif" />
Instead of moving the dependent_slice_enabled_flag upwards inside the
PPS, the dependent_slice_enabled_flag can be moved from PPS to SPS and / or VPS so that parameter sets which are lower in the hierarchy are not required to be tracked.
In other words, in accordance with the present embodiment, the position of the required syntax elements is changed for the purpose of reducing the number of parameter sets that a track needs to be kept on. This also reduces the complexity of analysis. The parameters required in this context mean the parameters involved which contribute to determining whether or not a cut is an interdependent cut. A first possibility directly applicable to the HEVC is to provide the dependency indication at the beginning of the dependent and unconditional cut header in the dependent cut enable flag which is included in a different parameter set than the cut header. A second possibility directly applicable to the HEVC is to provide the dependency indication in the dependent break header after the parameter set indication that identifies the parameter set in which the dependent break enable flag is included. The dependency indication can
ΙΜΡΙ institiit · mexican Γ »Ε IA prohWao INDUSTRIAL
<img file="MX339463B_D0067.tif" />
be conditioned on the indicator for enabling dependent cuts. The upward movement of the dependent break enable indicator within the PPS or the move of the dependent break enable indicator in the SPS can be beneficial to either of those possibilities. In particular, this is beneficial for the second possibility, in which it is necessary that the indicator of enabling of dependent cuts analyzes the indication of dependency.
As can be seen in FIGURE 10, the NAL unit header, along with the relevant portion of the cut header, has 18 bits (14 bits of the NALU header and 2 bits of the cut header). In accordance with this example, a media conscious network element can operate for a current outage pack as follows. If a previous cut is abandoned, which is a normal, entropic, or dependent cut, the network element checks the first two bits of the current cut header, which are the first_slice_in_pic_flag and (in the case where dependent cuts are allowed to the
<td>bit stream)</td><td colspan="2">the dependent_slice_flag.</td><td></td><td></td><td></td>
<td>When</td><td>the type</td><td>NAL unit</td><td>is</td><td>a</td><td>kind of</td>
<td colspan="2">VCL NAL unit and the</td><td>last two bits</td><td>of</td><td>the</td><td>18 bit</td>
<td>verified are </td><td>01, the</td><td>NAL unit is</td><td colspan="3">abandoned. In</td>
particular when the first bit of the cut header is
IMPI
MfclUCAN INSTITUTE »Dt THE INDUSTRIAL PROPERTY
<img file="MX339463B_D0068.tif" />
1, then it is the first cut in the visual representation that is not (according to the rules) a dependent cut. When the first bit of the cut header is 0 and the next bit of the cut header is also 0, the cut is not dependent. Therefore, only when the first two bits of the cut header are 01, the cut is dependent. Additionally, the cut must be abandoned since it cannot be decoded when the parent cut has already been abandoned. Therefore, the first_slice_in_pic_flag and dependent_slice_flag flags can be seen as an extension of the NAL header, even if they belong to the cut header syntax.
Accordingly, the present embodiment also provides as one of its aspects a network router to receive, analyze and send network packets to their destinations. The router includes a receiving unit for receiving a network packet that includes a packet destination address and a bitstream portion with encoded video data; an analyzer including the apparatus for analyzing a bit stream of an encoded video stream in accordance with any of the modalities cited above and below for the purpose of determining the dependence of the encoded video data on other packets; and a packet analyzer to analyze the destination address of received packets and the
IMPI
MEXICAN INSTITUTE Di LA MONEDAD INDUSTRIAL
<img file="MX339463B_D0069.tif" />
certain dependency and for network packet.
evaluate how to handle the
Mode 2
According to Mode 2, the dependent_slice_enabled_flag is dropped from the PPS. It should be noted that the dependent_slice_enabled_flag can be moved to the SPS, instead of being abandoned.
FIGURE 11 shows an example in which the dependent_slice_enabled_flag does not need to be parsed before accessing the first_slice_in_pic_flag and dependent_slice_flag.
In this example, the dependent_slice_enabled_flag is not used because it is not conditioned in the presence of the dependency indication. This example provides the ability to have the dependency indication at the beginning of the break header without causing parsing problems due to unknown identification of the current PPS set.
Effect of Mode 2, etc.
In Mode 1, for the purpose of parsing the dependent_slice_flag, the dependent_slice_enabled_flag has to be parsed. The dependent_slice_enabled_flag is signaled in a PPS. This may cause some parsing information as discussed above, when the dependent_slice_enabled_flag is located far from the start of the PPS and the preceding syntax elements are conditionally encoded.
IΜ ΡI Mexican Property Institute ”
INDIWHtlAL
Furthermore, signaling the dependent_slice_flag syntax element before the pic_parameter_set_id syntax element is parsed in the PPS can create parsing errors as follows. The presence of the dependent_slice_flag depends on the dependent_slice_enabled_flag which is signaled in the PPS. However, the identification of the currently active PPS is signaled after the dependent_slice_flag. Therefore, it is not possible to parse the dependent_slice_flag before accessing the previous elements.
Therefore, it is advantageous to remove the parsing condition in the dependent_slice_enabled_flag. It may be more beneficial when the following restriction applies. Specifically, if the dependent_slice_enabled_flag in the PPS is zero, then the dependent_slice_flag should be equal to zero.
However, these advantageous implementations do not limit the scope of the present invention.
Modification 1 of Modalities 1 and 2
As an alternative or in addition to removing the conditioning on the dependent_slice_enabled_flag, the dependent_slice_enabled_flag can be moved from the PPS to either the SPS and / or VPS.
On the other hand, instead of just moving the
ΙΜΡΙ
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0070.tif" />
dependent_slice_enabled_flag, the dependent_slice_enabled_flag can be duplicated in the SPS. In this case, the indicator in the SPS and the PPS could be forced to have the same value. 0, the PPS could be allowed to overwrite the indicator in the
SPS.
<td>By</td><td>example,</td><td>when</td><td>the</td>
<td>sps dependent</td><td colspan="2">slice enabled flag equals 1,</td><td>so he</td>
<td>pps dependent</td><td>slice enabled flag can</td><td>be 0 or</td><td>1. Then the</td>
<td>sps dependent</td><td>slice_enabled_flag is</td><td colspan="2">an indication of</td>
<td>rating -</td><td>dependent cuts</td><td>for one</td><td>sequence of</td>
<td colspan="2">designated visuals</td><td>at</td><td>SPS and the</td>
<td>pps dependent</td><td>slice enabled flag is</td><td colspan="2">an indication of</td>
<td>qualification</td><td colspan="2">dependent cuts</td><td>for one</td>
visual representation indicated in the PPS. However, when the value of the dependent_slice_enabled_flag can change in PPS, this means that PPS parsing is still required and the advantage of less frequent tracing and PPS parsing is prevented.
These modifications provide the advantage that the VPS and SPS have dependency structures. Bringing dependency structures by the VPS and SPS makes it possible for network elements to share bit streams, that is, to determine to drop dependent packets which cannot be decoded in any way or to drop outages
IMPI
INSTITUTO MEXICANO DE M NJOPIEDAO INDUSTRIai dependent preferably than independent courts. Therefore, the dependent_slice_enabled_flag in the VPS would cause the router to additionally or not verify the cut header.
It is noted that these modifications do not further reduce the complexity of analysis if the example of FIGURES 10 and 11 is applied. However, it provides a more beneficial structure of syntax for carrying dependency structures. In summary, according to this example, an indicator to indicate whether or not the dependent cuts are enabled for the bit stream is indicated in a set of video parameters. The video parameter set is a set of parameters that is applied to more than one cut in more than one visual representation.
There are two different advantages of dependent_slice_enabled_flag signaling: VPS and / or SPS. When the indicator is only moved or duplicated, the PPS is not required to be analyzed, reducing the accompanying analysis information. The other benefit is letting routers know about the prediction structure of the video stream. This advantage is present all the time. Usually, a router can check the content of a VPS / SPS in order to know what it will receive.
VPS is the highest parameter in the hierarchy. The
VPS can include information about multiple sequences
<img file="MX339463B_D0071.tif" />
IMPI
<img file="MX339463B_D0072.tif" />
MEXICAN INSTITUTE &
OF PROPERTY OR video, while SPS and PPS are specif<sup>></sup>l<sup>s</sup>e'OB an individual video sequence and a ™ · ι · ι · .ριιίΐ! ΐιΐηίαοΪΒ <ί. visual, respectively. The information in the VPS includes bit rate, temporal_layering structure of the video streams, and the like. It also includes information about inter-layer dependencies (dependencies between different video sequences). Consequently, the VPS can be viewed as a container for multiple video streams and provides an overview of each stream.
<td></td><td colspan="4">In the current HEVC version, the dependency between</td>
<td>cuts</td><td>in a plot</td><td>is</td><td>established both</td><td>by</td>
<td>dependent</td><td>slice flag like</td><td>by</td><td>first slice in_pic_flag</td><td>. Of</td>
Under current specifications, network entities cannot use inter-cut dependencies without applying highly complex analysis. A direct solution would be, if there is a packet loss discovered via a number of missing packets, drop all packets until the f irst_slice_in_pic_f lag which is equal to the value 1 is found. This is because the first cut in a visual representation is always a normal cut.
However, this solution leads to reduced coding efficiency. Therefore, as described above, inter-cut dependency signaling can be used which makes analysis possible.
IMPI
MEXICAN INSTITUTE Dt THE PROPERTY
<img file="MX339463B_D0073.tif" />
efficient. This is accomplished by pointing to the dependent_síi<sup>></sup>ce '¥ tlag the first_slice_in_pic_flag inside the doFté header<sup>1 </sup>immediately after the NAL heading.
Alternatively, or in addition, syntax elements that relate to inter-cut dependencies are encoded unconditionally, that is, independently of the other syntax elements which may be in the cut header or in the PPS.
Modification 2 of Modalities 1 and 2
FIGURE 12 illustrates Modification 2 alternative to Modification 1 set forth above. In particular, the NAL 1210 unit header is the same as the NAL unit header shown in FIGURE 10 (NAL 910 unit header shown in FIGURE 9A). However, the cut header 1220 and the cut header 1020 shown in FIGURE 10 are different because the dependent_slice_flag and first_slice_in__pic_flag cut header syntax elements are reversed with respect to their order. In particular, break header 1220 includes the dependent_slice_flag as a first syntax element and the first_slice_in_pic_flag syntax element as a second syntax element, conditioned in the presence of dependent_slice_flag.
As can be seen from this example, a first cut indication indicating whether the cut is or
ΙΜΡΙχ ^
MEXICAN INSTITUTE / Jiew l * E LA PWOPI «) AL) ΓΊ-Τ”, uinusTiUAL - not a first cut in a visual representation is included éTT the syntax. A first cut in a visual representation is always a cut for which variable length decoding does not depend on other cuts. On the other hand, the dependent cut indicator is included in the bitstream in front of the first cut indication. The first break indication is included in the bitstream only when the dependent break indicator does not signal a dependent break. This arrangement provides the same advantages as conditioning. In other words, the dependency indicator is conditioned on the first cut indication. As can be seen in FIGURE 12, both elements can be understood as the dependency indication and are included at the beginning of the cut heading.
Mode 3
In Mode 3, compared to Modes 1 and 2, the ordering method of syntax elements is changed from order to reduce parsing of unnecessary syntax elements.
In the modalities described above, the dependent_slice_flag is described in the case where the first_slice_in_pic_flag is included as a condition for the presence of the dependent_slice_flag. However, the first_slice_in_pic_flag and the dependent_slice_flag can be
IMPI
MEXICAN INSTITUTE OE LA PRÜPIgOAli
<img file="MX339463B_D0074.tif" />
both included in the bit stream without being c'Ó'í'lSÍ'ÓíoiT one in the presence of the other. For example the TfYÉtOdU 'encoding of the dependent_slice_flag is changed with respect to the order to be independent of the dependent_slice_enabled_flag syntax element according to one of the modifications described above.
FIGURE 13 is a diagram showing an example of a cut header according to the present embodiment. FIGURE 13 illustrates the case of including the conditioning of the dependency indication in the indicator of enabling of dependent cuts.
Specifically, in the cut header according to the present embodiment, the dependent_slice_flag is arranged before the slice_address compared to the existing cut header shown in FIGURE 6. Additionally, in the cut header according to the present embodiment , compared to the examples shown in FIGURES 10 through 12, the dependent_slice_f lag is arranged after the pic_parameter_set_id.
In the present embodiment, since the dependent_slice_flag is arranged before the slice_address, at least the SPS does not need to be parsed for parsing the dependent_slice_flag. As previously described, slice_address is a syntax element that indicates the start of a cut. Additionally, the slice_address only
<img file="MX339463B_D0075.tif" />
INSTITUTO MEXICaNí Dt IA RRORIEDAD INDUSTRIAL can be analyzed with the help of the syntax elements indicated within the SPS (pic_parameter_set_id).
Alternatively or additionally, the dependent_slice_enabled_flag is either moved upward within the PPS or is moved to the SPS and / or VPS. If the enabled flag is on the VPS and / or SPS, it may not be necessary to analyze and keep track of the PPS and SPS.
Modification of Mode 3, effect and the like (1) The example in FIGURE 13 may lead to the provision of an apparatus for analyzing a bit stream of a video sequence encoded at least partially with a variable length code and including data units carrying encoded sections of video images. In this case, the apparatus is configured to include an analyzer which extracts the following syntax elements from the bit stream:
a dependency indication which is a syntax element indicating a cut in the cut header whether or not the variable length decoding of the cut depends on other cuts;
a dependent break enable flag included within a set of parameters for a plurality of breaks and indicating whether or not the dependent breaks can be included within the bit stream; and a cutting direction indicating the position
IMPI
<img file="MX339463B_D0076.tif" />
within the bit stream at which the cut begins.
(2) On the other hand, in the present embodiment, the dependency indication is signaled within the break header before the break address and after a syntax element identifying the parameter set.
With this mode, it is possible without causing parsing errors to configure the dependency indication to be included in the bit stream only when the dependent break enable flag indicates that the dependent breaks can be included in the bit stream.
(3) In the present embodiment, the dependent break enable indicator is located within the bit stream in a common parameter set (PPS) for a plurality of breaks that form the same display frame and located at the beginning of the parameter set. However, it is not limited to this.
Alternatively (or in addition), the dependent break enable indicator is located within the bit stream in a common parameter set (SPS) for a plurality of breaks that form the same sequence of visuals. Still alternatively (or in addition), the dependent break enable indicator is located within the bit stream in a common set of parameters (VPS) for a plurality of breaks that form a plurality of visual frame sequences.
IMPI
MEXICAN INSTITUTE OF LA FRONEDAL · INDUSTRIAL
<img file="MX339463B_D0077.tif" />
(4) On the other hand, in the prestíTlLé Ikló'daiidad) he VPS_id and the SPS_id can be explicitly pointed out in the SEI message. When the dependent_slice_enabled_flag is reported in the SPS, the dependent_slice_flag must still follow the pic_parameter_set_id.
Otherwise the parsing dependency is introduced because the SPS_id is signaled in the PPS. With the signaling of the current SPS or VPS identification which carries the dependent_slice_enabled_flag, the dependency indication can also be included before the pic_parameter_set_id since then parsing of parameter sets is not necessary. On the other hand, this SEI message, which carries the VPS-id or SPS-id, is not necessary for the decoding operation since these identifications are also determined when parsing the PPS. In this way, the SEI message can be discarded without affecting the decoding operation after being used by the network elements.
Mode 4
In Mode 4, the inter-cut dependency information is duplicated (complementary to the information noted in the cut header and / or in a set of parameters) in another NAL unit such as a SEI message.
IMPI
MEXICAN INSTITUTE OF THE INDUSTRIAL NUmEIMD
<img file="MX339463B_D0078.tif" />
For example, a SEI message can be defined which carries inter-outage dependency information at each access unit or before each dependent outage. The term access unit refers to a data unit which is made up of a set of NAL units. An access unit includes encoded visual representation cuts, ie VCL NALUs. In particular, access units can define points for random access and can include NALUs of an individual visual representation. However, the access unit is not necessarily a random access point.
In current HEVC specifications, the access unit is defined as a set of NAL units that are consecutive in the order of decoding and contain exactly one encoded display. In addition to the encoded slice NAL units of the encoded display, the access unit may also contain other NAL units that do not contain scraps of the encoded display. Decoding an access unit always results in a decoded display. However in a future extension of the HEVC (such as Multiple View Coding (MVC) or Scalable Video Coding (SVC)), the definition of the access unit may be relaxed or modified. According to
<img file="MX339463B_D0079.tif" />
ΙΜΡΙ
MEXICAN INSTITUTE
OF THE MORETH
INDUSTRIAL current specifications, the access unit consists of an access unit delimiter, SEI messages and
VCL NALUs.
In accordance with the present embodiment, the dependency indication is located within the bitstream outside the header of a break to which the dependency indication refers. On the other hand, it can be beneficial when the dependency indication is located within the bitstream in a supplemental enhancement information message included in the bitstream before the dependent cut or once per access unit.
Mode 5
Pursuant to Mode 5, inter-court dependency information is indicated in the NAL heading as an indicator or implicitly as a type of reporting unit.
NAL with which it is associated.
As a general rule, parsing of syntax elements in the NAL header does not depend on any other syntax element. Each NAL unit header can be parsed independently. The NAL header is the usual place to point out dependency information. Therefore, according to the present modality, the inter-court dependency is also indicated within it.
In other words, the analysis apparatus can be adopted in a router or a decoder. The apparatus of
IMPI
ÍNSTÍT1 / TO MEXICANO OE THE INDUSTRIAL PROPERTY
<img file="MX339463B_D0080.tif" />
Analysis further includes a network adaptation layer unit for adding to a cut of encoded video data and to the cut header a network adaptation layer and a NAL header. Advantageously, the dependency indication is located within the bitstream in the NAL header and is encoded independently of the other syntax elements.
The dependency flag can be placed inside the NAL header since the NAL header in the current HEVC specifications includes some reserved bits which can be used for it. A single bit would be sufficient to signal the dependency indication.
Alternatively, the dependency indication is signaled by a NAL unit type and a predefined NAL unit type is reserved to carry dependency information.
Mode 6
It is observed that the five previous modalities can be arbitrarily combined with the purpose of making possible an efficient analysis of the dependency information in the network elements. Even when its use is redundant, the modalities are combinable. Therefore, duplication of dependency indication can be applied even when dependency indication
INRt'S
IMPI
MEXICAN INSTITUTE OF THE PROFlSüAI;
t'STKIAL
<img file="MX339463B_D0081.tif" />
it is also noted at the beginning of the cut heading.
FIGURE 14 shows an example of the NAL 1410 unit header in which the NAL unit header
910 shown in FIGURE 9A is modified. The NAL 1410 unit header includes the dependent_slice_flag.
On the other hand, for the purpose of moving the dependent_slice_flag within the NAL header and keeping the size of the NAL header fixed due to backward compatibility, the bit needed for the dependent_slice_flag is taken from the nuh_reserved_zero_6bits syntax element of the unit header from NAL. Consequently, the nuh_reserved_zero_6bits syntax element now only has 5 bits. The nuh_reserved_zero_6bits syntax element includes bits reserved for future use so that the reduction does not cause any problems and does not require any further modifications.
In general, a current VCL NAL unit depends on the previous VCL NAL unit which has the same temporary_layer_id. When the dependent_slice_flag is pointed to in the NAL header, one bit will be consumed for both VCL and non-VCL NAL units since each data unit such as a visual cut or parameter set has the same NAL. Therefore, although it seems that the dependent_slice_flag would also be pointed to for parameter sets or for
IMPI iNrrnyro mexicana o * LA RAOmiuD
INDUSTRIAL
<img file="MX339463B_D0082.tif" />
SEI messages, this is unnecessary. On the other hand, the dependent_slice_flag always needs to be pointed out even if the dependent breaks are disabled in the sequence parameter set. This leads to unnecessary attachment information.
In all of the above modes, the dependency indication may be a one-bit indicator.
Mode 7
In accordance with Mode 7, the dependency indication is signaled by a NAL unit type and a predefined NAL unit type is reserved to carry dependency information.
Therefore, a new (separate) VCL NAL type is defined with similar semiotics as the existing VCL NAL units. For example, when the NAL_unit_type equals 15 (or another predefined type or NALU which is not reserved for another particular type of NALU), then the current VCL NAL unit depends on the previous VCL NAL unit that has the same temporal_layer_id . Dependency refers to the dependency of the current cut in the cut heading of a preceding cut, as described above, ie the dependency in the analysis.
It may be advantageous in those cases to include the bit in the NAL header for the NAL unit types
ΙΜΡΙ ^>
MEXICANC DELA MONEDAD INDUSTRIAL INSTITUTE
<img file="MX339463B_D0083.tif" />
additional. This can be used to indicate whether or not the current break is a dependent break.
When dependency information is flagged in the cut header in addition to the NAL header, flagging in the NAL header becomes optional. Specifically, when the NAL_unit_type field in the NAL header is configured to signal that the current break is a dependent break, then it is not possible to point to any other type information. For example, in some cases it might be more beneficial to carry the information that a current cut is a first visual representation in the sequence (NAL__unit_type equal to 10 or 11). When the cross-cut dependency information in the NAL header is optional (since it is duplicated in the cut header), it can be selected to point to the most valuable information.
On the other hand, it may be advantageous to add two or more VCL NAL unit types, such as RAP-dependent display (required for analysis) or non-RAP-dependent display. RAP indicates the visual representation of random access. The random access visual is a visual encoded independently (in terms of prediction) from other visuals so that this visual can be used as a
MEXICAN INSTITUTE
OF THE CURRENCY »ζ *» β ^ 3
INDUSTRIAL starting point for encoding and decoding. With this, it is suitable in this way as a random access point.
In the dependent break header, the RapPicFlag syntax element is used in the parsing process. Specifically, the RapPicFlag syntax element is an indication that indicates whether or not the current visual representation is a random access visual representation.
The value of the RAPPícFlag depends on the NAL unit type as the following Expression 2.
Mathematical Equation 2
RapPicFlag = (nal __ unít _ type> 7 & & nal _ unit _ type <12) (Expression 2)
In other words, in the example shown in FIGURE 15, the random access visuals are carried by NALUs with a NALU type between 7 and 12. In order to enable correct analysis and to provide a cut dependency possibility for random access visuals, therefore, in the present invention, two different types of NAL unit are defined in order to ensure the correct analysis of the cut heading.
As a general rule, even when defining a new VCL NAL unit type, parsing the header
IMPI
MEXICAN INSTITUTE Γ> € THE INDIAN PROPERTY tSTRlAL
<img file="MX339463B_D0084.tif" />
Cutting should still be possible without any problem. Either multiple NAL types are defined as above or the dependent break header is changed such that there is no parsing problem.
When a new VCL NAL unit type is defined to indicate dependent break, the break header syntax structure can be changed as follows.
In the example above, the NAL unit type DS_NUT is used to indicate that the current nal VCL unit is a cutoff dependent. Compared to the state-of-the-art cut header syntax structure described in the Non-Patent Bibliography 3, the following two changes are introduced in the present embodiment.
(1) the no_output_of_prior_pics_flag is not noted in the dependent cut header. In other words, the presence of the no_output_of_prior_jpics_f lag is based on the condition that the current cut is not a dependent cut. (no_output_of_prior_pics_flag may be present in the cut header when the current cut is not a dependent cut).
(2) the first_slice_in_pic_flag is conditionally pointed at the value of the nal_unit_type. When the value of the nal_unit_type indicates that the current break is a dependent break, the syntax element
MEXICAN INSTITUTE
FROM PROPERTY first_slice_in_pic_flag is not explicitly pointedSfflin ^ e and it follows that it is 0. This saves speed of<sup>--</sup>bltü to the same quality.
According to the example, the no_output_of_prior_pics_flag is not signaled when the current cut is a dependent cut. Therefore, the RapPicFlag value is not required to be evaluated when the current cut is a dependent cut. Therefore, the cut header of a dependent cut can be analyzed without problem. More specifically, the cutoff header of the dependent cutoff can be parsed without reference to the NAL unit header of a preceding NAL unit header. A problem occurs when the preceding NAL unit header is not present at the time of decoding.
Second, the first_slice_in_pic_flag is signaled based on the value of the NAL_unit_type. This change is the same as that of the example described in FIGURE 12. In FIGURE 12, the first_slice_in_pic_flag is signaled in the cut header only when the current cut is not a dependent cut (which is indicated by the dependent_slice_flag). Similarly in the previous example the first_slice_in_pic_flag is signaled only when the nal_unit_type is not equal to DS_NUT, which means that the current cut is not a dependent cut.
Both changes are not required to be presented
ΙΜΡΙ
INSTnVT · MEXICAN OE LA INDUSTRIAL RRORIEDAD
<img file="MX339463B_D0085.tif" />
ant erio rment e be done together. It is also possible to carry out
<td>just one of</td><td>the</td><td>changes</td><td>in</td><td>the</td><td>cutting header.</td><td>The</td>
<td>benefit of</td><td>every</td><td>change</td><td>this</td><td colspan="2">associated with the cost of</td><td>the</td>
<td>check</td><td>if a</td><td>cut</td><td>that</td><td>not</td><td>a dependent cut.</td><td>Without</td>
<td colspan="3">However, when the two</td><td colspan="2">changes</td><td>are done together,</td><td>the</td>
Benefits of both changes can come from both the same costs as the benefit from each of the individual changes in the case where the two syntax elements first_slice_in_pic_flag and no_output_of_prior_pics_flag are coded consecutively. Thus, the application of both changes in combination with a consecutive coding of the two mentioned syntax elements provides an advantage over the direct application of each of the changes individually.
In any explanation in the modalities, it is also possible to remove the dependent_slice_enabled_flag from the bitstream when the dependent cut indication is not conditionally encoded in it. In other words, when for example a new type of NAL unit is used to indicate that the current cut is a dependent cut, then the dependent_slice_enabled_flag can be removed from the bit stream.
FIGURE 15 shows a NAL unit header
1510 which is the same as the NAL 910 unit header
<img file="MX339463B_D0086.tif" />
shown in FIGURE 9A and a cut header 1520 which is changed from cut header 92 0 shown in
FIGURE 9A. Cut header 1520 includes the termination of the dependent_slice_flag value according to the type of the NALU. In particular, the NAL_unit_type syntax element with values 15 and 16 defines dependent breaks. When the NAL_unit_type equals 15, the cut type is a cut dependent on a random access visual. If, on the other hand, the NAL_unit_type equals 16, the cut is a cut dependent on a non-random access visual representation. Therefore, a relation of the following Expression 3 is established.
Mathematical Equation 3
RapPicFlag = (nal _ unil _ type> 1 & & nal _ unit _ type <ΥΣ \\ ηαΙ _unil _ type = 15) (Expression 3)
It is observed that the values 15 and 16 were selected
<td>just like a</td><td colspan="2">example. How</td><td>is</td><td>clear for</td><td colspan="2">those people</td>
<td>experts in</td><td>the</td><td>field, it</td><td colspan="2">can adopt</td><td>any</td><td>number</td>
<td>predefined</td><td>the</td><td>which not</td><td>I know</td><td>uses</td><td>of other</td><td>way.</td>
<td colspan="2">Specifically,</td><td>a first</td><td>type</td><td>from NALU it</td><td colspan="2">must define for</td>
<td>identify</td><td>a</td><td>content</td><td>of</td><td colspan="2">dependent cut</td><td>of a</td>
Random access visual and a second type of NALU must be defined to identify a cut content dependent on a non-random access visual.
<img file="MX339463B_D0087.tif" />
ΙΜΡΙ ™ 'AZXWdac, INDllvrR |<sub>To the</sub>
<img file="MX339463B_D0088.tif" />
On the other hand, a restriction may apply that dependent cuts not for RAPs or only used for non-RAPs only a new type of NALU is required.
Mode 8 only used
In these cases,
FIGURE 16 is a diagram showing an alternative solution. A NAL 1610 unit header is the same as NAL 910 unit header. Cut header 1620 assumes the definition of NAL_unit_type with values 15 and 16 indicating dependent breaks as described above.
However, the NAL unit type is not used in the analysis of the dependent cutoff indicator. This makes it possible to use the NAL_unit_type which is optional for the encoder. Accordingly, the advantage of the present embodiment is only achieved when it is determined that the encoder adopts the new NALU types.
Then the router only needs to examine the NALU type. However, when the encoder does not use the new NALU types, the router would handle dependent outages as in the prior art.
In summary, the dependency indication can be signaled by a NAL unit type. A predefined NAL unit type can be reserved to carry coded breaks of the cut header on which the
<img file="MX339463B_D0089.tif" />
cut heading of a preceding cut. Advantageously, a separate NAL unit type indicating dependency is provided for random access visuals and non-random access visuals.
In summary, the modalities described above refer to the syntax of a bit stream carrying encoded video sequences. In particular, the modalities described above refer to the syntax related to dependent and entropic cuts, of which the cut header depends on the cut header of a preceding cut. In order to allow a media conscious network element to consider this kind of dependency without essentially increasing its complexity and delay due to parsing, the dependency indication is signaled at the beginning of packets or in other words in the vicinity of headers or parameters which must be analyzed. This is accomplished, for example, by including the dependency indication at the beginning of the break header (FIGURES 10-12), possibly after the parameter set identifier and before the break address, or by including the dependency indication before the cutting direction (FIGURES 10 and 11) or by providing the dependency indication in a NALU header (FIGURE 14), in a separate message or by a special NALU type for NALUs that carry dependent outages (FIGURES 15 and 16).
ΙΜΡΙ
MEXICAN INSTmjTti OF THE FROWgOAP INDI ISTRIAL
<img file="MX339463B_D0090.tif" />
Modifications of Modalities 1 to 8, effect and the like.
Various changes are possible without being limited by Modalities 1 to 8 and are obviously included within the scope of the present invention.
Each of the structural elements in each of the modalities described above can be configured in the form of a unique hardware product (processing circuit) or can be performed by running a software program that is suitable for the structural element. Each of the structural elements can be
<td colspan="2">make reality</td><td>by</td><td>middle of a</td><td>Unit</td><td>of execution</td><td>of</td>
<td>programs,</td><td>such</td><td>how</td><td colspan="3">a CPU and a processor, which reads</td><td>and</td>
<td>run the</td><td colspan="2">Program</td><td>of software</td><td>Recorded</td><td>in a medium</td><td>of</td>
<td>recording</td><td>such</td><td colspan="2">like a disc</td><td>Lasted</td><td colspan="2">or a memory</td>
semiconductor.
Although in Modes 1 through 8, the description assumes a wavefront, it is not limited to it.
However, in the case of the wavefront, not all subcurrents can start at the same time. As described above, with respect to each of the subcurrents except the initial subcurrent, the start of processing (encoding or decoding) is delayed by two LCUs of the preceding subcurrent. For the
100
IMPI
<img file="MX339463B_D0091.tif" />
<img file="MX339463B_D0092.tif" />
<img file="MX339463B_D0093.tif" />
additional processing. In the present embodiment, by locating the dependency indication (dependent_slice_flag) after the syntax which identifies the PPS and before the cutoff address, the number of syntax elements that are parsed can be reduced and thus processing is reduced. reduces.
On the other hand, in Modes 1 to 8 described above, by arranging the dependency indication upwards within the cut header (notably at the beginning) it is possible, for example, to verify whether each of the cuts is a dependent cut in or not. an early stage of visual representation processing.
In other words, at the start of processing in a visual representation (encoding or decoding), when a step consisting in verifying whether each of the cuts is a dependent cut or not, it is possible to extract a starting point of parallel processing at the start of processing in the visual representation. In other words, when the visual representation includes a plurality of normal cuts, it is possible to extract a start point of processing parallel to the time of processing in a visual representation or at an early stage of processing.
In this document, conventionally, when the
101
IMPI
MEXICAN INSTITUTE BE LA FROFIEDAD INDUSTRIAL
<img file="MX339463B_D0094.tif" />
Dependency indication is available after the cut direction, it is not possible to verify whether the cut is a dependent cut or a normal cut until the analysis of the cut direction is completed. In this case, the start of processing in the normal cut in the middle of the visual display is significantly delayed from the start of processing in the normal cut in the beginning of the display.
On the contrary, in Modes 1 to 8 described above, since it is possible to verify whether each of the cuts is a dependent cut at an early stage of processing in a visual representation, it is possible to accelerate the start of processing in a normal cut in half of the visual. In other words, it is possible to start processing in the normal cut in the middle of a visual representation at the same time as the normal cut in the beginning of the visual representation.
Mode 9
The processing described in each of the modalities can be simply implemented on a separate computer system, by recording, on a recording medium, a program to implement the configurations of the encoding method of moving visuals (image encoding method). ) and the method
102
IMPI
INSTITUTO MEXICANO PE LA PRORIEIJAD INDUSTRIAL for the decoding of visual representations in motion
<img file="MX339463B_D0095.tif" />
(method of decoding images) described in each of the modalities. The recording media can be any recording medium as long as the program can be recorded, such as a magnetic disk, an optical disk, a magnetic optical disk, an IC card and a semiconductor memory.
Hereinafter, the applications for the encoding method for moving visual representations (image encoding method) and the decoding method for moving visual representations (image decoding method) described in each of the modalities will be described and systems using the same. The system has a quality which is to have an image encoding and decoding apparatus including an image encoding apparatus using the image encoding method and an image decoding apparatus using the image decoding method. Other settings in the system can be changed appropriately depending on the situations.
FIGURE 17 illustrates a complete configuration of an exOO content delivery system for implementing content delivery services. The area to provide communication services is divided into cells
103
<img file="MX339463B_D0096.tif" />
ΙΜΡΪ
IWTmiTD MJftlCANl
DIA IA nTOFIEDAD
INDUSTRIAL of the desired size and the base stations ex! 06, exlO7, exl08, exl09 and exllO which are fixed wireless stations are placed in each of the cells.
The exlOO content delivery system connects to devices, such as an exlll computer, an exll2 personal digital assistant (PDA), an exll3 camera, an exll4 cell phone, and an exll5 video game console, via Internet exlOl, an internet service provider exl02, a telephone network exl04, as well as base stations exl06 to exllO, respectively.
However, the configuration of the exlOO content delivery system is not limited to the configuration shown in FIGURE 17 and a combination in which any of the elements is connected is acceptable. Furthermore, each device can be connected directly to the telephone network exl04, preferably via the base stations exl06 to exllO which are the fixed wireless stations. Additionally, the devices can be interconnected with each other via short-distance wireless communication and others.
The exll3 camera, just like a digital video camera, is capable of capturing video. An exll6 camera, such as a digital camera, is capable of capturing both still images and video. Additionally, the exll4 cell phone
104
IMPI
INSTITUTO MEXICANO DE LA MONEDAR industrial
<img file="MX339463B_D0097.tif" />
it can be one that complies with any of the standards such as the Global System for Mobile Communications (GSM) (registered trademark), Multiple Access by Code Division (CDMA), Multiple Access by Broadband Code Division (W-CDMA), Long-Term Evolution (LTE), and High Speed Packet Access (HSPA). Alternatively, the exll4 cell phone can be a Personal Telephone System (PHS).
In the exlOO content delivery system, an exl03 streaming server connects to the exll3 camera and others via the exl04 telephone network and the exl09 base station, making it possible to distribute images of a live show and others. In this distribution, content (for example, video of a live music show) captured by the user using the exll3 camera is encoded as described above in each mode (i.e. the camera functions as the encoding apparatus of images according to an aspect of the present invention), and the encoded content is transmitted to the uninterrupted transmission server exl03. On the other hand, the exl03 uninterrupted transmission server performs the distribution
105
IMPI
MEXICAN INSTITUTE BE LA INDUSTRIAL RRORIEDAD
<img file="MX339463B_D0098.tif" />
uninterrupted content data transmitted to customers upon request. Clients include the exlll computer, the exll2 PDA, the exll3 camera, the exll4 cell phone, and the exll5 video game console that are capable of decoding the encrypted data mentioned above. Each of the devices that have received the distributed data decodes and reproduces the encoded data (ie, functions as the image decoding apparatus in accordance with one aspect of the present invention).
The captured data can be encrypted by the exll3 camera or the exl03 streaming server that transmits the data, or the encryption processes can be shared between the exll3 camera and the exl03 streaming server. Similarly, the distributed data can be decoded by the clients or the exl03 streaming server, or the decoding processes can be shared between the clients and the exl03 streaming server. Additionally, the still image and video data captured not only by the exll3 camera but also the exll6 camera can be transmitted to the exl03 streaming server via the exlll computer. Coding processes can be done by exll6 camera, exlll computer or streaming server
106
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0099.tif" />
uninterrupted exl03, or they may be the same.
Additionally, the encoding and decoding processes can be performed by an LSI ex500 generally included in each of the exlll computer and devices. The LSI ex500 can be configured from a single chip or a plurality of chips. The software for encoding and decoding video can be embedded in some type of recording medium (such as a CD-ROM, floppy disk, and hard drive) that is readable by exlll and other computers, and the encoding and decoding processes can be perform using the software. Additionally, when the exll4 cell phone is equipped with a camera, the video data obtained by the camera can be transmitted. Video data is data encoded by the LSI ex500 included in the exll4 cell phone.
Additionally, the exl03 streaming server can be comprised of servers and computers and can decentralize data and process decentralized data, record or distribute data.
As described above, clients can receive and reproduce the encrypted data in the exlOO content delivery system. In other words, clients can receive and decode information transmitted by the user and can reproduce the data
107
IMPI
MEXICAN INSTITUTE OE INDUSTRIAL PROPERTY decoded in real time in the provision system of
<img file="MX339463B_D0100.tif" />
exlOO content, so that the user who does not have any particular rights and equipment can implement a personal broadcast.
In addition to the example of the exlOO content delivery system, at least one of the moving visuals encoding apparatus (image encoding apparatus) and the moving visuals decoding apparatus (image decoding apparatus) described in each of the modalities can be implemented in an ex200 digital broadcast system illustrated in FIGURE 18. More specifically, an ex201 broadcast station communicates or transmits, via radio waves to an ex202 broadcast satellite, multiplexed data obtained by multiplexing audio and other data into video data. The video data is data encoded by means of the moving visuals encoding method described in each of the modalities (i.e., the data encoded by the image encoding apparatus according to an aspect of the present invention). Upon receipt of the multiplexed data, the broadcast satellite ex202 transmits radio waves for broadcasting. Then, an ex204 home-use antenna with a satellite broadcast reception function receives the radio waves. Then a device such as
108 »
IMPI
MEXICAN INSTITUTE OF INDUSTRY PROPERTY!
<img file="MX339463B_D0101.tif" />
an ex300 television (receiver) and an external tuner (STB) ex217 decodes the received multiplexed data and reproduces the decoded data (i.e., functions as the image decoding apparatus according to one aspect of the present invention).
Additionally, an ex218 (i) reader / writer reads and decodes the multiplexed data that is recorded on an ex215 recording medium, such as a DVD and a BD, or (i) encodes video signals on the ex215 recording medium, and on In some cases, it writes data obtained by multiplexing an audio signal into the encoded data. The ex218 reader / writer may include the moving visuals decoding apparatus or the moving visuals encoding apparatus as shown in each of the embodiments. In this case, the reproduced video signals are displayed on the ex219 monitor and can be reproduced by another device or system using the ex215 recording medium on which the multiplexed data is recorded. It is also possible to implement the decoding apparatus for moving visuals on the external tuner ex217 connected to the cable ex203 for a cable television or to the antenna ex2 04 for satellite and / or terrestrial broadcasting, in order to display the video signals on the ex219 monitor on the ex300 television. The decoding apparatus of
109
IMPIAS
MEXICAN INSTITUTE
OF THE PROPERTY *'
INDUSTRIAL moving visuals may not be implemented on the external tuner but on the ex300 television.
The figure. 19 illustrates the ex300 television (receiver) utilizing the moving visuals encoding method and the moving visuals decoding method described in each of the embodiments. The ex300 television includes: an ex301 tuning device that obtains or provides multiplexed data obtained by multiplexing audio data into video data, through the ex204 antenna or the ex203 cable, etc. that receives a broadcast; an ex302 modulation / demodulation unit that demodulates the received multiplexed data or modulates data into multiplexed data to be supplied to the outside; and an ex303 multiplexing / demultiplexing unit that demultiplexes the modulated multiplexed data into video and audio data or multiplexes video and audio data encoded by an ex306 signal processing unit into data.
<td>Lh</td><td>ex300 television</td><td>It includes</td><td>also:</td><td>a</td><td>Unit</td><td>of</td>
<td>processing</td><td colspan="2">of ex306 signals that</td><td>It includes</td><td>a</td><td>Unit</td><td>of</td>
<td>processing</td><td>signals</td><td colspan="2">ex304 audio and</td><td>a</td><td>Unit</td><td>of</td>
ex305 video signal processing that decode audio data and video data and encode audio data and data <sup>110</sup> Mexican IMPI 'NSTrruT »and LA FROFIEDAD INDUSTRIAL of video, respectively (which function as the image encoding apparatus and the image decoding apparatus in accordance with aspects of the present invention); and an ex309 output unit that includes an ex307 speaker that provides the audio signal
<img file="MX339463B_D0102.tif" />
<td>decoded and a</td><td>ex308 display unit</td><td>than</td><td>exhibits</td><td>the</td>
<td>video signal</td><td>decoded, such as</td><td>a</td><td colspan="2">screen.</td>
<td>Additionally, the</td><td>ex300 television includes</td><td>a</td><td>Unit</td><td>of</td>
<td colspan="2">ex317 interconnect that includes a unit</td><td>of</td><td>entry</td><td>of</td>
operation ex312 receiving input from a user operation. Additionally, the ex300 television includes an ex310 control unit that collectively controls each constituent element of the ex300 television and an ex311 power supply circuit unit that supplies power to each of the elements. Unlike the ex312 operation input unit, the ex317 interconnect unit can include: an ex313 jumper that connects to an external device, such as the ex218 reader / writer; an ex314 slot unit to enable bonding of the ex216 recording medium, such as an SD card; an ex315 controller that is connected to an external recording medium, such as a hard drive; and an ex316 modem that is connected to a telephone network. In this document, the ex216 recording medium can electrically record information using a non-lll semiconductor memory element
MEXICAN INSTITUTE 1
MEXICAN INSTITUTE OF PHOftEBA »INDUSTRIAL
<img file="MX339463B_D0103.tif" />
volatile / volatile for storage. The constituent elements of the ex300 television are connected to each other via a synchronous busbar.
First, the configuration in which the ex300 television decodes the multiplexed data obtained from the outside through the ex204 antenna and others and reproduces the decoded data will be described. In television ex300, with the operation of a user through a remote controller ex220 and others, the multiplexing / demultiplexing unit ex303 demultiplexes the multiplexed data that is demodulated by the modulation / demodulation unit ex302, under the control of the ex310 control that includes a CPU. Additionally, the audio signal processing unit ex304 decodes the demultiplexed audio data and the video signal processing unit ex305 decodes the demultiplexed video data, using the decoding method described in each of the modalities, on television. ex300. The ex309 output unit provides the decoded video signal and audio signal to the outdoors, respectively. When the output unit ex309 provides the video signal and the audio signal, the signals can be temporarily stored in the ex3l8 and ex319 buffers and others so that the signals are reproduced in synchronization with each other. Additionally, television ex300
112
IMPI
MEXICAN INSTITUTE Ofc EA INDUSTRIAL PROPERTY
<img file="MX339463B_D0104.tif" />
it can read multiplexed data not through a broadcast and others but from the ex215 and ex216 recording media, such as a magnetic disk, an optical disk and an SD card. Next, a configuration will be described in which the ex300 television encodes an audio signal and a video signal and transmits the data to the outside or writes the data to a recording medium. In television ex3 00, with the operation of a user through the remote controller ex220 and others, the audio signal processing unit ex304 encodes an audio signal and the video signal processing unit ex305 encodes a video signal , under control of the ex310 control unit using the coding method described in each of the modalities. The multiplexing / demultiplexing unit ex303 multiplexes the encoded video signal and audio signal and supplies the resulting signal to the outside. When the multiplexing / demultiplexing unit ex303 multiplexes the video signal and the audio signal, the signals can be temporarily stored in the ex320 and ex321 buffers and others so that the signals are reproduced in synchronization with each other. In this document, the ex318, ex319, ex320 and ex321 buffers can be plural as illustrated or at least one buffer can be shared on the ex300 television. Additionally, data can be stored in a buffer memory so that
113
<img file="MX339463B_D0105.tif" />
IMPI
<img file="MX339463B_D0106.tif" />
ut LA rROrlEDAt) v \ fNDVSTKlAL X system overflow and overflow between modulation / demodulation unit ex3 02 and multiplexing / demultiplexing unit ex303, for example, can be avoided.
Additionally, the ex300 television may include a configuration to receive an AV input from a microphone or a different camera from the configuration to obtain audio and video data from a broadcast or recording medium and may encode the data obtained. Although the ex300 television may encode, multiplex, and provide data to the outside in the description, it may only be able to receive, decode, and provide data to the outside but not encode, multiplex, and provide data to the outside.
Additionally, when the ex218 reader / writer reads or writes multiplexed data from or to a recording medium, one of the ex300 television and the ex218 reader / recorder can decode or encode the multiplexed data and the ex300 television and the ex218 reader / recorder can share decoding or encoding.
As an example, FIGURE 20 illustrates a configuration of an ex400 information recording / reproducing unit when data is read from or written to or from an optical disc. The ex400 information recording / reproducing unit includes the constituent elements ex401, ex402, ex403, ex404, ex405, ex406 and ex407 which are described
114
IMPI
MEXICAN INSTITUTE 'OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0107.tif" />
later in this document. The optical head ex401 radiates a laser focus onto a recording surface of the recording medium ex215 which is an optical disk for writing information and detects the reflected light from the recording surface of the recording medium ex2l5 to read the information. The ex402 modulation recording unit electrically drives a semiconductor laser included in the ex401 optical head and modulates the laser light according to recorded data. The ex403 reproduction demodulation unit amplifies a reproduction signal obtained by electrically detecting the reflected light from the recording surface using a photodetector included in the ex401 optical head and demodulates the reproduction signal by separating a recorded signal component in the medium of ex215 recording to reproduce the necessary information. The buffer ex404 temporarily holds the information to be recorded on the recording medium ex215 and the information reproduced from the recording medium ex215. The ex405 disk motor spins the ex215 recording medium. The ex406 servo drive moves the optical head ex401 to a predetermined information track while controlling the rotary drive of the ex405 disk motor in order to follow the laser focus. The ex407 system control unit together controls the ex400 information recording / reproducing unit. The reading and writing processes can
115
<img file="MX339463B_D0108.tif" />
control system ex407 stored in memory adding new information to be implemented by the unit using various intermediate information ex404 and generating and as necessary and by means of the modulation recording unit ex402, the reproduction demodulation unit ex403 and the unit ex406 servo controls that record and reproduce information through the ex401 optical head while being operated in a coordinated manner. The ex407 system control unit includes, for example, a microprocessor and executes processing by causing a computer to run a program to read and write.
Although the ex401 optical head radiates a laser focus in the description, it can perform high-density recording using near field light.
FIGURE 21 illustrates the ex215 recording medium which is the optical disc. On the recording surface of the ex215 recording medium, the guide grooves are spirally formed and an ex230 information track, in advance, records direction information indicating an absolute position on the disc according to a change in the shape of the grooves guide. The address information includes information for determining ex231 recording block positions that are a unit for recording data. Playing the information track ex230 and reading the address information on a recording device and
116
MEXICAN INSTITUTE OF THE RflOPIEÜAD
INDUSTRIAL
<img file="MX339463B_D0109.tif" />
Playing back data can lead to the determination of the positions of the recording blocks. Additionally, the recording medium ex215 includes a data recording area ex233, an inner circumference area ex232 and an outer circumference area ex234. The ex233 data recording area is an area for use in recording user data. The inside circumference area ex232 and the outside circumference area ex234 which are the inside
<td>and the exterior</td><td>of the</td><td>area</td><td>of</td><td>recording</td><td>ex233 data,</td>
<td>respectively,</td><td>are</td><td>for</td><td>use</td><td>specific</td><td>except for the</td>
<td>Recording of</td><td>the</td><td>data</td><td>of the</td><td>user.</td><td>The unit of</td>
information playback / recording 400 reads and writes encoded audio data, encoded video data, or multiplexed data obtained by multiplexing the encoded audio and video data from and onto the ex233 data recording area of the ex215 recording medium.
Although an optical disc having one layer, such as a DVD and BD is provided as an example in the description, the optical disc is not limited to that type and may be an optical disc having a multi-layer structure and capable of be engraved on a different part of the surface. Additionally, the optical disc may have a structure for multi-dimensional recording / reproduction, such as recording information using colored light with different wavelengths.
117
ΙΜΡΙ
INSTITUTO MEXICANO DC LA PROTISDAP INDUSTRIAL
<img file="MX339463B_D0110.tif" />
on the same portion of the optical disc and to record information that has different layers from various angles.
Additionally, an ex210 car having an ex205 antenna can receive data from the ex202 satellite and others, and can play video on a display device such as an ex211 car navigation system established in the ex210 car, in the ex200 digital broadcast system. . In this document, a configuration of the ex211 car navigation system will be a configuration, for example, that includes a GPS receiver unit of the configuration illustrated in FIGURE 19. The same will be true for the configuration of the exlll computer, the exll4 cell phone and others.
<td></td><td>The</td><td>FIGURE 22A illustrates</td><td colspan="2">the phone</td><td>exll4 cell</td><td>than</td>
<td>uses</td><td>the</td><td colspan="2">encoding method</td><td>of</td><td colspan="2">representations</td>
<td>visual</td><td>in</td><td>movement and the</td><td>method</td><td>of</td><td>decoding</td><td>of</td>
visual representations in movement described in the modalities. The exll4 cell phone includes: an ex350 antenna for transmitting and receiving radio waves through the exllO base station; an ex365 camera unit capable of capturing moving and still images; and an ex358 display unit such as a liquid crystal display for displaying data such as decoded video that is captured by the ex365 camera unit or received by the ex350 antenna. The exll4 cell phone also includes: a
118
<img file="MX339463B_D0111.tif" />
main body unit including an ex366 operation key unit; an ex357 audio output unit such as a speaker for the audio output; an ex356 audio input unit such as a microphone for the audio input; an ex367 memory unit for storing captured video or still images, recorded audio, encoded or decoded data of the received video, still images, emails or others; and an ex364 slot unit which is an interconnect unit for a recording medium that stores data in the same manner as the ex367 memory unit.
Next, an example of an exll4 cell phone configuration will be described with reference to the
FIGURE 22B. On the exll4 cell phone, an ex360 main control unit designed to jointly control each main body unit including the ex358 display unit as well as the ex366 operation key unit connects to each other, via a bus bar synchronous ex370, to an ex361 power supply circuit unit, an ex362 operation input control unit, an ex355 video signal processing unit, one ex363 camera interconnect unit, one ex359 liquid crystal display (LCD) control unit, one ex352 modulation / demodulation unit, one
119 mexican institute »fm noneDAD
INDUSTRIAL
<img file="MX339463B_D0112.tif" />
ex353, one unit<sup>1</sup> of<sup>-</sup> audio ex354, the slot drive ex367.
multiplexing / demultiplexing signal processing of ex364 and memory unit
When an end of call key or a power key is turned ON by a user's operation, the ex361 power supply circuit unit supplies the respective units with power from a battery pack in order to activate the cell phone exll4.
In the exll4 cell phone, the ex354 audio signal processing unit converts the audio signals collected by the audio input unit ex356 in voice talk mode into digital audio signals under the control of the ex360 main control unit which includes a CPU, ROM and RAM. The ex352 modulation / demodulation unit then performs spread spectrum processing on the digital audio signals, and the transmitting and receiving unit ex351 performs the digital-to-analog conversion and the frequency conversion on the data in order to transmit the resulting data via the ex350 antenna. Also, on the exll4 cell phone, the ex351 transmit and receive unit amplifies the data received by the ex350 antenna in voice talk mode and performs frequency conversion and analog-to-digital conversion on the data. Then the ex352 modulation / demodulation unit performs the processing of
120
IMPI
INSTITUTC? MEXICAN DC THE PROPERTY
INDUSTRIAL
<img file="MX339463B_D0113.tif" />
Inverse spread spectrum in the data and the ex354 audio signal processing unit converts it into analog audio signals, in order to send them via the ex357 audio output unit.
Additionally, when an email in data communication mode is transmitted, the email text data entered through the operation of the ex366 operation key unit and other main body data is sent to the ex360 main control unit. via the ex362 operation input control unit. The ex360 main control unit causes the ex352 modulation / demodulation unit to perform spread spectrum processing on the text data and the ex351 transmit and receive unit to perform digital-to-analog conversion and frequency conversion on the resulting data to transmit the data to the exllO base station via the ex350 antenna. When an email is received, the processing which is roughly the reverse of the processing to transmit an email is performed on the received data and the resulting data is provided to the display unit ex358.
When video, still images, or video and audio are transmitted in the data communication mode, the ex355 video signal processing unit compresses and encodes
121 video signals supplied from the ex365 camera unit
IMPI
INSTITUTO MEXICANO OE (X PÍONiDAO industrial
<img file="MX339463B_D0114.tif" />
using representation encoding method
<td>visuals in</td><td>movement</td><td>shown</td><td>in c</td><td>each of</td><td>the</td>
<td>modalities</td><td>(that is to say,</td><td>works</td><td>how</td><td>the apparatus</td><td>of</td>
<td>coding</td><td>of pictures</td><td>agree</td><td>with</td><td>the look of</td><td>the</td>
<td colspan="2">present invention), and</td><td>transmit</td><td>the</td><td colspan="2">video data</td>
encoded to the multiplexing / demultiplexing unit ex353. In contrast, when the ex365 camera unit captures video, still images, and others, the audio signal processing unit ex3 54 encodes audio signals collected by the audio input unit ex356 and transmits the encoded audio data to the unit. multiplexing / demultiplexing ex353.
The multiplexing / demultiplexing unit ex353 multiplexes the encoded video data that is
<td>supplied</td><td>of</td><td>the</td><td>Unit</td><td>of</td><td>processing</td><td>signal</td><td>of</td>
<td>video ex355</td><td>and</td><td>the</td><td>data</td><td>of</td><td colspan="2">encoded audio that</td><td>are</td>
<td>supplied</td><td>of</td><td>the</td><td>Unit</td><td>of</td><td>processing</td><td>signal</td><td>of</td>
ex354 audio, using a default method. Then, the modulation / demodulation unit (modulation / demodulation circuit unit) ex352 performs the spread spectrum processing on the multiplexed data and the transmitting and receiving unit ex351 performs the digital-to-analog conversion and the frequency conversion on the data in order to transmit the resulting data via the
122
IMPI
INSTITUTO MEXICANO OE LA PROFIERA!) Inoustxial
<img file="MX339463B_D0115.tif" />
ex350 antenna.
When data is received from a video file which is linked to a Web page and others in the data communication mode or when an email with video and / or audio is received, with the purpose of decoding the multiplexed data that are received via the ex350 antenna, The ex353 multiplexing / demultiplexing unit demultiplexes the multiplexed data into a video data bitstream and an audio data bitstream and supplies the ex355 video signal processing unit with the encoded video data and the unit. audio signal processing ex354 with the encoded audio data, through the synchronous busbar ex370. The video signal processing unit ex355 decodes the video signal using a decoding method of moving visuals that corresponds to the encoding method of moving visuals shown in each of the modes (i.e., operates as the apparatus decoding device according to the aspect of the present invention), and then the ex358 display unit displays, for example, the video and still images that are included in the video file linked to the website via the ex359 LCD control unit. Additionally, the audio signal processing unit ex354
123
IMPI
MEXICAN INSTITUTE Ot THE PROPERTY
INDUSTRIAL
<img file="MX339463B_D0116.tif" />
it decodes the audio signal and the audio output unit ex357 provides the audio.
Additionally, similar to ex300 television, a terminal such as the exll4 cell phone may have 3 types of implementation configurations that include not only (i) a transmit and receive terminal that includes both an encoding apparatus and a decoding apparatus, but also (ii) a transmission terminal that includes only an encoding apparatus and (iii) a reception terminal that includes only a decoding apparatus. Although the ex200 digital broadcast system receives and transmits the multiplexed data obtained by multiplexing audio data into video data in the description, the multiplexed data may be data obtained by multiplexing not audio data but video-related character data in video data and may not be multiplexed data but video data itself.
As such, the moving visuals encoding method and the moving visuals decoding method in each of the modalities can be used in any of the described devices and systems. In this way, the advantages described in each of the modalities can be obtained.
124
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL PROPERTY
<img file="MX339463B_D0117.tif" />
Additionally, the present invention is not limited to the embodiments, and various modifications and revisions are possible without departing from the scope of the present invention.
Mode 10
Video data can be generated by switching, as necessary, between (i) the moving visuals encoding method or the moving visuals encoding apparatus shown in each of the modes and (ii) a method moving visuals coding apparatus or a moving visuals coding apparatus in accordance with a different standard, such as
MPEG-2, MPEG-4 AVC and VC-1.
In this document, when a plurality of video data that meet the different standards is generated and then decoded, the decoding methods need to be selected to meet the different standards. However, since it cannot be detected which standard each of the plurality of video data being decoded meets, there is a problem that an appropriate decoding method cannot be selected.
For the purpose of solving the problem, the multiplexed data obtained by multiplexing audio and other data into video data has a structure that includes
X
125
IMPI
MEXICAN INSTITUTE Say THE INDUSTRIAL PROPERTY
<img file="MX339463B_D0118.tif" />
Identification information indicating which standard the video data meets. The specific structure of the multiplexed data including the video data generated in the moving visuals encoding method and by the moving visuals encoding apparatus shown in each of the modalities will be described below. The multiplexed data is a digital stream in the MPEG-2 Transport Stream format.
FIGURE 23 illustrates a structure of multiplexed data. As illustrated in FIGURE 23, multiplexed data can be obtained by multiplexing at least one of a video stream, an audio stream, a presentation graphics stream (PG), and a stream of interactive graphics. The video stream represents primary video and secondary video of a movie, the audio stream (IG) represents a primary audio part and a secondary audio part that mixes with the primary audio part and the presentation graphics stream represents movie subtitles. In this document, the primary video is normal video that is displayed on a screen and the secondary video is video that is displayed in a smaller window on the primary video. Additionally, the interactive graphics stream represents an interactive screen that is
126
ΙΜΡΙ
MEXICAN INSTITUTE Ot LA NLOFIEDAD INDUSTRIAL
<img file="MX339463B_D0119.tif" />
generated by ordering GUI components on one screen.
The video stream is encoded in the moving visuals encoding method or by the moving visuals encoding apparatus shown in each of the modes, or in a moving visuals encoding method or by an apparatus encoding of moving visuals in accordance with a conventional standard such as MPEG-2, MPEG-4 AVC and VC-1. The audio stream is encoded according to a standard, such as Dolby-AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD and
Linear PCM.
Each stream included in the multiplexed data is identified by a PID. For example, 0x1011 is assigned to the video stream that is used for movie video, 0x1100 to OxlllF is assigned to audio streams, 0x1200 to 0xl21F is assigned to presentation graphics streams, 0x1400 to 0xl41F are assigned to interactive graphics streams, OxlBOO to OxlBlF are assigned to the video streams that are used for the secondary video of the movie and OxlAOO to OxlAlF are assigned to the audio streams that are used for the secondary audio that is mixed with the primary audio.
FIGURE 24 schematically illustrates how data is multiplexed. First, a stream of
IMPI
I MEXICANO re LA PHOFiEDAO INDUSTUAl video and a stream of
<img file="MX339463B_D0120.tif" />
audio transform into ex236 v a stream of
127 video ex235 made up of audio frames ex238 made up of frames a stream of PES packets PES packets ex239 and also in TS packets ex237 and TS packets ex240, respectively. Similarly, the data from an ex241 presentation graphics stream and the data from an ex244 interactive graphics stream are transformed into a PES ex242 packet stream and a PES ex245 packet stream and furthermore into TS ex243 packets and TS ex246 packets, respectively . These TS packets are multiplexed into a stream to obtain ex247 multiplexed data.
FIGURE 25 illustrates in greater detail how a video stream is stored in a PES packet stream. The first bar in FIGURE 25 shows a stream of video frames in a video stream. The second bar shows the PES packet stream. As indicated by the arrows designated yyl, yy2, yy3, and yy4 in FIGURE 25, the video stream is divided into visuals such as visuals I, visuals B, and visuals P each of which is a unit of Video presentation and visuals are stored in a payload of each of the PES packages. Each of the PES packets has a PES header and the header of
128
ÍMPI • NSTITUTO MEXICANO • í THE PROPERTY
INDUSTRIAL
<img file="MX339463B_D0121.tif" />
PES stores a Presentation Time Record (PTS) indicating a display time for the visual display and a Decoding Time Record (DTS) indicating a decoding time of visual representation.
FIGURE 26 illustrates a format of TS packets that are ultimately written to the multiplexed data. Each of the TS packets is a 188-byte fixed-length packet, which includes a 4-byte TS header that has information, such as a PID to identify a stream and a 184-byte TS payload to store data. PES packets are divided and stored in TS payloads, respectively. When using a BD ROM, each of the TS packets is provided with a 4-byte TP_Extra_Header (Additional TP Header), thereby resulting in 192-byte source packets. The source packets are written to the multiplexed data. The TP_Extra_Header stores information such as an Arrival_Time_Stamp (ATS). The ATS displays a transfer start time at which each of the TS packets must be transferred to a PID filter. The source packets are arranged in the multiplexed data as shown in the background of FIGURE 26. The numbers that increment from the head of the multiplexed data are called source packet numbers (SPNs,
129
IMPI
INSTnirro MIXICANO ° f LA PROMC0AD industrial
<img file="MX339463B_D0122.tif" />
for its acronym in English) . ____, __
Each of the TS packages included in the multiplexed data includes not only audio, video, subtitle, and other streams, but also a Program Association Table (PAT), a Program Map Table (PMT) , and a Program Clock Reference (PCR). The PAT shows what a PID indicates in a PMT used in multiplexed data and a PID from the PAT itself is recorded as zero. The PMT stores PIDs of the video, audio, subtitle and other streams included in the multiplexed data and stream attribute information corresponding to the PIDs. The PMT also has several descriptors that refer to multiplexed data. Descriptors have information such as copy control information that shows whether or not copying of multiplexed data is allowed. The PCR stores the STC time information that corresponds to an ATS that shows when the PCR packet is transferred to a decoder, in order to achieve synchronization between an Arrival Time Clock (ATC). which is an ATSs time axis and a System Time Clock (STC) which is a PTSs time axis and
DTSs.
FIGURE 27 illustrates in detail the structure
130
<img file="MX339463B_D0123.tif" />
of PMT data. A PMT header is placed at the top of the PMT. The PMT header describes the length of data included in the PMT and others. A plurality of descriptors that refer to the multiplexed data is placed after the PMT header. Information such as copy control information is described in the descriptors. After the descriptors, a plurality of pieces of current information is placed that refers to the currents included in the multiplexed data. Each piece of stream information includes stream descriptors each describing information, such as a stream type to identify a stream compression codec, a stream PID, and stream attribute information (such as a frame rate or a dimensional relationship). The current descriptors are equal in number to the number of currents in the multiplexed data.
When multiplexed data is written to a recording medium and others, it is recorded along with multiplexed data information files.
Each of the multiplexed data information files is multiplexed data management information as shown in FIGURE 28. The multiplexed data information files are in one-to-one correspondence with the multiplexed data and
131
IMPI
INSTITUTO MUUCANO m industrial property
<img file="MX339463B_D0124.tif" />
each of the files includes multiplexed data information, current attribute information, and an input map.
As illustrated in FIGURE 28, the multiplexed data information includes a system rate, a playback start time, and a playback end time. The system rate indicates the maximum transfer rate at which a target system decoder described below transfers the multiplexed data to a PID filter. The ranges of the ATSs included in the multiplexed data are set at a level no higher than a system rate. Playback start time indicates a PTS in a video frame at the head of the multiplexed data. An interval of one frame is added to a PTS in a video frame at the end of the multiplexed data and the PTS is set at the end of playback time.
<td>How</td><td>it shows</td><td>in FIGURE</td><td> 29,</td><td>a</td><td>piece</td><td>of</td>
<td>information of</td><td>attributes</td><td>register in</td><td>the</td><td colspan="2">information</td><td>of</td>
<td>attributes of</td><td>stream,</td><td>for each PID</td><td>of</td><td>every</td><td colspan="2">stream</td>
<td>included in</td><td>the data</td><td>multiplexed.</td><td colspan="2">Every</td><td>piece</td><td>of</td>
Attribute information has different information depending on whether the corresponding stream is a video stream, an audio stream, a presentation graphics stream, or an interactive graphics stream.
132
Mexicano Mexican iNSTmrro DE LA RROHEDAD industrial
<img file="MX339463B_D0125.tif" />
Each piece of video stream attribute information carries information including what kind of compression codee is used for compression of the video stream and the resolution, dimensional ratio and frame rate of the pieces of visual data being displayed. include in the video stream. Each piece of audio stream attribute information carries information including what kind of compression codec is used to compress the audio stream, how many channels are included in the audio stream, what language the audio stream supports, and how high is the sampling frequency. The video stream attribute information and the audio stream attribute information are used for the initialization of a decoder before the player repeats the information.
In the present embodiment, the multiplexed data used is of a stream type included in the PMT. Additionally, when the multiplexed data is recorded on a recording medium, the video stream attribute information included in the multiplexed data information is used. More specifically, the moving visuals coding method or the moving visuals coding apparatus described in each of the modalities includes a step or unit for assigning unique information that
133
I ΡI
INSTrnn »MEXICAN
FROM PROPERTY IN »USTWjAi indicates the video data generated by the method of encoding visual representations in motion or the apparatus for encoding visual representations in each of the modalities, at the type of stream included in the PMT or the information of video stream attributes. With the configuration, the video data generated by the moving visuals encoding method or the moving visuals encoding apparatus described in each of the modalities can be distinguished from video data that complies with another standard.
<td>Further,</td><td>the</td><td>FIGURE 30 illustrates</td><td>Steps</td><td>of the</td>
<td>decoding method</td><td>of</td><td>representations</td><td>visual</td><td>in</td>
<td>movement according to</td><td>the</td><td>present modality.</td><td>At</td><td>He passed</td>
exSlOO, the stream type included in the PMT or the video stream attribute information included in the multiplexed data information is derived from the multiplexed data. Then, in Step exSlOl, it is determined whether or not the stream type or video stream attribute information indicates that the multiplexed data is generated by the moving display encoding method or the display encoding apparatus. in motion in each of the modalities. When it is determined that the stream type or video stream attribute information indicates that
134 multiplexed data is generated by the
<img file="MX339463B_D0126.tif" />
encoding of moving visuals or the encoding of moving visuals in each of the modes, in Step exS102, the decoding is performed by the method of decoding of moving visuals in each of the modes. Additionally, when the stream type or video stream attribute information indicates compliance with conventional standards, such as MPEG-2, MPEG-4 AVC and VC-1, in Step exS103, decoding is performed by a decoding method of moving visuals in accordance with conventional standards.
As such, assigning a new unique value to the stream type or video stream attribute information makes it possible to determine whether the moving visuals decoding method or the moving visuals decoding apparatus being described in each one of the modalities they can carry out or not the decoding. Even when entering multiplexed data that conforms to a different standard, an appropriate decoding method or apparatus can be selected. In this way, it becomes possible to decode information without any error. Additionally, the visual representation encoding method or apparatus
<img file="MX339463B_D0127.tif" />
<sup>135</sup> IMPI 'NST'TjnVMiJuCA ^ t * LA nORIIDA The INDUSTIUAL in movement, or the method or apparatus for decoding visual representations in movement in the present modality can be used in the devices and systems described above.
Mode 11
Each of the moving visuals coding method, the moving visuals coding apparatus, the moving visuals decoding method, and the moving visuals decoding apparatus in each of the modalities is typically accomplished in the form of an integrated circuit or a Large Scale Integrated Circuit (LSI). As an example of the LSI, FIGURE 31 illustrates a configuration of the LSI ex500 that is done on a chip. The LSI ex500 includes the ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 elements described below, and the elements are connected to each other via an ex510 busbar. The ex505 power supply circuit unit is activated by supplying each of the elements with power when the ex505 power supply circuit unit is turned on.
For example, when encoding is done, the LSI ex500 receives an AV signal from an exll7 microphone, an exll3 camera, and others through a low ex509 AV IO.
136
IMPI
MEXICAN INSTITUTE OE LA Eft * INDUSTRIAL FLEDAD
<img file="MX339463B_D0128.tif" />
control of an ex501 control unit including an ex502 CPU, an ex503 memory controller, an ex504 current controller, and an ex512 drive frequency control unit. The received AV signal is temporarily stored in external memory ex511, such as an SDRAM. Under the control of the ex501 control unit, the stored data is segmented into data portions according to the amount and speed of processing that is transmitted to an ex507 signal processing unit. The ex507 signal processing unit then encodes an audio signal and / or a video signal. In this document, the encoding of the video signal is the encoding described in each of the modalities. Additionally, the signal processing unit ex507 sometimes multiplexes the encoded audio data and the encoded video data, and a current IO ex506 provides the multiplexed data to the outside. The provided multiplexed data is transmitted to the base station exl07, or written to the recording medium ex215. When data sets are multiplexed, the data must be temporarily stored in the ex508 buffer so that the data sets are synchronized with each other.
<td>Although the</td><td>ex511 memory</td><td>is</td><td>a</td><td>element</td><td>outside the</td>
<td colspan="2">LSI ex500, can be included in</td><td>the</td><td>LSI</td><td>ex500.</td><td>The memory</td>
<td>intermediate ex508 no</td><td>is limited</td><td>to</td><td>a</td><td>memory</td><td>intermediate,</td>
137
<img file="MX339463B_D0129.tif" />
it can be made up of buffers. Additionally, the LSI ex500 can be made on one chip or a plurality of chips.
Additionally, although the ex501 control unit includes the ex502 CPU, the ex503 memory controller, the ex504 current controller, the ex512 drive frequency control unit, the configuration of the ex501 control unit is not limited to these. For example, the ex507 signal processing unit may also include a CPU. Including another CPU in the ex507 signal processing unit can improve processing speed. Additionally, as another example, the ex502 CPU may serve as or may be a part of the ex507 signal processing unit and, for example, may include an audio signal processing unit. In this case, the ex501 control unit includes the ex507 signal processing unit or the ex502 CPU which includes a part of the ex507 signal processing unit.
The name used in this document is LSI, but it can also be called IC, LSI system, super LSI or ultraLSI depending on the degree of integration.
On the other hand, the ways to achieve integration are not limited to the LSI and a special circuit or a general-purpose processor and thus can also achieve integration. The Programmable Door Matrix
138 Field (FPGA) that can be
IMPI ινϊτγπ / το Mexican
Of LA «ΟΜΙΟΑΓ industrial
<img file="MX339463B_D0130.tif" />
Programming after the manufacture of LSIs or a reconfigurable processor that allows reconfiguration of the connection or the configuration of an LSI can be used for the same purpose. This programmable logic device can typically execute the moving visuals encoding method and / or the moving visuals decoding method according to any of the modalities described above when loading or reading, from a memory or the like, one or more programs that are included in the software or firmware.
In the future, with the advancement in semiconductor technology, a new technology may replace LSI. Function blocks can be integrated using this technology. The possibility is that the present invention is applied to biotechnology.
Mode 12
When the video data generated in the moving visuals encoding method or by the moving visuals encoding apparatus described in each of the modes is decoded, the amount of processing may increase compared to when video data that conforms to a conventional standard such as MPEG-2, MPEG-4 AVC, and VC-1 is decoded. In this way, the LSI
139
IMPI
MEXICAN INSTITUTE OE LA P »* ME» A (j INOtISTKIAL
<img file="MX339463B_D0131.tif" />
ex500 needs to be set to a higher drive frequency than that of the ex502 CPU that is used when decoding video data in accordance with the conventional standard. However, when the driving frequency is set higher, there is a problem that the power consumption increases.
For the purpose of solving the problem, the moving visuals decoding apparatus such as the ex300 television and the ex500 LSI are configured to determine which standard the video data meets and to switch between the driving frequencies according to the determined standard . FIGURE 32 illustrates an ex800 configuration in the present embodiment. An ex803 drive frequency switching unit sets a drive frequency to a higher drive frequency when the video data is generated by the moving visuals coding method or the moving visuals coding apparatus described in each of the modalities. Then, the drive frequency switching unit ex803 instructs a decoding processing unit ex801 that executes the decoding method of moving visuals described in each of the modes for decoding the video data. When the video data meets the standard
140
IMPI
MEXICAN INSTITUTE • F. LA FROM E DA Π INDUSTRIAL Conventional, the drive frequency switching unit ex803 sets a drive frequency at a lower drive frequency than that of the video data generated by the me moving visuals encoding or the encoding apparatus of visual representations in movement described in each of the modalities. The ex803 driver frequency switching unit then instructs the ex802 decoding processing unit that complies with the conventional standard for decoding video data.
More specifically, the ex803 drive frequency switching unit includes the ex502 CPU and the ex512 drive frequency control unit in FIGURE 31. In this document, each ex801 decoding processing unit that executes the moving visuals decoding method described in each of the modalities and the ex802 decoding processing unit that complies with the conventional standard corresponds to the processing unit of signals ex507 in FIGURE 31. The ex502 CPU determines which standard the video data meets. The drive frequency control unit ex512 then determines a drive frequency based on a signal from the ex502 CPU. Additionally, the ex507 signal processing unit decodes the video data
<img file="MX339463B_D0132.tif" />
141
IMPI
MEXICAN INSTITUTE OF INDUSTRIAL MOREDAD
<img file="MX339463B_D0133.tif" />
based on the ex502 CPU signal. For example, the identification information described in Mode 10 may be used to identify the video data. Identification information is not limited to that described in Mode 10 but can be any information as long as the information indicates which standard the video data meets. For example, when it can be determined which standard the video data meets based on an external signal to determine that the video data is used for a television or a disc, etc., the determination can be made based on this external signal. Additionally, the ex502 CPU selects a drive frequency based on, for example, a lookup table in which the video data standards are associated with the drive frequencies as shown in FIGURE 32. The drive frequency can be selected by storing the lookup table in the ex508 buffer and in an internal memory of an LSI and with reference to the lookup table by the ex502 CPU.
FIGURE 33 illustrates steps for executing a method in the present embodiment. First, in Step exS200,
<td>unit</td><td>of</td><td>processing</td><td>of</td><td>signs</td><td>ex507 gets</td>
<td>information</td><td>of</td><td>ID</td><td>of</td><td>the data</td><td>multiplexed.</td>
<td>After in</td><td>the</td><td colspan="4">Step exS201, CPU ex502 determines if the</td>
<td colspan="2">video data</td><td>are generated or</td><td>not</td><td>through</td><td>of the method of</td>
142
IMPI
MEXICAN INSTITUTE M LA MOPIEDAO
INDUSTRIAL
<img file="MX339463B_D0134.tif" />
coding and the coding apparatus described in each of the modalities, based on the identification information. When the video data is generated by means of the moving visuals encoding method and the moving visuals encoding apparatus described in each of the modes, in Step exS202, the ex502 CPU transmits a signal to establish the drive frequency at a higher drive frequency to the drive frequency control unit ex512. Then the drive frequency control unit ex512 sets the drive frequency to the highest drive frequency. On the other hand, when the identifying information indicates that the video data complies with the conventional standard, such as MPEG2, MPEG-4 AVC and VC-1, in Step exS203, CPU ex502 transmits a signal to set the driving frequency at a lower boost frequency to the boost frequency control unit ex512. Then, the drive frequency control unit ex512 sets the drive frequency to the lower drive frequency than that in the case where the video data is generated by means of the moving visuals encoding method and the rendering encoding apparatus. visuals in motion described in each of the modalities.
143
IMPI iNSTrnnv Mexican OE LA MONEDAD INDUSTRIAL
<img file="MX339463B_D0135.tif" />
Additionally, along with switching drive frequencies, the energy conservation effect can be enhanced by changing the voltage that is applied to the LSI ex500 or an appliance that includes the LSI ex500. For example, when the drive frequency is set lower, the voltage applied to the LSI ex500 or the device that includes the LSI ex500 may be set to a lower voltage than that in the case where the drive frequency is set more high.
Additionally, when the amount of processing for decoding is larger, the driving frequency can be set higher, and when the amount of processing for decoding is smaller, the driving frequency can be set lower as the method of setting the driving frequency. Thus, the method of establishment is not limited to those described above. For example, when the amount of processing for decoding video data in accordance with MPEG-4 AVC is greater than the amount of processing for decoding of video data generated by means of the encoding method for moving visuals and the apparatus. coding of moving visuals described in each of the modes, the drive frequency may be set in reverse order of
144
<img file="MX339463B_D0136.tif" />
establishment described above.
Additionally, the method of setting the drive frequency is not limited to the method of setting the lowest drive frequency. For example, when the identifying information indicates that the video data is generated by the moving visuals coding method and the moving visuals coding apparatus described in each of the modes, the voltage may that applies to the LSI ex500 or the device that includes the LSI ex500 is set higher. When the identifying information indicates that the video data complies with the conventional standard, such as MPEG-2, MPEG-4 AVC, and VC-1, it is possible that the voltage applied to the LSI ex500 or the device that includes the LSI ex500 is set lower. As another example, it is possible that when the identification information indicates that the video data is generated by the moving visuals coding method and the moving visuals coding apparatus described in each of the modalities, the ex502 CPU drive does not have to be suspended and when the identifying information indicates that the video data complies with the conventional standard, such as MPEG-2, MPEG-4 AVC and VC-1, ex502 CPU drive may be suspended in a while
145
<img file="MX339463B_D0137.tif" />
ΙΜΡΙ
IWCTfRiM »_
<img file="MX339463B_D0138.tif" />
determined because the ex502 CPU has additional processing power. Even when the identifying information indicates that the video data is generated by means of the moving visuals encoding method and the moving visuals encoding apparatus described in each of the modes, in the case where the CPU ex502 It has additional processing capacity, the ex502 CPU drive may be suspended in a certain time. In this case, the sleep time may be set shorter than that in the case when the identifying information indicates that the video data complies with the conventional standard, such as MPEG2, MPEG-4 AVC and VC-1.
<img file="MX339463B_D0139.tif" />
Accordingly, the energy conservation effect can be improved by switching between the driving frequencies according to the standard that the video data meets. Additionally, when the LSI ex500 or the device that includes the LSI ex500 is powered using a battery, the life of the battery can be extended with the effect of conserving energy.
Mode 13
There are cases where a plurality of video data that meets different standards is provided to devices and systems, such as a television and a
146 cell phone.
With the purpose of
IMPI
INSTITUI »MEXICAN • EU MOMEüAD industrial make possible
<img file="MX339463B_D0140.tif" />
Decoding the plurality of video data that meet the different standards, the LSI ex500's signal processing unit ex507 needs to meet the different standards. However, the problems of increasing the scale of the LSI ex500 circuit and increasing the cost arise with the individual use of the ex507 signal processing units that comply with the respective standards.
In order to solve the problem, what is devised is a configuration in which the decoding processing unit to implement the decoding method of moving visual representations described in each of the modalities and the decoding processing unit that It complies with the conventional standard, such as MPEG-2, MPEG-4 AVC and VC-1 are partially shared. Ex900 in FIGURE 35A shows an example of the configuration. For example, the MPEG-4 AVC compliant decoding method for moving visuals described in each of the modalities and the decoding method for moving visuals have, partially in common, the details of the processing, such as entropic coding, reverse quantification, unlock filtering, and motion compensation prediction. It is possible that a unit of
147
MEXICAN INSTITUTE ·
DC LA rROHCBAB
INDUSTRIAL MPEG-4 compliant ex902 decoding processing
AVC is shared by common processing operations and
<td>what a</td><td>Unit</td><td>of</td><td>processing</td><td>decoding</td>
<td colspan="2">specialized ex901</td><td>I know</td><td>use for the</td><td>processing which is</td>
<td>unique to</td><td colspan="2">an aspect</td><td>of the present</td><td>invention and not</td>
<td>complies with</td><td>MPEG-4</td><td colspan="2">AVC. it's possible</td><td>that a unit of</td>
ex902 decoding processing that complies with MPEG-4 AVC is shared by common processing operations and that a specialized decoding processing unit ex901 is used for processing that is unique to an aspect of the present invention and is not MPEG-compliant. 4 AVC. The decoding processing unit to implement the decoding method of moving visuals described in each of the modes can be shared so that the processing is shared and a specialized decoding processing unit can be used for the unique processing for that MPEG-4 AVC.
Additionally, exlOOO in FIGURE 35B shows another example where processing is partially shared. This example uses a configuration that includes an exlOOl specialized decoding processing unit that supports single processing for one aspect of the present invention, an exl002 specialized decoding processing unit that
148
<img file="MX339463B_D0141.tif" />
supports unique processing for another conventional standard and a decoding processing unit exl003 that supports processing that is shared between the decoding method of moving displays according to the aspect of the present invention and the decoding method of displays in conventional motion. In this document, the units of
<td>processing</td><td>of</td><td>specialized decoding</td><td>exlOOl and</td>
<td colspan="2">ex! 002 are not</td><td>necessarily specialized</td><td>for him</td>
<td>processing</td><td>of</td><td>according to the appearance of the</td><td>Present</td>
<td>invention and</td><td>the</td><td colspan="2">conventional standard processing,</td>
respectively, and may be those capable of implementing general processing. Additionally, the configuration of the present modality can be implemented by the LSI ex500.
As such, reducing the scale of the circuit of an LSI and reducing the cost are possible by sharing the decoding processing unit so that the processing is shared between the decoding method of moving visuals according to the aspect of the present invention and the method of decoding moving visual representations in accordance with the conventional standard.
Industrial Applicability
An image encoding method and a method
149
ΙΜΡΙ
<img file="MX339463B_D0142.tif" />
INSTITUTO MEXICANO Οϊ IA TRONÍO * r> INDUSTRIAL image decoding according to the present invention can be applied to various multimedia data. The image encoding method and the image decoding method according to the present invention are useful as an image encoding method and an image decoding method in storage, transmission, communication and the like using a mobile phone, a DVD device, personal computer and the like.
List of Reference Signs
101 encoder
105 subtractor
110 transformation unit
120 unit of quantification
130, 230 reverse transformation unit
140, 240 adder
150, 250 filter unlock
160, 260 Adaptive Loop Filter
170, 270 frame memory
180, 280 prediction unit
190 entropic encoder
200 decoder
290 entropic decoder
300, 400, 710 visual representation
31, 32, 3i, 41, 42 row of LCU
150
IMPI
<img file="MX339463B_D0143.tif" />
<td></td><td> 311,</td><td>312, 3il,:</td><td> 321</td><td>LCU</td>
<td></td><td> 500</td><td>header</td><td>of</td><td>package</td>
<td></td><td> 510</td><td>header</td><td>of</td><td>IP</td>
<td></td><td> 520,</td><td>550 field</td><td>of</td><td>extension</td>
<td> 5</td><td> 530</td><td>header</td><td>of</td><td>UDP</td>
<td></td><td> 540</td><td>header</td><td>of</td><td>RTP</td>
<td></td><td> 560</td><td>header</td><td>of</td><td>Useful load</td>
<td></td><td> 570</td><td>header</td><td>of</td><td>NAL</td>
<td></td><td>yes</td><td colspan="3">input signal</td>
<td> 10</td><td>s2</td><td colspan="3">prediction signal</td>
e, e 'prediction error signal s', s, s3 reconstructed signal
It is noted that in relation to this date, the best method known by the applicant to put the aforementioned invention into practice is the one that is clear from the present description of the invention.
151
Contents120
178 sheets
Sheet 1 Sheet 2 Sheet 3 Sheet 4 Sheet 5 Sheet 6 Sheet 7 Sheet 8 Sheet 9 Sheet 10 Sheet 11 Sheet 12 Sheet 13 Sheet 14 Sheet 15 Sheet 16 Sheet 17 Sheet 18 Sheet 19 Sheet 20 Sheet 21 Sheet 22 Sheet 23 Sheet 24 Sheet 25 Sheet 26 Sheet 27 Sheet 28 Sheet 29 Sheet 30 Sheet 31 Sheet 32 Sheet 33 Sheet 34 Sheet 35 Sheet 36 Sheet 37 Sheet 38 Sheet 39 Sheet 40 Sheet 41 Sheet 42 Sheet 43 Sheet 44 Sheet 45 Sheet 46 Sheet 47 Sheet 48 Sheet 49 Sheet 50 Sheet 51 Sheet 52 Sheet 53 Sheet 54 Sheet 55 Sheet 56 Sheet 57 Sheet 58 Sheet 59 Sheet 60 Sheet 61 Sheet 62 Sheet 63 Sheet 64 Sheet 65 Sheet 66 Sheet 67 Sheet 68 Sheet 69 Sheet 70 Sheet 71 Sheet 72 Sheet 73 Sheet 74 Sheet 75 Sheet 76 Sheet 77 Sheet 78 Sheet 79 Sheet 80 Sheet 81 Sheet 82 Sheet 83 Sheet 84 Sheet 85 Sheet 86 Sheet 87 Sheet 88 Sheet 89 Sheet 90 Sheet 91 Sheet 92 Sheet 93 Sheet 94 Sheet 95 Sheet 96 Sheet 97 Sheet 98 Sheet 99 Sheet 100 Sheet 101 Sheet 102 Sheet 103 Sheet 104 Sheet 105 Sheet 106 Sheet 107 Sheet 108 Sheet 109 Sheet 110 Sheet 111 Sheet 112 Sheet 113 Sheet 114 Sheet 115 Sheet 116 Sheet 117 Sheet 118 Sheet 119 Sheet 120 Sheet 121 Sheet 122 Sheet 123 Sheet 124 Sheet 125 Sheet 126 Sheet 127 Sheet 128 Sheet 129 Sheet 130 Sheet 131 Sheet 132 Sheet 133 Sheet 134 Sheet 135 Sheet 136 Sheet 137 Sheet 138 Sheet 139 Sheet 140 Sheet 141 Sheet 142 Sheet 143 Sheet 144 Sheet 145 Sheet 146 Sheet 147 Sheet 148 Sheet 149 Sheet 150 Sheet 151 Sheet 152 Sheet 153 Sheet 154 Sheet 155 Sheet 156 Sheet 157 Sheet 158 Sheet 159 Sheet 160 Sheet 161 Sheet 162 Sheet 163 Sheet 164 Sheet 165 Sheet 166 Sheet 167 Sheet 168 Sheet 169 Sheet 170 Sheet 171 Sheet 172 Sheet 173 Sheet 174 Sheet 175 Sheet 176 Sheet 177 Sheet 178
98 members in 22 offices
Priority claims5
| Document | Office | Kind | Date |
|---|---|---|---|
| 61705846 | United States of America | – | |
| 201261705846 | United States of America | P | |
| 61711892 | United States of America | – | |
| 201261711892 | United States of America | P | |
| 2013005541 | Japan | W |
Members98
| Document | Office | Kind | |
|---|---|---|---|
| CA2881221A1 | Canada | A1 | |
| US2014093180A1 | United States of America | A1 | |
| WO2014050038A1 | World Intellectual Property Organization (WIPO) | A1 | |
| TW201429253A | Taiwan Province of China | A | |
| AU2013322008A1 | Australia | A1 | |
| AU2013322008A2 | Australia | A2 | |
| PH12015500365A1 | Philippines | A1 | |
| PH12015500365B1 | Philippines | B1 | |
| US9014494B2 | United States of America | B2 | |
| US2015131738A1 | United States of America | A1 | |
| SG11201500846TA | Singapore | A | |
| KR20150063356A | Republic of Korea | A | |
| CN104737541A | China | A | |
| MX2015002889A | Mexico | A | |
| EP2903267A1 | European Patent Office (EPO) | A1 | |
| EP2903267A4 | European Patent Office (EPO) | A4 | |
| MX339463BThis record | Mexico | B | |
| US9357234B2 | United States of America | B2 | |
| US2016241879A1 | United States of America | A1 | |
| JPWO2014050038A1 | Japan | A1 | |
| AU2013322008B2 | Australia | B2 | |
| RU2015103543A | Russian Federation | A | |
| US9503755B2 | United States of America | B2 | |
| EP3122048A1 | European Patent Office (EPO) | A1 | |
| US2017034534A1 | United States of America | A1 | |
| EP2903267B1 | European Patent Office (EPO) | B1 | |
| TWI593274B | Taiwan Province of China | B | |
| JP6172535B2 | Japan | B2 | |
| ES2630359T3 | Spain | T3 | |
| PL2903267T3 | Poland | T3 | |
| JP2017192144A | Japan | A | |
| US9872043B2 | United States of America | B2 | |
| EP3122048B1 | European Patent Office (EPO) | B1 | |
| DK3122048T3 | Denmark | T3 | |
| TR2018002584T4 | Türkiye | T4 | |
| TR201802584T4 | Türkiye | T4 | |
| US2018084282A1 | United States of America | A1 | |
| EP3301923A1 | European Patent Office (EPO) | A1 | |
| CN104737541B | China | B | |
| ES2664361T3 | Spain | T3 | |
| PT3122048T | Portugal | T | |
| JP6317015B2 | Japan | B2 | |
| RU2653236C2 | Russian Federation | C2 | |
| PH12017501838A1 | Philippines | A1 | |
| PH12017501838B1 | Philippines | B1 | |
| CN108282655A | China | A | |
| PL3122048T3 | Poland | T3 | |
| JP2018125881A | Japan | A | |
| RU2018111944A | Russian Federation | A | |
| HK1253286A | Hong Kong, China | A | |
| HK1253286A1 | Hong Kong, China | A1 | |
| JP6558784B2 | Japan | B2 | |
| BR112015004140A2 | Brazil | A2 | |
| JP2019205183A | Japan | A | |
| EP3301923B1 | European Patent Office (EPO) | B1 | |
| KR102072832B1 | Republic of Korea | B1 | |
| KR20200013098A | Republic of Korea | A | |
| US10616605B2 | United States of America | B2 | |
| EP3654649A1 | European Patent Office (EPO) | A1 | |
| US2020195975A1 | United States of America | A1 | |
| ES2780006T3 | Spain | T3 | |
| MY176984A | Malaysia | A | |
| JP6758456B2 | Japan | B2 | |
| KR102169058B1 | Republic of Korea | B1 | |
| PH12019501972A1 | Philippines | A1 | |
| RU2018111944A3 | Russian Federation | A3 | |
| CA2881221C | Canada | C | |
| EP3654649B1 | European Patent Office (EPO) | B1 | |
| EP3876536A1 | European Patent Office (EPO) | A1 | |
| CN108282655B | China | B | |
| RU2756093C2 | Russian Federation | C2 | |
| BR112015004140A8 | Brazil | A8 | |
| US11632572B2 | United States of America | B2 | |
| EP3876536B1 | European Patent Office (EPO) | B1 | |
| US2023209095A1 | United States of America | A1 | |
| EP4221217A1 | European Patent Office (EPO) | A1 | |
| PL3876536T3 | Poland | T3 | |
| ES2953336T3 | Spain | T3 | |
| US11943484B2 | United States of America | B2 | |
| EP4351137A2 | European Patent Office (EPO) | A2 | |
| EP4351137A3 | European Patent Office (EPO) | A3 | |
| PH12022553022A1 | Philippines | A1 | |
| US2024196020A1 | United States of America | A1 | |
| EP4221217B1 | European Patent Office (EPO) | B1 | |
| PL4221217T3 | Poland | T3 | |
| US12200269B2 | United States of America | B2 | |
| EP4521745A2 | European Patent Office (EPO) | A2 | |
| ES3005448T3 | Spain | T3 | |
| US2025097473A1 | United States of America | A1 | |
| EP4351137B1 | European Patent Office (EPO) | B1 | |
| EP4521745A3 | European Patent Office (EPO) | A3 | |
| ES3030459T3 | Spain | T3 | |
| PL4351137T3 | Poland | T3 | |
| HUE071157T2 | Hungary | T2 | |
| EP4622263A2 | European Patent Office (EPO) | A2 | |
| EP4521745B1 | European Patent Office (EPO) | B1 | |
| EP4521745C0 | European Patent Office (EPO) | C0 | |
| EP4622263A3 | European Patent Office (EPO) | A3 |
1 legal event, as the office reported them to INPADOC
Events
| Event | Code | |
|---|---|---|
| Grant or registrationFG | FG |
Numbers
- Publication
- 339463
- Application
- 2889
Titles2
- Spanish
- METODO DE CODIFICACION DE IMAGENES, METODO DE DECODIFICACION DE IMAGENES, APARATO DE CODIFICACION DE IMAGENES, APARATO DE DECODIFICACION DE IMAGENES Y APARATO DE CODIFICACION Y DECODIFICACION DE IMAGENES.
- English
- IMAGE ENCODING METHOD, IMAGE DECODING METHOD, IMAGE ENCODING DEVICE, IMAGE DECODING DEVICE, AND IMAGE ENCODING/DECODING DEVICE.
Classification
- CPC, 9
- H04N19/174
- H04N19/70
- H04N19/30
- H04N19/597
- H04N19/436
- H04N19/423
- H04N19/46
- H04N19/80
- H04N19/52
- IPC, 4
- H04N19 00
- G06K9 36
- H04N19 30
- H04N19 70