Moving picture stream generation apparatus, moving picture coding apparatus, moving picture multiplexing apparatus and moving picture decoding apparatus

ES 2 388 397 T3

DESCRIPTION

Motion picture stream generation apparatus, motion picture coding apparatus, motion picture multiplexing apparatus and motion picture decoding apparatus

Description

Technical field

The present invention relates to an apparatus, and the like, which generates a stream of coded moving pictures, especially an apparatus, and the like, which generates a stream on which trick play such as skip play can be performed. , variable speed playback, reverse playback and the like.

Previous technique

Recently, the era of multimedia has arrived in which sound, images and other pixel values ​​are integrated into a medium, and conventional information media, such as communication tools such as newspapers, magazines, television, radio and phone are considered as the recipients of multimedia. Generally, multimedia is a form of simultaneous representation not only of characters but also of graphics, sound and especially images. In order to manage the conventional information media previously described as multimedia, it is a requirement to represent the information digitally.

However, it is not realistic to directly process a large amount of information digitally using the conventional information media described above because, when the amount of data of each information medium described above is calculated as the amount of digital data, the amount of data per character is 1 to 2 octets, while that of sound per second is not less than 65 kbits (telephone voice quality) and that of moving images per second is not less than 100 Mbits (current television reception quality). For example, a television telephone is already commercially available, thanks to the Integrated Services Digital Network (ISDN), with a transmission speed of 64 kbps to 1.5 Mbps, but it is impossible to transmit mobile television camera images when use ISDN.

For this reason, the information compression technique is necessary. For example, a moving picture compression technique standard of H. 261 or H. 263, recommended by the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) is used for television telephones. Similarly, with the information compression technique of the MPEG-1 standard, it is possible to store picture information, together with sound information, on a normal CD (compact disc) for music.

Here, the Moving Picture Experts Group (MPEG) is an international standard for digitally compressing moving picture signals, and has been standardized by ISO / IEC (the International Organization for Standardization / International Engineering Consortium). MPEG-1 is the standard for compressing motion picture signals up to 1.5 Mbps, that is, for compressing television signal information by approximately one hundredth. Similarly, the quality that meets the MPEG-1 standard is the average level that can be achieved at a transmission speed of approximately 1.5 Mbps. In this way MPEG-2 is standardized to meet the need for higher image quality, and compresses motion picture signals from 2 to 15 Mbps. Currently, the working group (ISO / IEC JTC1 / SC29 / WG11), which standardized MPEG-1 and MPEG-2, has standardized MPEG-4 with higher speed Of compression. The MPEG-4 standard (i) achieves a compression speed higher than the MPEG-1 standard and the MPEG-2 standard, (ii) allows to encode, decode and perform object-by-object operations and (iii) performs new functions necessary in this it was multimedia. The initial aim of the MPEG-4 standard is to standardize a low bit rate image coding method, but the aim is extended to a general purpose high bit rate interlaced image coding method. Following this, ISO / IEC and ITU / T, in combination, have standardized MPEG-4 AVC (Advanced Video Coding) as a next-generation image encoding procedure for images with a high compression rate. It is expected to be used for next generation optical disk related apparatus, or in broadcasting for mobile terminals.

Generally, in the coding of moving images, the amount of information is compressed reducing the temporal and spatial redundancies. In inter-image prediction coding, aimed at reducing temporal redundancies, motion estimation and prediction image generation are carried out block by block with reference to a later image or a previous image, and the coding is carried out on the differential value between the prediction image obtained and the image to be encoded. In this specification, the term "Image" used is a term that represents an image. In a progressive image, an image means a raster, but in an interlaced image, it means a raster or a field. An "interlaced image" described here means a raster made up of two fields with a slight time lag. In interlaced picture encoding and decoding procedures, it is possible to process a frame as is, as two fields, or frame by frame or field by field of each block in a frame.

The image for carrying out intra-prediction coding without referring to any reference image is called the Intra-coded Image (I-image). Similarly, the image for carrying out the interprediction coding that refers to only one image is called the coded predictive image (P-image). Similarly, the image for carrying out inter-prediction coding that refers to two reference images simultaneously is called Bipredictive Coded Image (B-image). A B image can refer to two selected images as an arbitrary combination of an after image and an earlier image at display time. Such two reference pictures can be specified block by block, the block being a basic unit of encoding and decoding. Those reference images are distinguished from each other in the following way: the reference image described above in the encoded bit stream is called

ES 2 388 397 T3 is the first reference image, and the other reference image, described later, is called the second reference image. Note that such reference pictures must have been previously encoded or decoded to encode or decode P-pictures and B-pictures.

Motion compensation inter-prediction coding is used to encode P-pictures and B-pictures. Motion compensation intra-prediction coding is an intra-prediction coding method in which motion compensation is applied. Motion compensation is a procedure for improving prediction accuracy and reducing the amount of data by evaluating the amount of motion (hereinafter referred to as motion vector) of each block of an image and carrying out prediction coding that considers the motion vector. For example, the amount of data is reduced by evaluating the motion vectors of the images to be encoded and encoding each prediction residual between each prediction value that is shifted by the amount of each motion vector and each current image to be encoded. In the case of this method, since the motion vector information is required in decoding, the motion vectors are also encoded, and recorded or transmitted.

Motion vectors are evaluated macroblock by macroblock. More specifically, motion vectors are evaluated by setting the macroblock of an image to be encoded, moving the macroblock of a reference image within the search range, and finding the location of the reference block that is closest to the standard block.

FIG. 1A and 1B are, respectively, conventional MPEG-2 stream structural diagrams. As shown in FIG. 1B, an MPEG-2 stream has a hierarchical structure as will be described hereinafter. A stream is made up of a Group of Images (hereinafter referred to as GOP). Using a GOP as the basic unit in encoding processing allows you to edit a moving image or perform random access. A GOP is made up of I-pictures, P-pictures, and B-pictures. A stream, a GOP, and an image further include a synchronous signal (sync) indicating a unit boundary and a header indicating the common data in the units, the units here being a stream, a GOP, and an image, respectively.

FIG. 2A and 2B respectively show examples indicating the way to carry out inter-picture prediction coding used in MPEG-2. The diagonally shaded images in the figure are the images to which other images should refer. As shown in FIG. 2A, in the prediction coding in MPEG-2, the P-pictures (P0, P6, P9, P12 and P15) can refer only to a single selected picture as an I-picture or a P-picture immediately later in the display time . Similarly, B images (B1, B2, B4, B5, B7, B8, B10, B11, B13, B14, B16, B17, B19, and B20) can refer to two selected images as a combination of an I image or image P immediately after and an I image or P image immediately before. Furthermore, the order of the images to be placed in a stream is determined. The I images and a P image are placed in the order of the display time, and each B image is placed immediately after an I image to be displayed immediately after the B image or immediately after a P image. As a structural example of a GOP, as shown in FIG. 2B, the images from I3 to B14 are grouped into a single GOP.

FIG. 3A is a structural diagram of an MPEG-4 AVC stream. There is no concept equivalent to a GOP in the MPEG-4 AVC. However, since it is possible to construct a randomly accessible unit equivalent to a GOP, segmenting the data on the basis of a special image that can be decoded without relying on other images, the unit will be referred to hereinafter as RAU (Random Access Unit ). In other words, a random access unit RAU is a group of coded pictures starting with an intracoded picture that can be decoded without depending on any picture.

Next, the access unit which is a basic unit in managing a flow (simply referred to as AU hereinafter) will be described later. An AU is the unit for storing encoded data equivalent to an image, and includes a set PS of parameters, segment data and the like. There are two types of PS sets of parameters. One of them is a PPS set of image parameters (hereinafter simply referred to as PPS) which are data equivalent to the header of each image. The other is an SPS set of sequence parameters (hereinafter simply referred to as SPS) that is equivalent to the header included in one unit of one or more GOPs in MPEG-2. An SPS includes the maximum number of reference images, an image size, and the like. On the other hand, a PPS includes a variable-length encoding type, an initial value of the quantization step, the number of reference pictures, and the like. Each image is assigned an identifier that indicates which of the PPS and SPS described above is referenced. Likewise, a frame number FN, which is the identification number for identifying an image included in segment data. Note that a sequence begins with a special image in which all the states necessary for decoding are reset as will be described later, and that it is made up of a group of images that begins with a special image and ends with an image that is placed immediately. before the next special image.

There are two types of I images in the MPEG-4 AVC. They are an Instant Decoder Refresh (IDR) and the rest. An IDR picture is the I picture that can decode all the pictures placed after the IDR picture in a decoding order, without referring to the pictures placed before the IDR picture in the decoding order; In other words, it is the I image in which the states necessary for decoding are reset. An IDR image corresponds to the top I image of a closed MPEG-2 GOP. A sequence in the MPEG-4 AVC begins with an IDR image. In the case of an I picture that is not an IDR picture, a picture placed after the I picture in the decoding order may refer to an image placed before the I picture in the decoding order. The respective image types will be defined hereinafter. An IDR image and an I image are the images that are composed only of I segments. A P image is the image that can be composed of P segments and I segments. A B image is the image that can be composed of B segments, segments. P and I segments. Note that the segments of an IDR image are

ES 2 388 397 T3 are stored in a NAL unit whose type is different from that of the NAL unit where the segments of a non-IDR image are stored. Here, a NAL unit is a sub-picture unit.

In an AU in the MPEG-4 AVC, not only the data necessary for decoding can be included, but also the supplementary information and the border information of the AU. Such supplemental information is called Supplemental Enhancer Information (SEI) and is unnecessary for decoding segment data. All data such as a PS set of parameters, segment data and a SEI, are stored in a unit of the Network Abstraction Layer (NAL), ie a NALU. A NAL unit is made up of a header and a payload. A header includes a field indicating the type of data to be stored (hereinafter referred to as the NAL unit type). The values ​​of the NAL unit types are defined respectively for data types such as a segment or an SEI. The reference to such a value of a type of NAL unit makes it possible to identify the type of data to be stored in the NAL unit. The header of a NAL unit includes a field called nal_ref_idc. A nal_ref_idc field is defined as a 2-bit field and takes a value of 0, 1 or more, depending on the types of NAL units. For example, the NAL unit of an SPS or a PPS takes the value 1 or more. In the case of the NAL unit of a segment, a segment referred to by the other segments takes the value 1 or more, while the segment not referenced takes the value 0. Likewise, the NAL unit of an SEI always takes the value 0.

One or more SEI messages can be stored in the NAL unit of an SEI. An SEI message consists of a header and a payload, and the type of information to be stored in the payload is identified by the type of an SEI message indicated in the header. Decoding an AU means decoding the segment data in an AU, and displaying an AU means displaying the result of decoding the segment data in the AU in the following.

Here, since a NAL unit does not include information to identify a NAL unit boundary, it is possible to add boundary information to the top of each NAL unit at the time of storing a NAL unit as an AU. In managing an MPEG-4 AVC stream in an MPEG-2 Transport Stream (TS) or an MPEG-2 Program Stream (PS), a start code prefix, displayed as 3 octets of 0x000001 , is added to the top of a NAL unit. It is also defined that a NAL unit indicating an AU boundary must be inserted on top of an AU in an MPEG-2 TS or PS, such as an AU called Access Unit Delimiter.

Conventionally, various types of techniques related, such as this one, to the coding of moving images have been proposed (For example, refer to Patent Document 1).

Patent Document 1: Japanese Patent Laid-Open Publication No. 2003-18549.

FIG. 4 is a block diagram of a conventional moving picture encoding apparatus.

The moving image encoding apparatus 1 is an apparatus that produces an encoded stream Str obtained by converting, through compression encoding, an input video InputV signal, to be input into a bit stream of a encoded stream of length variable, or similar. The moving picture coding apparatus includes a prediction structure determining unit PTYPE, a motion vector estimation unit ME, a motion compensation unit MC, a Subtract from subtraction unit, an orthogonal transform unit T, a quantization unit Q, an inverse quantization unit IQ, an inverse orthogonal transform unit IT, a Sum unit of sum, a MemImag image memory, a switch and a VLC variable length encoding unit.

The input video input V signal is input to the subtraction unit Subtract and the motion vector estimation unit ME. The Subtract From Subtraction unit calculates the differential value between the input video input signal V and the prediction image, and outputs it to the orthogonal transform unit. The orthogonal transform unit T converts the differential value into a frequency coefficient, and sends it to the quantization unit Q. The quantization unit Q performs the quantization on the entered frequency coefficient, and sends a quantization value Qcoef to the variable-length coding unit.

The inverse quantization unit IQ performs inverse quantization on the quantization value Qcoef to reconstruct the frequency coefficient, and sends it to the inverse orthogonal transform unit IT. The inverse orthogonal transform unit IT performs the inverse frequency transform to transform the frequency coefficient into a pixel differential value, and sends it to the Sum of Sum unit. The Sum of Sum unit adds the differential value of pixels to the prediction image, to be sent from the motion compensation unit MC, to make a decoded image. The switch SW is turned on when storing the decoded image is instructed, and the decoded image is stored in the image memory MemImag.

On the other hand, the motion vector estimation unit ME, into which an input video input V signal is input macroblock by macroblock, searches for the decoded image stored in the image memory MemImag, and estimates the image area that is closest to the input image signal, and consequently determines the motion vector MV indicating the position. The estimation of the motion vector is carried out block by block, the block being a segmented part of a macroblock. Since various images can be used as reference images at this time, identification numbers for specifying images to refer to (relative indexes) are needed block by block. It is possible to specify reference images by calculating the image numbers indicated by the relative indices, such image numbers being assigned to the respective images in a MemImag image memory.

The motion compensation unit MC selects the image area that is optimal as a prediction image from the decoded images stored in the image memory MemImag.

ES 2 388 397 T3

The prediction structure determination unit PTYPE instructs the motion vector estimation unit ME and the motion compensation unit MC to carry out intra-image coding on the target image, as a randomly accessible special image using its type Image TypeI, in the case where a random access unit boot image, InputRAU, indicates that the rAu random access unit starts with the current image, and instructs the VLC variable length encoding unit to encode the TypeI type of image.

The variable length encoding unit VLC performs variable length encoding on the quantization value Qcoef, the relative index Index, the image TypeI type and the motion vector MV to form an encoded stream Stream.

FIG. 5 is a block diagram of a conventional moving picture decoding apparatus 2. This moving image decoding apparatus 2 includes a variable length decoding unit VLD, an image MemImag memory, a motion compensation unit MC, a Sum-sum unit, an inverse orthogonal transform unit IT and an IQ unit. inverse quantization. Note that, in the figure, these processing units performing the same operations as the processing units in a conventional moving image encoding apparatus, as shown in the block diagram of FIG. 4, they are assigned the same reference numerals, and descriptions about them will be omitted.

The variable length _decoding unit VLD decodes a Stream encoded stream, and outputs the quantization value Qcoef, the Index relative index, the Image TypeI type, and the motion vector MV. The quantization Qcoef value, the relative index index, and the motion vector MV are input into the image memory MemImag, the motion compensation unit MC, and the inverse quantization unit IQ, respectively, and then the processing is performed. decoding on them. Such operations of a conventional moving picture coding apparatus have already been described using the block diagram of FIG. Four.

A random access unit RAU shows that decoding can be carried out starting with the upper AU in the random access unit. However, since a conventional MPEG-4 AVC stream allows very flexible prediction structures, a storage apparatus with an optical disk or a hard disk cannot obtain information to determine the AUs to decode or display at the time of recording. variable speed playback or reverse playback.

FIG. 6A and 6B are examples of AU prediction structures. Here, one image is stored in each UA. FIG. 6A is the AU prediction structure used in an MPEG-2 stream. Diagonally shaded figures in the figure are reference images for other UAs. In MPEG-2 the AUs of the P-pictures (P4 and P7) can perform the prediction encoding by referring only to a single AU selected as the AU of an I-picture or P-picture immediately later in the display time. Similarly, the AUs of the B pictures (B1, B2, B3, B5, and B6) can perform prediction coding by referring only to two selected AUs as a combination of the AUs of an immediately subsequent I-picture or P-picture and one I image or P image immediately preceding the display time. Furthermore, the order of the images to be placed in a stream is predetermined as follows: the AUs of an I image and the P images are placed in the order of the display time; and each of the B-image AUs is placed immediately after the I-image AUs or one of the P-images that is placed immediately after the B-image AUs. Consequently, decoding can be carried out in the following three ways: (1) all images are decoded; (2) only the AUs of an I-picture and P-pictures are decoded and displayed; and (3) only the AU of an I picture is decoded and displayed. Therefore, the following three types of playback can be easily carried out using: (1) normal playback, (2) medium speed playback, and (3) high speed playback.

In MPEG-4 AVC, prediction can be performed where the AU of a B picture refers to the AU of a B picture. FIG. 6B is an example of a prediction structure in an MPEG-4 AVC stream, and the AUs of B pictures (B1 and B3) refer to the AU (B2) of picture B. In this example, the following four types of decoding or display can be performed: (1) all images are decoded; (2) only the AUs, of an I-picture, P-pictures, and B-pictures, referred to, are decoded and displayed; (3) only the AUs of an I picture, and the P pictures are decoded and displayed; (4) only the AU of an I picture is decoded and displayed.

Furthermore, in MPEG-4 AVC, the AU of a P-picture can be referred to the AU of a B-picture. As shown in FIG. 7, the AU of a P-picture (P7) can be referred to the AU of a B-picture (B2). In this case, the AU of a picture P (P7) can be decoded only after the AU of a picture B (B2) has been decoded. Therefore, the following three types of decoding or display can be carried out: (1) all images are decoded; (2) only the AUs, of an I-picture, P-pictures, and B-pictures, referred to, are decoded and displayed; (3) only the AU of an I picture is decoded and displayed. Thus, since various prediction structures are allowed in the MPEG-4 AVC, the analysis of the segment data and the evaluation of the prediction structure must be performed to know the reference relationship between the AUs. This brings with it a problem: the UAs to be decoded or displayed cannot be determined based on a rule that is predetermined according to a playback speed at the time of skip playback, variable speed playback, and reverse playback, at Difference from the case of MPEG-2.

Document EP-A-756 281 discloses a moving image stream generation apparatus in which the position information of I and P frames, required for trick play modes, is recorded in an input sector formed in the part top of each GOP.

ES 2 388 397 T3

Disclosure of the invention

An object of the present invention is to provide (i) a moving image stream generating apparatus that can generate a moving image stream that can carry out trick play such as skip play, variable speed play and play inverse, even in the case of an encoding procedure such as MPEG-4 AVC, which allows flexible prediction structures

The present invention provides a moving picture stream generation apparatus, a moving picture stream generation method, a moving picture decoding apparatus, a moving picture decoding method, a human readable recording medium. computer, a recording method and a moving image decoding system, appropriately defined in the claims.

As described up to this point, with the present invention, the AUs to be decoded at the time of trick play, such as variable speed play and reverse play, can be determined by referring to a specific NAL unit in the upper AU of a random access RAU unit. Therefore, a moving picture decoding apparatus with excellent trick play function can be easily realized, and thus the present invention is highly practical.

Brief description of the drawings

These and other objects, advantages, and features of the invention will become apparent from the following description, taken in conjunction with the accompanying drawings, which illustrate a specific embodiment of the invention. In the drawings:

FIGS. 1A and 1B are diagrams showing, respectively, MPEG-2 stream structures in prior art;

FIGS. 2A and 2B are diagrams showing, respectively, MPEG-2 GOP structures in prior art;

FIGS. 3A and 3B are diagrams showing, respectively, MPEG-4 stream structures in prior art;

FIG. 4 is a block diagram showing the structure of a conventional coding apparatus;

FIG. 5 is a block diagram showing the structure of a conventional decoding apparatus;

FIGS. 6A and 6B are diagrams showing, respectively, examples of the prediction structure in a conventional MPEG-4 AVC stream;

FIG. 7 is a diagram showing another example of the prediction structure in a conventional MPEG-4 AVC stream;

FIGS. 8A and 8B are diagrams showing, respectively, MPEG-4 AVC stream structures of the present invention;

FIGS. 9A to 9D are diagrams of a first example showing AUs to be decoded in a random access unit RAU;

FIGS. 10A to 10D are diagrams of a second example showing AUs to be decoded in a random access unit RAU;

FIGS. 11A to 11C are diagrams of a third example showing AUs to be decoded in a random access unit RAU;

FIGS. 12A to 12F are diagrams of an example showing the procedure for specifying AUs to decode in a random access RAU;

FIG. 13A is a diagram showing a syntax example of a table indicating variable speed playback information, and FIG. 13B is a diagram showing a data storage unit;

FIG. 14 is a diagram of an example extension of a table indicating variable speed playback information;

FIGS. 15A to 15C are diagrams of an example showing the AUs of the I image and the P images in a random access unit RAU in the form of variable speed playback information;

FIGS. 16A to 16C are diagrams of an example where the buffer time is used as a priority indicator at the time of using AU priorities as variable speed playback information;

FIGS. 17A and 17B are diagrams respectively showing examples in which frame structure AUs and field structure AUs coexist in the respective RAUs; FIG. 17C is a diagram showing the syntax example of the first map (RAU_map1) showing the structure of each AU in the RAU; FIG. 17D is a diagram showing the RAU_map1 of the RAU of FIG. 17B; FIG. 17E is a diagram showing the RAU_map in terms of the random access RAU of FIG. 17B; FIG. 17F is a diagram showing

ES 2 388 397 T3 the syntax example of the second map (RAU_map2) showing the type of encoding of each frame or each image of a pair of fields;

FIGS. 18A to 18C are diagrams showing another exemplary map in the form of reproduction information;

FIG. 19 is a diagram of the procedure for indicating border information in a random access unit RAU;

FIGS. 20A and 20B are diagrams showing examples of image prediction structures in a random access RAU;

FIG. 21 is a block diagram showing the structure of a moving picture coding apparatus of the present invention;

FIG. 22 is a flow chart of a moving picture coding procedure;

FIG. 23 is a block diagram showing the structure of a moving image multiplexing apparatus of the present invention;

FIGS. 24A and 24B are diagrams showing exemplary HLP contents of the supporting information;

FIG. 25 is a diagram showing an example of a NAL unit in which trick play information is stored in the HLP of the support information;

FIG. 26 is a flow chart showing the operation of a moving image multiplexing apparatus;

FIG. 27 is a block diagram showing the structure of a moving picture decoding apparatus of the present invention;

FIG. 28 is a flow chart of a conventional image decoding procedure;

FIG. 29 is a flow chart of determining the AUs to decode in the moving picture decoding method of the present invention;

FIG. 30 is a flow chart showing the processing performed in the case in which the AUs to be decoded do not match the AUs to be displayed in the moving picture decoding method of the present invention;

FIG. 31 is a diagram showing a data hierarchy of an HD-DVD drive;

FIG. 32 is a structural diagram of the logical space on an HD-DVD drive;

FIG. 33 is a structural diagram of an information file of a video VOB object;

FIG. 34 is a diagram of a temporal map;

FIG. 35 is a structural diagram of a playlist file;

FIG. 36 is a structural diagram of a program file corresponding to the playlist;

FIG. 37 is a structural diagram showing a management information file of the entire Blue-Ray BD disc;

FIG. 38 is a structural diagram of a file for recording a global event manager;

FIG. 39 is a block diagram showing the outline of an HD-DVD player; and FIGS. 40A to 40C are diagrams showing a recording medium for storing the program to perform the moving picture encoding method and the moving picture decoding process of the present invention

Best mode of carrying out the invention

An embodiment of the present invention will be described hereinafter with reference to the figures.

(Structure of an AVC stream)

First, the structure of an AVC stream to be generated by a moving image stream generation apparatus, a moving image encoding apparatus and a moving image multiplexing apparatus of the present invention will be described, said of otherwise, an aVc stream to be input into a moving picture decoding apparatus of the present invention.

FIG. 8A and FIG. 8B show, respectively, the AVC stream structures of the present invention. Note that the border information to add to the top of a NAL unit is not shown in the figure. The AVC stream differs from a conventional AVC stream in that trick play information is added, the trick play information indicating the AUs to decode at the time of trick play, such as skip play, variable speed play and reverse playback. The information of 7

ES 2 388 397 T3 trick play is stored in a NAL unit for storing play information (FIG. 8A). In MPEG-4 AVC, the relationship between the information to be stored and the NAL unit type of a specific NAL unit can be set by application. More specifically, the values ​​from 0 and 24 to 31 can be used, and these NAL unit types are called user-adjustable NAL unit types. Accordingly, the trick play information is stored in the NAL unit having such user adjustable NAL unit types. Here, in the case where the specific NAL Unit types are reserved for storing the information other than the trick play information, the NAL unit types that are different from the NAL unit types are assigned to the trick play information. . The nAl units of trick play information are stored in the top AU of a random access RAU unit. Such a NAL unit is placed immediately after a PPS NAL unit, if present, in an AU, but can be placed in another position as long as the order satisfies the requirement of the MPEG-4 AVC or other standard. Similarly, in the case where it is impossible to interpret the trick play information unit NAL, the data of the NAL unit can be skipped and decoding restart from the top of the next NAL unit. Therefore, even a terminal that cannot interpret the trick play information unit NAL can carry out the decoding processing without fail.

Note that such a trick play information unit NAL can be included, not in the top AU of a random access unit RAU, but in another AU such as the last AU. Similarly, such a trick play information unit NAL may be included in each AU that constitutes a random access unit RAU.

FIG. 9 to 11 show examples of the AUs to be decoded at the time of variable speed playback. FIG. 9A shows the display order of UAs. Here, the diagonally shaded UAs are the UAs that other UAs refer to, and the arrows show referenced images. Negative reference numbers are assigned to AUs to display before 10, and positive reference numbers are assigned to AUs to display after B15. FIG. 9B shows the decoding order of the AUs shown in FIG. 9A, and 10 to B11 constitute a random access RAU. At this time, I0, -B14, P4, B2, P8, P6, P12 and B10 are decoded to perform double speed playback (FIG. 9C), while I0, P4, P8 and P12 are decoded to carry perform quad speed playback (FIG. 9d). FIG. 9C and 9D show that AUs with an * sign are to be decoded at the time of double speed playback and quad speed playback, and these pieces of information are stored in the trick play information unit NAL. In the example of FIGS. 10A to 10D, the pictures 10 to B11 in the decoding order constitute a random access unit RAU. Here, 10, -B13, P3, B1, P6, B4, P9, B7, P12, and B10 are decoded to perform 1.5-speed playback while 10, p3, P6, P9, and P12 are decode for triple speed playback. Similarly, in the example of FIGS. 11A to 11C, I0, P3, P6, P9 and P12 are decoded to perform triple speed playback.

The playback speeds are not required to be exact here because they are described as guidelines for playback speeds. For example, in the example in fIg. 11C, in the case where all the AUs shown as the AUs to be decoded at the time of triple speed playback are decoded, the speed is 3.2 times higher than that obtained from the expression: 16: 5; In other words, it's not exactly triple speed. Similarly, at the time of reproduction multiplied by M, in the case in which the smallest value on M is N among the reproduction speeds shown as trick play information, it is possible to decode the AUs necessary to be decoded at the time of Playback multiplied by N and determine how the rest of the AUs should be decoded, according to the implementation of the decoding apparatus. Similarly, it is possible to place high priorities on the AUs needed to be decoded in the case where the reproduction speed is fast, and determine the AUs to decode on the basis of the priorities.

Note that some AUs, among the AUs to be decoded at the time of variable speed playback, may not be displayed. For example, the Nth AU is displayed at the time of double speed playback, but the Mth AU is not displayed. At this time, in the case where there is a need to decode the M-th AU in order to decode the N-th AU, the M-th AU is decoded but not displayed at the time of double speed playback. .

Next, the procedure for specifying the AUs to be decoded at the time of variable speed playback will be described with reference to FIGS. 12A to 12F. FIG. 12A to 12F show the specification examples of the AUs to be decoded in the same random access unit RAU as that of FIG. 9. As shown in FIG. 12D, I0, -B14, P4, B2, P8, P6, P12, B10 are decoded at the time of double speed playback. These AUs are the first, second, fifth, sixth, ninth, tenth, thirteenth, and fourteenth AUs when counting the AUs from the top AU of the random access unit RAU. In this way, it is possible to uniquely specify the AUs to be decoded at the time of variable speed playback, displaying the ordinal numbers of the AUs in a random access unit RAU. An access unit delimiter is securely placed on top of an AU when multiplexing an AVC stream by an MPEG-2 transport stream (TS). When data is obtained from AUs to be decoded at the time of variable speed playback, the access unit delimiters are searched in sequence for the limits of the AUs. This form of lookup processing eliminates the need to analyze the payload of NAL units, such as segment data, and is therefore easier.

Note that it is possible to specify the AUs to be decoded by determining that the AUs to which other AUs will refer, such as the AUs of an I-picture and P-pictures (such AUs to be referred to are called reference UAs), are decoded at the time of variable speed playback, and specifying the ordinal numbers of the reference UAs in a random access RAU. In the random access unit RAU of FIG. 12B, as shown in FIG. 12C, I0, -B14, P4, B2, P8, P6, P12, B10, are AU for reference. Similarly, at the time of double speed playback, I0, -B14, P4, B2, P8, P6, P12, B10 are decoded, but when these AUs are indicated in the order of the reference AUs, they correspond to the first , second, third, fourth, fifth, sixth, seventh and eighth reference AU, as shown in FIG. 12F. Whether or not an AU is an AU

Reference ES 2 388 397 T3 can be evaluated by referring to a specific field in the header of the NAL unit in a segment. More specifically, in the case where the value of a nal_ref_idc field is not 0, the UA is a reference UA. Note that a reference Au to be decoded can be specified on the basis of a frame number, because it is possible to identify a reference AU on the basis of a frame number.

Furthermore, it is possible to specify the UAs to be decoded by specifying the offset value equivalent to the length in octets, from the start position of the upper UA of a random access RAU to the start position of the UA to be decoded. For example, in FIGS. 12A to 12F, in the case where I0 starts with the position away from the top of a stream by 10,000 bytes, and P4 starts with the position away from P4 by 20,000 bytes, the offset value to P4 is 10,000 octets, obtained from the expression: 20,000 - 10,000. In the case where a multiplexed stream is used in an MPEG-2 TS, it is possible to specify an offset value that includes the overhead of a TS packet or a PES packet (Packet Elementary Stream), or You can specify an offset value that includes it at the time of per-application data fill. Similarly, it is possible to specify an AU by a frame number FN.

Note that in the case of using multiplexed stream in MPEG-2 TS, it is possible to specify AUs based on the number of TS packets from (i) the TS packet to store the index number and the address information for identifying a TS packet including the upper data of the AUs to be decoded, or the upper data of the random access unit RAU up to (ii) the current TS packet. Here, it is possible to use the information about the Source Package to be used for a Blu-ray Disc (BD) recording format instead of a TS package. The Origin Packet is obtained by adding, to a TS packet, a 4-octet header that includes temporal information of the TS packet, copy control information and the like.

FIG. 13A is an example of table syntax indicating information for variable speed playback. In the syntax, nro_imag_en_RAU shows the number of AUs that make up a random access RAU, nro_speed shows the number of playback speeds at which UAs are to be decoded, playback_rate shows a playback speed, nro_imag_desc shows the number of playback speeds. AU to be decoded at the time of playback at a playback speed shown in playback_speed, imag_desc shows the ordinal numbers of the UAs to decode in the case of counting UAs from the upper UA in a random access RAU. FIG. 13B is an example in the case of storing information about the AUs to be decoded in a random access unit RAU shown in FIGS. 9A to 9D at the time of double speed playback and quad speed playback. Note that nro_imag_en_RAU is used when calculating an exact playback speed based on the number of AUs to decode and the total number of AUs in a random access RAU unit, or to jump based on random access RAU units in sequence. However, nro_imag_en_RAU can be omitted because the same information can be obtained by searching for the top AUs of the Random Access RAUs. Similarly, a field indicating the size of a table can be added to the table. Note that, in the syntax example of FIG. 13A, the ordinal number of an AU to be decoded, counting from the top of a random access RAU unit, is displayed directly, but whether or not it is necessary to decode each AU can be displayed by turning the corresponding bits for each AU on or off. For example, a random access RAU is made up of 16 AUs in the example of FIGS. 9A to 9D, and 16 bits are needed when 1 bit is assigned to an AU. At the time of quad-speed playback, the first, fifth, ninth, and thirteenth AUs are shown to be decoded by assigning 16-bit information that is represented as 0b1000100010001000 (0b indicates a binary number). Here, the upper bit and the last bit correspond, respectively, to the upper AU and the last AU of a random access RAU.

Note that the size of a table is variable in the syntax example of FIG. 13A. The maximum value of the table size is determined in the case where the maximum value of the number of AUs that constitute a random access unit RAU and the maximum value of speed_no are prescribed. Accordingly, it is possible to set the size of the table to the determined maximum value, and in the case where the size of the information for variable speed playback does not reach the maximum value, it is possible to carry out padding. Setting the size of the table in this way makes it possible to always obtain the data of a fixed size when the playback information is obtained at variable speed, which enables to speed up the information acquisition processing. Note that the table size, or the size of a NAL unit to store the table, is displayed as management information. Similarly, it is possible to predetermine the size of a NAL unit for storing trick play information and, in the case where the information cannot be stored in a single NAL unit, it is possible to store the information for variable speed playback in various NAL units per separated. At this time the padding is performed on the payload of the last NAL drive, so that the size of the NAL drive becomes the default size. Similarly, some prescribed values ​​are determined as the table size values, and the index number indicating a prescribed table size value can be displayed in the table, or use application management information.

Similarly, it is possible to display differential information instead of listing all AUs to be decoded at each playback speed. As the information at the time of reproduction at a rate multiplied by M (<N), only the AUs needed to decode are shown in addition to the AUs to be decoded at the time of reproduction at a rate multiplied by N. In the example of FIG. 13B, such as the second, sixth, tenth and fourteenth AU, in addition, of the AUs to be decoded at the time of quad-speed playback, they are decoded at the time of double-speed playback, it is possible to display only the second, sixth , 10th and 14th AU as the information for double speed playback.

Note that the AUs necessary to decode at the time of variable speed playback are shown in the above description, but in addition, it is possible to display the information indicating the display order of the AUs necessary to decode. For example, the information at the time of double speed playback and quad speed playback is shown in the example of FIGS. 9A to 9D, but there is an example of this triple speed random access RAU playback. The display of a part of the UA to be displayed at the time of double speed playback, in addition to the UA to be displayed at the time of playback.

ES 2 388 397 T3 quad speed playback, enables triple speed playback. Here, when considering the case where one or more AUs between 10 and P4 are displayed, which are to be displayed at the time of quad-speed playback, the information for double-speed playback shows that the candidates are -B14, B2, B6 and B10. However, the display order of these four UAs can be obtained only in the case where the header information of a segment is parsed. Here, since the information in the display order shows that only -B14 is displayed between I0 and P4, it is possible to determine that -B14 is decoded. FIG. 14 is an example of a syntax indicating the display order information, and is obtained by adding the display order information to the syntax of FIG. 13A. Here, ind_pts_dts shows whether or not the decoding order of the UAs to be decoded at the playback speed matches the display order of the UAs, and only in the case where the decoding order does not match the display order, display order information is displayed in a display_order field.

Note that, in the case of playback at a playback speed that is not displayed by the variable rate playback information, it is possible to determine the AUs to be decoded and the AUs to be displayed based on the rule that is predetermined in the terminal. For example, in the case of triple speed playback in the example of FIG. 9, it is possible to display I0, B3, B6, B9 and P12 in addition to the AUs to be displayed at the time of quad speed playback, instead of displaying a part of the AUs to be displayed at the time of double speed playback. Here, as in the B pictures, the B pictures in the reference AUs can preferably be decoded or displayed.

Similarly, there is a case where trick play, such as variable speed play, is performed by playing only the AU of an I-image or only the AUs of an I-image and P-images. Therefore, a playlist can be stored. an I image and P images as trick play information. FIG. 15A to 15C show another example. Here, the images from 10 to B14 are included in a random access RAU as shown in FIG. 15B, and among those, the AUs of an I image and P images are I0, P3, P6, P9, P12, and P15, as shown in FIG. 15C. Therefore, the information to identify I0, P3, P6, P9, P12 and P15 is stored. At this time, it is possible to add the information to distinguish the UA of an I image from the UA of a P image. Likewise, it is possible to display the information to distinguish the following images from each other, including the images: an I image, P images, B images to serve as reference (hereinafter referred to as reference B images), and B images to which no reference will be made (hereinafter referred to as non-reference B pictures).

Furthermore, it is possible to store the priority information of the respective AUs as trick play information, and decode or display the AUs according to the priorities at the time of variable speed playback. Image types can be used as priority information. For example, UA priorities can be assigned in the following order of enumeration: (i) an I-picture; (ii) P images; (iii) reference B images; and (iv) non-reference B images. Similarly, it is possible to set priority information as follows: the longer the time between the time after the decoding of an AU and the time of displaying the AU, the higher the priority becomes. FIG. 16A to 16C show an example of setting priorities according to buffer time. FIG. 16A shows the prediction structure of the AUs and P3 is also referenced by B7 and P9. At this time, in the case where the random access unit RAU is composed of AUs from I0 to B11 (FIG. 16B), the buffer time of each AU is as shown in FIG. 16C. Here, the buffer detection time is displayed based on the number of frames. For example, P3 is required until P9 is decoded, and the buffer time must be equal to six pictures. Therefore, the decoding of the AUs whose buffer time is 3 or more means the decoding of the entire I-picture and all the P-pictures, and triple speed playback is performed. Here, the buffer time of P3 is longer than that of I0, but it is possible to add an offset value to the AU of image I to place the highest priority over the AU of image I. Similarly, it is possible to place high priorities on the AUs needed to decode at the time of high-speed playback and to use, as priority information, N on the AUs needed to decode at the time of the N-rate playback. Note that In the case where one AU is referred to in other AUs after being decoded or displayed, it is possible to display the period of time during which the AU is referred to.

Note that trick play information can be stored in a SEI message (FIG. 8B). In this case, the SEI message type is defined for the trick play information, and the trick play information is stored in the SEI message of the defined type. The SEI message for the trick play information is stored in the NAL unit of the SEi, alone or together with other SEI messages. Note that it is possible to store trick replay information in SEI message user_data_registered_itu_t_t35 or SEI message user_data_unregistered, which are SEI messages for storing user-defined information. At the time of using these SEI messages, it is possible to show that the trick play information is stored, or the type of trick play information in the payload part of an SEI by adding identifying information of the information to be stored.

Note that it is possible to store trick play information in UAs other than the upper UA in a random access RAU. Likewise, it is possible to predetermine the values ​​to identify the AUs needed to decode at the time of reproduction at a specific reproduction speed and add the values ​​determined for each AU. For example, regarding AUs to be decoded at a reproduction rate that is a rate multiplied by N, or less, N is given as reproduction rate information. Likewise, it is possible to show the following in nal_ref_idc, and the like, of the NAL unit of a segment: the image structure in an AU, the structure being a frame structure or a field structure and, in addition, in the case in If the image has a field structure, it is possible to show the type of field, which is an upper field or a lower field. For example, since there is a need to alternately display the upper fields and the lower fields in the case of interlaced display, it is desirable that it can be easily assessed that the field to be decoded next is an upper field or a lower field at the time of decode fields, skipping some fields at playback time

ES 2 388 397 T3 at high speed. In the case where the field type can be evaluated from the header of a NAL unit, there is no need to analyze the segment header and the amount of processing required for such evaluation can be reduced.

Note that the information indicating whether each AU that constitutes a random access RAU is a field or a frame can be stored in the upper AU of a random access RAU. Likewise, it is possible to easily determine the AUs to be decoded at the time of trick play, even in the case where a field structure and a frame structure coexist, by storing such information in the upper AU of the random access unit. FIG. 17A and 17B are examples where the AUs having a frame structure and the AUs having a field structure coexist in the random access unit RAU, and shows the display order of the AUs and the decoding order of the AUs, respectively. The following images are encoded, respectively, as field pairs: B2 and B3; I4 and P5; B9 and B10; B11 and B12; P13 and P14; B15 and B16; B17 and B18; and P19 and P20. Similarly, the other AUs are coded as the AUs that have a frame structure. At this time, in the case of reproducing only the AUs of an I-picture and P-pictures, the following can be decoded and reproduced in the following numbered order: the field pair of I4 and P5; the plot of P8; the field pair of P13 and P14; and the field pair of P19 and P20. However, the addition of such information is effective because there is a need to evaluate whether each AU is one of the fields that make up a pair of fields or whether each AU is a frame at the time of determining the AUs to be decoded.

FIG. 17C is an example of first map syntax (RAU_map1) indicating whether an AU in a random access RAU is a frame or a field. The number of AUs that constitute a random access unit is displayed in #AU_en_RAU, and the information about each AU is displayed in the next loop in a decoding order. Here, frame_field_flag shows whether the image to be stored in an UA is a frame or a field. Similarly, type_imag shows information about the type of encoding of an image. The types of encoding that can be displayed include: an I image; an IDR image; an image P; a reference image B; a non-reference image B; and the like. Therefore, it is possible to determine the images to be decoded at the time of trick play by referring to this map. Note that it is possible to indicate whether each I image and each P image are reference objects or not. In addition, it is possible to indicate the information to evaluate whether a predetermined requirement is applied regarding the prediction structures.

FIG. 17D shows RAU_map1, which refers to a random access RAU of FIG. 17B. Here, the image_types of an I image, P images, reference B images, and non-reference B images are 0, 1, 2, and 3, respectively. Here, it is possible to store the information that indicates the types of image encoding on the bases listed above, because the images are reproduced frame by frame, or a pair of fields by pair of fields, at the time of trick playback.

FIG. 17F is an example of second map syntax (RAU-map2) that indicates frame-by-frame or field-to-field pair image encoding types. Here, frame_number_in_RAU shows the number of frames that make up a random access RAU and the number of field pairs. Similarly, frame_flag shows whether an image is a frame or not, and in the case where it is a frame, you put 1 there. In the case where 1 is set in frame_indicator, the information about the encoding type of a frame is displayed in frame_type. In the case where 0 is set in frame_indicator, in other words, the image is one of a pair of fields, and the type of encoding of each field that constitutes the pair of fields is shown in type_par_fields.

FIG. 17E shows the RAU_map2 for the random access RAU of FIG. 17B. In FIG. 17E, the values ​​indicating the frame_type of an I-picture, P-pictures, reference B-pictures, and no-reference B-pictures are 0, 1, 2, and 3, respectively. Likewise, the field_par_type displays the type of each field in decoding order. The types of fields are as follows: I for an I image; P for P images; Br for reference B images; and Bn for non-reference images. For example, it is displayed as IP in the case where the first field is an I image and the second field is a P image, and it is displayed as BnBn in the case where the first field and the second field are non-reference B images . Here, the values ​​for indicating combinations of IP, PP, PI, BrBr, BnBn and the like are previously set. Note that the following information can be used as the information indicating the type of encoding of a pair of fields: information as to whether the pair of fields includes an I-picture or one or more P-pictures; information as to whether the field pair includes one or more reference B pictures; and information as to whether the field pair includes one or more non-reference B images.

For example, the trick play information may be the map of a random access unit RAU, as a syntax shown in FIG. 18. This map includes image_structure, which indicates the structure of each of the images included in the random access RAU, and image_type, which indicates the type of image. As shown in FIG. 18B, image_structure shows the structure of each image, that is, a field structure or a frame structure, and the like. Likewise, as shown in FIG. 18c, picture_type shows the picture type of each picture, that is, an I picture, a reference B picture, a non-reference B picture, and a P picture. Thus, the moving picture decoding apparatus received This map can easily identify the UAs on which trick play is performed by referring to this map. As an example, it is possible to decode and play back, in high speed playback, only an I picture and P pictures, or reference B pictures, in addition to an I picture and P pictures.

Note that in the case where the information indicating the image structure, such as a 3-2 downstream, is included in an AU that constitutes a random access unit RAU, it is possible to include the information indicating the image structure in the first or second map previously described. For example, it is possible to show if each image has visualization fields equivalent to three images or if each image has visualization fields equivalent to two images, Also, in the case where it has visualization fields equivalent to three images, it is possible to display the information A that indicates whether the first field is displayed repeatedly, or the information that indicates whether the first field is a superior field. Likewise, in the case where you have visualization fields equivalent to two images, it is possible to show the information of whether the first field

ES 2 388 397 T3 is a higher field. Here, in MPEG-4 AVC, whether or not an image has an image structure, such as a 3-2 downstream, can be displayed using (i) the pic_struct_present_flag of a set (SPS) of sequence parameters or ( ii) the picture_to_display_conversion_flag, and the like, in the AVC timing and the HRD descriptor that is defined in the MPEG-2 system standard. Also, the structure of each image is displayed by a pic_struct field of an Image Timing SEI. Therefore, it is possible to display the image structure by setting a flag only in the case where a pic_struct field has a specific value, for example, an image has display fields equivalent to three images. In other words, indicating the following three types of information regarding each image is effective (i) in the case where the skip playback is performed in the middle of a random access RAU unit and (ii) when determining the field to display at a specific time or the frame in which a field is stored. The same can be said in the case of determining images to be displayed during variable speed playback. The three types of information are:

(i) field (ii) frame (used when not using a 3-2 downward advance, or used also when using a 3-2 downward advance. In the latter case, the plot has display fields equivalent to two images).

(iii) frame that has a field of view equivalent to three images at the time of using a 3-2 downward advance

Note that these types of information can be indicated in the image_structure of a RAU map shown in FIG. 18.

Indicating image type list information of the respective images that make up a RAU in this way makes it possible to easily determine images to decode or display at the time of trick play, such as variable speed play, fast play. jumps and reverse playback. This is especially effective in the following cases:

(i) where only an I-image and P-images are reproduced;

(ii) where high-speed reproduction is performed, in which there is an I-image, P-images, and B-images for reference; and (iii) where the images, on which requirements regarding prediction structures are established, are identified on the basis of the image types, the necessary images to decode are selected at the time of trick play, and the Selected images are played back in trick play.

Furthermore, it is possible to store a default value of trick play information in a region, which is different from AVC stream, such as application level management information, and include trick play information in a random access RAU. only in the case where the trick play information is different from the trick play information displayed by the default.

Trick play information in terms of variable speed playback has been described above, but it is possible to use similar information as supplementary information at the time of reverse playback. It is possible to complete decoding in one moment, at the time of reverse playback in the case where all the images to be displayed can be stored in a memory, and the processing load required for decoding can be reduced. Considering an embodiment of inverse reproduction in the enumerated order of P12, P8, P4 and I0 in the example of FIGS. 9A to 9D, provided that all the decoding results of the four AUs are stored, it is possible to decode I0, P4, P8 and P12 in this order at a time and perform reverse playback. Therefore, it is possible to evaluate whether or not all the decoded data of the AUs can be stored based on the number of the AUs to be decoded or displayed at the time of playback, at a rate multiplied by N, and determine the AUs to be displayed. at the time of performing the reverse reproduction based on the evaluation result.

Similarly, trick play information can be used as supplemental information at the time of skip play. Here, skip playback means fast-forwarding a motion picture and performing normal playback of the beginning of motion pictures with the randomly determined position. Determining fast-forward images using such supplementary information, even at the time of skip playback, makes it possible to determine the image at which skip playback starts.

Note that the AU to serve as a reference for each AU that constitutes a random access unit can be displayed directly in the faked information. In the case where there are multiple reference UAs, all of them are displayed. Here, in the case where a reference AU belongs to a random access unit other than the random access unit that includes an AU that refers to the reference AU, the AU can be indicated in the following specific way: the M -th AU of the random access unit that is placed before or after N numbers of random access units, or the AU can be indicated in the following simple way: the AU belonging to the random access unit that is placed before or after N numbers of random access units. Note that it is possible to display the ordinal number, in the decoding order, of the reference UA in the case of counting from the AU that refers to the reference UA. At that time, UAs are counted based on one of the following: all UAs; the reference UAs; the AUs of a specific image type such as I, P, and B. Similarly, it is possible to show that each AU can refer to AUs of only up to N AU numbers before and after, in a decoding order. Note that, in the case of referring to an AU that is not included in the AUs of up to N AU numbers before and after in the decoding order, it is possible to add the information indicating the fact.

ES 2 388 397 T3

Note that it is possible to use the above-described trick play information in a similar way also in a multiplexing format, such as MP4, where the size of a NAL unit is used instead of using a start code prefix as border information. of a NAL unit.

Note that at the time of receiving and recording an encoded stream that is packetized using an MPEG-2 TS (Transport Stream) packet, or an RTP (Real Time Transmission Protocol), packet loss occurs. In this way, in the case of recording the data received in an environment in which packet loss occurs, it is possible to store, in an encoded stream as supplementary information, or as management information, the information indicating that data in a stream has been lost due to packet loss. It is possible to show a data loss due to the loss of a packet by inserting the flag information that indicates whether or not the flow data has been lost, or a special error notification code to notify the lost part. Note that in the case of performing error concealment processing when data is lost, it is possible to store identifying information indicating the presence / absence, or procedure, of error concealment processing.

The trick play information for determining the AUs to decode or display at the time of trick play has been described up to this point. Here, the data structure to allow detection of the border of the random access RAUs will be described with reference to FIG. 19.

In the upper AU of a random access RAU, the NAL unit of an SPS to be referenced by an AU constituting a random access RAU is always stored. On the other hand, in the MPEG-4 AVC standard, it is possible to store the NAL unit of the SPS to be referenced by the Nth AU in a decoding order, in an AU that is arbitrarily selected from the Nth AU or AUs placed before the N-th AU in a decoding order. Such a NAL unit is stored so that the NAL unit of an SPS can be transmitted repeatedly, in preparation for the case where the NAL unit of an SPS is lost due to packet loss at the time of transmitting a stream in communication or broadcast. . However, the following rule is effective for the use of storage applications. Only a single NAL unit of the sPs to be referenced by all the AUs of the random access RAU unit is stored in the top AU of a random access RAU unit, and the NAL unit of the SPS is not stored in the following AUs in the random access unit. Doing so makes it possible to ensure that the AU is the top AU of the Random Access RAU if it includes the NAL of an SPS. The start of the Random Access RAU drive can be found by searching for the SPS NAL drive. The management information of a stream such as a temporal map does not guarantee the provision of access information as to all the random access RAUs. Therefore, it is especially effective that the starting position of each random access RAU unit can be obtained by searching for the NAL unit of an SPS in a stream in the case of, for example, performing skip playback on the image in the middle. of the random access RAU whose access information is not provided.

Here, in the case where the top AU of the random access RAU is the AU of an IDR image, the AU of the random access RAU does not refer to the AU in the previously placed random access RAU. in a decoding order. This type of random access RAU is called a closed type random access RAU. On the other hand, in the case where the top AU of a random access RAU is the AU of an I image that is not an IDR image, the AU of the random access RAU may refer to the AU in the RAU. random access that is previously placed in a decoding order. This type of random access RAU unit is called open type random access rAu unit. At the time when the angles are switched during playback on an optical disc or the like, the switching is performed from a closed-type random access RAU. Therefore, it is effective that the evaluation as to whether a random access RAU is an open type or a closed type random access RAU can be done on top of the random access RAU. For example, it is possible to display the indicator information to evaluate the type, that is, an open type or a closed type, in a nal_ref_idc field of the NAL unit of an SPS. Since the value of nal_ref_idc is defined to be 1 or more in the NAL unit of an SPS, the high-order bit is always set to 1 and the flag information is displayed by the low-order bit. Note that an AU in a random access RAU cannot refer to an AU in a random access RAU that is previously in a decoding order, even in the case where the upper AU is the AU of an I image that it is not an IDR. This type of random access RAU unit can be considered as a closed type random access RAU unit. Note that the indicator information can be displayed using a field other than nal_ref_idc.

Note that it is possible to specify the starting position of a random access RAU based on the NAL unit other than an SPS, to be stored only in the upper AU of a random access RAU. Similarly, it is possible to display the type, that is, the open type or the closed type, of each of the random access RAUs, using the nal_ref_idc field of each random access RAU.

Finally, FIGS. 20A and 20B show examples of prediction structures of the AUs that constitute a random access unit RAU. FIG. 20A shows the positions of the UAs in a display order, and FIG. 20B shows the positions of the AUs in the decoding order. As shown in the figures, B1 and B2, shown before I3, which is the top AU of a random access RAU, can refer to the AUs to be displayed after I3. In the figure, B1 refers to P6, Here, in order to ensure that the AUs of I3 and the following images in the display order can be decoded correctly, the AUs of I3 and the following images are prohibited in the display order refer to AUs before I3 in display order (Motion picture encoding apparatus)

FIG. 21 is a block diagram of the motion picture coding apparatus 100 that performs the motion picture coding method of the present invention. This moving image coding apparatus 100 generates a coded stream, shown in FIGS. 8 to 20, of an image in

ES 2 388 397 T3 motion that can be played using trick play such as skip play, variable speed play, and reverse play. The motion picture coding apparatus 100 includes a trick play information generation unit TrickPlay, in addition to the units of a conventional motion picture coding apparatus 1 shown in FIG. Four. Note that the processing units that perform the same operations as the processing units of a conventional moving image encoding apparatus, shown in the block diagram of FIG. 4, the same reference numbers are assigned in the figure, and their descriptions will be omitted.

The TrickPlay unit for generating trick play information is an example of a unit that generates, on the basis of a random access unit that includes one or more images, supplementary information to be referenced at the playback time of the units. random access. The trick play information generation unit TrickPlay generates trick play information based on Type I types of images, and reports the trick play information to the variable length encoding unit VLC.

The variable length encoding unit VLC is an example of a stream generation unit that generates a stream including supplementary information and images, adding the generated supplementary information to each corresponding random access unit. The variable length encoding unit VLC encodes and positions the NAL unit to store trick play information in the upper AU of a random access RAU unit.

FIG. 22 is a flow chart of how the motion picture encoding apparatus 100 (mainly the trick play information generation unit TrickPlay) shown in FIG. 21 performs the procedure of generating an encoded stream that includes trick play information.

First, in step 10, the moving picture encoding apparatus 100 evaluates whether or not the AU to be encoded is the top AU of a random access unit RAU. In the case where it is the upper AU, it goes to Stage 11, while in the case where it is not the upper AU, it goes to stage 12. In step 11, the motion picture encoding apparatus 100 performs the initial processing to generate trick play information from the random access unit RAU and also secures the area for storing the trick play information in the upper AU of the random access RAU unit. In Step 12, the motion picture encoding apparatus 100 encodes the AU data, and then proceeds to Step 13. In Step 13, the motion picture encoding apparatus 100 obtains the necessary information at the time of generating trick play information. Such information is: the AU image types, that is, an I image, a P image, a reference B image, or a non-reference B image; or if there is a need to decode the AU at the time of playback at a rate multiplied by N. After this, the moving image encoding apparatus 100 proceeds to Step 14. In Step 14, the moving image encoding apparatus 100 assesses whether the AU is the last AU of the random access unit RAU. In the case where it is the last AU, the moving picture encoding apparatus 100 proceeds to Step 15, while in the case where it is not the last AU, it proceeds to Step 16. In Step 15, the motion picture encoding apparatus 100 determines trick play information, generates the NAL unit to store the trick play information, and stores the generated NAL unit in the area secured in Step 11. After completion processing in Step 15, the moving picture encoding apparatus 100 proceeds to Step 16. In Step 16, the moving picture encoding apparatus 100 evaluates whether or not there is an AU to be encoded next. In the case where there is an AU to be encoded, it repeats Step 10 and the following steps, while in the case where there is no AU to be encoded, it completes the processing. Here, in the case where the motion picture encoding apparatus 100 assesses that there is no AU to encode in Step 16, it stores trick play information of the last random access unit RAU, and then completes the processing.

For example, when the motion picture encoding apparatus 100 generates trick play information shown in FIG. 18A, obtains the following in Step 13: the image type; if the image has a field structure or if the image has a raster structure; or / and the information indicating whether the field of view of the image is equivalent to two images or equivalent to three images in the case where the information regarding a 3-2 downward advance is included in the encoded stream. In Step 15, the moving picture encoding apparatus 100 sets the picture_structure and the picture_type of all pictures in the random access unit RAU in a decoding order.

Note that, in the case where the size of the NAL unit for storing trick play information is not known at the time of initiating the encoding of the upper AU of a random access RAU, the processing to secure the area for storing the trick trick play information will be skipped in the Stage

eleven. In this case, the NAL unit generated for storing trick play information is inserted into the upper Au in Step 15.

Likewise, storing or not storing trick play information can be toggled on a coded stream basis. Especially in the case where the prediction structure between the AUs that constitute a random access unit is prescribed by application, it is possible to determine that the trick play information is not stored. For example, in the case where an encoded stream has the same prediction structure as an MPEG-2 stream, there is no need to store trick play information. This is due to the fact that it is possible to determine the AUs needed to decode at the time of trick play without trick play information. Note that such a switchover can be carried out on the basis of a random access RAU.

(Motion picture multiplexing apparatus)

ES 2 388 397 T3

FIG. 23 is a block diagram showing the structure of the moving image multiplexing apparatus 108 of the present invention. This moving picture multiplexing apparatus 108 inputs moving picture data, encodes the moving picture data to compose an MPEG4 AVC stream, multiplexes the stream with the access information to the AUs that constitute the stream and the information of management that includes the supplementary information to determine operations performed at the time of trick play, and records the multiplexed stream. The moving picture multiplexing apparatus 108 includes a stream attribute determining unit 101, an encoding unit 102, a management information generating unit 103, a multiplexing unit 106, and a storage unit 107. Here, the encoding unit 102 has a function for adding trick play information to the motion picture encoding apparatus 100 shown in FIG. twenty-one.

The stream attribute determining unit 101 determines the requirements concerning the trick play performed at the time of encoding of an MPEG-4 AVC stream, and sends them to the encoding unit 102 and the information generation unit 105 playback support as TYPE attribute information. Here, the requirements concerning trick play include information indicating: whether or not the requirement to constitute a random access unit applies to an MPEG-4 AVC stream; whether or not the information indicating the AUs to be decoded or displayed at the time of variable speed playback or reverse playback is included in the stream; or whether or not there is a requirement on the prediction structure between UAs. The flow attribute determining unit 101 sends to the general management information generation unit 104 the general management information which is the information necessary to generate management information such as a compression format or a resolution. The encoding unit 102 encodes the video data input into the MPEG-4 AVC stream based on the TYPE attribute information, sends the encoded data to the multiplexing unit 106, and sends the access information in the stream to the unit. 104 generation of general management information. Here, in the case where the TYPE attribute information shows that the information indicating the AUs to decode or display at the time of variable speed playback or reverse playback is not included in the stream, the trick play information is not included in the encoded stream. Note that the access information indicates the information of an access unit that is the basic unit in accessing the stream, and includes the start address, the display time, and the like, of the upper AU in an access unit. The general management information generation unit 104 generates the table data to refer to at the time of accessing a stream, and the table data that stores attribute information, such as a compression format based on the information of the stream. access, and the general management information, and sends the table data to the multiplexing unit 106 as the INFO management information. The playback bearer information generation unit 105 generates HLP bearer information indicating whether the stream has a random access structure based on the entered TYPE attribute information, and sends the HLP bearer information to the multiplexing unit 106. The multiplexing unit 106 generates encoded data input through the encoding unit 102, the INFO management information, and the multiplexing data, multiplexing the HLP support information, and then sends it to the storage unit 107. The storage unit 107 records the multiplexing data input through the multiplexing unit 106 onto a recording medium, such as an optical disk, a hard disk, and a memory. Note that the encoding unit 102 can packetize the MPEG-4 AVC stream into, for example, an MPEG-2 TS (transport stream) or an MPEG-2 PS (program stream), and then send the TSs or packetized MPEG-2 PS. Similarly, the encoding unit 102 can packetize the stream using an application-prescribed format, such as a BD.

Note that the content of the management information is not bound to depend on whether the trick play information is stored in the encoded stream or not. At this time, the HLP support information can be ignored. Similarly, the moving picture multiplexing apparatus 108 may have the structure without a reproduction carrier information generating unit 105.

FIG. 24A and 24B show examples of the information displayed by the HLP support information. The HLP bearer information includes the procedure that directly indicates the information of a stream as shown in FIG. 24A, and the procedure indicating whether the flow satisfies the requirement prescribed by a specific application standard, as shown in FIG. 24B.

FIG. 24A shows the following as information concerning a stream: information as to whether the stream has a random access structure; information as to whether there is a requirement on the prediction structure between images stored in an AU; and information as to whether there is information indicating the UAs to decode or display at the time of trick play,

Here the information relating to the AUs to be decoded or displayed at the time of trick play can directly indicate the AUs to be decoded or displayed, or indicate the priorities at the time of decoding or displaying. For example, it can be stated that the information indicating which UAs to be decoded or displayed on the basis of the random access units is stored in a NAL unit having a special NAL unit type prescribed by application, an SEI message or Similar. Note that it is possible to indicate whether there is information indicating the prediction structure among the AUs that constitute a random access unit. Likewise, the information relating to the AUs to be decoded or displayed at the time of trick play can be added on the basis of one or more random access units, or to each of the AUs that constitute the random access unit.

Also, in the case where the information indicating the AUs to be decoded or displayed is stored in the NAL unit having a special type, it is possible to display the NAL unit type of the NAL unit. In the example of FIG. 25, in the HLP support information, the information regarding the AUs to be decoded or displayed at the time of trick play is included in the NAL unit whose NAL unit type is 0. At that time, it is possible to obtain the information regarding trick play by demultiplexing the NAL unit whose NAL unit type is 0 from the AU data of the stream. In the case where the information relating to trick play is stored using a SEI message, it is possible to indicate the information for identifying the SEI message.

ES 2 388 397 T3

Similarly, regarding the requirements on the prediction structures, it is possible to indicate whether one or more predetermined requirements are satisfied or not, or it is possible to indicate that the following respective requirements are satisfied independently:

(i) regarding the AUs of an I-picture and P-pictures, the decoding order should match the display order;

(ii) the AU of a P-picture cannot refer to the AU of a B-picture;

(iii) the AUs after the upper AU in a display order in a random access unit can refer only to the AUs included in the random access unit; and (iv) each AU can refer only to AUs placed up to N numbers before and after in the order of decoding. In this case, all AUs are fully counted or AUs are counted based on reference AUs, and the value of N can be displayed in the HLP support information.

Note that in MPEG-4 AVC, it is possible to use, as reference images, images on which filter processing (unblocking) is performed to remove block distortion after decoding, to improve image quality. , and it is possible to use, as images for display, images before unlocking. In this case, the moving picture decoding apparatus needs to keep the picture data before and after unlocking. Therefore, it is possible to store, in the support information HLP, the information indicating whether there is a need to keep the images before unlocking for the use of the display. The MPEG-4 AVC standard defines the maximum size of a buffer (DPB: Decoded Picture Buffer) required to store the reference pictures or the pictures to be displayed as the decoding results. Therefore, with a DPB buffer having the maximum size or a buffer having the maximum size prescribed by application, it is possible to indicate whether the decoding processing can be performed without fail even in the case of storing the images for display. from the reference images. Note that, in order to store the images before unlocking the reference images, it is possible to indicate the size of the buffer to be secured, in addition to the necessary size as a DPB, using the number of octets or the number of frames. Here, whether or not unlocking is performed on each image can be known from the information in the stream, or the information outside the stream such as the management information. In the case of obtaining the information in the flow, for example, it can be obtained from an SEI. Furthermore, in the case of decoding a MpEG4 AVC stream, it is possible to evaluate whether the images, before unlocking the reference images, can be used for display or not, based on the size of the temporary memory that can be used. use in the decoding unit and the information described above, and then it is possible to determine how to display the images.

Note that all or some of the information can be included as HLP support information. Likewise, it is possible to include necessary information based on a predetermined condition, for example, to include information as to the presence or absence of trick play information only in the case where there is no requirement regarding the prediction structure. Likewise, information other than the information described above may be included in the HLP support information.

FIG. 24B does not directly indicate information regarding the structure of a stream, but indicates whether a stream satisfies the requirements on stream structures prescribed by the Blu-ray Disc (BD-ROM) standard or the High Definition DVD ( HD), which is the standard for storing high definition images on a DVD. Likewise, in the case where various modes are defined as the requirements of a flow in an application standard such as the BD-ROM standard or the like, the information indicating the applied mode can be stored. For example, the following modes are used: mode 1, which indicates that there are no requirements; Mode 2, which indicates that the stream has a random access structure and includes the information to specify the AUs to decode at trick play time; and the like. Note that it is possible to indicate whether the stream satisfies the requirements prescribed in the communication service, such as stream download or broadcast, or a broadcast standard.

Note that it is possible to indicate both the information shown in FIG. 24A as the information shown in FIG. 24B. Similarly, in the case where the flow is known to satisfy the requirements in a specific application standard, it is possible to store the requirements in the application standard by converting the flow structure to the format for direct description, as shown in FIG. 24A, instead of indicating whether the flow satisfies the application standard

Note that it is possible to store the information indicating the AUs to be decoded or displayed at the time of trick play as management information. Similarly, in the case where the content of the HLP support information is switched in one flow, the HLP support information may be indicated section by section.

FIG. 26 is a flow chart showing the operations of the moving picture multiplexing apparatus 108. In Step 51, the stream attribute determining unit 101 determines the TYPE attribute information based on the values ​​set by the user or the predetermined conditions. In Step 52, the encoding unit 102 encodes a stream based on the TYPE attribute information. In Step 53, the playback bearer information generation unit 105 generates the HLP bearer information based on the TYPE attribute information. Accordingly, in Step 54, the encoding unit 102 generates the access information based on an access unit of the encoded stream, and the general management information generating unit 104 generates the INFO management information by adding the information access to other necessary information (general management information). In Step 55, the multiplexing unit 106 multiplexes a stream, the HLP bearer information, and the INFO management information. In Step 56, the storage unit 107 records the multiplexed data. Note that Step 53 can be performed before Step 52, or after Step 54.

ES 2 388 397 T3

Note that the encoding unit 102 can store the information displayed in the HLP bearer information in a stream. In this case, the information displayed in the HLP support information is stored in the NAL unit for storing trick play. For example, in the case where P pictures do not refer to B pictures, it is possible to decode only I picture and P pictures at the time of variable speed playback. Therefore, flag information is stored indicating whether only an I-picture and P-pictures can be decoded and displayed. Similarly, there is a case where some AUs to be decoded at the time of variable speed playback cannot obtain an SPS or a PPS from the AUs to which the respective AUs should refer. It is the case where the PPS referenced by a P-picture is stored only in the AU of a B-picture in the case of decoding only an I-picture and P-pictures. In this case, there is a need to obtain the necessary PPS for decoding. the P image from the AU of a B image. Therefore, it is possible to include flag information indicating whether the SPS or PPS referred to by each AU to be decoded at the time of variable speed playback can be safely obtained from one of the other AUs to be decoded. at the time of variable speed playback. Doing so in this way makes it possible to perform the operation such as detecting an SPS or a PPS also from the AU of an image not intended to be decoded at the time of variable speed playback only in the case where an indicator is not activated. Similarly, at the time it is shown that only an I-picture and P-pictures can be decoded and displayed, it is possible to adjust the reproduction speed by also decoding B-pictures, especially reference B-pictures that are referenced by other pictures.

Similarly, it is possible to store the pointer information in the header of another NAL unit such as an SPS, a PPS or a segment, instead of using any NAL unit to store trick play. For example, in the case where an SPS referenced by an AU constituting a random access RAU is stored in the upper AU in the random access RAU, the nal_ref_idc field of the NAL unit of an SPS may indicate the indicator information. Since the value of nal_ref_idc is defined as 1 or more in the NAL unit of an SPS, it is possible to always set the high-order bit to 1 and indicate the flag information by the low-order bit.

Note that the content of the HLP support information can be stored in a flow or in management information, or both. For example, the content can be displayed in management information in the case where the content of the HLP supporting information is fixed in a flow, while the content can be displayed in a flow in the case where the content is variable. It is also possible to store the flag information indicating whether or not the HLP support information is set in the management information. Likewise, in the case where the HLP support information is predetermined in an application standard such as a BD-ROM or RAM, or in the case where the HLP support information is provided separately by communication or broadcast, the information HLP media may not be stored.

(Motion picture decoding device)

FIG. 27 is a block diagram of the moving picture decoding apparatus 200 that performs the moving picture decoding method of the present invention. This moving picture decoding apparatus 200 reproduces an encoded stream shown in FIGS. 8A and 8b to 20. You can perform not only normal play, but also trick play, such as skip play, variable speed play, and reverse play. The moving image decoding apparatus 200 further includes a stream extraction unit EXT and a unit AUsel for selecting the AUs to be decoded, in addition to the units of a conventional decoding apparatus 2 shown in FIG. 5. Note that the processing units that perform the same operations as the respective processing units of the conventional decoding apparatus 2, shown in the block diagram of FIG. 5, they are assigned the same reference numbers, and their descriptions will be omitted.

The AU selection unit AUsel to decode determines the necessary AUs to be decoded based on the decoded trick play information GrpInf in the variable length decode unit VLD, according to a trick play instruction input from outside. Here, the instruction indicating trick play is input from the selection unit AUs to be decoded. In addition, the AU selection unit for the AUs to be decoded notifies the stream extraction unit EXT about DecAU, which is the information that indicates the determined AUs as the AUs needed to be decoded. The stream extraction unit EXT extracts only the stream corresponding to the AUs that are evaluated as the AUs needed to be decoded by the selection unit AUsel of the AUs to be decoded, and then transmits the stream to the variable length decoding unit VLD .

FIG. 28 is a flowchart of the manner in which the moving picture decoding apparatus 200 (mainly the AUsel selection unit of the AUs to be decoded) shown in FIG. 27 performs the decoding procedure of a stream including trick play information at the time of trick play.

First, in step 20, the AU selection unit AUs to be decoded evaluates whether the AU is the top AU of a random access unit RAU by detecting an SPS, or the like, in the stream. In the case where the AU is the upper AU, it goes to Step 21, while in the case where the AU is not the upper AU, it goes to Step 22. Here, the start position of the random access unit RAU it can be obtained from management information, such as a temporal map. Especially in the case where the playback start position is determined at the time of skip playback, or only the top image of the random access RAU is selected and high-speed playback is performed on the selected top image, It is possible to determine the starting position of the random access RAU referring to the temporal map. In Step 21, the AU selection unit AUsel to decode obtains the trick play information from the AU data, analyzes the AU data and determines the AUs to be decoded before proceeding to Step 22. In the step Step 22, the AU selection unit AUs to decode evaluates whether the AU is the AU that is determined in Step 21 as the AU to be decoded. In the case where AU is determined, the data decoding apparatus 200

ES 2 388 397 T3 moving pictures decodes the AU in Step 23, while in the case where it is not the determined AU, it goes to Step 24. In Step 24, the moving picture decoding apparatus 200 evaluates whether there is some AU left to decode. In the case where there is an AU, the moving picture decoding apparatus 200 repeats the processing of Step 20, and the following steps, while in the case where there is no AU, the processing is completed. Note that it is possible to skip the process in Step 21 and Step 22, or skip the determination processing in Step 21, and send the information indicating that all UAs are decoded at the time of normal playback, where all the AU are decoded and displayed in order.

FIG. 29 is a flow chart indicating the processing (the processing by the selection unit AUsel of the AUs to be decoded) in Step 21. First, the selection unit AUsel of the AUs to be decoded detects the start position of a NAL unit constituting an AU, searching the AU data for a start code prefix, starting with the upper octet in Step 30, and proceeding to Step 31. Note that you can search for a start code prefix from, not the upper octet of the AU data, but another position, such as the end position of an Access Unit Delimiter. In Step 31, the AUsel unit for selecting the AUs to be decoded obtains the type of NAL unit from a NAL unit, and proceeds to Step 32. In Step 32, the selection unit AUsel of the AUs to be decoded evaluates whether the type of NAL unit obtained in Step 31 is the type of NAL unit for storing trick play information. In the case where the trick play information is stored, it proceeds to Step 33, while in the case where the trick play information is not stored, it repeats the processing of Step 30 and the following steps. Here, in the case where the trick play information is stored in a SEI message, the AUsel selection unit of the AUs to be decoded obtains the NAL unit of an SEI first, and further evaluates whether the SEI message to store trick play information is included or not in the NAL unit. In Step 33, the selection unit AUsel of the AUs to be decoded obtains trick play information, and proceeds to Step 34. In Step 34, the AU selection unit AUsel to decode determines the necessary images to decode at the time of performing a specified trick play operation. For example, as long as double speed playback is specified. In the case where the trick play information indicates that it is possible to perform double speed playback by decoding and reproducing only an I-picture, P-pictures and reference B-pictures, it is determined that these three kinds of pictures are decoded and reproduced. Note that, in the case where the trick play information is not detected in the upper image of the random access unit RAU in the processing from Step 30 to Step 32, the images necessary to decode in order to perform the operation of Trick Play are determined according to a predetermined procedure. As an example, it is possible to evaluate whether the image is a reference image or not by referring to the field indicating the image type of an image in an Access Unit Delimiter, or by checking the nal_ref_idc of the NAL unit header. For example, it is possible to distinguish reference B images from non-reference B images by referring both to the field indicating the types of images and to nal_ref_idc.

FIG. 30 is a flow chart indicating the processing (the processing by the AUsel selection unit of the AUs to be decoded) in the case where all the AUs to be decoded are not always displayed. To the steps to perform the same processing as the steps in the flow chart of FIG. 28 are assigned the same reference numerals, and their description will be omitted. In Step 41, the AU selection unit AUsel to decode obtains and analyzes trick play information, determines the AUs to be decoded and the AUs to display in a specified trick play operation, and proceeds to Step 42. In the Step 41 Step 42, the unit AUsel for selecting the AUs to be decoded evaluates whether the AUs to be decoded completely coincide with the AUs to be displayed. In the case where there is a complete match, it goes to Step 22, while in the case where there is no complete match, it goes to Step 43. In Step 43, the AUsel unit for selecting the AUs to be decoded sends information on the list of AUs to display, and proceeds to Step 22. The list information of the sent AUs is used in one step (not shown in a figure) to determine the AUs to display among the decoded AUs.

Note that in MPEG-4 AVC, it is possible to use, as reference images, images on which filtering (unblocking) processing is performed to remove block distortion after decoding, in order to improve quality. images, and it is possible to use, as display images, images before unlocking. In this case, the moving image decoding apparatus 200 needs to maintain the image data before and after unlocking. Here, provided that the moving picture decoding apparatus 200 has a memory that can store post-decoding data equivalent to four pictures, in the case where it stores the picture data before and after unlocking in the memory, the memory You need to store data equivalent to two images, in order to keep images before unlocking the reference images. However, as described above, it is desirable that as many images as possible can be contained in memory at the time of reverse playback. Provided that the motion picture decoding apparatus 200 uses the pictures after unlocking also for display use, it may contain four-picture data in a memory because no pictures need to be stored before unlocking. Therefore, displaying images before unlocking, in order to improve the image quality at the time of playback in a normal direction, and displaying images after unlocking at the time of reverse playback, makes it possible to keep more images in a memory, and reduce the amount of processing at the time of reverse playback. For example, in the example of FIGS. 15A to 15C, which list the AUs of an I-picture and P-pictures as trick play information, all four-picture data can be kept in one memory at the time of reverse playback, while the following sets of two pictures, which are arbitrarily selected from I0, P3, P6 and P9, can be kept in memory at the same time at the time of playback in a normal direction: I0 and P3; P3 and P6; and P6 and P9.

(Example of a trick play recording format on an optical disc)

A trick play function is especially important in an optical disc apparatus that reproduces packet media. Here, an example of recording trick play information described above on a Blu-ray disc (BD) which is a next generation optical disc will be described.

ES 2 388 397 T3

First, a recording format of a BD-ROM will be described.

FIG. 31 is a diagram indicating the structure of the BD-ROM, especially the structures of a BD 114 disc, which is a disc medium, and the data 111, 112 and 113 stored on the disc. The data stored on the Bd disc 114 includes VA data 113, BD management information 112, such as management information relating to VA data and a VA playback sequence, and a BD playback program 111 performing interactivity. Here, for the sake of convenience, the description of the BD disc will focus on the VA application for playing movie audio and visual content, but a similar description can be made focused on another use.

FIG. 32 is a diagram showing the structure of a logical data directory file stored on the above-described BD disk. A BD disc has a recording area from its innermost radius to its outermost radius such as a DVD, CD and the like, and has logical address space to store logical data between the read input on the innermost radius. and the reading output on the outer radius. Likewise, inside the read input, there is a special area that can only be read by a controller called Burst Cut Area (BCA). As this area cannot be read from the application, it can be used, for example, for an intellectual property protection technique.

File system (volume) information is stored at the top of the logical address space, and application data such as video data is stored there as well. As described in the prior art, a file system is, for example, UDF or ISO9660, and allows the reading of stored logical data using a directory structure or a file structure as in the case of a computer. normal staff.

In this embodiment, like the directory structure and the file structure on the BD disk, the BDVIDEO directory is placed immediately below a root directory (ROOT). This directory is a directory that stores data such as VA content or management information (101, 102 and 103, which are described in FIG. 32) that is managed in the BD.

Below the BDVIDEO directory, the following seven files are recorded.

(i) BD. INFO (the file name is fixed) which is a fragment of the "BD management information" and is a file that stores the information regarding the entire BD disc. The BD player reads this file first.

(ii) BD.PROG (the file name is fixed) which is one of the "BD playback programs" and is a file that stores the playback control information relating to the entire BD disc.

(iii) XXX. PL ("XXX" is variable, and the extension "PL" is fixed) which is an item of "BD management information" and is a file that stores the playlist information, which is a script (play sequence ). Each playlist has a file.

(iv) XXX.PROG (“XXX” is variable, and the extension “PROG” is fixed) which is one of the “BD playback programs” and is a file that stores the playback control information prepared on the basis of of playlists. The corresponding playlist is identified on the basis of a file body name (based on the "XXX" match).

(v) YYY.VOB (“YYY” is variable, and the “VOB” extension is fixed) which is one of the “VA data” and is a file that stores the vOb (the same as the VOB described in the technique previous). Each VOB has a file.

(vi) YYY.VOBI ("YYY" is variable, and the extension "VOBI" is fixed) which is an element of the "BD management information" and is a file that stores the flow management information related to the VOB that are the data of Va. The corresponding playlist is identified based on a file body name (based on a match of "YYY").

(vii) ZZZ.PNG ("ZZZ" is variable, and the extension "PNG" is fixed) which is one of the "VA data" and is a file that stores PNG image data (which is a standard image format by the W3C and called "ping") to constitute subtitles and menus. Each PNG image has a.

The BD navigation data structure (BD management information) will be described with reference to FIGS. 33 to 38.

FIG. 33 is a diagram showing the internal structure of a VOB management information file ("YYY.VOBI"). The VOB management information has the information of the flow attribute (Attribute) of the VOB and a temporal map (TMAP). Stream attribute has video attribute (Video) and audio attribute (Audio # 0 to Audio # m) separately. Especially in the case of audio stream, since a VOB has several audio streams at the same time, the presence or absence of a data field is indicated by the number (Number) of audio streams.

The following are video attributes (Video) stored in fields respectively, and the values ​​that the respective fields can have:

(i) compression format (Encoding): MPEG-1; MPEG-2; MPEG-4; and AVC (Advanced Video Coding) of MPEG-4.

(ii) Resolution (Resolution): 1920x1080; 1440x1080; 1280x720; 720x480 and 720x565.

(iii) aspect ratio (Aspect): 4 to 3; and 16 to 9.

ES 2 388 397 T3 (iv) frame rate (Frame rate): 60; 59.94 (60 / 1.001), 50; 30; 29.97 (30 / 1.001), 25; 24; and 23.976 (24 / 1,001).

The following are Audio (Audio) attributes stored in fields respectively, and the values ​​that the respective fields can have.

(i) compression format (Encoding): AC3; MPEG-1; MPEG-2; and LPCM.

(ii) the number of channels (Ch): 1 to 8 (iii) language attribute (Language):

The temporal map (TMAP) is a table for storing the information for each VOBU, and it has the number of the VOBUs that the VOB has and the respective pieces of VOBU information (VOBU # 1 to VOBU # n). The respective VOBU information fragments include I_start, which is the address (the start address of an I image) of the top TS packet of a VOBU and an offset address (I_fin) to the end address of the I image, and the playback start time (PTS) of picture I.

FIG. 34 is a diagram illustrating the details of the VOBU information. As is widely known, since variable bit rate compression can be performed in MPEG video stream, in order to record the video stream in high quality, there is no proportionality between the playback time and the size of the files. data. On the other hand, since fixed bit rate compression is performed in AC3, which is an audio compression standard, the relationship between time and direction can be obtained from a primary expression. However, in the case of MPEG video data, each frame has a fixed display time, for example, a frame has a display time of 1 / 29.97 seconds in the case of NTSC, but the data size After compressing each frame changes greatly, depending on the characteristic of the image, or the type of image used in the compression, such as an I image, a P image, or a B image. Therefore, in the case of an MPEG video stream, it is impossible to represent the relationship between time and direction using a primary expression.

As can be expected, it is impossible to represent the relationship between time and data size using a primary expression in an MPEG system stream where the MPEG video data is multiplexed, i.e. a VOB. Therefore, a temporal map (TMAP) associates time with address in a VOB.

In this way, in the case where time information is given, the VOBU to which the time belongs is searched first (following the PTS of the VOBU in order), it jumps to the PTS immediately before that time in the VOBU that a TMAP has (the address specified by I_start), decoding starts with the top I picture of the VOBU, and display starts with the picture corresponding to that time.

Next, the internal structure of a playlist information ("XXX. PL") will be described with reference to FIG. 35. The playlist information includes a cell list (CellList) and an event list (EventList).

The cell list (CellList) is a sequence of playback cells in the playlist, and the cells are played in the order of description indicated in this list. The content of the cell list (CellList) is the number of cells (Number) and the information of each cell (Cell # 1 to Cell # n).

The cell information (Cell #) has a VOB file number (VOBname), start time (In) and end time (Out) in the VOB, and subtitles (SubtitleTable). The start time (In) and the end time (Out) are represented as a frame number in each VOB. It is possible to obtain the address of the VOB data needed for playback using the temporal map (TMAP) described above.

The subtitle table (SubtitleTable) is a table that stores subtitle information that is played in sync with the VOB. As in the case of audio, several languages ​​are included in the subtitles. The first information in the subtitle table (SubtitleTable) includes the number of languages ​​(Number) and the following tables (Language # 1 to Language # k) prepared on a one-language basis.

Each language table (Language #) includes language information (Language), the number (Number) of subtitle information fragments of the subtitles to be displayed separately, and subtitle information (Speech # 1 to Speech # j) of subtitles to be viewed separately. The subtitle information (Speech #) includes an image data file name (Name), the subtitle display start time (In), the subtitle display end time (Sal), and a display position of subtitle (Position).

The event list (EventList) is a table that defines each event that occurs in the playlist. The event list includes the number of events (Number) and the respective events (Event # 1 to Event # m). Each event (Event #) includes an event type (Type), an event identifier (ID), an event occurrence time (Time), and an event duration (Duration).

FIG. 36 is an event management table ("XXX. PROG") having an event manager (which is a temporary event and a user event for menu selection) prepared on the basis of a playlist. The event management table includes the number of defined event managers / programs (Number) and the respective event managers / programs (Program # 1 to Program # n). The content of each event handler / program (Program #) is the definition of the start of an event handler (event_handler tag) and the event handler identifier (ID) that is paired with the previously described event identifier, and then from it, the program described in “{}” that follows the word Function. The event (Event # 1 to Event # m) stored in the event list (EventList) of “XXX. PL "previously described is specified using an event handler identifier (ID) of" XXX. PROG ”.

ES 2 388 397 T3

Next, the internal structure of the information relating to the entire BD disc ("BD.INFO") will be described with reference to FIG. 37. Information regarding the entire BD disc includes a title list (TitleList) and an event table for global events (EventList).

The title list (TitleList) includes the number of titles on a disc (Number) and fragments of title information (Title # 1 to Title # 3) that follow the number of titles. The respective snippets of title information (Title #) include a playlist table in the title (PLTable) and a list of chapters in the title (ChapterList). The playlist table (PLTable) includes the number of playlists in the title (Number) and playlist names (Name) which are the playlist file names.

The list of chapters (ChapterList) includes the number of chapters included in the title (Number) and fragments of chapter information (Chapter # 1 to Chapter # n). Each chapter information fragment (Chapter #) includes a cell table (CellTable) included in the chapter, and the cell table (CellTable) includes the number of cells (Number) and cell input information fragments (CellInput # 1 to CellInput # k). The cell entry information (CellInput #) includes the name of the playlist that includes the cell and a cell number in the playlist.

The event list (EventList) includes the number of global events (Number) and pieces of global event information. It should be noted that the global event to be defined first is called the first event (FirstEvent), and it is the event invoked first after inserting the BD disc into the player. Event information for global events has only one event type (Type) and one event identifier (ID).

FIG. 38 is a table ("BD.PROG") of a global event manager program. The content of this table is the same as the content of the event management table described in FIG. 36.

In the case of storing the above-described trick play information in the BD-ROM format described so far, a VOBU is considered to include one or more random access RAUs, and the trick play information is included in the upper AU. of the VOBU. Note that a NAL unit is included in the MPEG4 AVC where trick play information is stored.

Note that trick play information can be stored in BD management information. For example, it is possible to store ready trick play information VOBU to VOBU, extending the temporal map of the VOB management information. Similarly, it is possible to define a new map to store trick play information.

Likewise, it is possible to store the trick play information either in the VOBU or in the management information of the BD.

Likewise, it is possible to store only the default value of the trick play information in the BD management information, and only in the case where the trick play information, in terms of the VOBU, is different from the default value, is it possible store trick play information on the VOBU.

Likewise, it is possible to store a set of one or more pieces of trick play information in the management information of the BD as the information that is common between the streams. The VOBU may refer to a trick play information chunk among the trick play information chunks stored in the BD management information. In this case, the index information of the trick play information referenced by the VOBU is stored in the management information of a VOBU unit or in the VOBU.

(Player for playing optical discs)

FIG. 39 is a block diagram broadly showing the functional structure of a player that plays a BD disc shown in FIG. 31, and the like. The data on the BD disc 201 is read by an optical player 202. The read data is transmitted to an exclusive memory depending on the respective data types. The BD playback program (the content of "BD.PROG" or "XXX.PROG") is transmitted to a program memory 203. Similarly, the BD management information ("BD.INFO", "XXX.PL" or "YYY.VOBI") is transmitted to a management information memory 204. Similarly, AV data ("YYY.VOB" or "ZZZ.PNG") is transmitted to AV memory 205.

The BD playback program recorded in the program memory 203 is processed by a program processing unit 206. Likewise, the BD management information recorded in the management information memory 204 is processed by the management information processing unit 207. Similarly, AV data recorded in AV memory 205 is processed by display processing unit 208.

The program processing unit 206 receives the playlist information to be played by the management information processing unit 207 and the event information, such as the timing of program execution, and performs program processing. Likewise, it is possible to dynamically change the playlists to be played by the program. This can be accomplished by sending an instruction to play the playlists to the management information processing unit 207. Program processing unit 206 receives an event from a user; In other words, it receives a request through a remote controller, and in the case where there is a program corresponding to the user event, it executes the program.

The management information processing unit 207 receives an instruction from the program processing unit 206, analyzes the playlists and the management information of the VOBs that 21

ES 2 388 397 T3 correspond to the playlists, and instructs the display processing unit 208 to reproduce the desired AV data. Similarly, the management information processing unit 207 receives the standard time information from the presentation processing unit 208, instructs the presentation processing unit 208 to stop playing the AV data based on the time information. . Similarly, the management information processing unit 207 generates an event to notify the program processing unit 206 of the timing of the program execution.

The presentation processing unit 208 has a decoder that can process video, audio, and subtitles / images (still images), respectively. It decodes and produces the AV data according to an instruction from the management information processing unit 207. In the case of video data, and subtitles / images, they are decoded and then rendered in the respective exclusive planes, that is, the video plane 210 and the image plane 209. After this, the synthesis processing unit 211 performs synthesis processing on the video, and outputs the video to a display device such as a television.

At the time of trick play, such as skip play, variable speed play, and reverse play, the display processing unit 208 interprets the trick play operation requested by the user, and notifies the unit 207 of management information processing on the information such as the playback speed. The management information processing unit 207 analyzes the trick play information stored in the upper AU of the VOBU and determines the AUs to be decoded and displayed so that the user-specified trick play operation can be performed safely. Note that the management information processing unit 207 can obtain the trick play information, send it to the presentation processing unit 208 and determine the AUs to decode and the AUs to be displayed in the presentation processing unit 208.

Note that a standalone computer system can easily execute the processing shown in this embodiment, recording the program to perform the moving picture encoding procedure and the moving picture decoding procedure shown in this embodiment, on a recording medium such as a floppy disk.

FIG. 40A to 40C are illustrations of how the computer system executes the motion picture encoding procedure and the motion picture decoding procedure of this embodiment using a program recorded on a recording medium such as a floppy disk.

FIG. 40A shows an example of a physical format of a floppy disk as a recording medium. FIG. 40B shows a flexible disk and the front view and the cross-sectional view of the appearance of the flexible disk. A flexible disk (FD) is contained in a housing F, a plurality of tracks (Tr) are concentrically formed on the surface of the disk from the outer radius to the inner radius of the disk, and each track is divided into 16 sectors (Se) in the angular direction. Therefore, in the case of the floppy disk storing the above-described program, the program is recorded in an area assigned to it on the floppy disk (FD).

Likewise, FIG. 40C shows the structure for recording and reproducing the program on the floppy disk. In the case of recording the above program to perform the moving picture encoding procedure and the moving picture decoding procedure on the floppy disk FD, a computer system Cs writes the program on the floppy disk through a drive. floppy disk. Similarly, in the case of constructing the above moving picture coding apparatus and the above moving picture decoding apparatus to perform the moving picture coding procedure and the moving picture decoding procedure using the program in the floppy disk, the program is read from the floppy disk through the floppy disk drive, and transmitted to the computer system.

Note that the above description has been made using a floppy disk as the recording medium, but the program can be recorded on an optical disk. Similarly, a recording medium is not limited to this, and another recording medium such as an IC card or ROM memory cassette can be used as long as the program can be recorded.

Up to this point, the moving image stream generation apparatus, the moving image coding apparatus, the moving image multiplexing apparatus and the moving image decoding apparatus of the present invention have been described on the basis of the embodiment, but the present invention is not limited to this embodiment.

For example, the present invention includes the following in this embodiment: (i) a moving image stream generation apparatus; an optical disc recording apparatus having one of a moving image encoding apparatus and a moving image decoding apparatus; a moving image sending apparatus; a digital television broadcast transmission apparatus; a web server; a communication apparatus; a mobile information terminal; and the like; and (ii) a moving image receiving apparatus having a moving image decoding apparatus; a digital television broadcast receiving apparatus; a communication apparatus; a mobile information terminal; and the like.

Note that the respective functional blocks shown in FIGS. 21, 23, 27 and 39 are typically realized as an LSI, which is a large scale integration circuit. Each of the functional blocks can be manufactured on a single chip, or part of, or all, the functional blocks can be integrated on a single chip (for example, functional blocks except for a memory can be manufactured on a single chip) . The integrated circuit is called LSI here, but it can be called IC, system LSI, super LSI, or ultra LSI, depending on the level of integration. Likewise, the manufacturing process of the same in an integrated circuit is not limited to the manufacturing process in an LSI; it can be done by a dedicated circuit or a generic processor. Likewise, it is possible to use (i) a reconfigurable processor where the connection or circuit cell configuration can be reconfigured or (ii) a programmable FPGA (Field Programmable Gate Formation), after

ES 2 388 397 T3 manufacture them on an LSI. In addition, in the case where the technique of its manufacture in an integrated circuit, instead of its manufacture in an LSI, appears when the semiconductor technique is further developed, or any derivative technique appears, in due course, blocks can be manufactured functional in an integrated circuit using such a new technique. Biotech application is likely. Similarly, between the respective functional blocks, a storage unit (an image memory) in which the image data to be encoded or decoded is stored, can be configured separately instead of being included in a single chip.

Industrial Applicability

The present invention can be applied as: a moving image stream generation apparatus that generates a moving image to be reproduced in trick play; a moving image encoding apparatus that generates, by encoding, a moving image to be reproduced in trick play; a moving image multiplexing apparatus that generates, by packet multiplexing, a moving image to be reproduced in trick play; and a moving picture decoding apparatus that reproduces the moving picture in trick play; and especially, as an apparatus for constructing the system for playing an MPEG-4 AVC stream using a trick play mode, such as variable speed play and reverse play, such an apparatus being, for example, an apparatus related to an optical disc on which its trick playing function is generally focused.

A list of additional accomplishments follows.

Embodiment 1. A moving image stream generation apparatus for generating a stream that includes images that constitute a moving image, said apparatus comprising: a supplementary information generation unit, usable to generate, for each random access unit, supplementary information to the one to refer to at the time of reproduction of each random access unit, each random access unit including one or more images; and a stream generation unit, usable to generate a stream that includes the generated supplementary information and the images, adding the supplementary information to each corresponding random access unit, in which, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying images to be decoded at the time when the images included in each random access unit are reproduced with trick play.

Embodiment 2. The moving image stream generation apparatus having the features of Embodiment 1, wherein the trick play includes at least one of: skip play; variable speed playback; and reverse playback.

Embodiment 3. The moving image stream generation apparatus with the characteristics of embodiment 2, in which each of the images is composed of sub-picture units, and said stream generation unit is usable to store the supplementary information in a first sub-picture unit, other than a second sub-picture unit, for storing a pixel value of each of the pictures.

Embodiment 4. The moving image stream generation apparatus with the characteristics of embodiment 3, in which each random access unit has one or more images, and said stream generation unit is usable to store the supplementary information in a top image included in each random access unit.

Embodiment 5. The moving image stream generation apparatus with the characteristics of embodiment 4, in which the supplementary information includes information for specifying images to be decoded at the time of reproduction of each random access unit at a specified rate .

Embodiment 6. The moving image stream generating apparatus with the features of Embodiment 4, in which the supplementary information includes information indicating image priorities, based on which each random access unit is reproduced.

Embodiment 7. The moving image stream generation apparatus with the characteristics of embodiment 4, in which the supplementary information includes pieces of information indicating image types of all images included in each random access unit, being placed the pieces of information in an order that corresponds to a decoding order of the images.

Embodiment 8. The moving image stream generating apparatus with the features of embodiment 7, in which the image types include: an I image on which intra-coding is carried out; a picture P on which inter-coding is carried out, with reference to one picture per block, the block being a basic unit in coding; a reference image B on which intercoding is carried out with reference to two images per block, the block being a basic unit in the encoding, and the reference image B being an image that is referenced by another picture; and a non-reference image B on which intercoding is carried out with reference to two images per block, the block being a basic unit in the encoding, and the non-reference image B being an image that does not it is the object of reference on the part of another image.

Embodiment 9. The moving image stream generation apparatus with the features of Embodiment 4, in which the supplementary information includes pieces of information indicating types of image structures of all images included in each random access unit, the information fragments being placed in an order corresponding to a decoding order of the images.

Embodiment 10. The moving image stream generating apparatus with the features of Embodiment 9, wherein the types of image structures include at least: a field structure; and a frame structure.

ES 2 388 397 T3

Embodiment 11. The moving image stream generation apparatus with the features of Embodiment 10, in which the types of image structures further include a frame structure with information indicating whether an image has a field of view equivalent to two images, or if the image has a field of view equivalent to three images, in the case where the image has a frame structure.

Embodiment 12. The moving image stream generation apparatus with the characteristics of embodiment 1, said apparatus further comprising a sequence parameter set addition unit, usable to add, to each random access unit, a sequence parameter set , that is, a group of parameters with respect to one or more images, in which the sequence is composed of images that start with a special image in which all the states necessary for decoding are reset, and that end with an image that is placed immediately before a next special image.

Embodiment 13. The moving image stream generation apparatus with the features of embodiment 12, wherein each random access unit consists of one or more images, and said sequence parameter set addition unit is usable for storing a set of sequence parameters that is referenced by each image in the random access unit, only in a top image included in each random access unit.

Embodiment 14. An apparatus for generating streams of moving images, for generating a stream that includes images that constitute a moving image, said apparatus comprising a unit for adding sets of sequence parameters, usable to generate a stream of moving images that includes sequence parameter sets, adding the sequence parameter sets, for each random access unit, each of the sets of sequence parameters being a group of parameters with respect to one or more images, in which the sequence is composed of images that start with a special image in which all the states necessary for decoding are reset , and ending with an image that is placed immediately before a next special image.

Embodiment 15. The moving image stream generation apparatus with the features of embodiment 14, wherein each random access unit consists of one or more images, and said sequence parameter set addition unit is usable for storing a set of sequence parameters, which is referenced by each image in the random access unit, only in a superior image included in each random access unit.

Embodiment 16. A procedure for generating streams of moving images to generate a stream that includes images that constitute a moving image, said procedure comprising: generating, for each random access unit, supplementary information to be referred to at the time of reproduction of each random access unit, each random access unit including one or more images; and generating a stream that includes the generated supplementary information and the images, adding the supplementary information to each corresponding random access unit, in which, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without relying on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 17. A program for use with a moving image stream generation apparatus to generate a stream that includes images that constitute a moving image, the program causing a computer to execute a moving image stream generation procedure, which It includes: generating, for each random access unit, supplementary information to be referenced at the time of reproduction of each random access unit, each random access unit including one or more images; and generating a stream that includes the generated supplementary information and the images, adding the supplementary information to each corresponding random access unit, in which, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without relying on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 18. A moving image encoding apparatus, for encoding images constituting a moving image, said apparatus comprising: a supplementary information generation unit, usable to generate, for each random access unit, supplementary information to be referred to in the time of reproduction of each random access unit, each random access unit including one or more images; and an encoding unit usable to encode the generated supplementary information and images, and usable to generate a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit, in which, in a upper end of each random access unit, an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 19. A moving image coding procedure, to encode images that constitute a moving image, said procedure comprising: generating, for each random access unit, supplementary information to be referred to at the time of reproduction of each access unit random, each random access unit including one or more images; and encoding the generated supplementary information and images, and generating a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit, in which,

ES 2 388 397 T3 at an upper end of each random access unit, an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information to specify the images to be decoded at the time when The images included in each Random Access Unit are played back with trick play.

Embodiment 20. A program for a moving image encoding apparatus for encoding images constituting a moving image, the program causing a computer to execute a moving image encoding procedure including: generating, for each random access unit, information supplementary to which to refer at the time of reproduction of each random access unit, each random access unit including one or more images; and encode the generated supplementary information and images, and generate a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit, in which, at an upper end of each random access unit , an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 21. A multiplexing apparatus for moving images, for encoding images that constitute a moving image, said apparatus comprising: a supplementary information generation unit, usable to generate, for each random access unit, supplementary information to be referred to in the timing of reproduction of each random access unit, each random access unit including one or more images; an encoding unit usable to encode the generated supplementary information and images, and usable to generate a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit, a packetizing unit usable for packetizing the encoded stream generated, and a multiplexer unit usable to generate management information, storing at least one of the following: image playback time information in the encoded and packetized stream; image size information; and start address information of each random access unit; and usable to multiplex the management information and the encoded and packetized stream in different areas, in which, at an upper end of each random access unit, an intra-encoded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 22. A multiplexing method of moving images, to encode images that constitute a moving image, said procedure comprising: generating, for each random access unit, supplementary information to refer to at the time of the reproduction of each random access unit , each random access unit including one or more images; encoding the generated supplementary information and images, and generating a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit; packetize the generated encoded stream; and generating management information by storing at least one of the following: image playback time information in the encoded and packetized stream; image size information; and start address information of each random access unit; and usable to multiplex the management information and the encoded and packetized stream in different areas, in which, at an upper end of each random access unit, an intra-encoded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 23. A program for a moving image multiplexing apparatus, for encoding images that constitute a moving image, the program causing a computer to execute a moving image multiplexing procedure that includes: generating, for each random access unit, supplementary information to the one to refer to at the time of reproduction of each random access unit, each random access unit including one or more images; encoding the generated supplementary information and images, and generating a stream including the encoded supplementary information and images, adding the supplementary information to each corresponding random access unit; packetize the generated encoded stream; and generating management information, storing at least one of the following: image playback time information in the encoded and packetized stream; image size information; and information of the starting address of each random access unit; and usable to multiplex the management information and the encoded and packetized stream in different areas, in which, at an upper end of each random access unit, an intra-encoded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 24. A moving image decoding apparatus, for decoding and reproducing a stream including images constituting a moving image, said apparatus comprising: an instruction obtaining unit, usable to obtain an instruction indicating that trick play should be carried out ; an analysis unit, usable to analyze, by demultiplexing, supplementary information for each random access unit, where each random access unit constitutes the stream; an image specification unit to be reproduced, usable to specify the images, among the images included in each random access unit, that are needed for the trick play indicated by the instruction obtained by said instruction obtaining unit, based on a analysis result obtained by said analysis unit; and a decoding unit usable to decode and reproduce the images specified by said image specification unit to be reproduced, in which, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without depend on none

ES 2 388 397 T3 image, and the supplementary information includes information for specifying the images to be decoded at the time when the images included in each random access unit are reproduced with trick play.

Embodiment 25. The moving picture decoding apparatus with the features of embodiment 24, wherein said specifying unit of pictures to be reproduced is usable to specify the pictures necessary for trick reproduction, based on a predetermined rule, in the case in which said analysis unit analyzes a random access unit and obtains as a result that the random access unit does not include supplementary information.

Embodiment 26. The moving image decoding apparatus with the characteristics of embodiment 24, said apparatus further comprising a unit for specifying random access units, usable to extract, from the stream, a set of sequence parameters, that is, a group of parameters with respect to one or more images, and usable to specify a random access unit that includes an image, such as a top image, in which the extracted sequence parameter set is included; in which said image specification unit to be reproduced is usable to specify the upper image included in the random access unit specified by said random access unit specification unit, and the sequence begins with a special image in which they are reset all states necessary for decoding, and the sequence is made up of images that start with a special image and end with an image that is placed immediately before a next special image.

Embodiment 27. A moving picture decoding method for decoding and reproducing a stream including pictures constituting a moving picture, said method comprising: obtaining an instruction indicating that trick play should be performed; analyzing, by demultiplexing, supplementary information for each random access unit, the stream constituting each random access unit; specifying the images, among the images included in each random access unit, that are needed for the trick play indicated by the instruction obtained in said obtaining, based on an analysis result obtained by said analysis; and decoding and reproducing the images specified by said specification, in which, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specify the images to decode at the time when the images included in each random access unit are played back with trick play.

Embodiment 28. A program for a moving image decoding apparatus, for decoding and reproducing a stream that includes encoded images constituting a moving image, the program causing a computer to execute a moving image decoding procedure including: obtaining an instruction indicating that trick play should be carried out; analyzing, by demultiplexing, supplementary information for each random access unit, the stream constituting each random access unit; specifying the images, among the images included in each random access unit, that are needed for the trick play indicated by the instruction obtained in said obtaining, based on an analysis result obtained by said analysis; and decode and reproduce the images specified by said specification; in which, at an upper end of each random access unit, an intracoded image is placed that can be decoded without depending on any image, and the supplementary information includes information to specify the images to be decoded at the time that the included images on each random access unit are played with trick play.

Embodiment 29. A stream comprising images constituting a moving image, in which each random access unit includes supplementary information to be referenced at the time of reproduction of the random access unit, each random access unit including one or more more images, at an upper end of each random access unit, an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 30. A computer-readable recording medium comprising a stream that includes images constituting a moving image, in which each random access unit includes the supplementary information to be referenced at the time of the random access unit's playback , each random access unit including one or more images, at one end of each random access unit, an intra-coded picture is placed that can be decoded without depending on any picture, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.

Embodiment 31. An integrated circuit for generating a flow that includes images that constitute a moving image, said integrated circuit comprising: a supplementary information generation unit, usable to generate, for each random access unit, supplementary information to be referred to in the timing of reproduction of each random access unit, each random access unit including one or more images; and a flow generation circuit unit, usable to generate a flow that includes the generated supplementary information and the images, adding the supplementary information to each corresponding random access unit, in which, at one end of each random access unit , an intra-coded image is placed that can be decoded without depending on any image, and the supplementary information includes information for specifying the pictures to be decoded at the time when the pictures included in each random access unit are reproduced with trick play.