Image coding apparatus and image decoding apparatus

ES 2 227 686 T3

DESCRIPTION

Image encoder and image decoder.

Technical field

The present invention is related to an image encoding apparatus and an image decoding apparatus, and more particularly to a moving image encoding apparatus for encoding image data in an efficient manner and an image decoding apparatus. motion picture to decode the encoded data generated by the motion picture coding apparatus, used in the field of digital image processing.

Art Background

In image coding, a method of superimposing different sequences of moving images has been considered. The article entitled "Image coding scheme using layered representation and multiple templates" (IEICE technical report, IE94-159, pages 99-106 (1995), describes a scheme for superimposing a sequence of moving images as background and another sequence of moving images of a component or partial image (for example, a video image of a human figure or a fish cut using the Cromakey technique as a background, to generate a new sequence of images.

Fig. 12 is a block diagram showing an encoding apparatus and a decoding apparatus in accordance with the conventional art. The data coding part of the pixels 1201 in FIG. 12 is a part for coding the data of the pixels representing intensity and color differences, and the coding part of the shape data 1202 is for coding the data. so that they represent a shape of the partial image. These parts constitute an apparatus for encoding a partial image.

The shape data is used to encode the pixel data. A pixel data decoding part 1203 in Fig. 12 is a part for decoding the pixel data, and the shape data decoding part 1204 is for decoding the shape data. These parts constitute an apparatus for decoding a partial image. To decode the pixel data, the decoded shape data is used.

The shape data encoding portion 1202 first expresses an outline of a shape using 8 directional string codes, for example, and then encoding the string codes by Huffman encoding. The pixel data encoding portion 1201 encodes the pixel data by the international standard moving picture encoding method, such as MPEG or H.261. When dividing the data into blocks, an arbitrarily shaped DCT technique or the like is used for the blocks including a partial image boundary.

Each partial image is decoded by a decoding apparatus, and then superimposed on an overlay part (not shown) using the shape data, and displayed on a display device such as a screen. For example, when a partial image p (i, j) is superimposed in an arbitrary shape on a rectangular background image b (i, j), the data of shape s (i, j) is generally used according to the following expression (1):

f (i, j) = p (i, j) s ((i, j) + b (i, j) [1-s (i, j)] ... (1) where (i, j) represents the coordinates of a pixel, and f (i, j) represents a value of the pixel. The value of s (i, j) assumes that the value "1" is inside a partial image, and that "0" is outside from image.

In the conventional art, however, a technique for setting up the spatial hierarchy for a partial image has not been proposed. The international MPEG2 standard runs a hierarchy (ie spatial hierarchy) across a complete image. In the method, the data in a lower layer has a low spatial resolution throughout the entire image, and the data in an upper layer to improve the resolution is co-decoded to achieve a high spatial resolution.

It is desirable to provide an image decoding apparatus and an image decoding apparatus that can realize a spatial hierarchy in a partial image.

To obtain the low resolution shape data, the high resolution shape data obtained by conventional art can be simply reduced. However, if the low resolution image thus obtained is displayed on a large screen monitor having a low resolution, the outline of a part will have a stepped appearance, which leads to deterioration in the quality of the physical image. The same problem arises when a low resolution image is enlarged for display on a large screen monitor that has a high resolution.

ES 2 227 686 T3

It is desirable to provide an image coding apparatus and an image decoding apparatus that can solve the problem.

Attention is invited to an article entitled "Coding arbitrarily shaped image segments based on a generalized orthogonal transform", by Michael Gilge et al., Elsevieer Science Publishers BV, vol. 1 number 2, of October 1, 1989, which refers to the representation of images oriented to areas.

Document EP-0588411 discloses a system with encoder and decoder switching means, which are designed for setting a ratio of signals present on the first and second switching contacts and for combining the signals set in this way.

Also, pages 3-49 of the 1966 Munich meeting for the MPEG4 working group report (XP002047798) describe encoder / decoder technology in relation to the planes of video objects.

According to a first aspect of the present invention, there is provided an image coding apparatus according to claim 1. According to a second aspect of the present invention, there is provided an image decoding apparatus according to claim 3.

Preferred features are set out in the dependent claims.

In order that the present invention may be more easily understood, the embodiments thereof will now be described, with reference to the accompanying drawings, in which:

Fig. 1 is a block diagram showing a coding apparatus according to a first embodiment that helps to understand the present invention, Fig. 2 is a block diagram showing a decoding apparatus according to the first embodiment, which helps to understand the present invention, Fig. 3 is a block diagram showing a coding apparatus according to a second embodiment, which helps to understand the present invention, Figure 4 is a block diagram showing a decoding apparatus according to the second embodiment, which helps to understand the present invention, Figure 5 is a block diagram showing an encoding apparatus according to a third embodiment which helps to understand the present invention, Fig. 6 is a block diagram showing a decoding apparatus for a lower layer according to the third embodiment that helps to understand the present invention, Fig. 7 is a diagram to be used in illustrating information about the difference between the upper layer shape data and the lower layer shape data, Figures 8A to 8D are diagrams for use in illustrating a technique for obtaining the low resolution shape data from the high resolution shape data, Figure 9 is a block diagram showing an encoding apparatus of according to another embodiment that helps to understand the present invention, Figure 10 is a block diagram showing an example of an apparatus for executing hierarchical coding of a partial picture that helps to understand the present invention, Figure 11 is a block diagram showing a lower layer decoding apparatus According to a further embodiment of the present invention, Fig. 12 is a block diagram showing a coding apparatus and a decoding apparatus of the conventional art.

In Fig. 10 an apparatus is shown as a means for setting the spatial hierarchy in a partial image. The downsampling portion 1001 in FIG. 10 shrinks the input pixel data pixels to a smaller spatial resolution of the pixel data. A second low sampling portion 1002 reduces the pixels of the input shape data to a lower spatial resolution of the shape data. The shape data shows a shape of a partial image, and is represented as a binary image having a pixel value of "1" within a part and a pixel value of "0" outside the part, for example.

The lower layer encoding portion 1004 encodes the pixel data in low resolution. For the

ES 2 227 686 T3 encoding, the international standard method of encoding moving images is used, such as MPEG or H.261. When an image is divided into blocks, an arbitrary shape DCT technique or the like is used for the block that includes a boundary of the partial image. In this case, the low-resolution shape data output from the second low-sampled portion 1002 is used as information at the boundary of the partial image.

The upper layer encoding portion 1003 encodes the high resolution pixel data, where the international standard moving picture encoding method such as MPEG or H.261 is employed. In this case, the prediction of the decoded images is used in the lower layer as well as in the upper layer. For the partial image boundaries, the high-resolution form data that has not been sampled is used.

The shape data encoding portion 1005 encodes the shape data in a high resolution corresponding to the resolution of the upper layer. The outline of a shape is expressed with 8 directional string codes, for example, and these string codes are encoded using Huffman encoding.

The encoded data is integrated by an embedded part (not shown) and transmitted or stored. In the decoding apparatus for decoding only a lower layer, as shown in Fig. 11, only the data necessary to decode the lower layer is collected, from the encoded data having the data of the upper and lower layers integrated. in them, and are used for the decoding process. In other words, the "encoded data of the lower layer pixel data" and the "encoded data of the shape data" are selected in a part of the section (not shown) and decoded.

At this time, a decoding part 1103 of the shape data in FIG. 11 decodes the high-resolution shape data. The low resolution shape data is obtained in a low sampling part 1102, using the same technique as in the second low sampling part 1002 of FIG. 10. The data thus obtained is used to decode the data of the pixels in a decoding part of the lower layer 1101. The decoding apparatus for decoding up to an upper layer uses all the encoded data to carry out decoding up to the upper layer.

However, the above-described apparatus suffers from the following problem. That is, although the conventional decoding of a lower layer only requires the data in low resolution form, the above apparatus has to use the data in high resolution form when decoding data in the lower layer, resulting in redundant data. In other words, the shape data in a higher layer that has a large amount of data should be used instead of the shape data in a lower layer with a smaller amount of codes. Consequently, it becomes difficult to encode a lower layer with good image quality with limited transmission or with limited storage capacity.

The first and second embodiments are helpful in understanding the present invention to solve this problem.

Referring to FIG. 1, a lower sampling portion 104 reduces the input pixel data to generate low resolution pixel data for a lower layer. A low-pass filter is used in reduction to prevent distortion. The lower layer encoding portion 105 encodes the low resolution pixel data.

For coding, the international standardized method of coding moving images is used, such as MPEG or H.261. When an image is divided into blocks, an arbitrary shape DCT technique or the like is used for a block that includes a boundary of the partial image. For the partial image boundary, the low resolution shape data obtained from a first generation part of the shape data 106 is used.

The first shape data generating part 106 generates low resolution shape data for a lower layer. Shape data is extracted by dividing the pixel data into areas, for example. Techniques used for area division include an edge detection technique using differential operation, and morphological segmentation.

When obtaining a shape of the moving object from a static background, dynamic area detection using intra-frame differences can be used. Alternatively, the shape data can be generated using the Chromakey technique. In Fig. 1, the low resolution shape data is obtained after reducing the target pixel data in a low sampling portion 104 to reduce the resolution thereof. However, another method can be used in which the shape data is obtained first for the high resolution pixel data, and the shape data is then reduced to obtain the low resolution shape data.

The shape data generated here is binary images or images that have two or more levels of gradation. In the latter case, the superposition of a partial image and a background image according to the previous expression (1) can be considered as a weighted average of the partial image and the background image with the data so that they represent a weighting . It will be observed that the data of form s (i, j) in expression (1) takes a value of "1" within the target partial image, "0" outside the partial image, and a value between "0" and "1" in the part of the boundary of the partial image.

ES 2 227 686 T3

A first shape data encoding part 107 encodes the low resolution shape data for a lower layer. If the shape data is binary data, you can use fixed-length encoding, MMR encoding, string encoding, or the like. If the shape data is expressed with two or more levels of gradation, an encoding scheme using DCT as in the MPEG system, an encoding scheme using a quaternary tree and vector quantization ("MPEG Video Verification Model- 4 Version 2.0 (ISO / IECTC1 / SC29 / WG11 N1260)), or similar.

A second shape data generation part 102 generates high resolution shape data for an upper layer from the high resolution pixel data. The method of generating the shape data is the same as in the first shape data generation part 106, and the description thereof will not be repeated accordingly. A second shape data encoding part 103 encodes the high resolution shape data generated by the second shape data generating part 102. The shape data encoding method is the same as in the first shape data encoding part 107, and the description thereof will not be repeated. It will be noted that these two shape data encoding parts 103, 107 do not have to use the identical encoding method; they can use independent methods for decoding.

The upper layer encoding portion 101 encodes the high resolution pixel data. The encoding is performed using the international standard method of encoding moving images, such as the MPEG or H.261 system, as in the technique described in the Background of the Art section. In this case, however, the encoding uses the prediction from the partially decoded images existing in the lower layer encoding portion 105, additionally for the prediction from the preceding or following decoded images. in the top layer. For the boundary of the partial image, the high resolution shape data output of the second shape data generation part 102 is used.

Next, the decoding apparatus according to the first embodiment will be described with reference to Fig. 2.

The lower layer decoding apparatus 205 comprises parts surrounded by a broken line in the figure

two. The first shape data decoding part 204 decodes the lower layer shape data encoded data to obtain the low resolution shape data. The decoded shape data is sent to a lower layer decoding part 203, and is also used to display a lower layer image. The lower layer decoding part 203 decodes the encoded data of the lower layer pixel data, and supplies the low resolution pixel data to a lower layer display part (not shown).

Next, the decoding of an upper layer of FIG. 2 will be described. A second shape data decoding part 202 decodes the encoded data of the upper layer shape data, to obtain the high resolution shape data. . The decoded shape data is sent to a decoding part 201 of the upper layer, and is also used to display an image of the upper layer.

The upper layer decoding part 201 decodes the encoded data of the upper layer pixel data, and supplies the high resolution pixel data to a display part of the upper layer (not shown). The decoding portion 201 of the upper layer uses, as in the encoding portion 101 of encoding of the upper layer shown in Fig. 1, the prediction of the images decoded in the decoding portion 203 of the lower layer, in addition to the prediction from the previous or next decoded images in time in the upper layer.

The second embodiment to aid in understanding the present invention will be described below.

In the first embodiment described above, the high resolution shape data used in an upper layer and the low resolution shape data used in a lower layer are encoded independently of each other. Thus, the information about the shape data in the lower layer is not reflected to encode the shape data in the upper layer, and consequently the amount of shape data in the upper layer becomes large. This leads to a problem where the total number of codes for the upper layer is increased more than in the case where the methods shown in Figures 10 and 11 were used. Consequently, this embodiment is directed to solve the above problem by encoding the difference between low resolution shape data and high resolution shape data.

The coding apparatus of Fig. 3 differs from the coding apparatus in Fig. 1 in that the low resolution shape data in a first shape data coding part 307 is sent to a shape data coding part 303, and in that the second shape data encoding part 303 encodes only the information about the difference between the low resolution shape data and the high resolution shape data.

The parts other than the upper layer coding part 101, the second shape data generation part 102, reduced sampling part 104, the lower layer coding part 105, and the first generation part 106 of generation of shape data, are the same as those of figure 1, and consequently their description is not repeated.

ES 2 227 686 T3

The decoding apparatus in FIG. 4 is different from the decoding apparatus of FIG. 2, in that the low resolution shape data decoded in a first shape data decoding part 404 is sent to a second shape decoding part 402. shape data, and wherein the second shape data decoding part 402 decodes the high resolution shape data together with the difference information.

The parts other than the upper layer decoding part 201, and the lower layer decoding part 203 are the same as in Fig. 2, and therefore their description will not be repeated. Additionally, the lower layer decoding apparatus 405 is equivalent to the lower layer decoding apparatus 205 in FIG. 2.

When a quaternary tree is used to represent data in binary form, for example, data in the lower hierarchies of the tree can be used as the difference information described above. Figure 7 shows binary form data and an exemplary representation of the quaternary tree. The highest value of the quaternary tree is "1" when there is at least a "1" pixel value within a 4 pixel by 4 pixel block, or "0" otherwise. The values ​​in the second and third hierarchies of the tree are also determined for the 2 pixel by 2 pixel and 1 pixel by 1 pixel blocks, respectively.

The blocks in the respective hierarchies are scanned from the upper left to the lower right in a frame scan sequence. In the example shown in Figure 7, the shape data in the top layer is expressed with three hierarchies, and the shape data in the bottom layer is expressed with two hierarchies. The data in the third hierarchy is expressed as the information about the difference between the upper and lower layers. These data expressed with the quaternary tree are encoded by arithmetic encoding, for example.

When shape data is expressed with two or more gradation layers, another encoding method can be used, in which the low resolution data is sampled up in resolution for a higher layer before taking the difference with the high resolution shape data, and the difference data is then encoded by transform encoding.

Next, the third embodiment to assist in understanding the present invention will be described.

In the method described with reference to Figures 10 and 11 above, the low resolution shape data is obtained by reducing the shape data. This can cause the problem that the outline of a partial image in the lower layer becomes a stepped shape, which leads to a degradation in image quality.

In the third embodiment, the above problem is solved by setting for the shape data with more gradation layers than in the original shape data, together with reducing the shape data.

Figure 8A shows a part of the data in binary form. Figure 8B shows the exemplary shape data transformed to fit a low resolution monitor, where the resolution is halved and the pixel dimension is doubled both vertically and horizontally. In this example, the 2 x 2 filter,

0,25 0,25

0.25 0.25 is applied to a block surrounded by a dashed line marked in bold in Figure 8A, thereby obtaining low resolution shape data having five gradation levels of 0.25, 0.5 , 0.75, and 1. When an element of the 2 x 2 filter above is represented as f (i, j) and an element within a block in Figure 8A is represented as d (i, j), the operation filter can be represented as follows:

Σ f (i, j) d (i, j) where Σ represents a sum for I = 1, 2 and j = 1,2. If the original shape data has N levels of gradation, filtering allows the generation of shape data with M (M> N) levels of gradation.

Figure 8C shows exemplary shape data used to enlarge a low resolution partial image for display on a monitor that allows viewing at the same resolution as in Figure 8A. The data can be obtained by simply enlarging the data in Figure 8B by a factor of 2 vertically as well as horizontally. By increasing the number of gradation levels of the low resolution shape data as described above, the appearance of the stepped shape of the boundary portion can be avoided.

The shape data generated in Figure 8C, for example, can be used to overlay images according to expression (1) described in the Background of the Art section. In this case, the background and the foreground blend into the contour portion of a partial image, thus making the boundary jagging pattern go unnoticed.

ES 2 227 686 T3

Although the above example has been described using a 2 x 2 filter, other filters can be used instead. For example, the 3 x 3 filter as follows can be used to transform the data vertically as well as a horizontally halved version.

1/60

1/6 2/61/6

1/60

In this case, the target pixels for subsampling are, for example, those shown by the · sign.

Figures 5 and 6 are block diagrams respectively showing an encoding apparatus and a decoding apparatus of the lower case according to the third embodiment. They are modifications of those shown in Figures 10 and 11, with the parts for the reduced sampling of the data so that they are replaced by the transformation parts 504 and 602, respectively.

The parts other than the first reduced sampling part 1001, the upper layer coding part 1003, the lower layer coding part 1004, the shape data coding part 1005, the lower layer decoding part 1101, and shape data decoding part 1103 are the same as those indicated in FIGS. 10 and 11, and therefore their description is not repeated.

Transform parts 504 and 602, which operate identically with each other, transform the shape data into those with low resolution, but with more levels of gradation, as shown in Figure 8.

Although a 2 x 2 filter has been used in Figures 8A-8C, many other filters, such as 3 x 3 or 2 x 5 can also be used for transformation.

Additionally, in the description of the coding apparatus according to the above embodiments, the shape data sent to the shape data coding part is also sent to the data coding part of the corresponding pixels. In such a case, however, the problem will arise when the non-reversible encoding is executed in the shape data encoding part, because the shape data used in a pixel data encoding part and the shape data used in the data decoding part of the pixels in a decoding apparatus become different from each other.

In such a case, the encoding apparatus may be provided with a shape data decoding part, such that the shape data encoded can be decoded in the shape data decoding part before being sent to the corresponding part. encoding of the pixel data. Figure 9 shows a variation of the coding apparatus in Figure 3, which has been modified in this way.

Referring to Fig. 9, the decoded data of the lower layer shape data that has been decoded by a first decoding part 909 of the shape data is sent to a lower layer coding part 905, as well as also to the second shape data encoding part 903, and to a second shape data decoding part 908. The decoding apparatus corresponding to the encoding apparatus shown in Fig. 9 also uses the decoded data of the lower layer shape data in an upper layer shape data decoding part, that is, a part equivalent to the second. decoding part 908 of the shape data in FIG. 9.

Finally, the fourth embodiment to aid in the understanding of the present invention will be described.

The present invention is based on the first and second embodiments, but in which an upper layer is configured to improve a spatial resolution of only a part of the partial image encoded in the lower layer. By means of this configuration, the area for the face of a figure can be encoded as a partial image in the lower layer, and only the part of the mouth can be encoded in high resolution in the upper layer, for example.

When the fourth embodiment is adapted to the first embodiment, the second shape data generating part 102 of the encoding apparatus shown in Fig. 1 can be used to generate the high resolution shape data corresponding to a part of the image. partial, and therefore the present invention can be implemented.

As a corresponding decoding apparatus, the apparatus shown in Fig. 2 can be used. Similarly, when this embodiment is adapted to the second embodiment, the high resolution shape data corresponding to a part of the partial image can be generated by the second generation part 102 of the shape data of the encoding apparatus shown in Figure 3, whereby the present invention can be implemented.

In this case, however, the signal line from the first shape data coding part 307 to the second shape data coding part 303 is not used. In accordance with the present invention a switch is provided for switching between the methods described in the fourth and second embodiments. This switch is provided between the first shape data coding part 307, and the second data coding part 303

ES 2 227 686 T3 of form. By using the method according to the fourth embodiment, the switch is turned off and the upper layer shape data (the second shape data) is independently encoded. By using the method as described in the second embodiment, the switch is connected and the information about the difference between the upper layer shape data (the second shape data) and the lower layer shape data ( the first shape data).

For the decoding apparatus according to a further aspect of the present invention, the apparatus shown in figure 4 is provided with a similar switch, which is arranged on a signal line connecting the first decoding part 404 of the data. shape data and the second decoding part 402 of the shape data in FIG. 4. The switch, such as with the switch in the encoding apparatus, controls the switching between the case in which the upper layer shape data (the second shape data) is independently decoded, and the case in which the upper layer shape data is decoded. data representing the difference between the upper layer shape data (the second shape data) and the bottom layer shape data (the first shape data), and in that the difference data is added to the lower layer shape data (the first shape data) to decode the upper layer shape data (the second shape data).

According to the moving picture coding apparatus and the moving picture decoding apparatus according to the above embodiments, the following favorable effects can be obtained:

(1) To be able to carry out the encoding and decoding of a partial image that has a spatial hierarchy.

(2) In the first embodiment, the low resolution shape data is encoded as the shape data for a lower layer. Consequently, the redundancy associated with encoding the data in high resolution form in the lower layer can be eliminated. Consequently, the image in a lower layer can be encoded and decoded in the desired image quality, even with limited transmission or limited storage capacity.

(3) In the second embodiment, the information about the shape data in the lower layer is used to encode the shape data in the upper layer, and the information about the difference between the low-resolution shape data and the low-resolution shape data is encoded. high resolution shape data. Consequently, the amount of data required to encode the shape data in the upper layer can be reduced. Consequently, efficient encoding can be performed in the upper layer.

(4) In the third embodiment, the upper layer data is transformed to generate lower layer shape data having more levels of gradation.

Consequently, the stepped boundaries in the contour portion of the lower layer shape data may be rendered imperceptible. Consequently, the outline of a partial image on the display can be made smooth, and thus a physically favorable image can be obtained.

(5) In another embodiment of the present invention, assuming that the upper layer shape data is smaller than the lower layer shape data, hierarchical encoding can be performed to improve the spatial resolution of a part of the layer. bottom using the data in the top layer. Consequently, if there is a need to search within the content of only one image in a part of the lower layer, or if only a part in the lower layer has a complex texture and has to be displayed in high resolution, it will be possible to execute a appropriate hierarchical coding.