Image encoding device and image decoding device

The present invention relates to an image coding device that divides a high-resolution image into a plurality of partial images and compress-codes them in parallel, and an image decoding device that decodes the compressed-coded image.

The moving image is composed of a plurality of consecutive frames. Codecs installed in broadcasting station or home recording devices compress and encode moving images, for example, frame by frame or by utilizing differences between a plurality of frames. The amount of processing of the codec required for compression coding increases as the number of pixels (spatial resolution) and frame rate (temporal resolution) constituting one frame increase.

In order to compress and code such a moving image in real time, a method of spatially dividing one frame or dividing a plurality of frames in time to parallelize the compression coding process is well known. ing. However, since the compression coding process of moving images is a technology that efficiently reduces the amount of data by utilizing the spatial and temporal correlation of frames, the division process for parallelization can reduce the compression efficiency. There is sex. Therefore, a parallelization method that minimizes a decrease in compression efficiency due to division processing has been conventionally proposed. Although there are various combinations of these proposals, the basic idea can be roughly divided into two divisions: spatial processing division and temporal processing division (for example, Patent Document 1).

In "spatial processing division", a high-resolution video signal is spatially divided into a plurality of partial images. The individual partial images are codec-coded in parallel to obtain compressed data. Finally, the individual compressed data are integrated. This codec does not have to have the ability to process high-resolution images, but only needs to have the ability to process lower-resolution images. According to the division of spatial processing, the efficiency of in-frame compression utilizing the correlation between spatially adjacent pixels is reduced at the boundary divided into partial images. However, conventionally, there has been a device that does not reduce the efficiency of inter-frame compression by dividing the high-resolution image so that the partial images overlap each other by the search range of motion compensation. Spatial processing division has the advantage that the processing delay can be reduced because the processing unit for parallelization can be limited to one frame.

In "Division of temporal processing", the moving image is divided by the playback period with one frame or multiple frames as one unit while keeping the resolution of one frame as it is. As a result, a plurality of frames having a relatively short duration can be obtained. The individual frames are processed in parallel by the codec to obtain compressed data. Finally, the compressed data is integrated. The division of time processing can make the best use of the correlation between adjacent pixels in the frame, and thus does not reduce the efficiency of in-frame compression. However, as a condition that the frame groups can be processed independently in parallel, it is essential that there is no reference relationship between the frames, and the efficiency of inter-frame compression is lowered. For example, in MPEG-2 of Non-Patent Document 1, when parallel processing is performed in units of GOP (Group Of Picture), it is necessary to make a Closed GOP that has no reference relationship between GOPs. Further, a large-capacity frame buffer memory is required to process a plurality of GOPs in parallel, which causes a problem that the processing delay increases. Since the increase in processing delay makes it difficult to control the code amount for controlling the generated code amount to a predetermined amount, the conditions under which temporal division can be used for parallel processing are limited.

Therefore, conventionally, when a high-resolution moving image is divided and compressed and coded, the image is often spatially divided and processed in parallel.

<p><patcit num="1"><text>JP-A-59-97272</text></patcit></p>

<p><nplcit num="1"><text>ISO / IEC 13818-2 Information technology --Generic coding of moving pictures and associated audio information: Video</text></nplcit><nplcit num="2"><text>ISO / IEC 14496-10 Information technology --Coding of audio-visual objects --Part10: Advanced video coding</text></nplcit></p>

<p> When the input video is spatially divided and compressed and coded in parallel by a conventionally known method, since the plurality of spatially divided partial images contain different patterns, the input video is compressed and coded for each partial image. Difficulty is different. Therefore, in many compression codings that employ variable-length codes, when compression coding is performed with the same compression parameters, the amount of generated code varies for each divided partial image. On the other hand, if the compression parameters for each partial image are adjusted so that the amount of code generated for each partial image is even, the compression distortion for each partial image will vary, making it easier to visually detect the division boundaries on the screen. .. It is very difficult to control the amount of code so that the sum of the codes finally generated in all the partial images becomes a predetermined amount while processing the individual partial images in parallel independently.</p><p> In particular, when spatial processing is divided in the MPEG-4 AVC / H.264 method (for example, Non-Patent Document 2), which has been attracting attention with high compression efficiency in recent years, the above-mentioned problem becomes more remarkable. The MPEG-4 AVC / H.264 method achieves a significant improvement in compression efficiency by referring to adjacent pixels in one frame. However, at the boundary between the partial images, the correlation between pixels belonging to different partial images cannot be used even if they are spatially adjacent to each other. Therefore, when spatial image division is applied to the MPEG-4 AVC / H.264 method, it is inevitable that the in-frame compression efficiency will be significantly reduced compared to the conventional compression method.</p><p> The present invention has been made to solve the above problems, and an object of the present invention is to eliminate the bias of compression distortion between partial images when dividing one frame into a plurality of partial images and encoding them, and to eliminate the bias of compression distortion between the partial images. Is to make it difficult to detect.</p>

<p> The image coding apparatus according to the present invention divides an image of one frame into a plurality of fragment images, divides the plurality of fragment images into N sets (N: 2 or more integers), and outputs them as N partial images. An image exchange unit that exchanges at least one fragment image between the division unit and the N partial images and outputs N partial images in which each fragment image is mixed, and an image exchange unit output from the image exchange unit. It includes a coding unit that encodes the N partial images.</p><p> Each of the N partial images may correspond to each of the cases where the one-frame image is divided into a plurality of regions.</p><p> The image division unit further divides each of the first image division unit that divides the image of the one frame into the N partial images and the N partial images divided by the first image division unit. It may be provided with a second image dividing unit for dividing and outputting.</p><p> The second image dividing unit may divide each of the N partial images into fragment images obtained by an integral multiple of N.</p><p> The image dividing unit may generate a fragment image having the same shape as the slice shape predetermined in the coding unit.</p><p> The image dividing portion may generate a partial image composed of a plurality of fragment images having the same shape.</p><p> The image dividing unit may generate a partial image composed of fragment images from a plurality of fragment images having different shapes.</p><p> The slice is composed of a plurality of macroblocks, and the image dividing unit may generate the fragment image so that a plurality of adjacent macroblocks are included with respect to any macroblock.</p><p> The image division unit may determine the size of each of the N partial images so as to have a size that can be processed by the coding unit.</p><p> When the size of the image of the one frame is not an integral multiple of the size that can be processed by the coding unit, the image coding apparatus has the size of the image of the one frame that can be processed by the coding unit. An image extension unit that adjusts to an integral multiple may be further provided.</p><p> The image expansion unit is added by adding a plurality of pixels to the image of the one frame so that the size of the image of the one frame is adjusted to be an integral multiple of the size that can be processed by the coding unit. Each pixel value of the plurality of pixels may have a value that cannot be taken by the pixel value of the pixels constituting the image of the one frame.</p><p> The image expansion unit sets the size of the image of the one frame to an integral multiple of the size that can be processed by the coding unit so that the plurality of pixels to be added are evenly included in each of the N partial images. You may adjust so that.</p><p> In the image coding apparatus according to the present invention, a decoding unit that decodes each of the N encoded partial images and a plurality of decoded N partial images each having a predetermined shape. At least one fragment image is exchanged between the N partial images based on a predetermined rule and an image dividing portion that divides into fragment images, and the N partial images in which each fragment image is exchanged are exchanged. It includes a decoded image exchange unit for output and a decoding integrated unit that integrates the N partial images output from the decoded image exchange unit into one image and outputs the decoded image.</p><p> In the coded N partial image data, as management information, information on the number of partial images divided from one frame image, information on the shape of the fragment image divided from each partial image, and fragment image Contains information on the number of images, at least one of the rules on how to exchange each fragment image at the time of encoding, the image coding apparatus said to be transmitted with data of N encoded partial images. A coded information acquisition unit for acquiring information is further provided, and at least one of the image division unit, the decoded image exchange unit, and the decoding integration unit operates based on the management information acquired by the coding information acquisition unit. You may.</p>

<p> According to the present invention, since at least one fragment image is exchanged between N sub-images, the bias of the pattern between the spatially divided sub-images is improved and the difficulty of compression is leveled between the sub-images. It becomes possible to change. Furthermore, even with a compression method that makes high use of the correlation of adjacent pixels in the frame, adjacent pixels are not referenced at the slice boundary, so compression by spatial image division is performed by matching the slice boundary and the division boundary of the fragment image. It is possible to prevent a decrease in efficiency.</p>

<figref num="1">It is a block diagram which shows the structure of the image coding apparatus 10 in Embodiment 1. FIG.</figref><figref num="2">It is a figure which shows typically the process of the 1st image division part 100.</figref><figref num="3">It is a figure which shows typically the process of the 2nd image division part 101a to 101d.</figref><figref num="4">It is a figure which shows typically the exchange method of the fragment image in the image exchange part 102.</figref><figref num="5">It is a figure which shows an example of the mixing rule of the image exchange part 102.</figref><figref num="6">It is a flowchart which shows the operation procedure of the image coding apparatus 10.</figref><figref num="7">It is a figure which shows the division method of the partial image of the 2nd image division part 101a to 101d in Embodiment 2. FIG.</figref><figref num="8">It is a figure which shows the division state of the input image by Embodiment 2.</figref><figref num="9">It is a figure which shows the partial image group 900 which the fragment image was mixed by the image exchange part 102 by Embodiment 2.</figref><figref num="10">It is a figure which shows the example when the input image is divided into 8 partial images of 2 rows and 4 columns by the 1st image division part 100.</figref><figref num="11">It is a figure which shows the structure of the image coding apparatus 20 according to Embodiment 3.</figref><figref num="12">It is a flowchart which shows the operation procedure of the image coding apparatus 20.</figref><figref num="13">It is a schematic diagram explaining the function of the image expansion part 900.</figref><figref num="14">(a) is a diagram showing an expanded input image 1300 in which invalid pixels are added around the input image, and (b) is an L-shaped and cross-shaped invalid pixel in the input image. It is a figure which shows the extended input image 1302 attached to the mold.</figref><figref num="15">It is a block diagram which shows the structure of the image decoding apparatus 30 according to Embodiment 4.</figref><figref num="16">It is a figure which shows typically the process of the decoded image division part 1402a to 1402d.</figref><figref num="17">It is a figure which shows typically the process of the decoded image exchange part 1403.</figref><figref num="18">It is a figure which shows the integration method of the partial image group 1800 in the decoding integration part 1404.</figref><figref num="19">It is a flowchart which shows the operation procedure of the image decoding apparatus 30.</figref><figref num="20">It is a figure which shows the structure of the image decoding apparatus 40 by Embodiment 5.</figref>

Hereinafter, embodiments of the image coding device and the image decoding device according to the present invention will be described with reference to the accompanying drawings.

In the following embodiments, the image coding device and the image decoding device will be described as recording devices and playback devices provided in the broadcasting station, respectively. However, this is an example as an embodiment. Only the image decoding device may be a home-use playback device, and the image coding device may be a home-use recording device.

(Embodiment 1) (1-1. Configuration of Image Coding Device) FIG. 1 is a block diagram showing a configuration of an image coding device 10 in the present embodiment. The image coding device 10 includes a first image dividing unit 100, a second image dividing unit 101a to 101d, an image exchange unit 102, and a coding unit 103a to 103d.

The first image dividing unit 100 receives the image data. The first image dividing unit 100 spatially divides the received image data into image data so that the image has an appropriate size. The data of the divided image (hereinafter referred to as "partial image") is coded in parallel in the coding units 103a to 103d through a later process.

FIG. 2 schematically shows the processing of the first image dividing unit 100. The first image dividing unit 100 receives the image data of the input image 300 and performs spatial division processing. As a result of the division process, four partial image groups 301 are obtained. In the present embodiment, the first image dividing unit 100 divides the input image into four rectangular partial images having the same size as each other. As illustrated, in the present specification, for convenience, each partial image shall be numbered 0, 1, 2, 3 in order from the upper left to distinguish them.

In the present embodiment, the input image is, for example, an image having a size of 8192 pixels horizontally × 4320 pixels vertically, so-called 8K4K. Then, it is assumed that each partial image 0 to 3 is an image having a size of horizontal 2048 pixels × vertical 1080 pixels.

The above-mentioned input image size and the image size after division are examples. As another example, the first image dividing unit 100 may receive data of an image of 7680 pixels horizontally × 4320 pixels vertically and divide it into a partial image of 1920 horizontal pixels × 1080 vertical pixels.

Further, in the present embodiment, the first image division unit 100 is divided into four rectangular regions of the same size, but the shape and the number of divisions are not limited to this. For example, the number of divisions may be at least 2.

Hereinafter, the processing of the first image dividing unit 100 is referred to as "first division processing" for convenience.

The second image dividing units 101a to 101d shown in FIG. 1 receive the partial images 0 to 3 of the partial image group 301 divided by the first image dividing unit 100, and further spatially divide each partial image. .. In the present embodiment, it is assumed that the number of the second image division units is equal to the number of partial images generated by the first image division unit 100. The second image division units 101a to 101d perform partial image division processing in parallel. The image obtained by further dividing the divided image is hereinafter referred to as a "fragment image" in the present specification.

FIG. 3 schematically shows the processing of the second image dividing portions 101a to 101d.

The second image division units 101a to 101d receive the partial images 0 to 3, respectively, and perform spatial division processing. Specifically, the second image division units 101a to 101d subdivide the partial images 0 to 3 into smaller fragment image groups 401. The number of fragment images constituting the fragment image group 401 is a multiple of 4, which is the number of partial images generated by the first image dividing unit 100.

In the present embodiment, the second image dividing units 101a to 101d divide each partial image into four strip-shaped fragment images. Each fragment image constituting the fragment image group 401 obtained by the division processing of the second image division portions 101a to 101d is numbered with hyphens 0, 1, 2, 3 after the number of each partial image 0 to 3. To distinguish.

Assuming that each partial image 0 to 3 is an image having a size of horizontal 2048 pixels × vertical 1080 pixels, the size of the fragment image is assumed to be an image having a size of horizontal 2048 pixels × vertical 256 pixels.

In the present embodiment, the second image division portions 101a to 101d divide the partial image into four strips, but the shape and the number of divisions are not limited to this. Further, the number of divisions of the second image division unit 101 does not have to be an integral multiple of the number of partial images divided by the first image division unit 100. As will be described in detail later, the leveling effect can be obtained if there are a sufficient number of fragment images that can be exchanged and mixed.

In the present embodiment, in order to process the partial images in parallel, it is assumed that there are as many second image division portions 101 as there are partial images. However, this is just an example. The second image dividing unit 101 may be at least one, and the dividing process may be performed in a time division manner.

Hereinafter, the processing of the second image dividing units 101a to 101d will be referred to as "second division processing" for convenience.

Further, in the present embodiment, the first image division unit 100 and the four second image division units are separately provided. However, these may be done in one component. That is, in the previous example, one input image may be spatially divided to generate 16 fragment images without generating partial images. FIG. 1 shows an image dividing unit 105 that divides one input image into 16 fragment images.

The image exchange unit 102 shown in FIG. 1 performs a process of exchanging fragment images constituting the partial image between the partial images.

FIG. 4 schematically shows a method of exchanging fragment images in the image exchange unit 102. The fragment image group 500 generated by the first image dividing unit 100 and the second image dividing units 101a to 101d includes four fragment images for each partial image. The total number of fragment images is 16.

The image exchange unit 102 exchanges fragment images between the partial images to generate a mixed fragment image group 501, and further groups the fragment images four by four to form partial images 0', 1', 2', and 3'. To generate. In the present embodiment, since the second image dividing portions 101a to 101d are divided into fragment images so as to be a multiple of the number of the partial images divided by the first image dividing portion 100, each partial image 0' , 1', 2', and 3'can be exchanged so that fragment images of all partial images are evenly included.

As an example, according to FIG. 4, the partial image 0'in which the fragment images are mixed is composed of 0-0, 1-1, 2-2, and 3-3. Similarly, the partial images 1', 2', and 3'in which the fragment images are mixed also include the fragment images of all the partial images 0 to 3.

FIG. 5 shows an example of the mixing rule of the image exchange unit 102. In this embodiment, the image mixing rule is defined by the following equation. Mixed Fragment Image [x]-[y] = Fragment Image [(x + y)% N]-[y]

Here, [x]-[y] are numbers (0-0, 0-1, 0-2, 0-3, 1-0, etc.) indicating fragment images in FIGS. 3 to 5. N is the number of divisions of the first image division unit 100. M is the number of divisions of the second image division portions 101a to 101d. Also, x is an integer from 0 to N-1, and y is an integer from 0 to M-1. In this embodiment, N = 4 and M = 4. The% in the formula means the modulo operation.

In the present embodiment, the embodiment in which the fragment images are exchanged is described based on the above equation, but the image exchange unit 102 only needs to be able to mix the fragment images of all the partial images as evenly as possible. The same effect can be obtained even if the mixed rule is not disclosed. In the present specification, when at least one fragment image of each partial image is included in the partial image in which the fragment images are mixed, it is said that the fragment images are evenly mixed.

However, it does not have to be mixed exactly evenly. For example, at least one fragment image constituting the partial image may be exchanged with the fragment image constituting the other partial image. This is because even in such a case, the effect of the present invention can be obtained as compared with the case where the mixture is not mixed.

Each of the coding units 103a to 103d encodes the exchanged partial image of the fragment image output by the image exchange unit 102. Coding is based on, for example, the MPEG-4 AVC / H.264 standard. In the present embodiment, the coding units 103a to 103d are provided as many as the number of partial images generated by the first image dividing unit 100, and the partial images are coded in parallel. The coding units 103a to 103d output the coded image data. Each image data may be output as separate data, or four data may be output as a set. In either case, a certain rule may be provided for the arrangement order of the data. For example, even if the image data generated by the coding unit 103a is first, the image data generated by the coding unit 103b, the image data generated by the coding unit 103c, and the image data generated by the coding unit 103d are output in sequence. Good.

In addition, information indicating which of the coding units 103a to 103d the coded image data has been processed by which of the coding units 103a to 103d may be added to each of the output coded image data. This information is referred to in the decoding process described later. This information may be embedded in the UMID information, for example as metadata in the coded data. The UMID information is identification information uniquely added to each coded data.

In this embodiment, in order to perform coding processing on the partial images in parallel, a mode in which the coding units 103a to 103d are provided as many as the number of the partial images will be described. Such a configuration is currently considered to be cost advantageous. The reason is that although the development of circuits that can encode 8K4K size images as they are without dividing them is underway, such circuits are still expensive, while divided size images can be displayed. This is because encodeable circuits are easily and relatively inexpensively available. However, this configuration is an example. At least one coding unit may be provided. The coding process may be performed in parallel by time division.

(1-2. Operation of Image Coding Device 10) The operation of the image coding device 10 in the present embodiment will be described below with reference to FIG.

FIG. 6 is a flowchart showing the operation procedure of the image coding device 10.

One frame of image data is input to the image coding device 10 (step 201). The first image dividing unit 100 spatially divides the input image data into image data having an appropriate image size (step 202). As a result, a plurality of partial images are obtained.

The second image dividing units 101a to 101d further spatially divide each of the plurality of partial images generated by the first image dividing unit 100 (step 203). As a result, a plurality of fragment images can be obtained from each partial image.

The image exchange unit 102 exchanges fragment images constituting each partial image between the partial images (step 204).

The coding units 103a to 103d receive the partial images 0', 1', 2', and 3'in which the fragment images are mixed, which are output by the image exchange unit 102, respectively, and are based on, for example, MPEG-4 AVC / H.264. And encode (step 205). The coding method may be an MPEG-2 method or the like. The image coding device 10 outputs the compressed image data encoded by the coding units 103a to 103d, respectively (step 206).

In this way, the coding units 103a to 103d do not directly encode the partial image divided by the first image dividing unit 100, but further convert the partial image into a fragment image by the second image dividing unit 101a to 101d. The partial image created by dividing the image and exchanging the fragment images with the image exchange unit 102 is encoded. This makes it possible to improve the bias of the pattern between the spatially divided partial images and to level the difficulty of compression between the partial images in which the fragment images are mixed.

(Embodiment 2) It is assumed that the second image division portions 101a to 101d according to the first embodiment generate a strip-shaped fragment image.

The second image dividing portions 101a to 101d according to the present embodiment generate fragment images having a shape other than a strip shape. Hereinafter, the shapes of the second image division portions 101a to 101d and their fragment images will be mainly described.

The configuration of the image coding device according to the present embodiment is the same as the configuration of the image coding device 10 according to the first embodiment. Therefore, the image coding device according to the present embodiment will be described as the "image coding device 10" shown in FIG.

Hereinafter, the configuration and operation of the second embodiment different from the first embodiment will be described. The components and their operations which are not particularly described are the same as those in the first embodiment.

FIG. 7 shows a method of dividing the partial image of the second image dividing portions 101a to 101d in the present embodiment. The second image dividing units 101a to 101d divide each partial image divided into rectangles as shown in FIG. 2 into 10 fragment images having a shape as shown in FIG. 7. The number of divisions of the fragment image is 10 because the number of slices of the coding units 103a to 103d is 10. The number of slices and the shape are predetermined in the coding units 103a to 103d. In each case, the second image dividing units 101a to 101d divide the partial image into fragment images according to the specifications of the coding units 103a to 103d.

When a plurality of coding units 103a to 103d are used for coding in parallel, the second image dividing parts 101a to 101d divide the partial image into fragment images according to the division boundary of the slices of the coding parts 103a to 103d. To do. The advantage of such division is that the compression efficiency does not decrease even in coding using the correlation between adjacent pixels (or adjacent macroblocks) as in the MPEG-4 AVC / H.264 method. In the MPEG-4 AVC / H.264 method, adjacent pixels are not referenced between slices. Therefore, dividing the fragment image according to the division boundary of the slice does not affect the compression efficiency. In the present embodiment, the second image division units 101a to 101d generate a fragment image so that a plurality of adjacent macroblocks are included with respect to any macroblock constituting the slice. As a result, coding is performed using adjacent macroblocks. The term "adjacent" as used herein includes not only horizontal adjacency but also vertical adjacency.

In FIG. 8, the first image division unit 100 divides the input image into four rectangular partial images of the same size, and the second image division units 101a to 101d align with the slice division boundaries of the coding units 103a to 103d. The division state when the image is subdivided into 10 fragment images is shown.

The fragment images divided by the second image dividing portions 101a to 101d do not all have the same shape. Therefore, the shape of the fragment images exchanged between the partial images by the image exchange unit 102 is restricted. The image exchange unit 102 according to the present embodiment exchanges fragment images having the same shape between the partial images to generate a partial image in which the fragment images are mixed.

FIG. 9 shows a partial image group 900 in which fragment images are mixed by the image exchange unit 102 according to the present embodiment.

In this embodiment, the image mixing rule is defined by the following equation. Mixed Fragment Image [x]-[y] = Fragment Image [(x + y)% N]-[y]

Here, [x]-[y] are numbers (0-0, 0-1, 0-2, 0-3, 1-0, etc.) indicating fragment images in FIGS. 8 and 9. N is the number of divisions of the first image division unit 100. M is the number of divisions of the second image division portions 101a to 101d. Also, x is an integer from 0 to N-1, and y is an integer from 0 to M-1. In the second embodiment, N = 4 and M = 10. The% in the formula means the modulo operation.

In the present embodiment, the number of divisions (10) of the fragment image by the second image division units 101a to 101d is not a multiple of the number of divisions (4) of the partial image by the first image division unit 100, so that it is complete. Fragment images cannot be mixed evenly. However, the effect of leveling can be fully expected by exchanging and mixing the fragment images between the partial images.

The exchange rule of the image exchange unit 102 described above is an example. Other mixed rules may be adopted. It is sufficient that at least one fragment image constituting the partial image is exchanged with the fragment image constituting another partial image having the same shape. Even in this case, the effect of the present invention can be obtained as compared with the case where the mixture is not mixed.

In the present embodiment, an example in which the number of slice divisions of the coding units 103a to 103d is 10, but the number of slice divisions does not have to be 10. Further, the number of divisions into partial images by the first image division unit 100 is not limited to 4.

Hereinafter, a modified example regarding the number of divisions into a partial image will be described.

For example, FIG. 10 shows an example when the input image is divided into eight partial images of 2 rows and 4 columns by the first image division unit 100. FIG. 10 also shows 10 fragment images obtained by dividing each partial image by the second image dividing portion. In this modification, it is sufficient that eight second image division portions are provided. However, the four second image division units 101a to 101d may process two partial images in a time division manner. Alternatively, one second image division unit may be provided, and eight partial images may be processed in parallel by time division. The shape of the fragment image constituting each partial image in this modification is as shown in FIG. 7.

After the fragment images are generated as shown in FIG. 10, the image exchange unit 102 exchanges fragment images having the same shape between the partial images. As a result, there is less bias in the pattern than in the example shown in FIG. By encoding such a partial image in the coding unit, it is possible to level the difficulty of compression between the partial images in which the fragment images are mixed.

(Embodiment 3) In the present embodiment, an image coding device in which an image expansion unit is added to the configuration of the image coding device 10 of the first embodiment or the second embodiment will be described.

FIG. 11 shows the configuration of the image coding device 20 according to the present embodiment. Among the components of the image coding device 20 shown in FIG. 11, those having the same functions as the components of the image coding device 10 of the first embodiment are given the same reference numerals, and the description thereof will be omitted.

The image expansion unit 900 has a function of expanding the size of the image of the input image data. The purpose of expanding the image size is to adjust the image size of the input image data so that a partial image having a size that can be processed by the coding units 103a to 103d can be obtained. If the size of the input image is not an integral multiple of the image size that can be processed by the coding units 103a to 103d, a partial image of a size that can be encoded by the coding units 103a to 103d cannot be obtained as it is. .. Therefore, in the present embodiment, the image expansion unit 900 is provided in front of the first image division unit 100, and the size of the image of the input image data is adjusted.

Hereinafter, the operation of the image coding apparatus 20 according to the present embodiment will be described with reference to FIGS. 12 and 13. FIG. 12 is a flowchart showing the operation procedure of the image coding device 20. Further, FIG. 13 is a schematic diagram illustrating the function of the image expansion unit 900. In the present embodiment, the difference from the operation of the image coding apparatus of the first embodiment is that the image expansion process (step 1101) exists, and the other processes are the same as those of the first embodiment. Omit.

First, the image expansion unit 900 determines whether or not the size of the input image 1200 is an integral multiple of the image size that can be processed by the coding units 103a to 103d. When it is an integral multiple, the image expansion unit 900 does not perform any further processing and sends the received input image data to the first image division unit 100. On the other hand, when it is determined that the image size is not an integral multiple, the image expansion unit 900 inserts invalid pixels until the size of the input image 1200 becomes an integral multiple of the image size that can be processed by the coding units 103a to 103d to expand the image size. To do. The "invalid pixel" is, for example, a pixel for padding in which the pixel values of all color components are null (for example, 0). The pixel value of the invalid pixel may be a value that cannot be taken by the pixel value of the pixels constituting the input image.

For example, in FIG. 13, it is assumed that the size of the input image 1200 is 7680 pixels horizontally × 4064 pixels vertically. On the other hand, it is assumed that the image size that can be processed by the coding units 103a to 103d is a size of horizontal 2048 pixels × vertical 1080 pixels.

At this time, the image expansion unit 900 determines that the size of the input image 1200 is not an integral multiple of the image size that can be processed by the coding units 103a to 103d. Then, as shown in FIG. 13, the image expansion unit 900 generates the input image 1203 expanded by inserting the invalid pixel 1201. The invalid pixel 1201 is added to the input image 1200 in an L shape to enlarge the input image by 512 pixels in the horizontal direction and 256 pixels in the vertical direction.

When the image expansion unit 900 sends the expanded input image 1203 to the first image division unit 100, the first image division unit 100 divides the image including the invalid pixel 1201 into a plurality of partial images. As a result, the output partial image group 1202 can be coded by using a plurality of existing coding units 103a to 103d (four in the above example) in parallel. The pixel value of the invalid pixel to be inserted may be any value.

However, in relation to the image decoding process described later, information on whether or not the decoded image is an image to which invalid pixels are added, and if invalid pixels are added, the number of pixels thereof. Is required. Therefore, the image expansion unit 900 can generate information indicating whether or not the image output from the image expansion unit 900 includes invalid pixels and information on the number of added pixels. For example, when an invalid pixel is added in common to a series of images constituting a moving image, the image expansion unit 900 may describe the above information in the management information accompanying the coded data. Alternatively, it may be embedded in the UMID information as management information (metadata) in the coded data. The UMID information is identification information uniquely added to each coded data.

The invalid pixel does not need to be added in an L shape. The method is arbitrary as long as it can be added so that invalid pixels are evenly included in the divided partial image. For example, FIG. 14 (a) shows an expanded input image 1300 with invalid pixels added around the input image. The number of pixels to be added in the horizontal direction and the vertical direction is divided into two, and each of them is added around the input image. In the example of FIG. 14A, the first image dividing unit 100 divides the expanded input image 1300 into four equal parts to generate a partial image group 1301.

Further, FIG. 14 (b) shows an expanded input image 1302 in which invalid pixels are added in an L-shape and a cross shape to the input image. The number of pixels to be added in the horizontal and vertical directions is divided into two parts, which are added to the horizontal center and right edge of the input image, and to the vertical center and bottom of the input image. Will be added. In the example of FIG. 14B, the first image dividing unit 100 divides the expanded input image 1302 into four equal parts to generate a partial image group 1303.

By adding the invalid pixels in this way, in the partial image group 1301 or the partial image group 1303, the insertion area of the invalid pixels can be made the same between the partial images, and the complexity of the pattern can be leveled.

(Embodiment 4) (4-1. Configuration of Image Decoding Device) Embodiments 1 to 3 have described an image coding device that encodes image data.

The present embodiment and the fifth embodiment describe an image decoding apparatus that decodes the encoded image data. More specifically, in the present embodiment, an apparatus for decoding an image encoded by the image coding apparatus according to the first embodiment will be mainly described.

FIG. 15 is a block diagram showing a configuration of the image decoding device 30 according to the present embodiment. The image decoding device 30 includes decoding units 1401a to 1401d, decoding image division units 1402a to 1402d, a decoding image exchange unit 1403, and a decoding integration unit 1404.

The decoding units 1401a to 1401d receive the encoded partial image and decode it. In the present embodiment, the decoding units 1401a to 1401d decode the encoded partial image data in parallel. Regarding the rules by which the decoding units 1401a to 1401d receive each encoded data, for example, the encoded image data is output from the encoding units 103a, 103b, 103c and 103d according to the first embodiment. When they are sequentially recorded on the recording medium, they may be read out in that order and sequentially sent to the decoding units 1401a, 1401b, 1401c and 1401d.

In this embodiment, in order to perform decoding processing of partial images in parallel, a mode in which decoding units 1401a to 1401d are provided as many as the number of partial images will be described. This configuration is currently considered to be cost effective. The reason is the same as the advantage of providing the coding units 103a to 103d as many as the number of partial images in the first embodiment. However, this configuration is an example. Only one decoding unit is required, and the decoding process may be performed in a time division manner.

Before encoding, the decoded image dividing units 1402a to 1402d spatially display the image of the image data output by the decoding units 1401a to 1401d according to the dividing method adopted by the second image dividing units 101a to 101d. Divide to generate multiple fragment images.

For example, when the present invention is implemented as one recording / playback device, the recording / playback device operates as an image coding device according to the above-described embodiment at the time of coding, and at the time of decoding, the present embodiment or the fifth embodiment described later. Operates as an image decoding device. Such a recording / playback device determines what division method was adopted at the time of coding, for example, based on the information indicating the division method held inside, or based on a preset division method. It can be easily known at the time of decoding. In addition, the image coding apparatus may attach the information indicating the division method to the coded image data, and the image decoding apparatus may specify the division method with reference to the information. The latter method is also effective when the image coding device and the image decoding device are separate bodies.

In the present embodiment, the division method at the time of coding is determined in advance, and the decoded image division units 1402a to 1402d perform the division process based on the division method.

FIG. 16 schematically shows the processing of the decoded image dividing units 1402a to 1402d.

The decoded image dividing units 1402a to 1402d receive the decoded partial image group 1600 output by the decoding units 1401a to 1401d. The decoded image dividing units 1402a to 1402d divide into fragment images 1601 by a predetermined dividing method. As a result of the division, each of the partial image groups 1600 is divided into four strip-shaped fragment images 1601. This division method is the same as the division method in the first embodiment, and the shape and size of each fragment image are the same as those of the fragment image in the first embodiment.

The fragment image group 1601 generated by the decoded image division units 1402a to 1402 has the same arrangement as the mixed fragment image group 501 generated by the image exchange unit 102, as shown in FIG. 4, for example. Please note.

In the present embodiment, the decoded image dividing units 1402a to 1402d generate fragment images by the same method as the dividing method according to the first embodiment. However, the shape of the division and the number of divisions may be matched to the division method divided by the second image division portions 101a to 101d at the time of coding. Therefore, the fragment image is not limited to the strip-shaped shape. For example, the division method adopted in the second embodiment as shown in FIG. 7 can also be adopted. When the second image division parts 101a to 101d generate a fragment image from a partial image according to a slice division that is not a simple horizontal division, the decoded image division parts 1402a to 1402d are divided by the same division method as this division method. The image will be divided into fragment images.

The decoded image dividing units 1402a to 1402d divide the partial images output by the decoding units 1401a to 1401d in parallel, and output a fragment image. In the present embodiment, in order to divide the partial images in parallel, a mode in which the decoded image dividing units 1402a to 1402d are provided by the number of the partial images will be described, but at least one of the decoded image dividing units 1402a to 1402d is described. Anyway, the division process may be performed in a time division manner. For example, FIG. 15 shows an image division unit 1402 in which the decoded image division units 1402a to 1402d are configured as one.

See FIG. 15 again.

At the time of coding, the decoded image exchange unit 1403 performs a process of exchanging fragment images constituting the partial image between the partial images according to the image exchange method performed by the image exchange unit 102. In the present embodiment, the decoded image exchange unit 1403 stores the image exchange at the time of encoding in advance, and performs the exchange process based on the image exchange.

FIG. 17 schematically shows the processing of the decoded image exchange unit 1403.

The decoded image exchange unit 1403 exchanges fragment images constituting the partial image between the partial images based on the image exchange method performed by the image exchange unit 102 at the time of coding. For example, in relation to the first embodiment, the image mixing rule of mixed fragment image [x]-[y] = fragment image [(x + y)% N]-[y] has been described. The decoded image exchange unit 1403 performs an operation from the right side to the left side of this calculation formula to exchange fragment images.

In the present embodiment, the partial image is composed of the fragment image group 1700 mixed by the image exchange unit 102. The fragment image group 1601 is a partial image 0', 1', 2', 3'generated by the image exchange unit 102. The decoded image exchange unit 1403 performs an exchange process opposite to the exchange process performed by the image exchange unit 102, and generates a fragment image group 1701 after the exchange. This fragment image group 1701 is output as partial images 0,1,2,3. The partial images 0,1,2,3 are the same as the partial images 0,1,2,3 generated by the second image dividing portions 101a to 101d in the first embodiment.

The decoding integration unit 1404 integrates the partial images output by the decoding image exchange unit 1403 and collectively outputs them as one decoded image data.

FIG. 18 shows an integration method of the partial image group 1800 in the decoding integration unit 1404. Decoding integrating unit 1404 receives a partial image 0,1,2,3 output from the decoded image replacement section 1403, its in a predetermined arrangement method to arrange the these. For example, the partial image 1 is placed to the right of the partial image 0, the partial image 2 is placed below the partial image 0, and the partial image 3 is placed to the right of the partial image 2.

The decoding integration unit 1404 constructs and outputs one image 1801 from the partial image group 1800. As a result, each partial image is integrated as an output image 1801 in a positional relationship corresponding to the arranged position.

The above-mentioned processing is the reverse of the processing in which the first image dividing unit 100 in the first embodiment generates a partial image group and outputs the partial images 0, 1, 2, and 3.

In the present embodiment, the decoding integration unit 1404 has described an example of integrating four rectangular regions of the same size into one image, but the shape of the division and the number of divisions are the same as the division method at the time of coding. It suffices, and is not limited to the example shown in FIG.

(4-2. Operation of Image Decoding Device) The operation of the image decoding device 30 in the present embodiment will be described below with reference to FIG.

FIG. 19 is a flowchart showing the operation procedure of the image decoding device 30.

In the present embodiment, it is assumed that the coded image data input to the image decoding device 30 is the coded image data generated by the image coding device in the first embodiment.

The coded image data is input to the image decoding device 30 (step 1501).

The decoding units 1401a to 1401d decode the input coded image data based on the coding method at the time of coding (step 1502). For example, the decoding units 1401a to 1401d decode the image based on the MPEG-4 AVC / H.264 method. The decoding method is not limited to the MPEG-4 AVC / H.264 method, and may be the same as the coding method.

The decoded image dividing units 1402a to 1402d spatially divide the partial image output by the decoding units 1401a to 1401d according to the dividing method divided by the second image dividing units 101a to 101d (step 1503).

The decoded image exchange unit 1403 exchanges fragment images constituting each partial image between the partial images (step 1504).

The decoding integration unit 1404 combines the partial images output by the decoding image exchange unit 1403 into one decoded image data (step 1505). The image decoding device 30 outputs the decoded image data collected by the decoding integration unit 1404 (step 1506).

In this way, the decoding units 1401a to 1401d improve the bias of the pattern between the spatially divided partial images at the time of coding, and decode the encoded data in which the difficulty level of compression is leveled between the partial images. By doing so, it is possible to level the processing at the time of decoding. Further, by performing image exchange corresponding to the image exchange at the time of coding, the image data before encoding can be correctly decoded.

Even when the image is divided by the division method shown in FIG. 10 and the fragment images are exchanged and encoded, the image decoding apparatus 30 is decoded by performing the processing according to the above processing. The previous input image can be constructed and output.

(Embodiment 5) In the fifth embodiment, an image decoding device in which a coding information acquisition unit is added to the configuration of the image decoding device 30 of the fourth embodiment will be described.

FIG. 20 shows the configuration of the image decoding device 40 according to the present embodiment. Among the components of the image decoding device 40 shown in FIG. 20, those having the same functions as the components of the image decoding device 30 of the fourth embodiment are given the same reference numerals, and the description thereof will be omitted.

In FIG. 20, the coding information acquisition unit 1900 acquires information on the division method and the exchange method at the time of coding from the input coded image data. For example, each image coding apparatus according to the first to third embodiments encodes the division method and the exchange method in the first image division unit 100, the second image division units 101a to 101d, and the image exchange unit 102 at the time of coding. Suppose it was stored in the data. For example, in the fifth embodiment, it is assumed that the division method and the exchange method are embedded in the UMID information as metadata in the coded data.

The coding information acquisition unit 1900 acquires information on the division method and the exchange method in the coded data from the UMID of the metadata. Then, the coding information acquisition unit 1900 outputs the information of the division method to the decoding image division unit 1202, outputs the information of the exchange method to the decoding image exchange unit 1203, and outputs the information of the division method to the decoding integration unit 1404. ..

In the fifth embodiment, the division method and the exchange method are embedded in the UMID information as metadata in the coded data, but the place where the information of the division method and the exchange method is embedded is the UMID in the coded data. It is not limited to. It may be stored in the management information accompanying the coded data.

Hereinafter, among the operations of the image decoding device 40 according to the present embodiment, the differences from the image decoding device 30 according to the fourth embodiment will be described.

In the fourth embodiment, it is assumed that the decoded image dividing units 1402a to 1402d store the division method at the time of coding in advance. In the present embodiment, the decoded image division units 1402a to 1402d perform division processing based on the division information at the time of coding input from the coding information acquisition unit 1900.

Further, in the fourth embodiment, it is assumed that the decoded image exchange unit 1403 stores the exchange method at the time of coding in advance. In the present embodiment, the decoded image exchange unit 1403 performs the exchange process based on the exchange information at the time of coding input from the coding information acquisition unit 1900.

Further, in the fourth embodiment, it is assumed that the decoding integration unit 1404 stores the division method at the time of coding in advance. In the present embodiment, the decoding integration unit 1404 performs the integration process based on the division information at the time of coding input from the coding information acquisition unit 1900.

In this way, by exchanging information on the division method and exchange method at the time of coding between the image coding device and the image decoding device as a part of the coded data, it is possible to flexibly change the division method and the exchange method. It is possible to deal with this, and it is possible to automatically and correctly decode and integrate the image before division.

The processing of the device according to each of the above-described embodiments and variations thereof may be realized by one or more processors, which are computers, executing a computer program composed of various execution codes on the memory. Such a computer program defines processing procedures based on the flowcharts shown in, for example, FIGS. 6, 12, and 19. As a product, it can be recorded on a recording medium typified by an optical disk such as a CD-ROM or a semiconductor memory such as a memory card and distributed on the market, or transmitted through a telecommunication line such as the Internet. The processor can also be realized as hardware such as a DSP, a chip circuit, and an optical disk controller in which a computer program is incorporated in a semiconductor circuit.

The image coding device and the image decoding device of the present invention make it possible to use a plurality of lower resolution codecs in parallel when compressing and coding a high resolution moving image, including a video camera. It can be applied to various image recording devices and is useful.

100 First image division 101a to 101d Second image division 102 Image exchange 103a to 103d Coding unit 104 Integration unit 900 Image expansion unit 1401a to 1401d Decoding unit 1402a to 1402d Decoded image division 1403 Decoded image exchange Part 1404 Decoding integration part 1900 Coding information acquisition part