Motion picture processing method and device

Description: TECHNICAL FIELD [Detailed description of the invention]

【0001】

[Industrial application field]

The present invention relates to a method and an apparatus for processing a moving image, and more particularly to a method and an apparatus for performing region division of a moving image and interpolation of a motion vector based on the region division and other processing.

【0002】

[Conventional technology]

Generally, motion compensation prediction coding is used in a moving image coding / decoding device that compresses and encodes moving image data and transmits the moving image data. In motion compensation prediction coding, an operation called motion detection is required to detect motion vector indicating the direction and magnitude of image motion between adjacent frames. This motion detection will be briefly described.

【0003】

Now, suppose that the frame image at a certain time is FIG. 18 (a) and the next frame image is the same figure (b). In this case, the animal body (ball in the figure) 82 moves from the lower left to the lower right in front of the background 81 where many horizontal lines are drawn. Although the image is schematically drawn as a binary image in FIG. 18, in reality, motion detection is mainly performed on a shading image.

【0004】

As a typical motion detection method, Reference 1: "Multidimensional signal processing of TV images" (written by Takahiko Fukibuki, published by Nikkan Kogyo Shimbun, 1988), "Measurement of motion vector by matching" described in pp.203-204 Are known. Explaining this method, first, in the frame image of FIG. 18B, the representative points 91 of the motion vector detection are arranged in a grid pattern at N pixel intervals as shown in FIG. 19 (here, N = 4). Then, an image of a square block area of 4 × 4 pixels centered on each representative point 91 is cut out from the frame image of FIG. 18 (b), and the pattern matching is performed on FIG. 18 (a) to obtain the same image pattern. Let the vector to a certain place be the motion vector 92 of the representative point. The motion vector 92 is represented by an arrow symbol, and a representative point without an arrow means that there is no motion (motion vector).

【0005】

Next, image composition using motion vectors will be described. In this image composition, the image one frame before shown in FIG. 18 (a) and the motion vector 92 of the representative point shown in FIG. 19 are given, and these are used to make the current frame image shown in FIG. 18 (b) as much as possible. Combine close images.

【0006】

As an image composition method, there is a method in which the operation at the time of vector detection is used as it is. That is, a region of 4 × 4 pixels one frame before, which is indicated by the motion vector of each representative point, is cut out and placed in a 4 × 4 block around the representative point to obtain a current frame image. As shown in FIG. 21, this corresponds to using the motion vector 92 of the representative point 91 of the block including the same pixel 93 as it is as the motion vector 94 of the pixel 93. In this way, if the motion vectors 92 of the adjacent representative points 91 are different, the distribution of the motion vectors (hereinafter referred to as the motion vector field) becomes discontinuous at the boundary of the block, and as shown in FIG. 20 (a), the background 81 Block-like distortion occurs in the portion.

【0007】

For the purpose of eliminating the discontinuity of the motion vector field at the block boundary, Reference 2: "16 kb / s coding of moving images" by Yuichiro Nakaya et al., Image Coding Symposium Proceedings pp.129-132, In 1992, motion compensation was devised based on the patch described in. In this document 2, the representative point of motion vector detection is at the apex of the triangular patch, but here, for comparison with the above method, the case where the same is applied to the four angles will be described.

【0008】

In the motion compensation based on this patch, the motion vector of each pixel 93 is obtained from the four representative points 91 around it by linear interpolation such as bilinear interpolation, so that the portion discontinuous in the vector field is obtained as shown in FIG. Does not occur. Therefore, the block-like distortion does not occur in the composite image, but the problem that the background is distorted occurs as shown in FIG. 20 (b).

【0009】

Originally, there is a discontinuity in the motion vector field of the pixel. In the example of FIG. 18 (b), the absolute value of the motion vector of the pixel of the animal body 82 is large, but the motion vector is zero in the background 81, and there is a discontinuity between the two. .. Since the conventional motion detector does not detect the boundary where such motion vectors are discontinuous, it is not possible to obtain an accurate motion vector in image composition. As a result, in the moving image coding / decoding apparatus using the motion compensation prediction coding, a lot of compressed reproduction distortion occurs in the reproduced image decoded by the image synthesis using the motion vector.

【0010】

On the other hand, if the movement region is set by fine undulating lines in pixel units, a huge amount of code is required to represent it. As a method for suppressing the amount of code, for example, in Reference 3: "Motion image coding method by three-dimensional motion estimation and arbitrary shape orthogonal conversion" by You Yokoyama, Image Coding Symposium Proceedings pp.77-80, 1992. As described, a method of expressing a motion region as a polygon is known.

【0011】

However, even in the method described in this document 3, the coordinates of the vertices of the polygon must be expressed in absolute coordinates, which requires a relatively large amount of code. For example, if there are 256 possible values for the x and y coordinates, then 8 bits are required to represent these x and y coordinates, and if a total of 20 vertices are represented, then (8 bits + 8 bits) x 20 = 320 bits. Further, in the method of Reference 3, since the part where the difference between frames is large is simply set as the motion region and approximated by the polygon, there is a problem that the region is not necessarily the polygon region where the motion prediction error is minimized. is there.

【0012】

[Problems to be Solved by the Invention]

As described above, in the conventional motion detection method, block distortion occurs in the image synthesized using the motion vector due to the discontinuity of the motion vector field, and in the method in which the discontinuity is eliminated, the background is distorted. was there.

【0013】

Further, the conventional motion region setting method has a problem that a large amount of code is required to represent the region and a motion prediction error is large.

【0014】

An object of the present invention is to provide a moving image processing method and a moving image processing apparatus capable of solving such conventional problems.

【0015】

[Means for solving problems]

In the first moving image processing method according to the present invention, a number of representative points smaller than the number of pixels is set in a frame of moving image data, a motion vector of the representative points is detected, and the movement of the representative points in the frame. A motion discontinuity point is set indicating a motion vector boundary in which at least one of the boundary or direction and the absolute value of the region where the vector is detected and the region where the vector is not detected is different, and the frame is formed by a line connecting the motion discontinuities. It is characterized by being divided into a plurality of areas.

【0016】

In the second moving image processing method according to the present invention, a number of representative points smaller than the number of pixels is set in the frame of the moving image data, the motion vector of the representative point is detected, and the motion vector of the representative point is used as the motion vector. The first motion image processing method is applied to the motion vector interpolation method for obtaining the motion vector of each pixel in the frame by interpolation, and the motion vector of each pixel is divided into a plurality of regions as described above. It is characterized in that it is obtained by interpolation from the motion vector of the representative point inside. Here, "interpolation" includes "replacement" which is a 0th-order interpolation. Therefore, the motion vector of each pixel may be replaced by the motion vector of the representative point in the same region.

【0017】

In the first and second moving image processing methods, the motion discontinuity point may be on the line connecting the representative points, or may be any other point. The method of setting the position of the representative point or the motion discontinuity and the value of the motion vector of the representative point is not particularly limited. Can be used.

【0018】

The first motion image processing apparatus according to the present invention includes a storage means for storing at least one frame of motion image data sequentially input in frame units, and motion image data of past frames read from the storage means. A motion vector detecting means for detecting the motion vectors of a plurality of representative points set in the current frame from the moving image data of the frame, and a region where the motion vector of the representative point is detected and a region where the motion vector of the representative point is not detected by the motion vector detecting means. A line connecting a motion discontinuity detecting means for detecting a motion discontinuity point indicating a boundary of a motion vector having a boundary or an absolute value different from each other and a motion discontinuity point detected by the motion discontinuity detecting means. It is characterized by providing a region dividing means for dividing the current frame into a plurality of regions.

【0019】

Further, the second moving image processing device according to the present invention is characterized in that the motion detecting device includes a transmission means for transmitting information on the motion vector and the motion discontinuity detected as described above.

【0020】

Further, the third motion image processing device according to the present invention is an image compositing device, in which a storage means for storing at least one frame of past motion image data and a motion vector of a plurality of representative points set in the current frame. And the information of the motion discontinuity indicating the boundary of the motion vector in which at least one of the direction and the absolute value is different from the boundary or the direction between the region where the motion vector of the representative point is detected and the region where the motion vector is not detected is input, and these motion vectors are input. And a synthesis means for synthesizing the motion image data of the current frame from the information of the motion discontinuity point and the past motion image data read from the storage means, and the synthesis means connects the motion discontinuity points. The current frame is divided into a plurality of regions by a line, the motion vector of each pixel in the current frame is obtained by interpolating from the motion vector of the representative point in the same region, and the position specified by this motion vector is obtained. It is characterized in that the pixel value is taken out from the motion image data of the frame synthesized in the past to obtain the pixel value of the pixel.

【0021】

[Action]

In the moving image processing method according to the present invention, by connecting the motion discontinuity points with a line and dividing the frame into a plurality of regions, a region boundary substantially along the actual motion discontinuity line can be obtained. Therefore, using this region division result, it is determined which region each pixel belongs to, and from the result, the motion vector of the pixel is interpolated using the motion vector of one or more representative points in the same region. Can be sought. As a result, a motion vector that matches the actual motion can be obtained without the motion vector field being affected by the motion vectors in different regions that are discontinuous as in the conventional case.

【0022】

Further, since the motion region is set using the information of the motion vector, it is not necessary to add the information of which representative point the vertex is to be placed to the code again, and the information indicating which position between the representative points is to be placed. Since it is only necessary, the region can be expressed with a very small amount of code. For example, if the number of vertices that can be placed is 16 between the representative points, 4 bits x 20 = 80 bits are sufficient to express 20 vertices, which is compared with the method shown in Reference 3 above. The amount of code is much smaller. Also, if the number of vertices that can be placed in the same example is one, there is no need for bits to represent the area.

【0023】

Further, in the first moving image processing device, the moving image data is input to the storage means, the motion vector detecting means, and the motion discontinuity detecting means. The moving image data read from the storage means is input to the motion vector detecting means and the motion discontinuity detecting means. In the motion vector detecting means, the motion vector of the representative point is detected from the motion image data of the current frame and the previous frame. The motion vector output from the motion vector detecting means is input to the motion discontinuity detecting means.

【0024】

In the motion discontinuity detecting means, the motion discontinuity is detected between the representative point where the motion vector is detected and the representative point where the motion vector is a zero vector, that is, the motion vector is not detected. In this case, by connecting the detected motion discontinuities with a straight line or a curve, when the object is moving in front of a stationary image such as the background, the area of the moving object is accurately represented. it can. Further, if a motion discontinuity point is detected between body surface points having different motion vector directions, each region of a plurality of objects having different motion directions can be individually represented. Further, if a motion discontinuity point is detected between representative points having different absolute values of motion vectors, each region of a plurality of objects having different motion speeds can be individually represented.

【0025】

In the motion discontinuity detection, first, the motion discontinuity is temporarily set at a predetermined position between the representative points, and then the motion image data of the past frame and the motion prediction image data synthesized by the motion discontinuity are obtained. The position of the temporarily set motion discontinuity point may be modified so that the error from the moving image data of the current frame becomes small. Further, if the motion vector detecting means and the motion discontinuity detecting means send each other's detection results, the discontinuity is set only between the representative points having different motion vectors, or the motion vector is set at the stage where the discontinuity is set. Can also be modified.

【0026】

Further, when the moving image processing device is applied to the image synthesizing device, the motion vector of the representative point and the information of the motion discontinuity point are input to the pixel motion vector calculating means in the synthesizing means. In the pixel motion vector calculation means, the motion vector for each pixel is obtained by performing 0th-order interpolation, 1st-order interpolation, or higher-order interpolation from the representative point vector, but the discontinuity obtained by connecting the motion discontinuities is obtained. By not using the motion vector of the representative point in the region beyond the continuous line, a motion vector close to the actual pixel motion vector can be obtained. The pixel motion vector calculated in this way is sent to the storage means. The storage means stores and holds the sequentially synthesized moving image data, and sends the pixel value at the position specified by the pixel motion vector from the synthesized moving image data one frame before to another storage means, thereby. The same image data synthesized at any time is output. In this case, since the motion vector of the pixels is close to the movement of the actual image, it is possible to synthesize an image without block-shaped distortion or large distortion of the background portion.

【0027】

[Example]

Hereinafter, examples of the present invention will be described with reference to the drawings.

【0028】

FIG. 1 is a diagram for explaining an embodiment of the motion interpolation method according to the present invention, and shows a part of the moving image data of a certain frame in an enlarged manner. In FIG. 1, the pixels 3 are arranged in a grid pattern at equal intervals. The representative points 1 are also arranged in a grid pattern at an interval four times that of the pixel 3. As shown in the figure, the representative point 1 may be arranged so as not to overlap with the pixel 3, or may be arranged so as to overlap with the pixel 3, and the interval may be many times the interval of the pixel 3.

【0029】

The detected motion vector 2 is given to the representative point 1. Generally, the size and direction of the motion vector 2 are different for each representative point 1, but in the figure, they are drawn as vectors having the same size and direction for easy viewing. A representative point without an arrow means that the movement is zero. Then, the motion discontinuity point 5 is set on the line 6 connecting the representative point with zero motion and the representative point with non-zero motion in this example.

【0030】

If the arrangement of the pixel 3 and the position of the representative point 1 are set to predetermined fixed values, only the information of the motion vector 2 of the representative point 1 and the information of the motion discontinuity point 5 (position information) can be used for image composition. May be output for each frame as additional information to be added to the coded data. At that time, in this embodiment, since it is determined to set the motion discontinuity point 5 between the representative point where the motion vector 2 is zero and the representative point where the motion vector 2 is non-zero, which representative point is the motion discontinuity point 5. No additional information about whether it is in between is required.

【0031】

The position information of the motion discontinuity point 5 is only the distance from the end point on the line connecting the representative points 1. Further, if it is decided that the motion discontinuity point 5 is placed at the midpoint on the line connecting the representative points 1, additional information as the position information of the motion discontinuity point 5 becomes unnecessary.

【0032】

In this embodiment, the motion vector 4 for each pixel 3 is obtained by interpolation from the motion vectors of these representative points and the information of the motion discontinuities. For this interpolation, the motion discontinuity point 5 is first connected by line 6, and the frame is divided into two regions, one on the right side and the other on the left side of line 6. Then, as a general rule, the motion vector 4 of the pixel is obtained by bilinear interpolation from the four representative points 1 around it as shown in FIG. 2 (a).

【0033】

However, as shown in Fig. 2 (b), if there is a line 6 meaning discontinuity in this quadrangle, the motion vector 2 of the representative point 1 beyond the line 6 is used for interpolation calculation. Not used. In the case of Fig. 2 (b), bilinear interpolation is performed assuming that the lower right representative point has a virtual mean value of the vectors of the upper right and lower left representative points. Then, as shown in FIG. 1, the value of the motion vector 4 of the pixel 3 obtained by interpolation becomes discontinuous with the line 6 as the boundary.

【0034】

When connecting the motion discontinuity points 5 with the line 6, the motion discontinuity points may be set on all four sides of the quadrangle whose apex is the representative point 1. In that case, there are two methods of connecting the motion discontinuity points 5 with the line 6 as shown in FIGS. 3 (a) and 3 (b). Set to widen the area of.

【0035】

Figure 4 shows the frame edge processing method. As shown in the figure, the line 6 connecting the motion discontinuity points 5 is drawn perpendicularly from the motion discontinuity point 5 to the frame edge. Also, for the interpolation of the motion vector 4 of pixel 3, consider a quadrangle that virtually protrudes outside the frame, and if the outside representative point has the motion vector of the closest representative point in the frame, the interpolation is performed. good.

【0036】

In the above embodiment, the vector discontinuity point 5 is arranged between the representative points having zero motion and the representative points having non-zero motion, but as shown in FIG. 5, the motion vector 2 is located between the representative points having different directions. It is also effective to arrange it. To determine the direction of the motion vector 2, for example, the inner product of the vectors is calculated, and if the value is negative, it is determined that the directions are different.

【0037】

In this case, when the angle θ formed by the two vectors is 90 ° or more, it is determined that the directions are different. For example, when it is desired to determine that the directions are different when θ is 45 ° or more, the two vectors are (x).₁, y₁), (X₂, y₂), θ is expressed by the following equation.

【0038】

cos θ = (x₁* x₂+ y₁* y₂) / (sqrt (x₁²+ y₁²) * (X₂²+ y₂²)) (1) Where * is multiplication, / is division, and sqrt () is square root operation. Therefore, sqrt (2) / (2) (x₁* x₂+ y₁* y₂) / (sqrt (x₁²+ y₁²) * (X₂²+ y₂²)) (2) It suffices that the vectors satisfying the above conditions have different directions. By using this equation, it is possible to divide the area more finely for each object having slightly different movements.

【0039】

Further, if the absolute value of the motion vector is equal to or larger than the predetermined value C, it is determined that the magnitude is different, that is, C | sqrt (x₁²+ y₁²)-sqrt (x₂²+ y₂²) | (3) If the size is different in the case of, and if a motion discontinuity line is set between them, objects having the same direction but different motion speeds can be divided into different regions.

【0040】

Further, it is also possible to set a boundary of a motion vector having a different direction and at least one of the absolute values as a motion discontinuity to perform region division.

【0041】

Next, an example of a specific algorithm for region division will be described with reference to the flowchart shown in FIG. This is an algorithm that groups motion vectors that have almost the same motion and are adjacent to each other as a group M.

【0042】

First, the group M is set as an empty set as an initial state (step S0). Next, find the motion vector with the maximum absolute value in the frame, and set it as Vo (step S1). Then add Vo to group M (step S2). The state of the processing up to this point is shown in FIGS. 24 (a) and 24 (b). That is, when the motion vector shown in FIG. 24 (a) is obtained, the vector corresponding to the block shown by the diagonal line in (b) is Vo. The diagonal line indicates that it belongs to group M.

【0043】

Returning to FIG. 23, after step S2, an angle formed with the vector Vo around the group M is added to the group M to be less than or equal to a predetermined value (step S3). As a result, it is checked whether there is a new vector added to the group M (step S4), and if so, the process returns to step S3, and the vectors around the group M are checked again. Then, when there are no new vectors to be added to M around the group M, a motion discontinuity is set at the boundary between M and the rest (step S5). The processing of steps S3 to S5 is shown in FIGS. 24 (c) to 24 (f). That is, in (c), eight vectors indicated by halftone dots around the diagonal line are compared with Vo, and as a result, seven new vectors are added to group M as shown in (d). Next, (f) is obtained by comparing Vo with the vector indicated by the halftone dots shown in (e). Next, we examine the part indicated by halftone dots in (g), but since not all of them join group M, the one shown in (h) finally becomes a group with equal movement, and others like (i). The boundary line with and will be set.

【0044】

With the algorithm described above, representative points with almost the same direction of movement can be stored in a single area. Naturally, a condition that the difference between the absolute value and the Vo is less than or equal to the predetermined value may be added to the condition to be added to the group M. In order to set two regions, the same processing may be performed using the representative points that did not belong to the group M by the above algorithm to form another group. Also, if the movement area is not set when the absolute value of Vo is smaller than the predetermined value, the movement area will not be set when there is almost no movement, and the code representing the area will be wasted. Is gone.

【0045】

Furthermore, if such region division is performed, vectors with almost the same orientation and absolute value will exist in the same region, so some of these vectors will be omitted and the surrounding vectors will be substituted for the omitted parts. Even if you do, you can get almost the correct movement. In this way, the motion vector can be coded with a smaller amount of code than when all the vectors are coded one by one. Further, even if the number of vectors to be encoded is reduced due to such omission, the area can be set correctly by encoding the group number of each representative point.

【0046】

In the above embodiment, the line 6 connecting the motion discontinuities 5 is a straight line as shown in FIG. 6 (a), but it may be a curved line as shown in FIG. 6 (b).

【0047】

Further, even when the position of the representative point 1 is set to be the apex of the triangle as shown in FIG. 7, the motion discontinuity point 5 is set between the representative points 1 and the area is divided in the same manner as described above. be able to.

【0048】

Next, an embodiment of the motion detection device according to the present invention will be described with reference to FIG. In FIG. 8, the signal line for transmitting the control signal between the first control unit 14 and the other components is omitted in order to avoid the figure becoming complicated.

【0049】

In FIG. 8, the moving image data 10 is sequentially input in frame units, and is alternately stored in the two frame memories 11 and 12. One of the frame memories 11 and 12 is for the current frame, and the other is for the frame one frame past, and the switching is performed frame by frame by the switch 13 controlled by the first control unit 14. Here, a case where the frame memory 11 is for the current frame will be described. The moving image data 10 is input to the frame memory 11 and stored and held. The moving image data one frame before is already stored and held in the frame memory 12.

【0050】

The motion vector of each representative point is detected as follows. Address data 20 indicating the addresses around the corresponding representative point is sent from the first control unit 14 to the switch 13. The switch 13 is a 2-input 2-output switch, and is set in advance to send the address data 20 as the address data 21 of the frame memory 11 to the frame memory 11 by a signal from the control unit 14. The moving image data 23 of the current frame around the representative point at the address indicated by the address data 21 is read from the frame memory 11 and sent to the switch 15. The switch 15 is preset to send the moving image data 23 from the frame memory 11 as the data 41 to the subtractor 32.

【0051】

On the other hand, the motion vector information 44 is sequentially output from the second control unit 33. The switch 37 is set in advance to output the motion vector information 44, and this motion vector information 44 is sent to the pixel motion vector calculation unit 39 as the motion vector information 48. At this time, the memory 36 that holds the information of the optimum motion discontinuity holds the information that there is no motion discontinuity by default, and the information 47 is sent to the switch 38. The switch 38 is set in advance to output the optimum discontinuity information 47, and the optimum discontinuity information 47 is sent to the pixel motion vector calculation unit 39 as the discontinuity information 49.

【0052】

The pixel motion vector calculation unit 39 performs the interpolation calculation as described in the embodiment of the motion vector interpolation method, obtains the motion vector for each pixel, and sets the address data 25 determined from the motion vector to the switch 13. send. The switch 13 sends the input address data 25 to the frame memory 12 as the address data 22. The moving image data 24 one frame before at the address indicated by the address data 22 is read from the frame memory 12 and sent to the pixel interpolation calculation unit 31 as data 42 via the switch 15.

【0053】

Since the motion vector obtained by the pixel motion vector calculation unit 39 is generally a non-integer, the pixel motion vector calculation unit 39 generates address data 25 having a plurality of integer values close to it. The pixel interpolation calculation unit 31 performs the interpolation calculation using the plurality of moving image data 42 one frame before specified by the address data 25, and sends the interpolation value to the subtractor 32. Further, information 40 such as a fractional part of the vector required for the pixel interpolation calculation is sent from the pixel motion vector calculation unit 39 to the pixel interpolation calculation unit 31. Note that this part can be simply represented by only one integer address close to the interpolated motion vector, in which case the pixel interpolation calculation unit 31 becomes unnecessary.

【0054】

In the subtractor 32, the moving image data 41 of the current frame and the moving image data 42 one frame before are subtracted from each other and the absolute value sum is obtained, and this is used as an error signal 43 as the second control unit. Send to 33. The control unit 33 switches the motion vector information 44 in various ways, sends the motion vector information in which the value of the error signal 43 becomes small to the memory 35 as the optimum motion vector information 44, and stores and holds the motion vector information 44. In this way, the motion vectors of all the representative points are determined, and the optimum motion vector information 44 of each representative point is stored and held in the memory 35.

【0055】

Next, the motion discontinuity point is detected as follows. When detecting the motion discontinuity point, the first control unit 14 switches switches 37 and 38 in the reverse direction, and generates address data 20 around the representative point where the zero vector and the non-zero vector are detected. The third control unit 34 outputs the information of the points between the representative points as the sequential motion discontinuity information 45, and the switch 38 sends this as the motion discontinuity information 49 to the pixel motion vector calculation unit 39. On the other hand, the motion vector information 46 held in the memory 35 is sent to the pixel motion vector calculation unit 39 as motion vector information 48 via the switch 37. The subsequent processing is the same as when obtaining the motion vector, the error signal 43 is sequentially sent to the control unit 34, and the optimum discontinuity information 45 is held in the memory 36.

【0056】

Here, as shown in FIG. 25, for example, the motion discontinuity information 49 is sequentially set in the direction of the other representative point from the vicinity of one representative point, and the error is obtained each time, and the error is set to the position where the error is the smallest. decide. However, since the part where the error can change in the movement range of the movement discontinuity point position is only the part shown by the diagonal line in FIG. 26, it is sufficient to check the error of the diagonal line part.

【0057】

If it is desired to detect the motion vector and the motion discontinuity with less error, the motion vector and the motion discontinuity may be detected alternately and a plurality of times. At this time, if you want to reduce the amount of calculation, instead of outputting all the motion vectors preset as motion vector information 44 and evaluating the error, a predetermined number of motion vectors whose error was reduced when the previous motion vector was detected The vector may be stored for each representative point, and the error evaluation may be performed again using only the stored motion vector at each representative point. Further, the amount of calculation can be suppressed by re-detecting the motion vector only for the representative point in which the vector having a large absolute value is detected without recalculating the motion vector for all the representative points.

【0058】

The motion vector information 46 and the motion discontinuity information 47 obtained as described above are output to the outside. In the next frame, the switch 13 and the switch 15 are switched, and the frame memory 12 is used for the current frame and the frame memory 11 is used for the frame one frame before, respectively.

【0059】

Next, an embodiment of the image synthesizer according to the present invention will be described with reference to FIG. In FIG. 9, the motion vector information 46 and the motion discontinuity information 47 obtained by the motion detection device of FIG. 8 are input from the outside and input to the pixel motion vector calculation unit 51. The pixel motion vector calculation unit 51 performs the interpolation calculation as described in the embodiment of the motion vector interpolation method, obtains the motion vector for each pixel, and stores the address data 54 determined from the motion vector in the frame memory 53. Send to.

【0060】

The contents of the frame memory 53 are initially set to predetermined initial image data, and then the composite image data 56 is sequentially stored and held. Therefore, the moving image data 55 one frame before the position indicated by the address data 54 is read from the frame memory 53, and this is held in the memory 52 as the value of the pixel of the current frame. In this way, the values of all the pixels of the moving image data of the current frame are obtained and output as the composite image data 56 to the outside.

【0061】

FIG. 11 is a block diagram showing an embodiment in which the motion detection device described in FIG. 8 and the image synthesizer described in FIG. 9 are incorporated into a moving image coding device. In FIG. 11, the image data 41 to be encoded is input from the outside and sent to the motion parameter detector 30 and the subtractor 61. Since the motion parameter detector 30 has the same configuration as the portion surrounded by the broken line in FIG. 8, the description thereof will be omitted. On the other hand, the moving image data 42 one frame before is read from the frame memory 53 and sent to the motion parameter detector 30.

【0062】

The motion vector information 46 and the motion discontinuity information 47 output from the motion parameter detector 30 as described above are sent to the synthesizer 50 and also output to the outside. Since the synthesizer 50 has the same configuration as the part surrounded by the broken line in FIG. 9, the description thereof will be omitted. The composite image data 56 output from the synthesizer 50 as described above is sent to the subtractor 61 and the adder 67. The subtractor 61 obtains the difference image data 62 between the coded image data 41 and the composite image data 56, and sends the difference image data 62 to the encoder 63.

【0063】

Various methods can be used as the coding method of the encoder 63. Specifically, transform coding such as DCT (discrete cosine transform), vector quantization, and predictive coding (Reference 1 above). (See pp.213 to 291)) can be used. The coded data 64 output from the encoder 63 is sent to the decoder (also referred to as a local decoder) 65 and is also output to the outside. The decoder 65 decodes and reproduces the difference image data 66 by the reverse conversion operation of the encoder 63, and sends the difference image data 66 to the adder 67. By adding the composite image data 56 and the difference image data 62 by the adder 67, the locally decoded image data 68 is created and stored and held in the frame memory 53.

【0064】

FIG. 10 is a block diagram showing an example in which the image synthesizer described with reference to FIG. 9 is incorporated into a moving image decoding device. In FIG. 10, the coded data 71 from the moving image coding device (not shown) is input to the decoder 72, and the motion vector information 46 and the motion discontinuity information 47 are input to the synthesizer 50, respectively. In the adder 74, the difference image data 73 from the decoder 72 and the composite image data 56 from the synthesizer 50 are added to obtain the decoded image data 75. The decoded image data 75 is output to the outside and stored and held in the frame memory 53.

【0065】

The data sent from the moving image coding device to the moving image decoding device are the coding data 64, the motion vector information 46, and the motion discontinuity information 47 as described with reference to FIG. Therefore, the transmission order of these data will be described below.

【0066】

The coding by the encoder 63 of FIG. 11 and the decoding by the decoder 65 of FIG. 11 and the decoder 72 of FIG. 10 can be performed in units of square blocks. FIG. 13 shows an example of the positional relationship between the square block and the representative point. Here, Pij is a motion vector representative point, and is arranged in a grid pattern with P11, P21, .... Bij is the coding unit block, Hij is the discontinuity between Pij and Pi + 1j, Vij is the discontinuity between Pij and Pij + 1, and Pij is located in the center of Bij. Also, Hij and Vij are only in the part judged to be discontinuous. The data to be transmitted in this case are the coded data 64 of the Eij: Bij part, the motion vector information 46 of Aij: Pij, and the above Hij and Vij.

【0067】

FIG. 14 shows an example of the transmission order of these data, in which all of the coded data 64 all of the motion vector information 46 all of the motion discontinuity information 47. Here, M is the number of blocks in the horizontal direction, and N is the number of blocks in the vertical direction. In this example, the data structure may be simple, but the image recovery device must temporarily hold all the data for one frame, and that much memory is required. In addition, the image reproduction cannot be started until the data of almost one frame is received, and the compression / reproduction delay becomes large. Therefore, an example of the transmission order for reducing the amount of memory as much as possible and the delay as much as possible is shown below.

【0068】

In FIG. B11 B21 ... BM1 B12 B22 ... BM2 ..................... BMN It is assumed that the processing is performed in the decoding device in the order of. Up to play the Bij part Eij, Ai-1j-1, Aij-1, Ai + 1j-1, Ai-1j, Aij, Ai + 1j, Ai + 1j + 1, Aij + 1, Ai + 1j + 1 Hi-1j-1, Hij-1, Hi-1j, Hij, Hi-1j + 1, Hij + 1 Vi-1j-1, Vij-1, Vi + 1j-1, Vi-1j, Vij, Vi + 1j Information is needed. However, since the blocks to the left and above Bij have already been reproduced, much of this information has already been transmitted, and the new information required for Bij reproduction is Eij, Ai + 1j + 1, Hij + 1, Vi + 1j Only. That is, as shown in FIG. 15, if these four pieces of information are transmitted as a group in close time in time, Bij can be reproduced at the stage when this part is received, and the compression / reproduction delay is small. Become. Then, if the blocks are played back in sequence and the information that is no longer used is deleted from the memory, the amount of memory can be saved.

【0069】

However, about the edge of the screen When j = 1 and i = 1 E11, A11, A21, A12, A22, H11, H12, V11, V21 When i = M EMj When j = 1 and i = 1 and i = M Eij, Ai + 11, Ai + 12, Hi1, Hi2, Vi + 11 When j = N EiN When j = 1 and j = N and i = 1 E1j, A1j + 1, A2j + 1, H1j + 1, V1j, V2j Is a group.

【0070】

FIG. 12 shows the case where the vector representative point 1 is on the intersection of the block boundaries. Again, for block Bij Eij, Ai + 1j + 1, Hij + 1, Vi + 1j Can be transmitted together.

【0071】

Next, an example of a data structure in a form close to CCITT Recommendation H.261, which is an international standard for image coding for video telephones and video conferencing, will be described in relation to the order of data transmission. In H.261, the data forms a layered structure consisting of a frame layer, a GOB (group of block) layer, an MB (macroblock) layer and a block layer. Of these, only the MB layer is changed in this embodiment. Since the other layers are the same as H.261, the description is omitted, and the MB layer will be described with reference to FIG.

【0072】

First, the data common to H.261 is shown below.

【0073】

-MBA (Macroblock Address): Relative address from the macroblock that had information before it -MTYPE (type information): Information on the type of macroblock and which data element appears. -MQUANT (quantization characteristics): Quantization characteristics for subsequent macroblocks -MVD (motion vector information): Difference from the motion vector of the previous macroblock CBP (Significant Block Pattern): Indicates which of the four luminance blocks and two color difference blocks contained in the macroblock is significant. However, unlike H.261, a macroblock without information means that Eij (the sign of Bij) is zero and Ai + 1j + 1 (the motion vector of Pi + 1j + 1) is zero and Pi + in Fig. 22. Bij when there is no discontinuity between 1j and Pi + 1j + 1 and there is no discontinuity between Pij + 1 and Pi + 1j + 1. In addition, the type with motion compensation in MTYPE means that Ai + 1j + 1 is non-zero. Similarly, the motion vector of the MVD is the motion vector Ai + 1j + 1 of the lower right point of the corresponding macroblock Bij.

【0074】

Next, the data not specified in H.261 are shown below.

【0075】

-HV (second type information): Indicates which of the following four types.

【0076】

(1) There are also discontinuity points Hij + 1 and Vi + 1j (2) Only Hij + 1 is available (3) Only Vi + 1j is available (4) No discontinuity This can be before MTYPE.

【0077】

H (first discontinuity information): Position information of Hij + 1 (distance from Pi + 1j, etc.) -V (second discontinuity information): Vi + 1j position information (distance from Pij + 1 etc.) However, the decoding device can also determine the HV information from the information on whether the motion vector is zero or non-zero, which is included in the MTYPE, and in that case, the HV is unnecessary.

【0078】

When HV is omitted and a discontinuity is set between motion vectors with different directions, it is necessary to determine the presence or absence of the discontinuity after receiving the MVD, so the positions of H and V are shown in Fig. 17. It is better to do so.

【0079】

In the above embodiment, if there is a motion discontinuity point on any of the four sides of the quadrangle surrounded by the four representative points, the motion discontinuity line is always included in the quadrangle region. However, in some cases, the error may be smaller if the quadrilateral region is interpolated without the discontinuity line described in the conventional example. For such cases, it is possible to switch between discontinuous interpolation and continuous interpolation for a quadrangle that may be discontinuously interpolated, and it is also possible to add a switching bit. When this method is used and both adjacent quadrilaterals are continuously interpolated, the discontinuity point is unnecessary on the line segment sandwiched between the two quadrilaterals. This switching bit may be included in the HV of FIG. 16 or FIG.

【0080】

Next, as another application example of the domain decomposition method according to the present invention, an example applied to the image cropping device will be described. FIG. 27 is a block diagram showing an embodiment of the image cropping device. In the figure, the input moving image data 41 of the current frame is input to the frame memory 101, the motion parameter detector 30 shown in FIG. 8, and the cutting unit 102. From the frame memory 101, the data 104 one frame before is input to the motion parameter detector 30. From the motion parameter detector 30, motion discontinuity point information 47 is input to the cutting unit 102.

【0081】

The cutout unit 102 outputs only the image data in the region set by the motion discontinuity point 47 out of the moving image data 41 of the current frame as the cutout image data 103. As described above, since the motion parameter detector 30 divides the area into the same area with substantially the same motion, it is possible to cut out image data for each subject from the motion image data 41. Further, when a plurality of regions exist, it is possible to cut out only the image data of the selected specific region by giving the region selection signal 105 to the cutting unit 102.

【0082】

The cut-out image data 103 can be stored as, for example, an image database, and if necessary, the database can be searched to read out the image data in a desired region, and a desired image can be synthesized. It is also possible to transmit only the cut out image data to a remote location. In this case, only the image data of the animal body is transmitted, but it is sufficient for monitoring the animal body, and is effective as, for example, a security system or other remote abnormality monitoring system. In this case, there is an advantage that the amount of information transmission can be saved as compared with the transmission of the entire image data of the frame. Further, as another application of the image cropping device, it is also possible to recognize an animal body from the shape and image quality of the cropped image.

【0083】

Furthermore, the present invention is also effective for computer graphics using motion compensation.

【0084】

[Effect of the invention]

As described above, according to the present invention, it is possible to synthesize an image without block-shaped distortion or large distortion of the background portion by using the motion vector interpolation method. Further, the motion detection device according to the present invention has the advantages that the amount of code required to represent the motion region is small and the motion prediction error is also small. As described above, the present invention is particularly useful for a moving image coding / decoding device, and is also effective for an image cropping device, computer graphics, and the like.

[Simple explanation of drawings]

[Figure 1]

The figure for demonstrating one Example of the motion vector interpolation method according to this invention. [Figure 2]

The figure which shows the method of interpolating the motion vector of a pixel from the motion vector of a representative point in the same Example. [Fig. 3]

The figure which shows the connection example of the motion discontinuity point in the same Example [Fig. 4]

The figure which shows the processing method of the frame edge in the same Example [Fig. 5]

The figure which interpolated discontinuously between different motion vectors in the same Example [Fig. 6]

The figure which shows the connection example of the discontinuity point in the same Example [Fig. 7]

Diagram when representative points are arranged in a triangle [Fig. 8]

A block diagram showing an embodiment of a motion detection device according to the present invention. [Fig. 9]

A block diagram showing an embodiment of an image synthesizer according to the present invention. [Fig. 10]

Block diagram showing another embodiment of the image synthesizer according to the present invention [Fig. 11]

A block diagram showing an example in which a motion detection detection device and an image composition device according to the present invention are combined. [Fig. 12]

The figure which shows an example of the coding block [Fig. 13]

Diagram showing another example of a coded block [Fig. 14]

The figure which shows an example of the data transmission order [Fig. 15]

Diagram showing another example of data transmission sequence [Fig. 16]

Diagram showing an example of a data structure [Fig. 17]

Diagram showing other examples of data structures [Fig. 18]

The figure which shows an example of the frame image for demonstrating the principle of motion vector detection. [Fig. 19]

Diagram showing representative points and motion vectors on a frame image to explain the principle of motion vector detection [Fig. 20]

The figure which shows an example of the motion compensation composite image by the conventional motion vector detection. [Fig. 21]

The figure for demonstrating an example of the conventional motion vector interpolation method. [Fig. 22]

Diagram to illustrate other examples of conventional motion vector interpolation methods [Fig. 23]

A flowchart for explaining the algorithm of the region division method in another embodiment of the present invention. [Fig. 24]

The figure which shows the specific example of the area division method of the same Example [Fig. 25]

The figure for demonstrating the setting method of the motion discontinuity point information in this invention. [Fig. 26]

The figure for demonstrating the setting method of the motion discontinuity point information in this invention. [Fig. 27]

Block diagram showing an embodiment of the image cropping apparatus according to the present invention. [Explanation of symbols]

1 ... Representative point 2 ... Motion vector of representative point 3 ... Pixel 4 ... Pixel motion vector 5 ... motion discontinuity 6 ... line