JPH099263A - Encoding method and encoder for motion compensation prediction of dynamic image - Google Patents

Abstract

PURPOSE: To detect a spatially and temporally continuous motion vector in the motion vector detection method of a block matching type. CONSTITUTION: An encoding object image 1 is inputted to a frame memory 66 and connected with the plural encoding object images timewisely continued to the encoding object image 1 already stored inside the frame memory 66 and space-time images 61 are prepared. The space and time images 61 are inputted from the frame memory 66 to a luminance value continuation information detector 65 and the line element of a part indicated by a direction for which a luminance value is similar to a time direction is extracted. The encoding object image is inputted to a motion detection part 63 along with luminance value continuation information 62 and the motion vector 64 of respective blocks is obtained. In the motion detection part 63, as estimated search position for the respective blocks within the encoding object image 1 is obtained from the curve formula, only around the estimated search position is searched and the motion vector 64 is detected.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a digital compression encoding method for image signals used in image communication, image recording, etc., and more particularly to a motion compensation predictive encoding method for moving images by area division.

[0002]

2. Description of the Related Art In digital compression coding of moving images, motion-compensated interframe prediction is often used as a means for suppressing temporal redundancy of moving image signals. In this inter-frame prediction, the encoding target image is usually 16 pixels × 1.
The difference between the prediction image generated by dividing the reference image by a motion vector and detecting the amount of motion (motion vector) from the reference image for each block by dividing the block into rectangular blocks of 6 lines The (motion compensation prediction error) signal is encoded. By this motion-compensated inter-frame prediction, the inter-frame correlation of the moving image is dramatically improved, and a large amount of information compression can be obtained as compared with the simple inter-frame prediction. Furthermore, a discrete cosine transform (DCT: Disc) is applied to the motion compensation prediction error signal.
By performing rete cosine transform) or sub-band division, the redundancy in the spatial direction is suppressed and further information compression is achieved. For this reason, video coding for videophone / conference video ITU-T H.264. 261, video encoding ISO / IEC 11172 for storage (MPEG
In -1) and the like, a hybrid coding configuration for DCT-coding a residual signal by motion-compensated interframe prediction is adopted.

ITU-T (formerly CCITT) Recommendation H.264 26
1 is entitled “p × 64 kb / s video coding method for audiovisual service”, and is 64 kb / s (p
= 1) to 2 Mb / s (p = 30), which is a video coding standard for communication. The standardization work started in December 1984, and the recommendation was established in December 1990.
It is the moon. Examples of the application include a videophone and a video conference. H. 261 suppresses the temporal redundancy of the moving image signal by motion compensation prediction, and suppresses the spatial redundancy of each frame by discrete cosine transform (DCT) coding. Hereinafter, with reference to FIG. The encoding algorithm of H.261 will be briefly described.

First, the image to be coded 1 is input to the motion detecting section 51 together with the square pattern 50, and 16 pixels × 16.
It is divided into square blocks called line macroblocks. In the motion detection unit 51, the amount of motion with respect to the reference image is calculated for each macroblock in the encoding target image 1,
By using a full search or the like, the most similar block in the search range is detected as a motion vector 52 by using the absolute value difference or the sum of squared differences as an evaluation function, and is sent to the block motion compensation unit 53. . Here, the motion vector of each macroblock is represented as a displacement between the coordinate of the block having the highest degree of matching with the macroblock of interest and the coordinate of the macroblock of interest in the reference image. The motion vector search range is limited to the coordinates of the macroblock of interest and ± 15 pixels × ± 15 lines around it.

Next, the block motion compensation unit 53 generates a motion compensation predicted image 15 from the motion vector 52 of each macroblock and the locally decoded image 6 of the immediately preceding frame stored in the frame memory 5. The motion compensation prediction image 15 obtained here is input to the subtractor 16 together with the encoding target image 1. The difference between the two, that is, the motion compensation prediction error 17 is D
The CT / quantization unit 54 performs DCT conversion, and further quantizes the compressed difference data 19. Where DCT
Has a block size of 8 × 8. Compressed difference data 19
The (quantization index) is data-compressed in the differential data encoding unit 20 and becomes differential image encoded data 21. On the other hand, the motion vector 52 is coded in the motion vector coding unit 26, and the obtained motion vector coded data 2
7 is multiplexed with the differential image coded data 21 by the multiplexing unit 28 and transmitted as multiplexed data 29.

Since the same decoded image as the decoder is obtained in the encoder, the compressed difference data 19 (quantization index) is returned to the quantized representative value by the inverse quantization / inverse DCT unit 55, and further the inverse DCT is performed. After conversion, the decoded difference image 23 is obtained. The decoded difference image 23 and the motion-compensated predicted image 15 are added by the adder 24 to form a locally decoded image 25. This locally decoded image 25 is stored in the frame memory 5 and used as a reference image when the next frame is encoded.

[0007]

When a motion vector is obtained using a motion estimation method in the conventional digital compression encoding of a moving image, the motion search is usually 16 × 16.
The position where the prediction error power (mean squared error) in the block is the smallest is selected as the matching target, so the difference in the average luminance value has a large influence on the motion search, and due to changes in the light source, etc. Although different, there is a problem in that there is often a case where there is no continuity between the motion vector values of each block in the space / time direction because it matches with the part with the minimum error power.

An object of the present invention is to solve the above-mentioned problems and to provide a motion compensation predictive coding method and a coder for a moving image, which detects a spatially and temporally continuous motion vector.

[0009]

In order to achieve the above object, the present invention divides an image to be encoded into polygonal patches, and sets the amount of motion between the image to be encoded and a prediction reference image to the above-mentioned multiple amount. A motion-compensated predictive coding method for a moving image, which detects each of the rectangular patches, performs motion compensation to generate a predicted image, and encodes a difference between the predicted image and the target image to be encoded. When calculating the vertex motion vector of the polygon patch of, the continuous information of each brightness value in a plurality of coded reference images that are continuous in the time direction, that is, the direction of the line element with similar brightness values, as a pre-process of motion vector detection. Is calculated for each pixel, and the direction of the line element is referred to when detecting the motion amount, whereby the motion prediction search range is constrained.

The motion-compensated predictive encoder for moving pictures according to the present invention accumulates the image to be encoded, the newly input image to be encoded, and the previously-encoded image to be encoded. A frame memory that connects a plurality of temporally continuous encoding target images to form a spatiotemporal image and the spatiotemporal image is input, and line element extraction is performed at a location that represents a direction in which luminance values are similar in the temporal direction. The present invention is characterized by further comprising luminance value continuation information extraction means for performing and outputting luminance value continuity information, and motion detection means for inputting the image to be encoded and the luminance value succession information to obtain a motion vector of each block.

[0011]

In the moving image sequence, the region shape of the object has temporal correlation. For example, the area shape of the background portion of the screen hardly changes with time. Further, if the time is short, it may be considered that there is no temporal change in the area shape of the object.

However, in reality, it is difficult to predict the background and even the static portion from the area shape of the prediction reference image because the area shape changes subtly due to the noise in the image and the influence of the light source. Also, regarding the moving area, if the motion is correctly obtained, the current area shape can be estimated from the area shape of the prediction reference image and the motion information. Since the position where the electric power is the minimum is selected as the matching destination, the difference in the average luminance value has a great influence on the motion search, so that the portion with the minimum error electric power that does not match the actual motion is matched.

Therefore, as described in claim 2 and claim 3, as a pre-process of motion vector detection, a line element (luminance value similar in the time direction in a plurality of encoded reference images continuous in the time direction) In obtaining the direction of the trajectory of the object drawn in the spatiotemporal image, a function approximation is performed using a three-dimensional Hough transform for directions with similar brightness values, and the direction of the line element is constrained during motion vector search. As a result, by narrowing the search range of the motion vector, it is possible to reduce the amount of calculation at the time of search and maintain the continuity of the detected motion vector in the space / time direction.

Further, since the motion-compensated prediction method in the above-mentioned prior art is a block-based prediction in which a rectangular block of 16 pixels × 16 lines is regarded as one rigid body, block-like discontinuous distortion occurs in the predicted image. appear. This discontinuous distortion is particularly noticeable in a portion having a large amount of motion, and becomes a large visual obstacle during low root coding in which a sufficient coding amount cannot be allocated for coding a prediction error image.

As means for solving the above problems,
A method has been proposed in which an image to be encoded is divided into polygonal, for example, triangular or quadrangular patches, and the motion vectors of the vertices of each patch are interpolated by spatial transformation to perform motion compensation for each pixel. As a typical example, Gary J. Sullivan
Et al. "Motion Compensation for Video Compressio
n Using Control Grid Interpolation "(IEEE ICASSP '
91, pp. 2713-2716, 1991) will be briefly described with reference to FIG.

First, the image 1 to be encoded is 16 pixels × 16.
It is divided into square patches such as lines, and the motion detection unit 3 uses the polygon pattern 2 to obtain the motion vector 4 of the vertex of each patch. Next, the motion vector interpolation unit 12 calculates a motion vector 13 for each pixel from the four vertex vectors 4 for each patch. As shown in FIG. 4, vertex A,
The motion vectors in B, C and D are

[0017]

[Outside 1] Then, the interpolation vector at the coordinates (x, y) in the square ABCD

[0018]

[Outside 2] Is calculated by the following formula.

[0019]

[Equation 1] This spatial transformation method is based on bi-linear interpolation.
interpolation). As a result, the motion vector value changes smoothly for each pixel, and the motion vectors are smoothly connected even at block boundaries. The pixel-based motion compensation unit 14 uses the pixel-based motion vector 13 thus obtained to perform motion-compensated prediction for each pixel, thereby giving the same motion vector value to all the pixels in the block. Compared with the prediction method, there is an advantage that block-like discontinuous distortion does not occur in the predicted image.

However, when a motion vector is obtained by using a motion prediction method in the conventional digital compression encoding of a moving image, in the motion search, as described above, the position where the prediction error power is the minimum is matched. Since it is selected as the destination, it differs from the actual matching destination, but it matches the part with the minimum error power, and there are many cases where there is no continuity between the motion vector values of each block in the space / time direction. By performing the motion vector interpolation across the discontinuous parts of the motion vector, not only the quality of the predicted image but also the prediction efficiency is reduced.

Therefore, as described in claims 2 and 3, as a pre-process of motion vector detection, a direction in which the luminance value is similar in the time direction is obtained in a plurality of encoded reference images continuous in the time direction. At this time, by performing a function approximation using a three-dimensional Hough transform or the like on the directions having similar brightness values, and using the continuous information as a constraint in the search for a motion vector, the detected motion vector is continuous in the space / time direction. It is possible to maintain the property, and it is possible to reduce the distortion of the predicted image after the motion vector interpolation.

[0022]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing the structure of an encoder for realizing a motion-compensated predictive coding method for a moving image according to an embodiment of the present invention, and corresponds to the first and second aspects of the present invention.

In the encoder, the image to be encoded 1 is first input to the frame memory 66, and then the frame memory 6
The encoding target image 1 already stored in 6 is connected to a plurality of temporally continuous encoding target images to create a spatiotemporal image 61.

The spatio-temporal image 61 is input from the frame memory 66 to the luminance value continuous information detector 65, and the line element is extracted in the direction where the luminance values are similar in the time direction. On this occasion,
For each pixel of the current frame, the squared error from the luminance values of the pixels in the other plurality of frames is taken, and the direction is approximated by the spline curve. As a means for extracting line elements, as an example of using a local operator, Vander Brug or P
There are line detection operators such as aton and Kasvand, and other various algorithms such as a relaxation method and a Hough transform method can be applied. The method of region segmentation is described in "Image Analysis Handbook" (supervised by Mikio Takagi and Yohisa Shimoda. The University of Tokyo Press, 1
Details (January 991).

The image 1 to be coded is also input to the motion detecting section 63 together with the luminance value continuous information 62, and the motion vector 64 of each block is obtained. At this time, the brightness value continuation information 62 uses a coefficient obtained by approximating a direction in which the brightness values are similar as a curve formula.

The motion detection unit 63 obtains an estimated search position for each block in the image to be coded 1 from the curve expression, searches only in the vicinity of the estimated search position, and calculates the motion vector 6
4 is detected.

The subsequent operation is exactly the same as in the case of FIG.

Next, the spatiotemporal image will be described with reference to FIGS.

FIG. 5 shows a moving image sequence vertically arranged in the time direction. In the moving image, the object 71 is stationary, and the object 72 is moving to the right while moving away (reducing) in the distance.

FIG. 6 is a diagram obtained by arranging and connecting FIG. 5 in the time direction, which is called a spatiotemporal image.

As is apparent from FIG. 6, in the spatiotemporal image, the objects existing in the moving image sequence leave a three-dimensional trajectory in the spatiotemporal image depending on the movement of the object. FIG. 7 is a sectional view of the spatiotemporal image shown in FIG. 6 taken along a plane (y = a) perpendicular to the y-axis.

Hough exchange will be described as an example of the extraction of line elements in a spatiotemporal image. For simplification, a spatiotemporal sectional view of a two-dimensional image, that is, a method of using the two-dimensional Hough transform in FIG. 7 and extracting a straight line from line elements will be described.

The Hough transform is a means for detecting a figure (for example, a straight line, a circle, an ellipse, a parabola) that can be expressed by parameters from an image. Here, typically Duda an
A straight line detection method based on the dHart method will be described.

The straight line is expressed by the equation ρ = x ₀ cos θ + y ₀ sin θ, and (ρ,
θ) is used. Here, ρ is a length of a perpendicular line drawn from the origin to a straight line, and θ is an angle formed by the perpendicular line and the x axis. If this straight line passes through the point (x ₀ , y ₀ ) on the image, the following equation (2)

[0036]

[Equation 2] The relationship is established. This function is a sine curve on the parameter space ρ-θ. That is, one point in the xy space is ρ
Corresponding to a single locus of −θ, conversely ρ expressed by equation (2)
The locus in −θ space is (x ₀ , y ₀ ) in xy space.
This means that it represents all the straight lines passing through. Therefore, when a point on a straight line in the xy space is copied in the ρ-θ space, the locus in the ρ-θ space created from these points intersects at one point.

Specifically, first, a pixel which is a candidate for a linear element is extracted from the current image by the line detection operator. If the x-y coordinates of these pixels are (x _i , y _i ), all the loci ρ = x _i cos θ + y _i sin θ
Are counted in the ρ-θ space. If a straight line ρ ₀ on the image
= Xcos θ ₀ + ysin θ ₀ exists, all pixels on the straight line count up with respect to the point (ρ ₀ , θ ₀ ) on the ρ-θ space (ρ ₀ , θ ₀ ). There should be a peak at. A straight line is detected by detecting this peak.

[0038]

As described above, according to the present invention, when the vertex motion vector of the polygon patch in the image to be encoded is obtained, the direction of the line element whose luminance value is similar to the time direction is obtained,
By referring to this when detecting the motion amount, the search range of the motion vector is narrowed, the amount of calculation at the time of search is reduced, and at the same time, the continuity of the motion vector in the space / time direction is maintained, so decoding The edges of the image become more natural and close to the original. Furthermore, according to the present invention, it is possible to maintain the continuity of the detected motion vector in the space / time direction, and by performing the motion vector interpolation in the motion vector field, the distortion of the predicted image after the motion vector interpolation is performed. Can be reduced.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of an encoder according to an image encoding method in an embodiment of the present invention.

[Fig. 2] Fig. 2 is a diagram illustrating the configuration of an encoder of a conventional block-based motion compensation predictive encoding method.

FIG. 3 is a diagram showing a configuration of an encoder of a conventional motion compensation predictive coding method by motion vector interpolation.

FIG. 4 is a diagram showing a method of motion vector interpolation.

FIG. 5 is a diagram in which a moving image sequence is arranged vertically in the time direction.

FIG. 6 is a diagram in which the moving image sequences of FIG. 5 are arranged in the time direction and connected.

7 is a cross-sectional view of the spatiotemporal image shown in FIG. 6 taken along a plane perpendicular to the y-axis.

[Explanation of symbols]

 1 image to be encoded 2 polygon pattern 3 motion detection unit 4 vertex motion vector 5 frame memory 6 locally decoded image 7 region division unit 8 region shape reference image 9 shape motion compensation unit 10 polygon pattern 11 motion compensation prediction shape 12 adaptive motion Vector interpolation unit 13 Pixel unit motion vector 14 Pixel unit motion compensation unit 15 Motion compensation prediction image 16 Subtractor 17 Motion compensation prediction error 18 Spatial redundancy compression unit 19 Compressed differential data 20 Differential data encoding unit 21 Differential encoded data 22 Differential data expansion unit 23 Expanded differential image 24 Adder 25 Local decoded image 26 Motion vector coding unit 27 Motion vector coded data 28 Multiplexing unit 29 Multiplexing data unit 30 Region dividing unit 31 Region shape image 50 Square pattern 51 Motion detection Part 52 motion vector 53 block motion compensation part 61 space-time Image 62 luminance values continuous information 63 motion detection section 64 motion vector 65 luminance value continuous information detector 66 frame memory 71 still object 72 moving objects

Claims (4)

[Claims] 1. An image to be encoded is divided into polygonal patches, a motion amount between the image to be encoded and a prediction reference image is detected for each polygonal patch, and motion compensation is performed to obtain a predicted image. A motion compensation predictive coding method for generating and coding a difference between the predicted image and a coding target image, in detecting a vertex motion vector of a polygon patch in the coding target image, detecting a motion vector As pre-processing of, the continuous information of each luminance value in a plurality of encoded reference images continuous in the time direction, that is, the direction of a line element having a similar luminance value is obtained for each pixel, and the direction of the line element is the motion amount. A motion compensation predictive coding method for a moving image, characterized in that the motion prediction search range is constrained by referring to it when detecting. 2. In obtaining a direction of a line element in which a pixel has a similar luminance value in the time direction in a plurality of coded reference images, the luminance values in a plurality of coded reference images continuous in the time direction have similar luminance values. 2. The motion compensation predictive coding method according to claim 1, wherein the approximated direction is approximated by a function. 3. The motion-compensated prediction code according to claim 2, wherein a three-dimensional Hough transform is used for functionally approximating directions of line elements having similar pixel luminance values in the time direction in a plurality of coded reference images. Method. 4. An image to be encoded is divided into polygonal patches, a motion amount between the image to be encoded and a prediction reference image is detected for each polygonal patch, and motion compensation is performed to obtain a predicted image. A motion compensation predictive encoder that generates and encodes the difference between the predicted image and the encoding target image, accumulates the encoding target image, and stores the newly input encoding target image and the already accumulated image. A frame memory that forms a spatiotemporal image by connecting the encoding target image and a plurality of temporally continuous encoding target images to each other and inputs the spatiotemporal image, and the luminance values are similar in the temporal direction. The brightness value continuous information extracting means for extracting the line element of the portion representing the different direction and outputting the brightness value continuous information, and the motion for inputting the encoding target image and the brightness value continuous information to obtain the motion vector of each block Having detection means Wherein, the motion compensated prediction coder image video.