JP3235453B2 - Image coding device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates are those which relate to the image code KaSo location for transmitting storing the image with a small coding amount.

[0002]

2. Description of the Related Art If the movement of the entire screen caused by shaking the camera right and left and zooming can be compensated, it can be expected that the coding efficiency of an image is improved. Hitoue Uekura, Yutaka Watanabe: "Global Motion Compensation Method in Video Coding", E-Informatics
B-1, Vol. J76-BI, No. 12, pp. 944-952, 1993.

In this method, the motion of the entire screen of an image caused by a pan / zoom operation of a camera is estimated and compensated.
Hereinafter, the motion compensation for the entire screen is referred to as global motion compensation. FIG. 5 shows a block diagram of an image coding apparatus that executes the conventional motion compensation method. In FIG. 5, reference numeral 501 denotes a residual encoder, 502 denotes a residual decoder, 503 denotes a frame memory, 504 denotes a motion predictor 1, 505 denotes a three-parameter predictor, 506 denotes a global motion compensator, and 507 denotes local motion. Compensation units 1 and 508 are motion prediction units 2, 509 is a motion compensation unit 2, 510 is a comparator, 511 is a subtractor, 512 is an adder, and 513 is a motion information multiplexing unit.

In order to facilitate understanding, 501, 50
The structure of 2, 512, 503, 504, 507, and 511 is MPEG1 ("A Video Compression Standard for Multimedia Applications") Le Gall, D .: "MPEG: A Video Compression Standa
rd for Multimedia Applications ", Trans. ACM, 199
1, described in April).

An image coding apparatus and a decoding apparatus which will be described later as embodiments of the present invention are also apparatuses based on MPEG1. Therefore, the residual encoding unit 501 includes a digital cosine transform (DCT) compliant with MPEG1, a quantizer, and a Huffman encoder. The residual decoding unit is M
It is composed of an inverse quantization unit based on PEG1 and a digital cosine inverse transformer. In this conventional example, a motion vector is obtained by block matching as in the embodiment described later. This is shown in FIG. FIG. 6 is a diagram for explaining block matching common to the conventional example and an embodiment described later.

In FIG. 6, a region R is a block region of 16 × 16 pixels. The motion prediction unit 1 (504) performs block matching with respect to the input image one frame before as a reference image, and the motion prediction unit 2 (508) uses a global motion-compensated image described later as a reference image. In block matching, the sum of absolute error values shown in Equation 1 is calculated, and as shown in Equation 2, the deviation (p, q) that has the minimum error absolute value
Is the actually measured motion vector of the block. Equation 1
, Gt (x, y) represents the luminance value at the frame t and the screen position (x, y). And equation 3
As shown in (1), the deviation with the smallest value is searched to obtain the original estimate (p, q). This (p, q) is determined with half-pixel accuracy.

[0007]

(Equation 1)

[0008]

(Equation 2)

In MPEG1, a motion vector is obtained between a decoding result of one frame before and a current input frame.
Motion compensation is performed by parallel movement of a 16 × 16 block (hereinafter referred to as a macroblock). This is executed by the local motion compensator 1 (507). In addition to this, in the conventional example, global motion compensation is realized by the left / right / up / down motion of the camera and the zoom.

If the zoom coefficient of the entire screen is Z, the translation amount is H0 horizontally, V0 vertically, and the center of zoom is (x0, y0),
The global motion amount at an arbitrary position (x, y) on the entire screen in the input image can be expressed by Expressions 3 and 4.

[0011]

(Equation 3)

[0012]

(Equation 4)

Further, assuming that the center position of the macroblock at the j-th row and the i-th column is (xij, yij), the motion vector (uij, vij) at the center of the macroblock can be expressed by the following equation (5) and (6).

[0014]

(Equation 5)

[0015]

(Equation 6)

[0016]

(Equation 7)

In the conventional example, the motion vectors of two or more macroblocks are extracted from the output of the motion prediction unit 1 (504), and Equation 8 is executed by the three-parameter prediction unit (505).

Equation 8 is a calculation equation that minimizes the sum of squared errors between the motion vector observed in each macroblock and the predicted motion vector.

P in Equation 8 is a real symmetric matrix obtained from the position of the extracted macroblock as shown in Equation 9, and can be calculated in advance.

[0020]

(Equation 8)

[0021]

(Equation 9)

The calculated global parameters (Z,
H, V), the global motion compensator (506) performs the modification shown in Expression 3. This transformation is performed at each pixel position of the input image.
For (x, y), calculate a motion vector according to Equation 3,
This is executed by obtaining the pixel value at the pixel position indicated by the motion vector obtained from the frame memory (503). The motion vector of the global motion compensation is obtained with 1/4 pixel precision, and is obtained as linear interpolation of four pixel values at neighboring integer positions. Further, the motion prediction unit 2 (508) performs motion prediction on a macroblock basis on the transformed image. The comparator (510) is a macroblock-based MPEG unit.
A prediction method having a small error is selected from the same macroblock motion-compensated image as that of No. 1 and the macroblock motion-compensated image of the previous frame after global motion compensation.

In FIG. 5, the motion vector information includes a selection result SW in units of macro blocks, a motion vector mv1 or mv2,
Then, the global parameter g is stored in the motion information multiplexing unit (5
It is multiplexed and sent in 13). In this conventional example, a motion vector for each block calculated by a block matching method is used, and parameters of the entire screen are calculated only by a simple product-sum operation without performing repetitive calculations. When applying global motion compensation to video coding,
An adaptive ON / OFF control method for each block is proposed.

[0024]

[Problems to be solved by the invention]

[Problem 1] In the conventional example described above, it is possible to compensate for the movement caused by the horizontal and vertical swinging and zooming of the camera. However, in the above-described configuration of global motion compensation, there is a problem that the spatial resolution of a predicted image decreases for each frame unless residual coding is performed.

A pixel-based motion vector obtained by global motion compensation is generally not an integer value. For this purpose, a pixel value is calculated from neighboring pixel values by interpolation. As a result, unless the pixel value is updated by the residual coding,
The spatial resolution of the predicted image decreases every time the frame is updated. If the residual encoding is performed, a high resolution can be maintained, but the code amount for that increases. In order to efficiently encode a moving image by global motion compensation, a motion compensation method with a small reduction in resolution is required.

[Problem 2] In predictive image coding, predicted images must match on the encoding side and the decoding side. In the above-mentioned conventional example, the motion vectors of the global motion compensation have to match with a 1/4 pixel precision for each pixel position. For this reason, the encoding accuracy of the term Z relating to the zoom in Expression 3 is required to the extent that the pixel positions coincide with each other by 1/4 pixel accuracy. However, this accuracy depends on the size of the image because it is a multiplier related to the position. This is not desirable given the future expansion of the coding device. Also, address calculation for global motion compensation requires high calculation accuracy as the image size increases.

[0027]

Means for Solving the Problems To solve the above problems,
Therefore, the present invention provides a short-term frame memory that stores a pixel block of n × n dots as an encoding target frame and stores decoded frames that are temporally preceding and succeeding as a short-term reference frame; between the reference frame, short motion generating a local motion vector detection means for detecting as a local motion vector local response, a prediction image by reading an image from the short-term frame memory by using a pre-Symbol station plant motion vector and compensation means, a long-term frame memory interval that is updated than the short-term reference frame with respect to the encoding target frame is stored as a long-term reference frame long decoded frame, the encoding target
The absolute value error between the frame and the image one frame before is u
(x , y), v (x , y) (x, y are arbitrary points on the screen; u, v
Is the motion vector at the pixel position (x, y)) as a variable
U, v and affine transformation parameters
A relational expression with the meter a is created, and the quadratic function is
Suppose that the partial derivative of the bin transformation parameter a is a zero vector
By obtaining a value that minimizes the absolute value error,
Estimate the motion of the entire screen as an affine transformation parameter a
An affine parameter inter-frame estimator;
Before and after the output from the parameter
An affine parser that synthesizes affine parameters between frames
Parameter updating means, and the affine parameter updating means
An image coding apparatus that includes a full screen motion compensation means for generating a prediction image by reading the image from the long-term frame memory using the affine parameters.

[0028]

[0029]

[0030]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention solves the above-mentioned problem 1, in which a frame sequence of an image is input, and a decoded frame temporally before and after a frame to be encoded is set as a short-term reference frame. A short-term frame memory for storing; a local motion vector detecting means for detecting a local correspondence between the encoding target frame and the short-term reference frame as a local motion vector; and the short-term frame using the local motion vector. A short-term motion compensator for reading an image from a memory to generate a predicted image, and a long-term frame storing a decoded frame having a longer update interval than the short-term reference frame with respect to the encoding target frame as a long-term reference frame Correspondence of the entire screen between the memory and the encoding target frame and the long-term reference frame is determined at an arbitrary position on the screen. Full-screen motion detecting means for detecting the motion vector of the above as a parameter of a polynomial function whose position is a variable, and full-screen motion compensation for generating a predicted image by reading an image from the long-term frame memory using the parameters of the polynomial function Means for adaptively selecting the short-term motion compensation means and the full-screen motion compensation means to perform predictive encoding.

[0031]

[0032]

In the image coding apparatus according to the present invention, the short-term frame memory stores a decoded frame that is temporally before or after the frame to be coded as a short-term reference frame. The local motion vector detecting means detects a local correspondence between the short-term reference frame and the encoding target frame as a local motion vector. And the short-term motion compensation means,
An image is read from the short-term frame memory using a local motion vector to generate a predicted image.

On the other hand, the long-term frame memory stores a decoded frame whose update interval is longer than that of the short-term frame memory for the encoding target frame as a long-term reference frame. The full-screen motion detection means detects the correspondence between the long-term reference frame and the entire frame of the encoding target frame as a parameter of a polynomial function using a motion vector at an arbitrary position on the screen as a variable.

The full-screen motion compensation means reads an image from the long-term frame memory using the parameters of the polynomial function to generate a predicted image. The image encoding device adaptively selects a local motion compensated image from the short-term memory and a motion compensated image from the long-term memory and performs predictive encoding. Full-screen motion compensation is performed using frame data with a long update interval.

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048]

(First Embodiment) An embodiment of the present invention will be described below with reference to the drawings. An embodiment of the present <br/> onset bright image encoding apparatus with reference to FIG. 1,
An embodiment of the present onset bright image decoding apparatus will be described with reference to FIG.

FIG. 1 is a block diagram of an image encoding apparatus.
1 is a residual encoding unit, 102 is a residual decoding unit, 103 is a frame memory, 104 is a motion prediction unit 1, 105 is an affine parameter inter-frame estimating unit, 106 is an affine parameter updating unit, 1
07 is the template memory, 108 is the affine motion compensator, 1
09 is a local motion compensator 1, 110 is a motion predictor 2, 111 is a motion compensator 2, 112 is a comparator, 113 is an adder, 114 is a differentiator, 1
Reference numeral 15 denotes a switch, and 116 denotes a motion information multiplexing unit.

FIG. 2 is a block diagram of the image decoding apparatus.
1 is a residual decoding unit, 202 is a frame memory, 203 is a local motion compensation unit 1, 204 is an affine motion compensation unit, 205 is a local motion compensation unit 2, 206 is a motion information separation unit, 207 is a selector, and 208 is a selector. A template memory, 209 is a switch, and 210 is an adder.

In the present embodiment, an encoding device and a decoding device based on MPEG1 are used as the basic configuration as in the conventional example.

In the configuration of the image encoding apparatus shown in FIG. 1, the residual encoder (101), the residual decoder (102), the adder (11)
3), the frame memory (103), the motion prediction unit 1 (103), the local motion compensation unit 1 (109), the difference unit (114), and the residual decoding unit ( 201), adder (2
10), frame memory (202), local motion compensator 1 (20
3) has the same configuration as the encoder and decoder based on MPEG1 cited in the conventional example. Therefore, the residual encoding unit (103) includes a digital cosine transform (DCT) compliant with MPEG1, a quantizer, and a Huffman encoder. Also, the residual decoding unit (102, 201) uses MPEG1
It consists of a compliant inverse quantization unit and a digital cosine inverse transformer.

The input image of the encoding device and the output image of the decoding device are composed of 240 × 352 pixels in length and width. A block for performing a correlation operation for detecting a motion vector is composed of 16 vertical pixels and 16 horizontal pixels.

The first embodiment configured as described above will be described by showing differences from MPEG1 and the conventional example.

The local motion estimator 1 (104) reads out the block information R of the input image as shown in FIG. 6 and simultaneously changes the displacement (u, v) for each pixel, thereby obtaining the frame memory (103). , The block information R ′ of the previous frame is read out from, and the correlation operation shown in Expression 1 is performed. This is the same operation as the local motion prediction unit 1 (504) in the conventional example. As a result, as shown in Expression 2, the deviation having the smallest value is searched to obtain the original estimation (p, q). This (p, q) is determined with half-pixel accuracy.

The above operation is performed for each block, and the result is sent to the local motion compensator (109), where macro block motion compensation similar to MPEG1 is performed.

In this embodiment, unlike the conventional example, the expression 1
Global motion compensation is performed by affine transformation expressed by six parameters of 0. In Equation 10, a0 to a5
Is the affine parameter, the pixel position of the x, y input image, (u∧
(x, y,), v∧ (x, y)) is a motion vector at a pixel position. Hereinafter, the process of obtaining the affine parameters will be described.

[0059]

(Equation 10)

In the process of obtaining the minimum deviation for each block by the correlation operation shown in Expression 1, the sequence S of absolute value errors near the minimum deviation shown in Expression 11 is calculated. () ^T represents the transpose of the matrix. (P, q) and S of each block are sent to the affine parameter inter-frame estimator (105).

[0061]

[Equation 11]

The affine parameter inter-frame estimator (10
In 5), the calculation of Expressions 12 to 17 is performed to calculate the correlation parameter.

[0063]

(Equation 12)

[0064]

(Equation 13)

[0065]

[Equation 14]

[0066]

(Equation 15)

[0067]

(Equation 16)

[0068]

[Equation 17]

The above calculation of the correlation parameter is performed for each block. The correlation parameter has the following meaning. By obtaining the coefficients of Expressions 12 to 17, the absolute value error of each block from the input image to the image one frame before is represented by the motion vector (u,
It can be expressed as a quadratic function with v) as a variable.

Ei, j shown in Expression 18 is expressed as a Taylor expansion at the position (p, q) of the absolute value error of the block. The function is used to evaluate the correlation.

[0071]

(Equation 18)

Here, the affine motion compensator (108) transforms the entire screen by the affine transformation expressed by the equations (19) and (20). (u (x, y), v (x, y)) is a motion vector at the position (x, y), and Expression 20 represents a vector composed of a sequence of affine transformation parameters.

[0073]

[Equation 19]

[0074]

(Equation 20)

Since the motion vector of each block is described by the affine parameters as shown in equations (19) and (20), the necessary condition for minimizing the sum of the absolute value error functions according to the variation principle is shown in equation (21). An Euler equation in which the partial derivative of the affine transformation parameter a must be a zero vector can be derived. This can be represented by the matrix of Equation 22.

The affine parameter inter-frame estimator (10
In (5), the matrices of Expression 23 (6 × 6 matrix) and Expression 24 (6 × 1 matrix) are further obtained from Expression 18, and finally the affine transformation parameters are obtained by Expression 25.

In equations 23 and 24, (xj, yi) is the center position of block i, j.

[0078]

(Equation 21)

[0079]

(Equation 22)

[0080]

(Equation 23)

[0081]

(Equation 24)

[0082]

(Equation 25)

On the other hand, the switch (115) closes every N (N ≧ 1) frames and copies the contents of the frame memory (103) to the template memory (107).

In this embodiment, N = 10. That is, while the frame memory (103) is updated every frame, the template memory (107) is updated every 10 frames. The operation of the affine parameter inter-frame estimating unit (105) described above obtains the corresponding position of the decoded image one frame before viewed from the input image as the affine parameter. The affine motion compensation unit (108) is stored in the template memory (107) from the viewpoint of the input image.
Since the decoded image in the range from the previous frame to the previous nine frames is transformed, the output of the affine parameter inter-frame estimator (105) cannot be used as it is.
Therefore, the affine parameters are obtained by combining the affine parameters between the preceding and succeeding frames using the fact that the affine transformation is linear for the pixel position of the input image as shown in Expression 10. This is realized by the affine parameter update unit (106).

The affine parameter update unit (106) is a template memory (107) viewed from the input image one frame before.
The affine transformation parameters for the images in are stored as a to 0 to a to 5. This parameter is reset to 0 every time a new image is stored in the template memory (107) in conjunction with the switch (115). a ~ 0 ~ a ~ 5 and a0 ~ obtained from the affine parameter inter-frame estimation unit (105)
Using a5, the affine transformation parameters a'0 to a'5 for the decoded image stored in the template memory (107) as viewed from the input image are obtained by Expressions 26 to 31. The obtained a '
Repeated updating is performed by substituting 0 to a'5 into a to 0 to a to 5 for each frame.

[0086]

(Equation 26)

[0087]

[Equation 27]

[0088]

[Equation 28]

[0089]

(Equation 29)

[0090]

[Equation 30]

[0091]

(Equation 31)

In the affine motion compensator (108), a'0 to a'5
Is used to calculate the motion vector shown in Equation 10 for all pixel positions of the input image, and obtain the pixel value of the pixel position at which the motion vector has been obtained on the image in the template memory (107). . The motion vector for global motion compensation is obtained with 1/4 pixel precision, and is obtained as linear interpolation of four pixel values at neighboring integer positions. Further, the motion prediction unit 2 (110) performs motion prediction on a macroblock basis on the transformed image. Local motion compensator (111),
The comparator (112), the difference unit (114), the residual encoding unit (10
1) The operations of the residual decoding unit (102) and the adder (113) are the same as in the conventional example.

As the motion vector information, the selection result SW in units of macroblocks, the motion vector mv1 or mv2, and the affine parameter a are multiplexed by the motion information multiplexing unit (116) and transmitted.

Next, the operation of the decoding device corresponding to the encoding device of FIG. 1 will be described with reference to FIG. Motion information separation unit (20
6) receives the output of the motion information multiplexing unit (116) as an input,
Selection result SW and motion vector mv1 in macro block units
Alternatively, mv2 and the affine parameter a are separated and decoded.

The operation of the local motion compensator 1 (203) or the affine motion compensator (204) and the local motion compensator (20)
5) The combination operation is selected. The selector (207) selects one of the outputs based on the SW.

In another configuration of the image decoding apparatus, the residual decoding unit (201), the frame memory (202), the local motion compensating unit 1
(203), affine motion compensator (204) local motion compensator 2
(205) template memory (208), switch (209),
The operation of the adder (210) is the same as that of the block of the same name in FIG. The switch (209) updates the contents of the template memory (208) every 10 frames in synchronization with the switch (115) on the encoding device side.

The present embodiment described above has the following features as compared with the conventional example. (1) In the conventional example, the output of the motion prediction unit 1 (504) is only a motion vector, and parameters describing the motion of the entire screen are calculated based on the motion vector. However, in a local correlation operation, a local motion vector is undefined in a region where there is no change in luminance. Even if there is a change in luminance, the motion vector has a degree of freedom in the tangential direction of the contour around the monotonous contour, and the local motion vector is similarly undefined. Therefore, a large estimation error is feared in the two-stage parameter estimation method shown in the conventional example. On the other hand, in the method shown in the present embodiment, the degree of freedom is quantitatively expressed by a quadratic function, and by minimizing the sum of the quadratic functions, the influence of a local motion vector which is otherwise undefined is suppressed. It is expected that parameter estimation can be performed more stably.

(2) Since the global motion compensation is performed based on a template updated at intervals of 10 frames, the problem that the spatial resolution of a predicted image not subjected to residual coding is gradually reduced is different from that of the conventional example. It is reduced in comparison.

It is difficult to directly obtain an affine parameter between an input image and an image separated by a maximum of 9 frames on the time axis by block correlation because a motion vector in each pixel may become large. Therefore, in the present embodiment, affine parameters are obtained between the preceding and succeeding frames, and the affine parameters are synthesized to obtain affine parameters at long frame intervals.

[0100] (Embodiment 2) Next, FIG an embodiment of an image encoding device 3 will be described with reference to FIG.

FIG. 3 is a block diagram of the image coding apparatus.
301 is a residual encoding unit, 302 is a residual decoding unit, 303 is a frame memory, 304 is a motion prediction unit 1, 305 is a representative point motion vector inter-frame estimating unit, 306 is a representative point motion vector updating unit, and 307 is Template memory, 308 is a representative vector interpolation motion compensator, 309 is a local motion compensator 1, 310 is a switch, 31
1 is a motion information multiplexing unit, 312 is a comparator, 313 is an adder, 314
Is a differentiator.

FIG. 4 is a block diagram of the image decoding apparatus.
1 is a residual decoding unit, 402 is a frame memory, 403 is a local motion compensation unit 1, 404 is a representative vector interpolation motion compensation unit, 405 is a template memory, 406 is a motion information separation unit, 407 is a selector, and 408 is A switch 409 is an adder.

Except for the motion information multiplexing unit (311), the operation of the components in FIG. 3 is the same as that in FIG. 1 if they have the same name. 4 except for the motion information separation unit (406).
2 are the same as those in FIG.

The differences between the second embodiment and the first embodiment are as follows: (1) Local motion compensation after global motion compensation is not performed. (2) Update and transmit affine parameters using motion vectors at three representative points. (3) Global motion compensation is performed by interpolation of motion vectors at three representative points. These are the three points. The differences are described below.

First, in the configuration shown in FIGS. 3 and 4, in the coding apparatus, the motion information multiplexing unit (311) uses the motion vector mv,
Multiplex the representative expression (s0, t0), (s1, t1), (s2, t2) of the representative point motion vector that is quadrupled and quantized to an integer value, and multiplex the selection result SW in macroblock units. Then, in the decoding device of FIG. 4, the motion information separation unit (406) outputs mv, (s0, t0), (s1, t1),
The operation of separating and decoding (s2, t2) and three pieces of information of SW is different.

The affine parameters are estimated by the following method. As shown in Expression 32, the affine parameter a can be obtained if the movement of three points o, h, and v that are not on the same straight line is determined. Po is a matrix related to the position of o shown in Equation 33, and Vo is Equation 3
6 is a motion vector at a position o shown in FIG. The same applies to h and v.

[0107]

(Equation 32)

[0108]

[Equation 33]

[0109]

(Equation 34)

In this embodiment, the positions of o, h, and v are set to the positions of (0, 0), (H, 0), and (0, V) on the input image.

H and V are the image sizes, H = 352 and V = 240. Next, as shown in FIG. 7 on the screen, TL, TR, C, B
The five points L and BR are selected as observation positions, and the representative point motion vector inter-frame estimator (305) selects the motion vector predictor 1 (3
Affine parameters are calculated using the results of the above five block correlations in the output of 04). This procedure is shown in FIG. A combination of three points is selected in order from the names of the five points, and the affine parameter a is obtained by Expression 25. Then, the motion vector at the position of o, h, v is calculated as 1 /
Obtain and store with 4-pixel accuracy.

For all combinations, the motion vectors are represented by o,
The positions of h and v are obtained, and an intermediate value is obtained as a result. This is V
Let o, Vh, Vv be another expression of the affine parameters for one frame. In the first embodiment, the affine parameters of the input image and the template are obtained by synthesizing the affine parameters.

In the present embodiment, the representative point motion vector updating unit (306) determines the correspondence between the input image one frame before and the image in the template memory (307) with respect to the representative point motion vectors Voo, V〜. h, V to v are stored.

This representative point motion vector is reset to zero vector every time an image is newly stored in the template memory (307) in conjunction with the switch (310). V ~ o, V
~ H, V ~ v and representative point motion vector inter-frame estimation unit (305)
Using the obtained Vo, Vh, Vv, the affine transformation parameters V′o, V′h, V ′ for the decoded image stored in the template memory (307) viewed from the input image according to Expressions 35 to 37. Find v. The obtained V'o, V'h, and V'v are substituted into V to o, V to h, and V to v for each frame to repeatedly update.

Then, the sequence of values (U4o, V4o), (U4h, V4h), (U4v, V4v) obtained by quadrupling V'o, V'h, V'v and converting them to integers is represented by the following difference format ( s0, s1, s2, t0, t1, t2).

[0116]

(Equation 35)

[0117]

[Equation 36]

[0118]

(37)

Here, the difference formats are as follows. s0 = U4o s1 = U4h-U4o s2 = U4v-U4o t0 = V4o t1 = V4h-V4o t2 = V4v-V4o This difference data is converted into a variable length code shown in Table 1 and encoded.

This includes the still image coding standard JPEG "JPEG Technical Specification" (ISO / IEC JTC1 / SC29 / WG10: JPEG Techn) by the International Standards Organization (ISO).
icalSpeicification, JPEG8-R8, August 1998). In this encoding method, the category SSSS and the Huffman code for the difference data are obtained. Here, SSSS is a variable for grouping the difference data. When the difference is positive, when the difference is expressed in binary, it is the number of the least significant bit in which “1” is first displayed from the upper bit. If the difference is negative, it is the number of the least significant bit in which "0" first appears when viewed from the upper bits when the difference value -1 is expressed in binary.

By transmitting the Huffman code of (Table 1) corresponding to the category, the category of the difference value can be known. However, since the difference data cannot be specified by this alone, an additional bit is added after the Huffman code. SSSS
Corresponds to the bit length of this additional bit. When the difference is positive, the SSSS bit is added from the least significant bit of the difference, and when the difference is negative, the SSSS bit is added from the least significant bit of the difference value -1. Becomes

In this way, for each of (s0, s1, s2, t0, t1, t2), the Huffman code of category SSSS and additional bits are transmitted as variable length codes. This coding is performed by the motion information multiplexing unit (311).

[0123]

[Table 1]

The motion information separation unit (406) finds a Huffman code corresponding to SSSS from the bit stream and decodes the category SSSS. Subsequently, the additional bits for the SSSS bits are read, and when the leading bit of the additional bit is 1, the difference is determined to be positive, and when the leading bit is 0, the difference is determined to be negative.

If the value is negative, the additional bits are expressed in binary notation, and 1 is added thereto. Then, 1 is added upward from the SSSS + 1st bit, and converted to a negative number.

In the first embodiment, the affine parameter a
And the motion vector of each pixel position on the input screen is calculated by Expression 10. In this embodiment, equivalently (U4o, V
4o), (U4h, V4h) and (U4v, V4v) are calculated by interpolation.

According to Equation 32, the affine parameter o is obtained from Equations 38 and 39 from the motion vectors of the three points o, h, and v.

[0128]

(38)

[0129]

[Equation 39]

When expanded, the motion vectors are obtained as Expressions 40 and 41. This means that the motion vectors of the three points o, h, and v are obtained by dividing the motion vectors according to the positions.

[0131]

(Equation 40)

[0132]

[Equation 41]

For each pixel position, 1/4 accuracy is guaranteed,
In addition, in order to make the prediction images based on the global motion compensation on the encoding side and the decoding side the same, it is conceivable to perform the expressions shown in Expressions 42 and 43 as integer operations.

[0134]

(Equation 42)

[0135]

[Equation 43]

In equations 42 and 43, (X, Y) is a pixel position on the input screen, V4 (X, Y), V4 (X, Y) are quadrupled integer motion vectors at that position, and (U4o , V4o) is the quadrupled position o
, The integer motion vectors (U4h, V4h) and (U4v, V4v) are the motion vectors at the upper right and lower left corners of the screen, respectively.

In the present embodiment, the actual calculation is (U4o, V
4o), (U4h, V4h), (U4v, V4v) are in the difference format (s0, s1, s2, t0,
It uses the fact that it is transmitted at t1, t2).

That is, in the representative vector interpolation motion compensator (308), equations (42) and (43) are realized as equations (44) and (45).

[0139]

[Equation 44]

[0140]

[Equation 45]

At this time, if it is found that all of s1, s2, t1, and t2 are zero, the calculation of Expressions 44 and 45 is simplified. That is, s1, s2,
Calculations for global motion compensation for t1 and t2 are omitted.

The features of this embodiment are summarized below. (1) Compared to the first embodiment, the second embodiment omits the motion prediction after global motion compensation, so that the image coding apparatus performs relatively heavy block matching processing. It is of the same size as MPEG1 described in the prior art.

(2) The processing load is light because the motion of the entire screen is obtained by five block correlations. Also, by selecting three block correlations from among the five block correlations, calculating the motion vectors of the representative points o, h, and v and selecting the intermediate values of the components, even if there is a motion different from the entire screen, The configuration is less susceptible to that effect.

(3) By selecting the representative points o, h, and v at orthogonal positions and transmitting the motion vectors at these points, the coordinate calculation for motion compensation is simplified. Thereby, high calculation accuracy can be compensated.

(4) By subjecting the representative points to variable-length coding in a difference format, if the affine transformation is a simple parallel movement, (s
The parallel movement is represented by (0, t0), and the other s1, s2, t1, and t2 are all zero. Therefore, if the parallel movement is one pixel horizontally, the affine transformation parameters can be transmitted with a total of 15 bits.

In Table 1, the movement of 1/4 pixel precision is -5.
Although it can be expressed in the range of 11.75 to +512 pixels, if this is performed by fixed-length coding, 12 bits × 6 = 72 bits are required. In comparison with this, the amount of generated bits can be suppressed in the range of a normal motion vector. Also, it can be easily determined from the difference expression whether the image deformation can be performed only by parallel movement or whether general affine transformation needs to be performed. Thus, there is an advantage that address calculation for global motion compensation can be performed in a simplified manner.

Although the global motion data transmission method described in this embodiment has been described in the case where global motion compensation is affine transformation, it is extended to a plane motion model under bilinear interpolation or perspective transformation. be able to.
In each case, the conversion is eight parameters, and can be realized by transmitting a new 1/4 pixel precision motion vector (U4hv, V4hv) at the coordinate position (H, V) of the lower right point of the screen. In the case of bilinear interpolation, as shown in Equations 46 and 47, 8
One parameter describes the movement of the entire screen.

[0148]

[Equation 46]

[0149]

[Equation 47]

S0 = U4o s1 = U4h-U4o s2 = U4v-U4o s3 = U4o-U4h-U4v + U4hv t0 = V4o t1 = V4h-V4o t2 = V4v-V4o t3 = V4o-V4h-V4v Then, in the case of bilinear interpolation, the motion vector is

[0151]

[Equation 48]

[0152]

[Equation 49]

Can be expressed by Where (s3, t3)
If = (0,0), the same processing as the affine transformation is performed. If all other than (s0, t0) are zero, the same processing as the parallel movement is performed. In the case of a plane motion model in perspective transformation, the calculation is somewhat complicated, but in each case (s3, t3) = (0, 0), whether (s0, t0) is zero other than Please affine transformation,
Since the automatic switching with the parallel movement can be performed, the advantage of the motion vector transmission of the image end point in the above-described difference format is great.

[0154]

As described above, according to the present invention, a decrease in resolution of a predicted image caused by global motion compensation can be reduced as compared with the prior art. This compensates for the movement caused by the camera's swiveling and zooming, so that an image can be transmitted with a smaller amount of information transmission.

[0155] Further, since performing global motion compensation and interpolating the motion vectors, it is possible to maintain a high operation accuracy reduces the complexity of address calculation. Therefore, it is possible to easily match the prediction images on the encoding side and the decoding side, and as a result, it is possible to perform efficient prediction image encoding by global motion compensation.

[0156]

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an image encoding device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of an image decoding apparatus according to the first embodiment.

FIG. 3 is a configuration diagram of an image encoding device according to a second embodiment of the present invention.

FIG. 4 is a configuration diagram of an image decoding apparatus according to the second embodiment.

FIG. 5 is a configuration diagram of a conventional image encoding device.

FIG. 6 is a principle diagram of a block correlation method.

FIG. 7 is a diagram showing representative points and observation points in the second embodiment.

FIG. 8 is an operation diagram of a representative point motion vector inter-frame estimating unit according to the second embodiment.

[Explanation of symbols]

101, 301 Residual coding unit 102, 201, 302, 401 Residual decoding unit 103, 202, 303, 402 Frame memory 104, 110, 304 Local motion prediction unit 105 Affine parameter inter-frame estimation unit 106 Affine parameter update unit 107, 208, 307, 405 Template memory 108, 204 Affine motion compensator 109, 110, 203, 205, 309, 403 Local motion compensator 112, 312 Comparator 113, 210, 313, 409 Adder 114, 314 Differentiator 116, 311 motion information multiplexing section 115, 209, 310, 408 switch 206, 406 motion information separating section 207, 407 selector 305 representative point motion vector inter-frame estimating section 306 representative point motion vector updating section 308, 404 representative vector Interpolation motion compensation unit SW Prediction selection result in macroblock units SW mv1, mv2 Macroblock motion vector g Global motion compensation parameter a Affine motion compensation parameter U4, V4 Representative point motion vector multiplied by 4 and quantized to an integer value

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N ^7/ 24-7/68

Claims (1)

(57) [Claims] 1. A short-term frame memory for storing a pixel block of n × n dots as an encoding target frame, and storing decoded frames preceding and succeeding in time as a short-term reference frame, and the encoding target frame and the short-term reference Local motion vector detecting means for detecting a local correspondence between frames as a local motion vector; and short-term motion compensating means for reading an image from the short-term frame memory using the local motion vector and generating a predicted image. A long-term frame memory for storing, as a long-term reference frame, a decoded frame whose update interval is longer than that of the short-term reference frame with respect to the encoding target frame; and an image one frame before the encoding target frame. Between
The absolute value error is expressed as u (x , y), v (x , y) (x, y
U, v are motion vectors at pixel position (x, y)
ト ル) as a quadratic function with variables as u, v and
A relational expression with the fin transformation parameter a is created,
The function is defined as follows:
And a value that minimizes the absolute value error is obtained.
Affine transformation parameters
Affine parameter inter-frame estimator for estimating as a
And the output from the affine parameter inter-frame estimator
Affine parameters between frames before and after
An image coding apparatus comprising: an affine parameter update unit; and a full-screen motion compensation unit that reads an image from the long-term frame memory using the affine parameters of the affine parameter update unit and generates a predicted image.