JPH05199514A - System for estimating two-dimensional moving field from time sequential image - Google Patents

Abstract

(57) [Abstract] [Purpose] The two-dimensional velocity field with high accuracy is obtained by adapting to the observation noise that changes depending on the local property of the image and performing appropriate control for minimizing the evaluation function. [Structure] A two-dimensional motion field estimation system that estimates an appropriate apparent motion field of a moving image based on an evaluation condition consisting of a spatial and temporal brightness gradient and a motion field of an image between two screens has a space and time. A variance estimation circuit that extracts the variance value of the noise of the observation system based on the brightness gradient, the extraction of the motion field based on this variance value and the space / time brightness gradient, and the extracted motion field, the variance value, and the space value. A motion field extraction circuit for extracting an appropriate apparent motion field by repeating the determination of the validity of the extracted motion field based on the evaluation condition using the temporal brightness gradient is provided.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for estimating an apparent motion field (motion vector field) on a two-dimensional plane from a moving image projected on a two-dimensional plane such as a television, and more particularly, The present invention relates to a two-dimensional motion field estimation system from a time series image suitable for obtaining a highly accurate two-dimensional velocity field by adapting to the local property of observation noise.

[0002]

2. Description of the Related Art Estimating a two-dimensional apparent motion field from a moving image sequence is a basic technique in computer vision and image recognition. If the motion field is accurately obtained, it is possible to estimate the three-dimensional motion and the three-dimensional structure including the depth from the two-dimensional image. This is the basic technology needed to achieve the "machine eye."

Conventional techniques for obtaining a two-dimensional motion field include, for example, B. Horn and B. Schunkk,
"Determining optical flo
w ", Artificial Intelligenc
e 17, 1981, pp. 185-203, there is a gradient method that uses the relationship between the spatial and temporal gradients of image brightness and motion. That is, the brightness of the point (i, j) at time t on one screen composed of the xy plane is I
Described as (i, j, t). Here, i indicates the spatial position on the x axis and j indicates the spatial position on the y axis. If the amount of movement in the time interval T is u, v (the movement in the x direction is u, the movement in the y direction is v), the position of the corresponding pixel at time t + T is (i + u, j + v, t + T). Given,

[Equation 1] Becomes

Taylor expansion of the brightness I yields the following equation.

[Equation 2] However, the brightness gradient in the x direction of the image at the point (i, j, t) is Ix = ∂I (i, j) / ∂x, and the brightness gradient in the y direction is Iy = ∂I (i, j). / ∂y Also, O
(2) represents an error term of second order or higher. On the other hand, the difference between frames,

[Equation 3] As a first-order approximation,

[Equation 4] Can be expressed as Since Ix, Iy, and It can be calculated from the image, the equation of Equation 4 becomes a linear equation with two unknowns.

However, since there are two unknowns with respect to one equation, in principle, only the movement of the luminance in the gradient direction can be obtained only by this equation. Therefore, in order to obtain the component perpendicular to the gradient direction, a method has been proposed which assumes smoothness of spatial variation of the two-dimensional motion field. This is performed by using two frames separated by the time interval T, the evaluation function Ep of the power of the spatial variation (smoothness) of the two-dimensional motion field in the screen, the luminance gradient of the two-dimensional motion field, and the motion field The evaluation function Ed of the power of the error from the equation of the constraint condition (4) is defined, and the motion field where the sum of Ep and Ed is minimized is obtained. That is, the motion field u (i, j), v
The differential of (i, j) in the x and y directions is ux = ∂u (i,
j) / ∂x and vx = ∂v (i, j) / ∂x and uy =
∂u (i, j) / ∂y and vy = ∂v (i, j) / ∂y
Then, the evaluation function E at this time is

[Equation 5] Can be formulated as a problem to obtain u (i, j) and v (i, j) that minimize

In the prior art, the weighting factor λ of the equation (5) is selected so as to be proportional to the S / N of the observation data of the Ed term. Ep represents the condition that the solution changes spatially and smoothly, and Ed describes the condition that the final solution is close to the measurement result. Therefore, when Ed is unreliable, the weighting factor λ is reduced in order to increase the smoothing effect by Ep. Conversely, if the data is reliable, the weighting factor λ is increased.

The observation system will be examined below in order to investigate the formulation of the regularization of the equation (5) in motion field estimation. The process of observing the gradient is

[Equation 6] Given in. Here, r is observation noise. Now, assuming that the variance of the observation noise r is constant over the entire screen, a good estimation result can be obtained by the conventional equation (5). That is, in the prior art, it is assumed that the variance of observation noise is constant.

However, since the nature of noise generally changes locally, this assumption cannot be said to be correct.
For example, in the region where the brightness gradient is small and the change is small, the expected value of the variance of the observation noise r is considered to be small.
On the other hand, in the region where the gradient is large, each observed value I of the equation (6) is
The error in the values of x, Iy, and It becomes large, and the variance of the observation noise r also becomes large. In the region of the extreme value of the brightness, the spatial gradient of the brightness changes from frame to frame, so the observation error is considered to be very large. in this way,
It can be seen that the observation noise changes due to the local nature of the image. Therefore, as in the past, the penalty functional and
If a single coefficient λ indicating the balance of the stabilizing functional is used, it cannot be said that appropriate control for minimizing the evaluation function E is performed.

[0009]

The problem to be solved is that the observation noise changes due to the local nature of the image. Therefore, in the conventional technique, an appropriate control for minimizing the evaluation function E is performed. It is not possible to obtain a precise two-dimensional motion field. The object of the present invention is to solve these problems of the prior art, to enable to obtain an accurate two-dimensional motion field from a moving image projected on a two-dimensional plane such as a television, and to determine the speed of an object in robot vision. A two-dimensional motion field estimation system from a time-series image that can improve the accuracy of detection and motion compensation prediction in interframe coding of television signals.

[0010]

In order to achieve the above object, a two-dimensional motion field estimation system from a time-series image of the present invention uses a spatial luminance gradient of an estimation target and a spatial luminance gradient of an estimation target based on luminance data of two screens of a moving image. It is equipped with a gradient calculation circuit that calculates the temporal brightness gradient, and based on the evaluation conditions consisting of the spatial brightness gradient and temporal brightness gradient calculated by this gradient calculation circuit, and the apparent motion field of the image between the two screens, two-dimensional In a two-dimensional motion field estimation system that estimates an appropriate apparent motion field on this two-dimensional plane from a moving image projected on a plane,
Based on the spatial brightness gradient and temporal brightness gradient calculated by the gradient calculation circuit, a variance estimation circuit that extracts the variance value of the noise of the observation system that changes depending on the local characteristics of the image, and this variance estimation circuit and gradient calculation circuit Extraction of the motion field based on the extracted variance value, spatial brightness gradient, and temporal brightness gradient, and extraction of the motion field based on the evaluation conditions using the extracted motion field, variance value, spatial brightness gradient, and temporal brightness gradient. It is characterized by providing a motion field extraction circuit for repeating appropriateness determination and extracting an appropriate apparent motion field.

[0011]

According to the present invention, in the conventional gradient method, the observation error r of the equation (6) is small in the region where the luminance gradient is gentle, but in the region where the luminance gradient is large or the region around the extreme value, Focusing on the fact that the observation error becomes large, the variance of the observation error is estimated based on the value of the gradient, and then the evaluation function is minimized. This makes it possible to estimate a motion field with higher accuracy.

[0012]

Embodiments of the present invention will now be described in detail with reference to the drawings. FIG. 1 is a block diagram showing an embodiment of a configuration according to the present invention of a two-dimensional motion field estimation system from time series images according to the present invention. In the figure, 1 is a moving image input terminal for inputting moving image data that is a target of motion estimation, and 2 is a sampling for converting analog data (moving image data) from the moving image input terminal 1 into digital data. Quantization circuit (denoted as A / D in the figure), 3 is a first frame memory for storing the moving image data digitally converted by the sampling and quantization circuit 2 for each screen, and 4 is a first frame memory. The second frame memory 5 for storing the data of the previous screen transferred from the frame memory 3 of No. 2 is based on the data of the first frame memory 3 and the data of the second frame memory 4, and the spatial luminance gradient to be estimated. And a gradient calculation circuit 6 for calculating the temporal brightness gradient, and 6 relates to the present invention, and a variance estimation circuit 7 for estimating the observation noise that changes depending on the local property of the image based on the brightness gradient value from the gradient calculation circuit 5. ,
A two-dimensional motion field memory that sequentially switches and stores motion fields obtained based on data between two screens, and 8 is an estimated variance value from the variance estimation circuit 6 and a brightness gradient value from the gradient calculation circuit 5. And a two-dimensional motion field arithmetic circuit that corrects the two-dimensional motion field based on the two-dimensional motion field from the two-dimensional motion field memory 7. Reference numeral 9 denotes the estimated variance value from the variance estimation circuit 6 and the gradient arithmetic circuit 5. The evaluation value is calculated based on the brightness gradient value of 1 and the two-dimensional motion field from the two-dimensional motion field calculation circuit 8, and the validity of the evaluation value is calculated based on the comparison between the calculated evaluation value and a predetermined threshold value. Based on the determination result of the evaluation value calculation circuit 9, the evaluation value calculation circuit 10 that determines the two-dimensional motion field from the two-dimensional motion field calculation circuit 8 is taken in as the final calculation result and output via the output terminal 13. Latch 11 for the two-dimensional motion field memory 7, Initial motion field input terminal for inputting an initial value of the dimension motion field, 12, two-dimensional motion field input destination toggle its switch to the memory 7, and, 13, the latch 1
This is a motion field output terminal that outputs a two-dimensional motion field in which 0 is captured. The two-dimensional motion field memory 7, the two-dimensional motion field calculation circuit 8, the evaluation value calculation circuit 9, and the latch 10 constitute a motion field extraction circuit of the present invention.

With such a configuration, the two-dimensional motion field estimation system of this embodiment minimizes the evaluation function after estimating the variance of the observation error based on the gradient value calculated by the gradient calculation circuit 5. By doing so, highly accurate estimation of the motion field is performed. The operation of the two-dimensional motion field estimation system having such a configuration will be described below.

First, at the start of calculation, a two-dimensional initial motion field is input from the initial motion field input terminal 11 and written in the motion field memory 7 via the switch 12. The initial values may be all "0" or a motion field obtained based on data between two screens. Also, the switch 12 is switched to the motion field calculation circuit 8 side after the initial data is input. An analog moving image is input from the input terminal 1 and subsequently converted into digital data by the sampling and quantizing circuit 2. The data converted by the sampling and quantizing circuit 2 is written in the first frame memory 3 for each screen, and when one frame is written, the data is transferred to the second frame memory 4. Therefore, the first frame memory 3 stores the brightness data of the current frame, and the second frame memory 4 stores the brightness data of the previous frame. The luminance data of the two screens respectively stored in the first frame memory 3 and the second frame memory 4 are input to the gradient calculation circuit 5, and the gradient calculation circuit 5 calculates the spatial luminance gradient and the difference between the two screens. The brightness difference (temporal brightness gradient) is calculated. The brightness gradient value calculated in this way is distributed to the variance estimation circuit 6, the motion field calculation circuit 8,
It is also input to the evaluation value calculation circuit 9.

In the conventional gradient method, the observation error r of the equation (6) is small in the region where the luminance gradient is gentle, but the observation error becomes large in the region where the luminance gradient is large or the region around the extreme value. .. Therefore, in the two-dimensional motion field estimation system of the present embodiment, as described below, based on the value of the gradient, the variance of the observation error is estimated, and then the evaluation function is minimized to improve the accuracy. Estimate a high motion field.

First, the variance estimation circuit 6 estimates the observation noise based on the brightness gradient value. That is, in a region where the brightness gradient is small and the spatial change is small, the variance value is small, and in a region where the gradient is large, the variance is large,
Further, in the region of the extreme value of the luminance, the spatial gradient of the luminance changes between frames, so the variance is increased. This correspondence is designed off-line and can be realized by a read-only memory when the circuit is operating.

Next, in the motion field calculation circuit 8, the estimated variance value thus obtained from the variance estimation circuit 6, the brightness gradient value calculated by the gradient calculation circuit 5, and the two-dimensional motion field memory. The two-dimensional motion field is corrected based on the two-dimensional motion field input from 7. This motion field calculation circuit 8
Is realized only by the addition, subtraction, multiplication and division calculation, and the obtained two-dimensional motion field correction value is stored in the evaluation value calculation circuit 9 and the latch 10.
It is input to the two-dimensional motion field memory 7 via the switch 12. Hereinafter, the operation of the motion field calculation circuit 8 will be described in more detail.

The observation system for image I is

[Equation 7] Given in. Note that r is noise of the observation system regarding the image I. Here, the variance σr of the observation noise r can be estimated based on the brightness gradient value. That is, in a region where the brightness gradient is small and the spatial variation is small, the variance value is small, in a region where the gradient is large, the variance is large, and in the region where the brightness is extreme, the spatial gradient of the brightness changes between frames. Therefore, it can be estimated that the variance is large. Then, the formula of Equation 5 becomes
Generalized evaluation function

[Equation 8] It is represented by. Here, λ is a parameter determined based on prior knowledge in the scene.

For the two-dimensional motion fields u and v, after the parameter λ is given in advance, the variance σr of the observation noise r is evaluated, and the above-mentioned evaluation function Ev is minimized while observing the brightness value I. Is obtained by This minimization problem is
It suffices to partially solve the two-dimensional motion fields u and v to solve the simultaneous equations that are zero. This can be determined using the relaxation method. That is, the x component A and the y component B of the motion field of the pixel (i, j, t) in the n-th step are

[Equation 9] By repeatedly using, the x component A and the y component B of the motion field approach the final solution, and the estimated value is obtained. However, here

[Equation 10] Is.

In the evaluation value calculation circuit 9, the gradient value obtained from the gradient calculation circuit 5, the variance value calculated in the variance estimation circuit 6, and the two-dimensional motion field value corrected in the motion field calculation circuit 8. Is used to calculate the evaluation value represented by the equation (8). The evaluation value calculation circuit 9 is composed of a multiplier and an accumulator. Then, the calculated evaluation value is compared with a predetermined threshold value, and when the evaluation value becomes smaller than the threshold value, the iterative calculation is stopped and a timing signal is output to the latch 10. Then, the latch 10 inputs the two-dimensional motion field from the motion field calculation circuit 8 and outputs the two-dimensional motion field data as the final calculation result to the output terminal 13 according to the timing signal from the evaluation value calculation circuit 9. Output through.

In this way, in the two-dimensional motion field estimation system of this embodiment, the variance of the observation error that changes depending on the local property of the image is estimated based on the value of the gradient, and then the evaluation function is minimized. By adapting it, it is possible to obtain a highly accurate two-dimensional velocity field by adapting to the local property of observation noise. Then, it can be applied to speed detection of an object in robot vision, motion compensation prediction in interframe coding of a television signal, and the like.

FIG. 2 is a flow chart showing an embodiment of the operation of the two-dimensional motion field estimation system in FIG. 1 according to the present invention. First, the initial value of the two-dimensional motion field on the screen is set (step 201). Subsequently, for each point (i, j) on the screen, spatial brightness gradients Ix, Iy and temporal brightness gradient It are calculated, and the variance (σr) ² of the observation noise is calculated based on these Ix, Iy, It. Calculate (step 202). After that, the repetitive calculation is started. Each step of this iterative operation can be divided into two processes. First, using the formula of the formula 9, the evaluation value of the formula of the formula 8 is
U and v, which are smaller than those in the previous step, are obtained for all pixels (step 203). Next, the evaluation formula of Expression 9 is calculated (step 204), and when the degree of the phenomenon of the value becomes smaller than the set level (step 20).
At 5), the calculation ends. Otherwise, the process returns to the first step (step 206).

As described above with reference to FIGS. 1 and 2, in the two-dimensional motion field estimation system from time series images of this embodiment, the variance of the observation error is calculated based on the value of the brightness gradient. Estimate and then minimize the evaluation function. This makes it possible to obtain a highly accurate two-dimensional velocity field by adapting to the local property of observation noise. The present invention is
The present invention is not limited to the embodiment described with reference to FIG. 1 and FIG. 2, and if the variance of the observation error is estimated based on the value of the brightness gradient and then the evaluation function is minimized. good.

[0024]

According to the present invention, it is possible to perform appropriate control for minimizing the evaluation function E in consideration of the observation noise that changes depending on the local property of the image. It is possible to obtain an accurate 2D motion field from a moving image projected on a 2D plane, and it is possible to use various methods such as speed detection of an object in robot vision and motion compensation prediction in interframe coding of television signals. It is possible to improve the performance of the applied technology.

[0025]

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a configuration according to the present invention of a two-dimensional motion field estimation system from a time series image according to the present invention.

FIG. 2 is a flowchart showing an embodiment of the operation of the two-dimensional motion field estimation system in FIG. 1 according to the present invention.

[Explanation of symbols]

 1 Moving Image Input Terminal 2 Sampling Quantization Circuit 3 First Frame Memory 4 Second Frame Memory 5 Gradient Calculation Circuit 6 Variance Estimation Circuit 7 Two-dimensional Motion Field Memory 8 Two-dimensional Motion Field Calculation Circuit 9 Evaluation Value Calculation Circuit 10 Latch 11 Initial motion field input terminal 12 Switch 13 Motion field output terminal

Claims (1)

[Claims] 1. Based on luminance data of two screens of a moving image,
The spatial brightness gradient and the temporal brightness gradient to be estimated are provided, and the spatial brightness gradient and the temporal brightness gradient calculated by the gradient calculation means and the apparent motion field of the image between the two screens are used. In a two-dimensional motion field estimation system from a time-series image that estimates an appropriate apparent motion field on the two-dimensional plane from a moving image projected on the two-dimensional plane based on the evaluation condition Based on the calculated spatial brightness gradient and temporal brightness gradient, a variance estimation unit that extracts the variance value of the noise of the observation system that changes depending on the local properties of the image, and the variance extracted by the variance estimation unit and the gradient calculation unit. Extraction of the motion field based on the value, the spatial brightness gradient, and the temporal brightness gradient, and the extraction based on the evaluation condition using the extracted motion field, the dispersion value, the spatial brightness gradient, and the temporal brightness gradient. Repeat the determination of validity of the feeder field, the appropriate apparent two-dimensional motion field estimation system from the image sequence and providing a motion field extracting means for extracting the motion field of.