JP6854629B2 - Image processing device, image processing method - Google Patents

Description

The present invention relates to a technique for obtaining an optical flow between images.

In recent years, the importance of a technique for associating each pixel between images has increased. Correspondence is the relationship with the pixels of the reference image which are considered to be the same as the pixels of the reference image, and can be expressed by the coordinates of two points. When a stereo image or a multi-viewpoint image is input, the depth of the subject can be calculated from the correspondence of pixels, so that it can be applied to three-dimensional image processing. Further, if images (moving images) captured continuously are input and the correspondence is expressed as relative coordinates, it becomes a motion vector. By using a motion vector for each pixel (hereinafter referred to as an optical flow), it is possible to track a moving object, isolate a moving image, and the like.

The gradient method is a typical method for acquiring optical flow. In the gradient method, the optical flow is calculated from the direction and magnitude of the spatiotemporal luminance change of the pixel. The gradient method can be roughly divided into two types. In the first gradient method, it is assumed that the pixels around the pixel of interest have the same movement, and the optical flow is calculated from the average spatiotemporal brightness change of the pixels in the patch centered on the pixel of interest (hereinafter). , Called a patch-based method). In the second gradient method, the brightness difference between images and the smoothing term representing the smoothness of the optical flow are weighted and added for each pixel, and the sum of all the pixels is used as energy. (Hereinafter referred to as energy optimization method).

A typical patch-based method is the Lucas-Kanade method (hereinafter referred to as the LK method) described in Non-Patent Document 1, and the same concept is used in Patent Document 1. In Patent Document 2, an energy optimization method is used.

International Publication No. 06/075394 Japanese Unexamined Patent Publication No. 9-178764

Pyramidal Implementation of the Lucas Kanade Footure Tracker Description of the algorithm Jean-Yves Bouget [online] [retrieved on 2016-11-07] Retrieved from the Internet stanford. edu / cs223b04 / algo_tracking. pdf>

However, the energy optimization method represented by Patent Document 2 requires iterative calculation for energy optimization, and has a problem that the amount of calculation increases. On the other hand, the patch-based method represented by Non-Patent Document 1 does not require iterative calculation, so that the optical flow can be estimated at high speed. However, since the constraint conditions are not clearly considered, there is a high possibility that a flow vector deviating from the correct answer value is estimated, and there is a problem that the estimation becomes unstable.

The method of Patent Document 1 is an improvement of the patch-based method so as to smooth the estimated optical flow in the hierarchical processing. As a result, the appearance of the flow vector deviating from the correct answer can be suppressed, but there is a problem that the estimated value becomes unstable in the region where the texture is small.

The present invention has been made in view of such a problem, and provides a technique for estimating an optical flow with high accuracy with a small amount of calculation.

The uniformity of the present invention comprises a first set of elements of a first image and a plurality of reduced images obtained by recursively reducing the first image at a specified reduction ratio, a second image, and the second image. A second set having a plurality of reduced images recursively reduced at the specified reduction rate, an acquisition means for acquiring the image, and an image belonging to the second set are selected in ascending order of image size. The selection means to be selected and each pixel of the selected image selected by the selection means this time are moved according to the converted optical flow obtained by converting the optical flow corresponding to the image previously selected by the selection means according to the size of the selected image. The generation means for generating the moved selected image, the first difference between the image belonging to the first set and the image having the same size as the selected image and the moved selected image, and the converted image. The second difference, which is the difference between the optical flow and the processed optical flow obtained by smoothing the converted optical flow, and the optical flow that minimizes the evaluation value based on the second difference correspond to the selected image. It is characterized by including a calculation means obtained as an optical flow and an output means for outputting the optical flow corresponding to the second image obtained by the calculation means.

According to the configuration of the present invention, the optical flow can be estimated with high accuracy with a small amount of calculation.

A block diagram showing a hardware configuration example of a computer device. The figure explaining the optical flow. The block diagram which shows the functional configuration example of an image processing apparatus. Flowchart of processing for generating optical flow. The block diagram which shows the functional configuration example of an image processing apparatus. Flowchart of processing for generating optical flow. Reference The figure explaining the process for obtaining an optical flow. The block diagram which shows the functional configuration example of an image processing apparatus. The block diagram which shows the functional configuration example of an image processing apparatus.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In addition, the embodiment described below shows an example when the present invention is concretely implemented, and is one of the specific examples of the configuration described in the claims.

[First Embodiment]
In this embodiment, an example of an image processing apparatus having the following configuration will be described. This image processing device includes a first set having a first image and a plurality of reduced images recursively reduced at a predetermined reduction ratio as elements, a second image, and the second image. A second set having a plurality of reduced images obtained by recursively reducing the images at the above-specified reduction ratio is acquired. Then, the image processing device selects the images belonging to the second set in ascending order of image size. Then, the image processing device generates a moved selected image in which each pixel of the selected image selected this time is moved according to the converted optical flow in which the optical flow corresponding to the previously selected image is converted according to the size of the selected image. To do. Then, the image processing device relates to the first difference, which is the difference between the image having the same size as the selected image and the moved selected image among the images belonging to the first set, and the converted optical flow and the converted optical flow. The second difference, which is the difference from the processed optical flow that has undergone the smoothing process, and the optical flow that minimizes the evaluation value based on the difference are obtained as the optical flow corresponding to the selected image (calculation). Then, the image processing device outputs the optical flow corresponding to the second image obtained by this calculation.

First, various definitions used in each of the following embodiments including the present embodiment will be described. The "optical flow" (hereinafter, may be referred to as OF) in the following description is a map image in which the motion vector of the target image with respect to the reference image is registered for each pixel. That is, the optical flow has the same resolution (the number of pixels in the vertical and horizontal directions) as the target image, and the element of the optical flow corresponding to each pixel of the target image is a two-dimensional vector.

In the following, when the image is expressed as I, the pixel value at the pixel position (x, y) on the image is expressed as I (x, y). In the optical flow, the element corresponding to the pixel position (x, y) on the reference image I is expressed as (u (x, y), v (x, y)). u (x, y) represents the pixel position of the reference image I (x, y) the horizontal component (X component) of the motion vector corresponding to, v (x, y) is the pixel position of the reference image I ( It represents the vertical component (Y component) of the motion vector corresponding to x, y).

The optical flow will be described by taking FIG. 2 as an example. FIG. 2 describes the optical flow of the image 202 with respect to the image 201. The image 201 is an image of the N (N is an integer of 1 or more) frame in the moving image of the scene in which the person 203 is moving while moving the imaging device, and the image 202 is the (N + N'in the moving image. ) (N'is an integer of 1 or more) This is the image of the frame. Images 201 and 202 include a person 203 and a house 204 as subjects. The motion vector 205 represents a motion vector from the person 203 in the image 201 to the person 203 in the image 202, and the motion vector 206 represents a motion vector from the house 204 in the image 201 to the house 204 in the image 202. Represents. Generally, the motion vectors for each pixel constituting the area of the person 203 (house 204) in the image are not exactly the same, but in FIG. 2, for the sake of simplicity, the motion vector of each pixel in the object is shown. Are all the same. That is, in FIG. 2, the motion vectors of each pixel in the region of the person 203 in the image 201 are all motion vectors 205, and the motion vectors of each pixel in the region of the house 204 in the image 201 are all vector 206. Here, the component of the motion vector 205 is (10, 5), and the component of the motion vector 206 is (-5, 0). At this time, when the pixel position (x, y) on the image 201 is included in the area of the person 203, the element (u (x, y) corresponding to the pixel position (x, y) in the optical flow with respect to the image 201). , V (x, y)) = (10, 5). Further, when the pixel position (x, y) on the image 201 is included in the area of the house 204, the element (u (x, y), which corresponds to the pixel position (x, y) in the optical flow with respect to the image 201). v (x, y)) = (-5,0). When the pixel position (x, y) on the image 201 is included in the background area (area other than the person 203 and the house 204), the element corresponding to the pixel position (x, y) in the optical flow with respect to the image 201. (U (x, y), v (x, y)) = (0,0).

In the present embodiment, the first image and the second image (the imaging time of the first image is earlier than the second imaging time) captured at different times by a single imaging device are acquired, and the image is taken. Generate an optical flow of the second image with respect to the first image. The first image and the second image are not limited to images captured at different times by a single imaging device, and may be images captured at the same time by a plurality of imaging devices. Images may be captured at different times by a plurality of image pickup devices.

Next, regarding the functional configuration example of the image processing device and its operation according to the present embodiment, FIG. 3, which is a block diagram showing the functional configuration example of the image processing device, the processing performed by the image processing device to generate the optical flow. This will be described with reference to FIG. 4, which shows a flowchart. In the drawings, OF represents an optical flow. Further, the process according to the flowchart shown in FIG. 4 is a process for obtaining an optical flow for one image. However, for example, when obtaining an optical flow for each of a plurality of images, processing may be performed for each of the plurality of images according to the flowchart of FIG.

In step S401, the image data acquisition unit 301 acquires the first image and the second image described above. Hereinafter, the case where only two images are acquired will be described, but a plurality of images or moving images may be acquired. If there are three or more images or a moving image, select the two target images or frames and proceed with the subsequent processing.

In step S402, the image reduction unit 302 _{recursively reduces the first image I 1 according} to the reduction ratio scale_factor (hereinafter referred to as sf: 0 <sf <1) to generate a plurality of reduced images. Further, the image reduction unit 302 _{recursively reduces the second image I 2} according to the reduction ratio sf to generate a plurality of reduced images. Specifically, the image reduction unit 302 first acquires the maximum number of layers (max_lv), which is the number of reduced images generated from _{I 1} and I _2. The maximum number of layers max_lv may be set in advance in the image processing device 100, or may be input by the user. In the present embodiment, _{the size (vertical and / or horizontal size) of the reduced image obtained by reducing I 1} (I ₂ ) max_lv times according to the reduction ratio sf is the size (vertical and / or horizontal size) of I ₁ (I ₂ ). It was decided to reduce the size until it became 5% or less. However, in this case, max_lv = 5 as shown in the following equation 1.

In the following, _{a reduced image obtained by reducing I 1} by lv (where lv is an integer satisfying 0 to max_lv) according to the reduction ratio sf is referred to as I ₁ [lv]. Further, a reduced image obtained by reducing lv times according reduction ratio sf the _{I 2} is expressed as _I 2 [lv]. That is, I ₁ = I ₁ [0] and I ₂ = I ₂ [0]. Reduction ratio of _I 1 [lv] for I ₁ s (reduction ratio s of _I 2 [lv] for _{I 2)} is expressed by the following equation 2.

That is, I ₁ [lv] is obtained by reducing I ₁ according to the reduction ratio s, and I ₂ [lv] is obtained by _{reducing I 2 according to the reduction ratio s.} In the following, as an example, it is assumed that sf = 0.5, but any value may be used as long as the value of sf is greater than 0 and less than 1. The size of I ₁ [max_lv] (I ₂ [max_lv]) should be set smaller as the movement of the motion detection target between images is larger, but the optimum setting should be made according to various factors such as the processing time. It is desirable to do. Further, in the image reduction processing, a bicubic method may be used, or a method such as the Lanczos 3-loved method may be used.

In step S403, the control unit 399 sets max_lv as the value of the variable lv. In the following, I ₁ [lv] and I ₂ [lv] are images in the hierarchy lv, I ₁ [0] and I ₂ [0] are images in the minimum hierarchy, and I ₁ [max_lv] and I ₂ [max_lv] are maximum. Sometimes referred to as an image in the hierarchy.

In step S404, the control unit 399 determines whether or not lv <max_lv. As a result of this determination, if lv <max_lv, the process proceeds to step S405, and if lv = max_lv, the process proceeds to step S408.

In step S408, the OF initialization unit 303 initializes the values of all the elements of the optical flow in the layer max_lv to 0. In the following, the optical flow in the layer lv is referred to as OF [lv]. Resolution OF [lv] is _I 1 _[lv], is the same as the resolution of I 2 [lv]. Then, the process proceeds to step S409.

On the other hand, in step S405, the OF enlargement unit 307 multiplies the value (component value of the motion vector) of each element of the recently obtained optical flow (OF [lv + 1]) by 1 / sf, and then increases the vertical and horizontal sizes of the optical flow. Generate OF'[lv + 1] magnified 1 / sf times. In the enlargement, the X component and the Y component of the motion vector are processed independently in the same manner as the R, G, and B components are processed independently in the enlargement of the RGB image. Bilinear interpolation may be used for this expansion, or another method such as a bicubic method may be adopted. The relationship between the elements u (x, y) and v (x, y) in OF [lv + 1] and the elements u'(x, y) and v'(x, y) in OF'[lv + 1] is expressed by the following equation 3 Shown in.

In step S406, the image transforming unit 305 _{selects I 2} [lv] from _{I 2} [0] to I ₂ [max_lv], and sets _{each pixel in the selected I 2} [lv] to OF'[lv + 1]. ], The image I _2w [lv] moved (warping) is generated. That is, as shown in the following equation 4, _{the pixels at the pixel positions (x, y) in I 2} [lv] are the u'(x, y), v'(x, y) in OF'[lv + 1]. _{The image I 2w} [lv] is generated by moving the motion vector in the direction defined by the motion vector by the length of the motion vector.

In step S407, the OF smoothing unit 304 applies a smoothing filter to the optical flow OF'[lv + 1] generated in step S405 to generate a smoothed optical flow OF "[lv + 1]. As the smoothing filter, for example, an average filter, a joint bilateral filter, or the like can be used. When a joint bilateral filter is used, the subject boundary is reproduced by referring to the pixel value of _{I 1 [lv].} In addition, a non-linear filter such as a median filter may be used. That is, any method may be used as long as it can smooth the optical flow OF'[lv + 1]. In the embodiment, it is assumed that the smoothing process for the optical flow OF'[lv + 1] is performed using an average filter having a filter size of 7x7.

In step S409, the energy function generation unit 306 is the _first difference, which is the difference between I 1 [lv] and I _2w [lv], and the difference between OF'[lv + 1] and OF "[lv + 1]. An energy function, which is a function based on the difference between 2 and 2, is generated. Details of the processing in step S409 will be described later.

In step S410, the OF calculation unit 308 generates an optical flow OF [lv] that minimizes the energy function generated in step S409. The details of the process in step S410 will be described later.

In step S411, the control unit 399 determines whether or not the value of the variable lv is 0. As a result of this determination, if the value of the variable lv is 0, the OF calculation unit 308 outputs the optical flow OF [0] generated in step S410 as the optical flow of the image I _{2 with reference to the image I 1.} .. The output destination of the optical flow OF [0] by the OF calculation unit 308 is not limited to a specific output destination such as a memory in the image processing device 100, an external memory, or an external device. Then, the process according to the flowchart of FIG. 4 is completed.

On the other hand, if the value of the variable lv is not 0, the process proceeds to step S412. In step S412, the control unit 399 decrements the value of the variable lv by one, and then the process proceeds to step S404.

Next, the details of the process in step S409 will be described. The method of estimating the optical flow so as to minimize the energy function is generally called the gradient method. The basis is a term called a data term, and the data term is defined by the following formula.

f is a function for obtaining the difference between _{I 1} and _{I 2w,} may be a function for obtaining the absolute value of the difference between _{I 1} and _{I 2w,} obtains the square of the difference between _{I 1} and _{I 2w} It may be a function. The energy functions of the gradient method can be classified into two main types.

The first is a type in which the sum of data terms in a certain patch range is defined as an energy function, which is defined by the following equation 6. Hereinafter, this method will be referred to as a patch-based method. In the patch-based method, the optical flow that minimizes the following energy function is calculated for each pixel.

Here, B represents a patch region centered on the pixel position (x, y). For example, when considering a 7 × 7 patch, p is from x-3 to x + 3, and q is from y-3 to y + 3. Takes an integer value up to. The advantage of this method is that the minimum optical flow can be obtained analytically when, for example, the difference square is adopted as ρ. On the other hand, the estimated optical flow often deviates from the correct answer, and it is difficult to estimate it with high accuracy.

Second, in order to solve the above problem, a smoothing term for smoothing the optical flow is added as a constraint condition. The energy function is often defined by the following equation.

Here, λ is an appropriate weighting coefficient, and ∇u and ∇v are the gradients of the optical flow. In the patch-based method, Σ is the sum within the patch area, but here it is the sum of all pixels. g is a smoothing term, and a TV norm or an L2 norm is often used. The gradient is calculated by, for example, the following formula.

In the method using the smoothing term, the optical flow of all pixels is optimized so as to minimize the energy function of the entire image as represented by Equation 7. Hereinafter, this method is referred to as an energy optimization method. While the energy optimization method can obtain an accurate optical flow, it has a problem that iterative calculation is required to perform the optimization and the amount of calculation increases.

In this embodiment, in view of the problems of the patch-based method and the energy optimization method, a pseudo-smoothing term is added to the patch-based method to incorporate the concept of the energy optimization method, and the patch-based method is used. Estimate the optical flow with almost the same amount of calculation as the method. The energy function according to this embodiment is shown in Equation 9 below.

The energy function of Equation 9 is for pixel positions (x, y). In Equation 9, the sum in the patch is not calculated for φ (), but the sum in the patch may be calculated in the same manner as in ρ (). Ρ () and φ () in Equation 9 are shown in Equation 10 below.

In Equation 10, p and q indicate the x-coordinate value and the y-coordinate value in the patch region centered on the pixel position (x, y). In step S410, du [lv] (x, y) and dv [lv] (x, y) in which E (x, y) becomes the minimum (minimum) are set in the optical flow corresponding to the _{image I 2 [lv].} , Obtained as the X and Y components of the motion vector with respect to the pixel position (x, y) in the image I _{2 [lv].}

ρ (p, q) is a motion vector defined by du [lv] (x, y) and dv [lv] (x, y) from the pixel position (p, q) in the image I _{2w [lv].} It represents the square of the difference between the pixel value of the pixel position moved by the minute and the pixel value _{at the pixel position (p, q) in the image I 1 [lv].} Note that ρ () is not limited to the square of the difference, but the absolute value of the difference, etc., is "du [lv] (x, y), dv [lv] from the pixel position (p, q) in the _{image I 2w [lv].} ] (X, y), the difference between the pixel value of the pixel position moved by the motion vector and the pixel value at the pixel position (p, q) in _{the image I 1 [lv] ”.} Expression may be applied.

In Equation 10, φ (x, y) is the sum of u'(x, y), which is the X component in OP'[lv + 1], plus du [lv] (x, y), and in OP'[lv + 1]. _{The square of the difference between u ave} (x, y), which is the X component, and v'(x, y), which is the Y component in OP'[lv + 1], plus dv [lv] (x, y). It represents the sum of and the square of the difference between _{v ave} (x, y), which is the Y component in OP ”[lv + 1]. Note that φ () is not limited to the sum of squares of differences, and may be, for example, the sum of the absolute value of the former difference and the absolute value of the latter difference.

By adding the term φ () to the energy function, u _ave (x, y) and v _ave (x, y) are smoother than the original optical flow, resulting in suppressed outliers. Estimated values are calculated so that the values of u'and u _ave do not deviate, and this term serves as a smoothing term. This also applies to v.

When λ = 0 in the above equation 9, it is reduced to the hierarchical Lucas-Kanade method. Here, assuming that the above du and dv are small, the Taylor expansion of ρ gives the following equation 11.

_{Here, I 2Xw} are those calculated by applying 1 Tsugihen differential image in the x direction of the image _{I 2w} instead of _{I 2} in Formula 4, the _{I 2Xw} instead of _{I 2w} in Equation 4. _{Similarly, I 2Yw} are those calculated by applying 1 Tsugihen differential image in the y direction of the image _{I 2w} instead of _{I 2} in Formula 4, the _{I 2Yw} instead of _{I 2w} in Equation 4. The first partial differential of the image I can be obtained, for example, by the following equation 12.

In addition to that, a horizontal or vertical Sobel filter or the like may be applied to obtain the result. The analytical solutions du and dv to be obtained satisfy the following simultaneous equations. Since equations 14 and 15 do not depend on the hierarchy, the hierarchy notation is omitted.

By multiplying both sides of Equation 13 by the inverse matrix of A, du and dv can be obtained. As described above, according to the present embodiment, the amount of calculation is increased by minimizing the energy so that the difference between the smoothed result of the optical flow of the previous layer and the calculated optical flow becomes small. The accuracy can be improved without causing the problem.

[Second Embodiment]
In the following, the differences from the first embodiment will be mainly described, and unless otherwise specified below, the same as the first embodiment. In the first embodiment, the optical flow used for the energy function is the optical flow in the layer (lv + 1) one level higher than the current layer lv. On the other hand, in the present embodiment, the optical flow obtained for the image of the frame immediately before the current frame is used for the energy function. In the following, an example will be described in which the optical flow for _{the image I 2 of the current frame is obtained by using the optical flow obtained for the image I 1} one frame before the frame.

An example of the functional configuration of the image processing apparatus according to the present embodiment and the processing _{performed by the image processing apparatus 100 for obtaining the optical flow for the image I 2} will be described with reference to the block diagram of FIG. 5 and the flowchart of FIG. In FIG. 5, the same functional unit as the functional unit shown in FIG. 3 is assigned the same reference number, and the description relating to the functional unit will be omitted. Further, in the flowchart of FIG. 6, the same processing step as the processing step shown in FIG. 4 is assigned the same step number, and the description relating to the processing step will be omitted. The process according to the flowchart shown in FIG. 6 is a process for obtaining an optical flow for one image. However, for example, when obtaining an optical flow for each of a plurality of images, processing may be performed for each of the plurality of images according to the flowchart of FIG.

In step S601, the OF deformer 501 converts the previously obtained optical flow for _{image I 1} into a reference optical flow used for the energy function to generate the optical flow for _{image I 2.} Various methods can be considered for this conversion method.

For example, the optical flow obtained for the image I ₁ is the optical flow of the image I _{1 with} respect to _{the image I 0} one frame before the _{image I 1} , and the element of the optical flow represents the motion vector from the _{image I 0.} ing. Here, when the time interval between frames is sufficiently short, the movement of the object in the image can be regarded as a constant velocity linear motion. Therefore, _{each element of the optical flow obtained for the image I 1} is the movement indicated by the element of the optical flow. The one moved according to the vector can be used as the above reference optical flow. Due to this movement, there may be an element in the reference optical flow in which the motion vector is not stored. Therefore, such an element is filled in from the surrounding motion vector by filtering or the like.

In the case where the optical flow of the image I ₀ relative to the image I ₁ has been obtained, those in which the sign of the elements of this optical flow in the opposite may be the above references optical flow.

Reference The process for obtaining the optical flow will be described by taking FIG. 7 as an example. Images 701 to 703 are images I _{0 to} I ₂ , respectively, and each image includes a person 203 and a house 204.

By the amount motion vector obtained by the movement of the motion vector 713 a-out animal vector 713 of a person 203 in the image _{I 1} on the basis of the image _{I 0,} as the motion vector 707 of a person 203 in the image _{I 2} relative to the image _{I 1} Ask. If the motion vector 705 of the person 203 in the image I ₀ with respect to the image I ₁ is obtained, the motion vector 707 may be an inverted version of the motion vector 705. The motion vector obtained by moving the motion vector 704 of the house 204 in the image I ₁ with respect to the image I ₀ by the amount of the motion vector 704 is used as the motion vector 708 of the house 204 in the image I _{2 with respect to the image I 1.} Ask. If the motion vector 706 of the house 204 in the image I ₀ with respect to the image I ₁ is obtained, the motion vector 708 may be an inverted version of the motion vector 706. The motion vectors 707 and 708 obtained in this way serve as the above-mentioned reference optical flow.

Returning to FIG. 6, next, in step S602, the OF smoothing unit 304 performs a smoothing process on the optical flow described in the first embodiment with respect to the reference optical flow generated in step S601.

In step S603, the OF reduction unit 502 multiplies the value of each element of the reference optical flow smoothed in step S602 by sf ^lv , and then reduces the vertical and horizontal size of the reference optical flow to sf ^{lv times.} Generate a flow.

After that, the u (x, y) and v (x, y) of the optical flow generated in step S603 are used as u _ave (x, y) and v _ave (x, y) to construct an energy function. Other than that, it is the same as that of the first embodiment. In the flowchart of FIG. 6, u (x, y) and v (x, y) of the optical flow generated in step S603 are set to u _ave (x, y) and v _ave (x, y) for all layers. To construct an energy function. However, for a specific layer, for example, a layer other than the final layer, the energy function is configured in the same manner as in the first embodiment, and for the final layer, u (x, y) of the optical flow generated in step S603. , V (x, y) may be used as u _ave (x, y), v _ave (x, y) to construct an energy function.

As in the first embodiment, the result of smoothing the optical flow of the previous layer may be added to the energy function. Each of u (x, y) and v (x, y) of the optical flow generated in step S603 is u _ave1 (x, y), _vave1 (x, y), the X component in OP "[lv + 1], Y. _Assuming that each of the components is uave2 (x, y) and _vave2 (x, y), the energy function is as follows.

In Equation 16, _{the sum in the patch is not calculated for φ 1} () and φ ₂ (), but the sum in the patch may be calculated in the same manner as in ρ (). According to this embodiment, it is possible to calculate the optical flow with high accuracy while suppressing the amount of calculation while considering the temporal continuity of the optical flow. Note that all the steps shown in FIGS. 4 and 6 are not limited to being executed in order from the top as described above, and the order may be changed in some processing steps, or some processing steps may be performed in parallel. You may execute it.

[Third Embodiment]
The optical flow generated by the optical flow generation process described in the first and second embodiments can be worn for various purposes. By calculating the optical flow, it is possible to identify a moving subject and estimate the direction in which the camera is moving. This makes it possible to apply it to various applications such as tracking a subject and vibration isolation of a moving image. It is also possible to add a video effect to the captured image or moving image. For example, by adding a blur in the direction of the optical flow to the captured image, it is possible to generate a dynamic image that emphasizes a moving subject. In the following, vibration isolation of moving images and a case of applying motion-based blur to a specific frame will be described.

An example of a functional configuration of an image processing device that uses an optical flow for vibration isolation of moving images will be described with reference to the block diagram of FIG. The image processing device 800 of FIG. 8 may be a device housed in the above-mentioned image processing device 100.

The OF data acquisition unit 801 acquires the optical flow generated and output by the image processing apparatus 100. The method of acquiring the optical flow by the OF data acquisition unit 801 is not limited to a specific acquisition method. For example, the optical flow may be acquired from the image processing device 100 via a wireless or wired network, or a network formed by a combination of wired and wireless, or the optical flow stored in an external storage device may be acquired. good.

The calculation unit 802 calculates the global motion using the optical flow acquired by the OF data acquisition unit 801. Global motion is the most dominant direction of movement for the entire image and is represented by a single vector. Global motion can be calculated, for example, by generating a histogram of optical flow and acquiring the mode. If the movement of the entire image can be calculated, another method may be used for calculation.

The smoothing portion 803 removes high frequency components in the time direction of the global motion. This is to eliminate the vibration of the moving image in the time direction. For example, it can be realized by Fourier transforming in the time direction to remove high frequencies or by operating a smoothing filter in the time direction.

The vibration isolation unit 804 electronically shifts and aligns the image at the corresponding time among the images of each frame acquired by the image data acquisition unit 805 based on the global motion at each time.

Next, an example of a functional configuration of an image processing device that imparts blur based on movement will be described with reference to the block diagram of FIG. The image processing device 900 of FIG. 9 may be a device housed in the above-mentioned image processing device 100. In FIG. 9, the same functional unit as in FIG. 8 is assigned the same reference number, and the description of the functional unit will be omitted. In the following, the image to be processed will be described as image 1.

When k = 1 to n-1, the image deformation unit 901 uses a motion vector obtained by multiplying each element (component of the motion vector) in the optical flow acquired by the OF data acquisition unit 801 by k / n. A shift image in which the image 1 is shifted according to 4 is generated. For example, assuming that n = 10, shift images for n-1 images are generated for k = 1-9. The image synthesizing unit 902 generates a composite image obtained by synthesizing n-1 deformed images and image 1 for each pixel, and divides the pixel value of each pixel of the composite image by n to add blur. Generate an image. The larger the movement, the larger the optical flow vector, and for a stationary subject, the optical flow vector becomes 0. Therefore, the larger the movement, the more blurred the image is generated. In the present embodiment, a fixed value is used as n, but it may be determined from the maximum value of the length of the optical flow in the image. For example, if the maximum value of the optical flow length is 50 pix, n = 50. If the user can specify the intensity of the blur, the optical flow may be rescaled according to the intensity and the same processing may be performed. For example, if you want to increase the effect of blurring, you can multiply the original optical flow by several times. According to the present embodiment, by using the optical flow, it is possible to increase the speed and accuracy of the camera function and to add a video effect. Further, in the case of images taken at the same time by different imaging devices, it is possible to calculate the depth of the subject from the optical flow.

[Fourth Embodiment]
Each of the functional units constituting the image processing apparatus 100 shown in FIGS. 3 and 5 may be implemented by hardware, but may also be implemented by software (computer program). In the latter case, a computer device having a processor capable of executing this computer program can be applied to the image processing device 100 described above. An example of a hardware configuration of a computer device applicable to the image processing device 100 will be described with reference to the block diagram of FIG.

The CPU 101 executes various processes using computer programs and data stored in the RAM 102 and the ROM 103. As a result, the CPU 101 controls the operation of the entire computer device, and executes or controls each of the above-described processes as performed by the image processing device 100.

The RAM 102 has an area for storing computer programs and data loaded from the ROM 103 and the storage unit 104. Further, the RAM 102 has a work area used by the CPU 101 when executing various processes. As described above, the RAM 102 can appropriately provide various areas. The ROM 103 stores setting data and a boot program that do not need to be rewritten.

The storage unit 104 is a large-capacity information storage device typified by a hard disk drive device. The storage unit 104 stores an OS (operating system) and computer programs and data for causing the CPU 101 to execute each of the above-described processes as performed by the image processing device 100. The computer program stored in the storage unit 104 includes a computer program for causing the CPU 101 to execute the functions of the functional units shown in FIGS. 3 and 5. Further, the data stored in the storage unit 104 includes the data described as known information in the above description, and the data of the image or moving image to be processed. The computer programs and data stored in the storage unit 104 are appropriately loaded into the RAM 102 according to the control by the CPU 101, and are processed by the CPU 101.

In addition to the hard disk drive device, the storage unit 104 can also be applied to a device that reads information from a storage medium such as a CD-ROM or a DVD-ROM, or a memory device such as a flash memory or a USB memory.

A display device 109 is connected to the output interface 106. The display device 109 is composed of a CRT, a liquid crystal screen, a projector device, and the like, and can display or project the processing result by the CPU 101 with images, characters, and the like.

The CPU 101, RAM 102, ROM 103, storage unit 104, and output interface 106 are all connected to the bus 107. The configuration shown in FIG. 1 is only an example of the configuration of a computer device applicable to the image processing device 100.

The same applies to the functional units of the image processing devices 800 and 900 shown in FIGS. 8 and 9, and all of them may be implemented by hardware, but may be implemented by software (computer program). In the latter case, a computer device having a processor capable of executing this computer program functions as the above-mentioned image processing devices 800, 900, and it goes without saying that the configuration shown in FIG. 1 can be applied to this computer device. Nor. Further, when the image processing device 800 and the image processing device 900 are housed in the image processing device 100, the computer device of FIG. 1 also realizes the functions of the image processing device 800 and the image processing device 900.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

301: Image data acquisition unit 302: Image reduction unit 303: OF initialization unit 304: OF smoothing unit 305: Image deformation unit 306: Energy function generation unit 307: OF enlargement unit 308: OF calculation unit

Claims (11)

A first set having a first image and a plurality of reduced images recursively reduced at a specified reduction rate as elements, and a second image and the second image reduced by the specified reduction. A second set whose elements are a plurality of reduced images recursively reduced by a rate, an acquisition means for acquiring, and an acquisition means.
A selection means for selecting images belonging to the second set in ascending order of image size, and
A moved selection image in which each pixel of the selected image selected by the selection means is moved according to a converted optical flow in which the optical flow corresponding to the image previously selected by the selection means is converted according to the size of the selected image. And the generation means to generate
Among the images belonging to the first set, the first difference, which is the difference between the image having the same size as the selected image and the moved selected image, and smoothing with respect to the converted optical flow and the converted optical flow. A calculation means for obtaining the second difference, which is the difference from the processed optical flow that has undergone the conversion process, and the optical flow that minimizes the evaluation value based on the optical flow, as the optical flow corresponding to the selected image.
An image processing apparatus including an output means for outputting an optical flow corresponding to the second image, which is obtained by the calculation means. The converted optical flow obtained by converting the optical flow corresponding to the image previously selected by the selection means according to the size of the selected image is a motion vector which is an element of the optical flow corresponding to the image previously selected by the selection means. The image processing apparatus according to claim 1, wherein the component value and the size of the optical flow corresponding to the image selected last time by the selection means are converted according to the size of the selected image. A first set having a first image and a plurality of reduced images recursively reduced at a specified reduction rate as elements, and a second image and the second image reduced by the specified reduction. A second set whose elements are a plurality of reduced images recursively reduced by a rate, an acquisition means for acquiring, and an acquisition means.
A selection means for selecting images belonging to the second set in ascending order of image size, and
A moved selection image in which each pixel of the selected image selected by the selection means is moved according to a converted optical flow in which the optical flow corresponding to the image previously selected by the selection means is converted according to the size of the selected image. And the generation means to generate
Among the images belonging to the first set, the first difference, which is the difference between the image having the same size as the selected image and the moved selected image, and the converted optical flow and the optical flow for the first image are shown. The second difference, which is the difference from the processed optical flow that has been converted according to the size of the selected image and then smoothed, and the optical flow that minimizes the evaluation value based on the second difference correspond to the selected image. The calculation method to be calculated as the optical flow to be performed,
An image processing apparatus including an output means for outputting an optical flow corresponding to the second image, which is obtained by the calculation means. The image processing apparatus according to any one of claims 1 to 3, wherein any one of an average filter, a joint bilateral filter, and a median filter is used for the smoothing process. In addition
Any one of claims 1 to 4, wherein the global motion in the image is obtained by using the optical flow output by the output means, and the vibration isolating means for shifting the image based on the obtained global motion is provided. The image processing apparatus according to the section. In addition
A means for generating a plurality of optical flows from the optical flows output by the output means, generating a plurality of shift images obtained by shifting the images using the plurality of optical flows, and synthesizing the images and the plurality of shift images. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus is characterized by the above. Any one of claims 1 to 6, wherein each of the first image and the second image is an image captured at the same time by a plurality of image pickup devices or at different times from each other. The image processing apparatus according to. The image according to any one of claims 1 to 6, wherein each of the first image and the second image is an image captured at different times by a single image pickup apparatus. Processing equipment. This is an image processing method performed by an image processing device.
The acquisition means of the image processing apparatus includes a first set including a first image and a plurality of reduced images in which the first image is recursively reduced at a predetermined reduction rate, a second image, and the like. An acquisition step of acquiring a second set whose elements are a plurality of reduced images obtained by recursively reducing the second image at the specified reduction ratio.
A selection step in which the selection means of the image processing device selects images belonging to the second set in ascending order of image size.
The generation means of the image processing apparatus has converted each pixel of the selected image selected this time in the selection step into an optical flow corresponding to the image previously selected in the selection step according to the size of the selected image. A generation process that generates a moved selected image that has been moved according to the flow,
The calculation means of the image processing apparatus includes a first difference, which is a difference between an image having the same size as the selected image and the moved selected image among the images belonging to the first set, and the converted optical flow. The second difference, which is the difference between the converted optical flow and the processed optical flow that has been smoothed, and the optical flow that minimizes the evaluation value based on the second difference, are used as the optical flow corresponding to the selected image. The required calculation process and
An image processing method characterized in that the output means of the image processing apparatus includes an output step of outputting an optical flow corresponding to the second image obtained in the calculation step. This is an image processing method performed by an image processing device.
The acquisition means of the image processing apparatus includes a first set including a first image and a plurality of reduced images in which the first image is recursively reduced at a predetermined reduction rate, a second image, and the like. An acquisition step of acquiring a second set whose elements are a plurality of reduced images obtained by recursively reducing the second image at the specified reduction ratio.
A selection step in which the selection means of the image processing device selects images belonging to the second set in ascending order of image size.
The generation means of the image processing apparatus has converted each pixel of the selected image selected this time in the selection step into an optical flow corresponding to the image previously selected in the selection step according to the size of the selected image. A generation process that generates a moved selected image that has been moved according to the flow,
The calculation means of the image processing apparatus includes a first difference, which is a difference between an image having the same size as the selected image and the moved selected image among the images belonging to the first set, and the converted optical flow. The evaluation value based on the second difference, which is the difference between the optical flow for the first image and the processed optical flow that has been subjected to the smoothing process after being converted according to the size of the selected image, is minimized. A calculation process for obtaining the optical flow as an optical flow corresponding to the selected image, and
An image processing method characterized in that the output means of the image processing apparatus includes an output step of outputting an optical flow corresponding to the second image obtained in the calculation step. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

Images