CN116486107B - Optical flow calculation method, system, equipment and medium - Google Patents

Abstract

The invention discloses an optical flow calculation method, system, equipment and medium, and relates to the field of optical flow processing. The method includes: acquiring a target image; the target image includes: two consecutive frames of images, respectively the first image and the second image; using a motion feature extraction network to extract the motion features of the target image; extracting the motion features and features of the target image The number of channels, determine the feature map of the first image and the feature map of the second image, and calculate the matching cost volume of the two feature maps; use the context encoder to extract the context features of the first image; based on the matching cost volume and context features, use The global-local cyclic optical flow decoder performs loop iterative solution to obtain the optical flow field of the target image; the global-local cyclic optical flow decoder is based on a depth-separable residual block, a multi-layer perceptron block, and a depth-separable convolution module built with a multi-head attention module. The invention can improve the accuracy and robustness of optical flow estimation.

Description

Optical flow calculation method, system, device and medium

technical field

The present invention relates to the field of optical flow processing, in particular to an optical flow calculation method, system, equipment and medium.

Background technique

Optical flow refers to the two-dimensional motion vector of moving objects and pixels on the surface of the scene in the image sequence, which not only provides the motion vectors of the objects and scenes in the image, but also carries rich shape and structure information. Therefore, optical flow estimation is a research hotspot in the field of image processing and computer vision. As a fundamental building block, it provides valuable motion correlation cues in many high-level vision tasks, such as action recognition, video interpolation, video segmentation, and object tracking.

In recent years, with the rise of deep learning, the optical flow estimation model based on Convolutional Neural Network (CNN) has achieved great success. This type of method first utilizes data-driven learning-based optimization strategies, and uses a modeled feature encoder to extract image features. Then calculate the similarity of all feature vectors between the feature maps, and regard the pair of feature vectors with the highest similarity as matching points, and finally decode the displacement field between consecutive frames of images. Since the encoding and decoding process requires that the features have sufficient resolution to reduce the matching error caused by large displacement motion and local ambiguity (occlusion, weak texture, illumination changes, etc.), how to efficiently and accurately decode motion features has become an improvement. The key to the accuracy and robustness of optical flow estimation. However, the existing deep learning optical flow calculation models usually use local convolution operations with limited receptive fields for optical flow decoding, which leads to insufficient ability of the model to extract and express image features, which in turn affects the overall performance of optical flow estimation.

Contents of the invention

Based on this, embodiments of the present invention provide an optical flow calculation method, system, device, and medium, so as to improve the accuracy and robustness of optical flow estimation.

In order to achieve the above purpose, the embodiment of the present invention provides the following solutions:

A method for calculating optical flow, comprising:

Acquiring a target image; the target image includes: a first image and a second image; the first image and the second image are two consecutive frames of images;

Using a motion feature extraction network to extract the motion features of the target image; the motion feature extraction network includes a plurality of convolutional layers of different sizes;

Determine the feature map of the first image and the feature map of the second image according to the motion feature of the target image and the number of feature extraction channels, and calculate the feature map of the first image and the feature map of the second image The matching cost volume of the feature map;

Using a context encoder to extract the context feature of the first image; the structure of the context encoder is the same as that of the motion feature extraction network;

Based on the matching cost volume and the context feature, a global-local cyclic optical flow decoder is used to iteratively solve the loop to obtain the optical flow field of the target image;

Wherein, the global-local cyclic optical flow decoder includes: a local motion information encoder, a global motion information encoder and a global-local motion information decoder connected in sequence; the output of the global-local motion information decoder is connected an input to the local motion information encoder;

Both the local motion information encoder and the global-local motion information decoder include: sequentially connected depth separable residual blocks and multi-layer perceptron blocks; the global motion information encoder includes: sequentially connected depth Separable convolution module and multi-head attention module;

The local motion information encoder is used to encode according to the matching cost volume and the residual optical flow of the last iteration to obtain local motion features;

The global motion information encoder is used to encode according to the local motion feature and the context feature to obtain global motion information;

The global-local motion information decoder is used to decode according to the local motion feature, the global motion information and the context feature to obtain the residual optical flow of the current iteration; the residual optical flow of the last iteration is used to determine The optical flow field of the target image.

Optionally, the motion feature extraction network specifically includes: a first convolutional layer, a convolutional residual block, and a second convolutional layer connected in sequence;

The convolution kernel size of the first convolutional layer is 7×7; the convolutional residual block includes: the third convolutional layer and the fourth convolutional layer connected in sequence; the second convolutional layer The size of the convolution kernel is 1×1; the size of the convolution kernel of the third convolution layer is 3×3, and the step size is 2; the size of the convolution kernel of the fourth convolution layer is 3×3 , with a step size of 1.

Optionally, according to the motion feature of the target image and the number of feature extraction channels, determine the feature map of the first image and the feature map of the second image, and calculate the feature map of the first image and the The matching cost volume of the feature map of the second image, specifically including:

Determining the first half of the motion features of the target image as the feature map of the first image, and determining the second half of the motion features of the target image as the feature map of the second image;

performing a dot product similarity operation on the feature map of the first image and the feature map of the second image to obtain matching cost information of the feature map of the first image and the feature map of the second image;

The matching cost information is down-sampled by using a pooling operation to obtain a matching cost volume of the feature map of the first image and the feature map of the second image.

Optionally, the depth-separable residual block specifically includes: a first depth-separable convolution layer, a first activation function, a second depth-separable convolution layer, and a second activation function connected in sequence;

The convolution kernel size of the first depth-separable convolution layer is 7×7; the connection mode of the second depth-separable convolution layer is dense connection and the convolution kernel size is 15×15; the first Both the activation function and the second activation function are GELU activation functions.

Optionally, the multilayer perceptron block specifically includes: a fifth convolutional layer, a third depthwise separable convolutional layer, a third activation function, and a sixth convolutional layer connected in sequence;

The convolution kernel size of the fifth convolution layer and the sixth convolution layer is 1×1; the convolution kernel size of the third depth separable convolution layer is 3×3; the third activation The function is the GELU activation function.

The present invention also provides an optical flow computing system, including:

An image acquisition module, configured to acquire a target image; the target image includes: a first image and a second image; the first image and the second image are continuous two-frame images;

The motion feature extraction module is used to extract the motion features of the target image using a motion feature extraction network; the motion feature extraction network includes a plurality of convolutional layers of different sizes;

A matching cost calculation module, configured to determine the feature map of the first image and the feature map of the second image according to the motion feature of the target image and the number of feature extraction channels, and calculate the feature map of the first image a matching cost volume with the feature map of the second image;

The context feature extraction module is used to extract the context feature of the first image by using a context encoder; the structure of the context encoder is the same as that of the motion feature extraction network;

The optical flow field solving module is used to obtain the optical flow field of the target image by using a global-local loop optical flow decoder to iteratively solve based on the matching cost volume and the context feature;

Wherein, the global-local cyclic optical flow decoder includes: a local motion information encoder, a global motion information encoder and a global-local motion information decoder connected in sequence; the output of the global-local motion information decoder is connected an input to the local motion information encoder;

Both the local motion information encoder and the global-local motion information decoder include: sequentially connected depth separable residual blocks and multi-layer perceptron blocks; the global motion information encoder includes: sequentially connected depth Separable convolution module and multi-head attention module;

The local motion information encoder is used to encode according to the matching cost volume and the residual optical flow of the last iteration to obtain local motion features;

The global motion information encoder is used to encode according to the local motion feature and the context feature to obtain global motion information;

The global-local motion information decoder is used to decode according to the local motion feature, the global motion information and the context feature to obtain the residual optical flow of the current iteration; the residual optical flow of the last iteration is used to determine The optical flow field of the target image.

The present invention also provides an electronic device, including a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above optical flow calculation method.

The present invention also provides a computer-readable storage medium, which stores a computer program, and implements the above optical flow calculation method when the computer program is executed by a processor.

According to the specific embodiments provided by the invention, the invention discloses the following technical effects:

The embodiment of the present invention aims at the problem of insufficient feature extraction ability in the existing optical flow estimation model, introduces a depth separable residual block and a multi-layer perceptron block to increase the receptive field, and utilizes the local characteristics of the depth local motion information encoder and the global characteristics of the global motion information encoder to construct a global-local cyclic optical flow decoder. Accuracy and robustness of optical flow estimation for regions.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some of the present invention. Embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without paying creative labor.

FIG. 1 is a flowchart of an optical flow calculation method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of a first image provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a second image provided by an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a global-local loop optical flow decoder provided by an embodiment of the present invention;

Fig. 5 is an optical flow field visualization image provided by an embodiment of the present invention;

FIG. 6 is a structural diagram of an optical flow computing system provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Embodiment one

Referring to Fig. 1, the optical flow calculation method of this embodiment includes:

Step 101: Acquire a target image; the target image includes: a first image and a second image; the first image and the second image are two consecutive frames of images.

In this embodiment, the 30th and 31st consecutive frames of images in the bamboo_3 image sequence are selected, and the two consecutive frames of images are input, wherein the 30th frame of images is used as the first image I ₁ , as shown in FIG. 2 shown; the thirty-first frame image is used as the second image I ₂ , as shown in FIG. 3 .

Step 102: Using a motion feature extraction network to extract motion features of the target image; the motion feature extraction network includes multiple convolutional layers of different sizes.

Specifically, in this step, first construct a motion feature extraction network: combine multiple stacked continuous convolutions to perform motion feature extraction on the first image I ₁ and the second image I ₂ , and the input of the motion feature extraction network is the first a stacked image F after image I ₁ and second image I ₂ are stacked, , H represents the height of the input image, W represents the width of the input image, and the output is the motion feature F _M of two consecutive frames of images.

The above motion feature extraction network specifically includes: a first convolutional layer, a convolutional residual block and a second convolutional layer connected in sequence. The convolution kernel size of the first convolutional layer is 7×7; the convolutional residual block includes: the third convolutional layer and the fourth convolutional layer connected in sequence; the second convolutional layer The size of the convolution kernel is 1×1; the size of the convolution kernel of the third convolution layer is 3×3, and the step size is 2; the size of the convolution kernel of the fourth convolution layer is 3×3 , with a step size of 1.

The motion feature extraction network is divided into 3 stages (Stage), namely Stage 1, Stage 2 and Stage 3, Stage 1 is 1/2 resolution, Stage 2 is 1/4 resolution, and Stage 3 is 1/8 resolution Rate. First, pass the 7×7 first convolutional layer of Stage 1 to control downsampling and feature extraction, and then pass through two 3×3 convolutional residual blocks stacked consecutively in Stage 2 and Stage 3, each passing through a Stage It means that the image is down-sampled twice, and finally, the number of channels is adjusted through a 1×1 second convolutional layer, and the motion feature F _M of two consecutive frames of images is output. The specific calculation formula is as follows:

;

The above formula represents the process of feature extraction by the motion feature extraction network. Conv _7×7 ( ) and Conv _1×1 ( ) represent the pair of the first convolutional layer of 7×7 and the second convolutional layer of 1×1 respectively. Image feature extraction; ConvBlock _3×3 (·) means using a third convolutional layer with a step size of 2 and a convolution kernel size of 3×3 and a step size of 1 and a convolution kernel size of 3×3 The convolutional residual block composed of the fourth convolutional layer extracts features from the image.

in,

;

;

f ₁ represents the output result after the third convolutional layer with a step size of 2 and a convolution kernel size of 3×3 performs downsampling and feature extraction on the input image x1, and then passes the residual connection and activation function relu; f Indicates the output result after the third convolution layer with a step size of 1 and a convolution kernel size of 3×3 performs downsampling and feature extraction on the input f ₁ , and then passes the residual connection and the activation function relu.

Step 103: Determine the feature map of the first image and the feature map of the second image according to the motion feature of the target image and the number of feature extraction channels, and calculate the feature map of the first image and the feature map of the second image Matching cost volume for feature maps of two images.

This step specifically includes:

(1) Divide the motion feature _FM into two according to the number of feature extraction channels. The first half of the motion features in the motion feature _FM are determined as the feature map F ₁ of the first image, and the second half of the motion features in the motion feature _FM are determined as The feature map F ₂ of the second image.

(2) Perform a dot product similarity operation on the feature map of the first image and the feature vector on the feature map of the second image, so that the matching cost information between all relevant point pairs on the two feature maps can be obtained , that is, the matching cost information of the feature map of the first image and the feature map of the second image is obtained.

(3) Downsampling the matching cost information by using a pooling operation, thereby converting the matching cost information of a large displacement into matching cost information of a small displacement, and obtaining the feature map of the first image and the feature map of the second image The matching cost volume of the feature map. The matching cost volume represents the form of the multi-scale matching cost pyramid. The calculation formula is as follows:

;

Among them, Cost represents matching cost information; Represents a matrix multiplication operation; AvgPool represents an average pooling operation; Indicates the layer number of the multi-scale matching cost pyramid; /> Indicates that the first multi-scale matching cost pyramid is obtained after downsampling the matching cost information The matching cost volume for the layer. when /> When a change occurs, the size of each feature map in Cost will change, and the size change will be determined by the step size of the average pooling operation. In this embodiment, the matching cost volume of each layer is obtained, so as to better estimate the optical flow when a large displacement or a small displacement occurs.

Step 104: Using a context encoder to extract context features of the first image; the structure of the context encoder is the same as that of the motion feature extraction network.

The context encoder in this step and the motion feature extraction network in step 102 have a parallel structure. The context encoder combines multiple stacked continuous convolutions to perform context feature extraction on the first image I ₁ in the image sequence, the input of the context encoder is the first image I ₁ , , the output is the context feature F _C of the first image.

Specifically, the context encoder is divided into three stages (Stage), namely Stage 1, Stage 2 and Stage 3, where Stage 1 is 1/2 resolution, Stage 2 is 1/4 resolution, and Stage 3 is 1/2 resolution. 8 resolutions. First, pass the 7×7 first convolutional layer of Stage 1 to control downsampling and extract features, and then pass through two 3×3 convolutional residual blocks stacked consecutively in Stage 2 and Stage 3, each passing a Stage means to downsample the image twice, and finally, adjust the number of channels through a 1×1 second convolutional layer, and output the context feature F _C of the first image. Calculated as follows:

.

Step 105: Based on the matching cost volume and the context feature, a global-local loop optical flow decoder is used to iteratively solve the loop to obtain the optical flow field of the target image.

Referring to Fig. 4, the global-local cyclic optical flow decoder includes: a sequentially connected local motion information encoder, a global motion information encoder and a global-local motion information decoder; the global-local motion information decoder The output is connected to the input of the local motion information encoder.

Each part of the above-mentioned global-local loop optical flow decoder will be further described in detail below in conjunction with FIG. 4 .

(1) Local motion information encoder.

The local motion information encoder includes: sequentially connected depthwise separable residual blocks and multilayer perceptron blocks (Multilayer Perceptron, MLP); the global motion information encoder includes: sequentially connected depthwise separable convolution modules and a multi-head attention module.

The depth-separable residual block specifically includes: a first depth-separable convolution layer, a first activation function, a second depth-separable convolution layer, and a second activation function connected in sequence. The convolution kernel size of the first depth-separable convolution layer is 7×7; the connection mode of the second depth-separable convolution layer is dense connection and the convolution kernel size is 15×15; the first Both the activation function and the second activation function are GELU activation functions. In this embodiment, the second depth separable convolution layer specifically includes: the seventh convolution layer with a convolution kernel size of 1×1, the eighth convolution layer with a convolution kernel size of 15×15, and convolution The ninth convolutional layer with a kernel size of 1×1, in which the GELU activation function is used to activate the feature after each layer of convolution operation, and the residual connection operation is performed on the feature after each layer of convolution operation.

The multi-layer perceptron block specifically includes: a fifth convolutional layer, a third depthwise separable convolutional layer, a third activation function, and a sixth convolutional layer connected in sequence. The convolution kernel size of the fifth convolution layer and the sixth convolution layer is 1×1; the convolution kernel size of the third depth separable convolution layer is 3×3; the third activation The function is the GELU activation function.

The local motion information encoder is used for encoding according to the matching cost volume and the residual optical flow of the last iteration to obtain local motion features. Specifically, the input of the local motion information encoder is the motion feature composed of the motion information in the matching cost volume and the initial optical flow field, these features are input into the local motion information encoder, and the circular iterative encoding of the motion feature is performed, and finally The output is the local motion feature of this iteration ,Calculated as follows:

;

in, Represents the residual optical flow of each iteration, DLP( ) represents a depth-separable MLP block; Cat( ) represents the operation of connecting feature maps, that is, splicing multiple feature maps with the same resolution according to the channel dimension; /> Indicates that a 1×1 convolutional layer is used to input /> Perform feature extraction; F _f represents the feature extracted from the optical flow of this iteration; F _cost represents the local motion feature indexed from the matching cost volume through the optical flow of this iteration, which is obtained according to the matching cost volume; F _m represents the local motion feature enhanced by the DLP block; /> Indicates the local motion features of this iteration. RDSCBlocks( ) denotes depthwise separable residual blocks; MLP( ) denotes multilayer perceptron blocks implemented by convolution; DwConv _7×7 (x2) denotes the first depthwise separable The separable convolutional layer performs feature extraction on the input image x2; DwConv _15×15,d ( ) indicates that the second depth separable convolutional layer with a convolution kernel size of 15×15 is used to extract features from the image; GELU(· ) indicates the GELU activation function; Conv _1×1 (x3) indicates that the fifth convolutional layer with a convolution kernel size of 1×1 is used to extract features from the input image x3; DwConv _3×3 ( ) indicates that the convolution kernel is used The third depthwise separable convolutional layer of size 3×3 performs feature extraction on the image.

(2) Global motion information encoder.

The global motion information encoder includes: a sequentially connected depthwise separable convolution module and a multi-head attention module.

The global motion information encoder is configured to perform encoding according to the local motion feature and the context feature to obtain global motion information. Specifically, by constructing a depthwise separable convolution module and a multi-head attention module with local position encoding, the local motion features output by the local motion information encoder and the context feature F _C output by the context feature encoder are input to the global motion information encoder, and finally output the global motion information F _g , the calculation formula is as follows:

;

The global motion information encoder encodes the context feature F _C to obtain the query vector q _c and the key vector k _c . i represents the abscissa of a certain point in the query vector, and j represents the ordinate of a certain point in the query vector. u represents the abscissa of a certain point in the key vector, and v represents the ordinate of a certain point in the key vector. q _c (i, j) represents the eigenvalue of a certain point in the query vector, and k _c (u, v) represents the eigenvalue of a certain point in the key vector. v _m represents the value vector constructed according to the local motion feature F _l ; v _m (u, v) represents the feature value of a certain point in the value vector. γ is a learnable factor; f(·) denotes the dot-product attention function. F _l (i, j) represents the feature value of a certain point in the local motion feature.

(3) Global-local motion information decoder.

The structure of the global-local motion information decoder is the same as that of the local motion information encoder, and will not be repeated here.

The global-local motion information decoder is used to decode according to the local motion feature, the global motion information and the context feature to obtain the residual optical flow of the current iteration; the residual optical flow of the last iteration is used to determine The optical flow field of the target image.

Specifically, the input of the global-local motion information decoder is the local motion feature output by the local motion information encoder and the global and local motion information aggregated by another part of the hidden state features output by the global motion information encoder and the context feature encoder , the output is the residual optical flow of this iteration ,Calculated as follows:

;

F _a represents the aggregated feature aggregated from global motion information, local motion features and contextual features; Indicates the global-local motion feature of this iteration; /> Represents the global-local motion features of the previous iteration.

The global-local cyclic optical flow decoder in this embodiment optimizes the optical flow field cyclically and iteratively for a total of n iterations, and upsamples the optical flow optimized in the last iteration to the same resolution as the input image, so as to obtain the final Optical flow field. Among them, in the first iteration, the initial optical flow field f is set to 0, and the residual optical flow is 0, through iterative optical flow, the optical flow field after each iteration is updated /> , the final optical flow field can be determined by calculating the residual optical flow of the last iteration. The final optical flow field visualization image is shown in Figure 5.

This embodiment utilizes the local modeling capability of the depthwise separable convolutional residual block with a large convolution kernel and the long-distance modeling capability with a local position encoding Transformer to improve the ability to capture motion features and capture more The motion relationship between pixels optimizes the accuracy of optical flow estimation in large displacement image areas and weak texture image areas, reduces errors caused by local information, and ensures the reliability and robustness of optical flow estimation.

Embodiment two

In order to implement the method corresponding to the first embodiment above to achieve corresponding functions and technical effects, an optical flow computing system is provided below.

Referring to Figure 6, the system includes:

The image acquisition module 601 is configured to acquire a target image; the target image includes: a first image and a second image; the first image and the second image are two consecutive frames of images.

The motion feature extraction module 602 is configured to use a motion feature extraction network to extract the motion feature of the target image; the motion feature extraction network includes multiple convolutional layers of different sizes.

The matching cost calculation module 603 is configured to determine the feature map of the first image and the feature map of the second image according to the motion feature of the target image and the number of feature extraction channels, and calculate the feature map of the first image map and the matching cost volume of the feature map of the second image.

The context feature extraction module 604 is configured to extract the context feature of the first image by using a context encoder; the structure of the context encoder is the same as that of the motion feature extraction network.

The optical flow field solving module 605 is configured to use a global-local loop optical flow decoder to perform loop iterative solution based on the matching cost volume and the context feature, so as to obtain the optical flow field of the target image.

Wherein, the global-local cyclic optical flow decoder includes: a local motion information encoder, a global motion information encoder and a global-local motion information decoder connected in sequence; the output of the global-local motion information decoder is connected Input to the local motion information encoder.

Both the local motion information encoder and the global-local motion information decoder include: sequentially connected depth separable residual blocks and multi-layer perceptron blocks; the global motion information encoder includes: sequentially connected depth Separable convolutional modules and multi-head attention modules.

The local motion information encoder is used for encoding according to the matching cost volume and the residual optical flow of the last iteration to obtain local motion features. The global motion information encoder is configured to perform encoding according to the local motion feature and the context feature to obtain global motion information. The global-local motion information decoder is used to decode according to the local motion feature, the global motion information and the context feature to obtain the residual optical flow of the current iteration; the residual optical flow of the last iteration is used to determine The optical flow field of the target image.

Embodiment three

This embodiment provides an electronic device, including a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the optical flow calculation method of Embodiment 1.

Optionally, the above-mentioned electronic device may be a server.

In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the optical flow calculation method of the first embodiment is implemented.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for the related information, please refer to the description of the method part.

In this paper, specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; meanwhile, for those of ordinary skill in the art, according to the present invention Thoughts, there will be changes in specific implementation methods and application ranges. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (7)

1. An optical flow calculation method, comprising:

acquiring a target image; the target image includes: a first image and a second image; the first image and the second image are two continuous frames of images;

extracting the motion characteristics of the target image by adopting a motion characteristic extraction network; the motion feature extraction network comprises a plurality of convolution layers with different sizes;

determining a feature map of the first image and a feature map of the second image according to the motion feature and the feature extraction channel number of the target image, and calculating the matching cost volume of the feature map of the first image and the feature map of the second image;

extracting a context feature of the first image using a context encoder; the structure of the context encoder is the same as that of the motion feature extraction network;

based on the matching cost volume and the context characteristics, adopting a global-local loop optical flow decoder to carry out loop iteration solution to obtain an optical flow field of the target image;

wherein the global-local loop optical flow decoder comprises: a local motion information encoder, a global motion information encoder and a global-local motion information decoder connected in sequence; the output of the global-local motion information decoder is connected to the input of the local motion information encoder;

the local motion information encoder and the global-local motion information decoder each include: the depth separable residual block and the multi-layer perceptron block are connected in sequence; the global motion information encoder includes: a depth separable convolution module and a multi-head attention module which are connected in sequence;

the local motion information encoder is used for encoding according to the matching cost volume and the residual light stream of the last iteration to obtain local motion characteristics;

the global motion information encoder is used for encoding according to the local motion characteristics and the context characteristics to obtain global motion information;

the global-local motion information decoder is used for decoding according to the local motion characteristics, the global motion information and the context characteristics to obtain a residual light stream of the current iteration; the residual optical flow of the last iteration is used to determine the optical flow field of the target image;

according to the motion characteristics and the characteristic extraction channel number of the target image, determining the characteristic diagram of the first image and the characteristic diagram of the second image, and calculating the matching cost volume of the characteristic diagram of the first image and the characteristic diagram of the second image, wherein the method specifically comprises the following steps:

determining the first half of the motion characteristics of the target image as a characteristic diagram of the first image, and determining the second half of the motion characteristics of the target image as a characteristic diagram of the second image;

performing dot product similarity operation on the feature images of the first image and the feature images of the second image to obtain matching cost information of the feature images of the first image and the feature images of the second image;

and downsampling the matching cost information by adopting pooling operation to obtain the matching cost volumes of the feature map of the first image and the feature map of the second image.

2. The optical flow computing method according to claim 1, characterized in that the motion feature extraction network specifically comprises: the first convolution layer, the convolution residual block and the second convolution layer are sequentially connected;

the convolution kernel size of the first convolution layer is 7×7; the convolution residual block includes: the third convolution layer and the fourth convolution layer are sequentially connected; the size of the convolution kernel of the second convolution layer is 1×1; the size of the convolution kernel of the third convolution layer is 3 multiplied by 3, and the step length is 2; the size of the convolution kernel of the fourth convolution layer is 3×3, and the step size is 1.

3. The optical flow computing method according to claim 1, characterized in that the depth separable residual block comprises: the first depth separable convolution layer, the first activation function, the second depth separable convolution layer and the second activation function are connected in sequence;

the convolution kernel size of the first depth separable convolution layer is 7 x 7; the second depth separable convolution layers are connected in a dense manner, and the convolution kernel size is 15 multiplied by 15; the first activation function and the second activation function are both GELU activation functions.

4. The optical flow computing method according to claim 1, wherein the multi-layer perceptron block specifically comprises: the fifth convolution layer, the third depth separable convolution layer, the third activation function and the sixth convolution layer are sequentially connected;

the convolution kernel sizes of the fifth convolution layer and the sixth convolution layer are 1×1; the convolution kernel size of the third depth separable convolution layer is 3×3; the third activation function is a GELU activation function.

5. An optical flow computing system, comprising:

the image acquisition module is used for acquiring a target image; the target image includes: a first image and a second image; the first image and the second image are two continuous frames of images;

the motion feature extraction module is used for extracting the motion features of the target image by adopting a motion feature extraction network; the motion feature extraction network comprises a plurality of convolution layers with different sizes;

the matching cost calculation module is used for determining a feature map of the first image and a feature map of the second image according to the motion feature and the feature extraction channel number of the target image, and calculating the matching cost volume of the feature map of the first image and the feature map of the second image;

a context feature extraction module for extracting context features of the first image using a context encoder; the structure of the context encoder is the same as that of the motion feature extraction network;

the optical flow field solving module is used for carrying out loop iteration solving by adopting a global-local loop optical flow decoder based on the matching cost volume and the context characteristics to obtain an optical flow field of the target image;

wherein the global-local loop optical flow decoder comprises: a local motion information encoder, a global motion information encoder and a global-local motion information decoder connected in sequence; the output of the global-local motion information decoder is connected to the input of the local motion information encoder;

the local motion information encoder and the global-local motion information decoder each include: the depth separable residual block and the multi-layer perceptron block are connected in sequence; the global motion information encoder includes: a depth separable convolution module and a multi-head attention module which are connected in sequence;

the local motion information encoder is used for encoding according to the matching cost volume and the residual light stream of the last iteration to obtain local motion characteristics;

the global motion information encoder is used for encoding according to the local motion characteristics and the context characteristics to obtain global motion information;

the global-local motion information decoder is used for decoding according to the local motion characteristics, the global motion information and the context characteristics to obtain a residual light stream of the current iteration; the residual optical flow of the last iteration is used to determine the optical flow field of the target image;

according to the motion characteristics and the characteristic extraction channel number of the target image, determining the characteristic diagram of the first image and the characteristic diagram of the second image, and calculating the matching cost volume of the characteristic diagram of the first image and the characteristic diagram of the second image, wherein the method specifically comprises the following steps:

determining the first half of the motion characteristics of the target image as a characteristic diagram of the first image, and determining the second half of the motion characteristics of the target image as a characteristic diagram of the second image;

performing dot product similarity operation on the feature images of the first image and the feature images of the second image to obtain matching cost information of the feature images of the first image and the feature images of the second image;

and downsampling the matching cost information by adopting pooling operation to obtain the matching cost volumes of the feature map of the first image and the feature map of the second image.

6. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the optical flow calculation method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the optical flow calculation method according to any one of claims 1 to 4.