CN111583345B - Method, device and equipment for acquiring camera parameters and storage medium - Google Patents

Abstract

The present application discloses a method, device, device and storage medium for acquiring camera parameters, including: collecting original continuous frame images captured by a monocular camera; constructing a DepthNet model and a MotionNet model; the DepthNet model includes a single-channel depth map output. network; the MotionNet model includes a backbone network for camera motion prediction and pixel confidence masks and a branch network for camera intrinsic parameter prediction; the original continuous frame images are preprocessed and input into the constructed model, and through joint The loss function is used for unsupervised training and hyperparameter tuning; the image to be tested is processed by the trained model, and the depth map of each frame of image, camera motion, camera internal parameters and pixel confidence containing scene motion information are output. mask. In this way, there is no need to calibrate the camera, and the video can be directly input to obtain the camera internal parameters, camera motion and depth map of each frame.

Description

A method, device, device and storage medium for acquiring camera parameters

technical field

The present invention relates to the fields of computer vision and photogrammetry, in particular to a method, device, device and storage medium for acquiring camera parameters.

Background technique

As one of the main tools of computer vision, the camera and the various algorithms surrounding the camera have an important position. Among them, photogrammetry mainly studies the imaging principles of cameras, and focuses on how to obtain real-world information from pictures taken by cameras. For various applications of computer vision and photogrammetry, such as industrial control, automatic driving, and robot navigation and pathfinding, camera intrinsic parameters, camera motion and depth of field are of great value, and a large number of computational processes involving photogrammetry and camera imaging properties are involved. All three types of information are required as input.

The internal parameters of the camera include information such as the focal length of the camera. The self-motion of the camera is also called ego-motion, which includes the position transformation information of the camera itself. . Generally, the process of obtaining camera internal and external parameters is called camera calibration (Camera Calibrate), and the process of obtaining ego-motion is called visual odometry (Visual Odometry, VO).

For camera intrinsics, ego-motion, and depth-of-field information, existing methods based on non-deep learning usually use separate techniques to obtain them separately. The method used to obtain the internal parameters needs to use the camera to take several (usually about 20) images of the calibration board at different angles. When the camera needs to be adjusted frequently, it has to be calibrated frequently, and for applications that are not accessible to the camera device scenarios, this calibration method is not available. The methods used to obtain ego-motion and depth-of-field information have similar flaws: these methods require some assumptions (i.e., scene static, scene consistency, and Lambertian surface assumptions) to work properly, and any operating conditions that violate these assumptions will affect The corresponding method works fine. The technology based on deep learning can get rid of the dependence on the premise to varying degrees, and can simultaneously obtain ego-motion and depth of field information, and the convenience of use is improved. However, it is still necessary to input the internal parameters of the camera, so it cannot completely get rid of the inconvenience brought by the camera calibration method.

Therefore, how to solve the problems that the existing solutions require camera calibration and a large amount of supervised learning data, etc., is a technical problem to be solved urgently by those skilled in the art.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a method, device, device and storage medium for acquiring camera parameters, which can perform unsupervised learning without calibrating the camera, take continuous frames captured by a monocular camera as input, and output each The depth map of the frame, the motion of the camera during shooting, and the camera's intrinsic parameters.

Its specific plan is as follows:

A method for obtaining camera parameters, including:

Collect the original continuous frame images captured by the monocular camera;

Build a DepthNet model and a MotionNet model; the DepthNet model includes a network for outputting a single-channel depth map; the MotionNet model includes a backbone network for camera motion prediction and a pixel confidence mask and a network for camera intrinsic parameter prediction. branch network;

The original continuous frame images are respectively input into the constructed DepthNet model and the MotionNet model after preprocessing, and unsupervised training is performed on the DepthNet model and the MotionNet model through a joint loss function, and a super parameter tuning;

The image to be tested is processed by the trained DepthNet model and the MotionNet model, and the depth map of each frame of the image to be tested, the motion of the camera, the internal parameters of the camera and the pixel confidence mask containing scene motion information are output.

Preferably, in the above-mentioned method for obtaining camera parameters provided by the embodiment of the present invention, the DepthNet model is composed of a first encoder and a first decoder;

The original continuous frame images are input into the DepthNet model for training after preprocessing, specifically including:

Obtaining a preprocessed three-channel image by the first encoder, and encoding the three-channel image into features of various granularities successively;

decoding using the first decoder in conjunction with features of different granularities;

A single-channel depth map of the same size as the input three-channel image is output through the first decoder.

Preferably, in the above-mentioned camera parameter acquisition method provided by the embodiment of the present invention, the three-channel image is sequentially encoded into features of various granularities by using the first encoder, and the first decoder is used to combine different granularities. The features of , are decoded, including:

In the first encoder, through a 2D convolution with a convolution kernel size of 7×7, after batch normalization and linear rectification units, the first-level feature encoding is formed;

Connect a max pooling layer and two first residual modules to form a second-level feature encoding;

alternately connecting the second residual module and the first residual module to form the third-level feature encoding, the fourth-level feature encoding and the fifth-level feature encoding respectively;

inputting the first-level feature encoding, the second-level feature encoding, the third-level feature encoding, the fourth-level feature encoding, and the fifth-level feature encoding to the first decoder;

In the first decoder, 2D transposed convolution and 2D convolution are used alternately, and the feature codes of five levels are combined step by step, and the softplus activation function is used in the output layer for output.

Preferably, in the method for acquiring the camera parameters provided in the embodiment of the present invention, the backbone network is composed of a second encoder and a second decoder;

The original continuous frame images are input into the MotionNet model for training after preprocessing, specifically including:

Obtaining two adjacent frames of images after preprocessing by the second encoder;

In the second encoder, 7 cascaded 3×3 2D convolutional layers are used, and a 1×1 convolutional layer is connected at the bottleneck part to compress the number of output channels to six, and the first three channels are output The translation of the camera, the rotation of the camera is output by the last three channels;

In the second decoder, two parallel convolution paths and short-cut connections are used, the convolution output and the bilinear interpolation output are combined to form the output of the Refine module, and a pixel-level confidence mask is output. code is used to determine whether each pixel participates in the calculation when the joint loss function is calculated, and at the same time, a penalty function is added to the pixel-level confidence mask to prevent training degradation;

The internal parameter matrix of the camera is output through the branch network connected to the bottommost encoder of the backbone network.

Preferably, in the above-mentioned camera parameter acquisition method provided by the embodiment of the present invention, outputting an internal parameter matrix of the camera specifically includes:

In the branch network, multiplying the network prediction value by the width and height of the image is the actual focal length;

Add 0.5 to the predicted value of the network, and multiply by the width and height of the image to obtain the pixel coordinates of the principal point;

Diagonalize the focal length into a 2×2 diagonal matrix, connect the column vectors formed by the coordinates of the principal point, and then add row vectors to form a 3×3 internal parameter matrix.

Preferably, in the method for acquiring the camera parameters provided in the embodiment of the present invention, the preprocessing of the original continuous frame image includes:

Adjusting the resolution of the original continuous frame images, and arranging and splicing them into multiple triple frame images;

When each of the triple frame images is input into the DepthNet model, the depth map of each frame image is output;

When each triple frame image is input into the MotionNet model, the camera motion between each adjacent two frame images, the camera's internal parameters and the pixel confidence mask are output four times.

Preferably, in the method for obtaining the camera parameters provided in the embodiment of the present invention, the joint loss function is calculated by using the following formula:

为深度平滑损失，b为所述深度平滑损失的权值，Λ为所述像素置信度掩码的正则化惩罚函数，c为所述惩罚函数的权值。Among them, L _total is the joint loss function, L _R is the reprojection error function, a is the weight of the reprojection error function,

is the depth smoothing loss, b is the weight of the depth smoothing loss, Λ is the regularization penalty function of the pixel confidence mask, and c is the weight of the penalty function.

The embodiment of the present invention also provides a device for acquiring camera parameters, including:

The image collection module is used to collect the original continuous frame images captured by the monocular camera;

A model building module for constructing a DepthNet model and a MotionNet model; the DepthNet model includes a network for outputting a single-channel depth map; the MotionNet model includes a backbone network for camera motion prediction and giving pixel confidence masks and A branch network for camera intrinsic parameter prediction;

The model training module is used to input the original continuous frame images into the constructed DepthNet model and the MotionNet model respectively after preprocessing, and perform a free-flow analysis on the DepthNet model and the MotionNet model through a joint loss function. Supervised training and tuning of hyperparameters;

The model prediction module is used to process the image to be tested by the trained DepthNet model and the MotionNet model, and output the depth map of the image to be tested in each frame, the motion of the camera, the internal parameters of the camera and the image that contains scene motion information. Pixel confidence mask.

An embodiment of the present invention further provides a device for acquiring camera parameters, including a processor and a memory, wherein, when the processor executes a computer program stored in the memory, the above-mentioned camera parameter acquisition as provided by the embodiment of the present invention is realized method.

Embodiments of the present invention further provide a computer-readable storage medium for storing a computer program, wherein, when the computer program is executed by a processor, the above-mentioned method for acquiring camera parameters provided by the embodiments of the present invention is implemented.

It can be seen from the above technical solutions that the method, device, device and storage medium for obtaining camera parameters provided by the present invention include: collecting original continuous frame images captured by a monocular camera; constructing a DepthNet model and a MotionNet model; a DepthNet model Including a network for outputting a single-channel depth map; the MotionNet model includes a backbone network for camera motion prediction and giving pixel confidence masks and a branch network for camera intrinsic parameter prediction; the original continuous frame images are preprocessed. Input into the built DepthNet model and MotionNet model, and perform unsupervised training on the DepthNet model and MotionNet model through the joint loss function, and perform hyperparameter tuning; through the trained DepthNet model and MotionNet model The image to be tested is processed, Output the depth map of each frame of the image to be tested, the motion of the camera, the intrinsic parameters of the camera and the pixel confidence mask containing scene motion information.

The present invention does not need to calibrate the camera, and has no additional restrictions on the use scene. Any video shot by the monocular camera can be directly input to obtain the camera motion track, the depth map of each frame and the internal parameters of the camera during shooting. When the camera internal parameters are unknown, using the joint loss function for unsupervised learning can ensure normal training; in addition, it provides a less constrained front-end solution for computer vision applications that require camera internal parameters, camera motion, and depth maps. Has good application value.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the accompanying drawings required for the description of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are only the For the embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without any creative effort.

FIG. 1 is a flowchart of a method for acquiring camera parameters provided by an embodiment of the present invention;

2 is a schematic structural diagram of a first encoder in a DepthNet model provided by an embodiment of the present invention;

3 is a schematic structural diagram of a first residual module in a first encoder according to an embodiment of the present invention;

4 is a schematic structural diagram of a second residual module in a first encoder according to an embodiment of the present invention;

5 is a schematic structural diagram of a first decoder in a DepthNet model provided by an embodiment of the present invention;

6 is a schematic structural diagram of a backbone network in the MotionNet model provided by an embodiment of the present invention;

7 is a schematic structural diagram of a Refine module in a backbone network provided by an embodiment of the present invention;

8 is a schematic structural diagram of a branch network in the MotionNet model provided by an embodiment of the present invention;

9 is a schematic diagram of an arrangement of training data provided by an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of an apparatus for acquiring camera parameters according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The present invention provides a method for acquiring camera parameters, as shown in FIG. 1 , comprising the following steps:

S101. Collect the original continuous frame images captured by the monocular camera; it should be noted that the collected original continuous frame images can be extracted from the KITTI data set;

S102, build a DepthNet model and a MotionNet model; the DepthNet model includes a network for outputting a single-channel depth map; the MotionNet model includes a backbone network for camera motion prediction and pixel confidence mask giving and a branch network for camera intrinsic parameter prediction ; It should be noted that the function of internal reference prediction enables the present invention to extract accurate camera motion and depth of field information from any video from unknown sources without camera calibration;

S103, input the original continuous frame images into the constructed DepthNet model and MotionNet model after preprocessing, and perform unsupervised training on the DepthNet model and MotionNet model through a joint loss function, and perform hyperparameter tuning; Yes, the joint loss function is composed of the reprojection error, the depth smoothing loss and the regularization penalty function of the pixel confidence mask. The joint loss function uses the relationship between consecutive frames captured by the monocular camera as the source of the supervision signal, which is the depth The model provides training power, which is the key to realizing unsupervised learning; in addition, the model proposed by the present invention mainly contains hyperparameters such as learning rate, loss function weight, and batch size, which need to be tuned in order to obtain the best combination;

S104 , process the image to be tested by the trained DepthNet model and the MotionNet model, and output the depth map of each frame of the image to be tested, the motion of the camera, the internal parameters of the camera, and a pixel confidence mask containing scene motion information.

In the method for obtaining the above camera parameters provided by the embodiment of the present invention, the DepthNet model and the MotionNet model are unsupervised deep learning models, which do not need to calibrate the camera, and have no additional restrictions on the use scene. Video, you can obtain the camera motion trajectory during shooting, the depth map of each frame and the camera's internal parameters, and when the camera's internal parameters are unknown, use the joint loss function for unsupervised learning to ensure normal training; in addition, for Computer vision applications that require camera intrinsics, camera motion, and depth maps provide a less constrained front-end solution with good application value.

In practical applications, the above model can be implemented using PyTorch, using a deep learning workstation for training, the CPU is two Intel Xeon E5 2678v3, 64GB main memory, four NVIDIA GeForce GTX 1080Ti, each graphics card has 12GB video memory. The invention optimizes this machine in parallel, specifically setting the epochs to be a multiple of 4. In the data reading stage, the two CPUs each load half of the data and store them in their corresponding main memory. A graphics card means that each CPU is directly connected to two graphics cards through a PCI-E channel. Therefore, the method of the present invention that the two CPUs respectively load data can maximize the use of the bandwidth of each PCI-E channel and help reduce data transmission. time. After the data is transferred from the main memory to the video memory, the four graphics cards start their respective gradient processes. When the four graphics cards each exhaust this batch of data, they reach the synchronization point of the program and report their gradient information to the CPU, which is carried out by the CPU. Gradient summarization and model update, then enter the next cycle. The final effect is that when the GPU performs gradient operations, the CPU is reading and preparing data, thereby reducing the downtime of the GPU as much as possible and improving the overall operating efficiency.

In specific implementation, in the above-mentioned method for obtaining camera parameters provided by the embodiment of the present invention, the DepthNet model may be composed of a first encoder and a first decoder; the input of the DepthNet model is a three-channel picture captured by a monocular camera , through the encoder-decoder structure, outputs a single-channel depth map of the same size as the input. This is equivalent to encoding every pixel of the input image. In addition, due to the large complexity of the model, in order to ensure the effective transfer of gradients and enable the deep network to receive good training, the model uses a large number of residual blocks (Residual BuildingBlocks).

The above-mentioned step S103 inputs the original continuous frame image into the DepthNet model for training after preprocessing, which may specifically include: first, obtaining the preprocessed three-channel image through the first encoder, and sequentially encoding the three-channel image into a variety of granular features; then, use the first decoder to combine features of different granularities for decoding; finally, output a single-channel depth map with the same size as the input three-channel image size through the first decoder. As shown in Figure 2, the first encoder can output features with five granularities.

Further, in the specific implementation, in the above steps, the first encoder is used to sequentially encode the three-channel image into features of various granularities, and the first decoder is used to combine the features of different granularities for decoding, which may specifically include: in the first encoding In the device, firstly through a 2D convolution with a convolution kernel size of 7×7, after batch normalization and linear rectification unit, the first-level feature encoding is formed; then a maximum pooling layer and two first-level feature encoding are connected. A residual block (residual_block_A), which forms the second-level feature encoding; finally, the second residual block (residual_block_B) and the first residual block are alternately connected to form the third-level feature encoding, the fourth-level feature encoding, and the fifth-level feature encoding, respectively. Feature encoding; Next, input the first-level feature encoding, the second-level feature encoding, the third-level feature encoding, the fourth-level feature encoding, and the fifth-level feature encoding to the first decoder; in the first decoder, 2D transposed convolution and 2D convolution are used alternately, and the feature encoding of five levels is combined step by step, and the softplus activation function is used in the output layer for output.

It should be noted that the first encoder contains two residual modules, residual_block_A and residual_block_B. Among them, as shown in Figure 3, residual_block_A is mainly composed of two 3×3 convolutional layers. The number of output channels of these two convolutional layers is the number of input channels of the residual module, so residual_block_A will not change the tensor number of channels. The part of two consecutive convolutional layers in the residual module is called the main branch, and the branch path directly connected from the input to the output of the main branch is called a short-cut. The main branch of residual_block_B is similar to residual_block_A, but the form of short-cut contains some conditional judgment logic; as shown in Figure 4, when the number of input channels and the number of output channels are not equal, short-cut is performed on the basis of the input tensor A 1×1 convolution, and the number of input and output channels is adjusted to be consistent through this layer of convolution; when the number of input channels and the number of output channels are equal, the size of the step size is further judged. When the step size is 1, the short-cut is the input tensor, and when the step size is not 1, the dimension of the output tensor will not be equal to the dimension of the input tensor. In order to compensate for the difference in dimension, an additional one is added to the input tensor. max pooling layer. As shown in Figure 2, out_channels and stride are input from the outside of the module, and the three outs represent the outputs in three cases respectively.

In addition, it should be noted that, as shown in FIG. 5 , the first decoder takes the five-level feature encoding of the first encoder as input, uses 2D transposed convolution and 2D convolution alternately, and combines them step by step. The five-level feature encoding uses a softplus activation function in the output layer, and finally outputs a single-channel depth map with the same size as the encoder input. The concat_and_pad in Figure 5 is a compound operation that first concatenates the output from the 2D transposed convolution with the next-level encoder output in the third dimension, and then performs a continuation to input the result to the subsequent 2D volume accumulate. The final output depth map size is [B,h,w,1], where B represents the batch size, h and w represent the height and width of the image, and 1 represents the number of channels of the depth map is 1.

During specific implementation, in the above-mentioned camera parameter acquisition method provided by the embodiment of the present invention, the backbone network in the MotionNet model may be composed of a second encoder and a second decoder.

The above-mentioned step S103 inputs the original continuous frame images into the MotionNet model for training after preprocessing, which may specifically include: first, obtaining the preprocessed adjacent two frame images through the second encoder; then, as shown in FIG. 6 , In the second encoder, 7 concatenated 2D convolutional layers with kernel size of 3×3 are used, and a 1×1 convolutional layer is connected to the bottleneck part of the second encoder to compress the number of output channels To six, the first three channels output the translation of the camera, and the last three channels output the rotation of the camera; after that, in the second decoder, two parallel convolution paths are used and short-cut connections like residual modules are used , the convolution output and the bilinear interpolation output are combined to form the output of the Refine module, and a pixel-level confidence mask is output to determine whether each pixel participates in the calculation when the joint loss function is calculated. The excluded pixels Due to the translation, rotation, occlusion and other factors of the scene, it cannot participate in the calculation of the reprojection loss, and at the same time, a penalty function is added to the pixel-level confidence mask to prevent training degradation; The branch network outputs the camera's intrinsic parameter matrix.

It should be understood that in the backbone network of MotionNet, the second decoder is composed of the Refine module. As shown in Figure 7, conv_input represents the input of the decoder side, and Refine_input represents the output of the previous stage of Refine. To address the difference in resolution, the present invention uses bilinear interpolation to resize the output of the previous stage of Refine.

In specific implementation, in the above-mentioned camera parameter acquisition method provided by the embodiment of the present invention, outputting the internal parameter matrix of the camera specifically includes: in the branch network, multiplying the network predicted value by the width and height of the image is the actual focal length; Add 0.5 to the predicted value of the network, and multiply it by the width and height of the image to obtain the pixel coordinates of the main point; diagonalize the focal length into a 2×2 diagonal matrix, connect the column vector formed by the coordinates of the main point, and then add the row vector , forming a 3×3 internal parameter matrix.

Specifically, as shown in Figure 8, the "bottleneck" on the left represents the lowest level feature output by the encoder of the backbone network, and its size is [B, 1, 1, 1024], B represents the batch size, and two 1's in parallel The ×1 convolution is used to predict the focal length f _x , f _y and the principal point coordinates c _x , _cy in the internal parameter matrix respectively. In order to facilitate network learning, in fact, the two convolutions predict a smaller number. For the focal length, the ratio of the actual focal length to the image width and height is predicted here, so the present invention multiplies the network predicted value by the image width and height. For the coordinates of the main point, the predicted coordinate value here accounts for the ratio of the width and height of the image. Since the position of the main point tends to the center of the image, the present invention adds 0.5 to the predicted value of the network, and then multiplies the width and height to obtain Get the pixel coordinates of the principal point. Finally, the focal length is diagonalized into a 2×2 diagonal matrix, the column vector formed by the coordinates of the principal point is connected, and the row vector [0,0,1] is added, and finally a 3×3 internal parameter matrix is formed.

In specific implementation, in the above-mentioned camera parameter acquisition method provided by the embodiment of the present invention, the preprocessing of the original continuous frame images in step S103 may include: adjusting the resolution of the original continuous frame images, and arranging and splicing them. , to be spliced into multiple triple frame images; when each triple frame image is input into the DepthNet model, the depth map of each frame image is output; when each triple frame image is input into the MotionNet model, the output is four times for each adjacent two Camera motion between frame images, camera intrinsics and pixel confidence masks.

It should be understood that, according to the model and training method adopted in the present invention, each training of the DepthNet model requires a monocular color image (that is, a three-channel image), and each training of the MotionNet model requires two consecutive images ( That is, two adjacent frames of images), in order to improve the reading efficiency of the data, the present invention needs to preprocess the original continuous frame images, as shown in FIG. The concentrated original continuous frame images, (b) represents the triple frame images spliced together after preprocessing, after such processing, for each triple frame image taken, two pairs of adjacent pictures can be obtained. In order to reduce the computational pressure, at the same time of splicing, the original image is scaled down, and the final resolution of all images can be unified to 416×128, and the resolution of a single triple frame image can be 1248×128.

Specifically, the training process is performed in units of triple frames. Every time a triple frame is read out, first use the DepthNet model to generate the depth map of each frame, and then use the MotionNet model to generate the camera motion, pixel confidence mask and camera internal parameters for the first to second frames, and similarly, generate the first frame 2 frames to 3 frames, 3 frames to 2 frames, and 2 frames to 1 frames, so that four predictions of the camera's internal parameters are obtained, and the average of these four internal parameters is taken as this group of triple associations internal reference. Then, the joint loss function can be used once between every two adjacent frames, so a total of four times (ie 1-2, 2-3, 3-2, 2-1) can be used, and the four loss function values can be accumulated, as the loss function value associated with this set of triples. In actual training, each time the data is read, batch size triple frames will be obtained, these frames will be calculated in parallel, and then backpropagated to update the model.

During specific implementation, in the above-mentioned method for obtaining camera parameters provided by the embodiments of the present invention, the joint loss function can be calculated by using the following formula:

为深度平滑损失(也称深度值的L₁范数，深度图的离群点和尖锐点越多，平滑损失越大)，b为深度平滑损失的权值，Λ为像素置信度掩码的正则化惩罚函数，c为惩罚函数的权值。Among them, L _total is the joint loss function, L _R is the reprojection error function, a is the weight of the reprojection error function,

is the depth smoothing loss (also known as the L ₁ norm of the depth value, the more outliers and sharp points in the depth map, the greater the smoothing loss), b is the weight of the depth smoothing loss, and Λ is the pixel confidence mask. The regularization penalty function, c is the weight of the penalty function.

In formula (1), _LR is:

为像素置信度掩码，表示(i，j)处像素的置信度，φ函数表示双线性插值，

为重投影视图，I_s(i，j)为真实视图。Among them, i, j are pixel coordinates,

is the pixel confidence mask, representing the confidence of the pixel at (i, j), the φ function represents bilinear interpolation,

is the reprojected view, and Is ( _i , j) is the real view.

The reprojection method in formula (1) is:

Among them, K is the camera internal parameter, R is the camera rotation matrix, t is the camera translation vector, _ps is the pixel coordinate after reprojection, _pt is the pixel coordinate before projection, D _s ( _ps ) is the pixel coordinate at _ps The corresponding depth value after reprojection, D _t ( _pt ) is the depth value corresponding to the pixel coordinate _pt before reprojection.

Λ in formula (1) is:

Its meaning is to take the average of all H(i, j). H(i,j) is defined as:

H(i,j)=-∑ _i,j M(i,j)log(s(i,j)) (5)

Its meaning is to take cross entropy for S(i, j), and S(i, j) is defined as:

取交叉熵。Its meaning is the pixel confidence mask

Take the cross entropy.

都取0，这种情况下损失函数的最主要部分L_R会直接为0，因为深度学习倾向于使损失函数尽量小，在不采用惩罚函数的情况下，很容易陷入这种“全部不可信”的情况，这时虽然损失函数很小，但是却没有实际任何意义)。惩罚函数Λ会在置信度掩码中“不可信像素”越多的时候取得越大的值。The penalty function Λ prevents the network from predicting all pixel confidence masks as "untrustworthy" (i.e. for each pixel position (i, j),

All take 0. In this case, the most important part of the loss function, _LR , will be directly 0, because deep learning tends to make the loss function as small as possible. Without the penalty function, it is easy to fall into this "all untrustworthy" ”, although the loss function is small, it has no practical significance). The penalty function Λ will take a larger value when there are more "untrustworthy pixels" in the confidence mask.

Based on the same inventive concept, an embodiment of the present invention also provides a device for acquiring camera parameters. Since the principle of the device for acquiring camera parameters for solving problems is similar to the aforementioned method for acquiring camera parameters, the device for acquiring camera parameters has a For the implementation, please refer to the implementation of the method for acquiring the camera parameters, and the repetition will not be repeated.

During specific implementation, the device for acquiring camera parameters provided by the embodiment of the present invention, as shown in FIG. 10 , specifically includes:

The image collection module 11 is used to collect the original continuous frame images captured by the monocular camera;

Model building module 12, for constructing DepthNet model and MotionNet model; Described DepthNet model includes the network that is used for outputting single-channel depth map; Described MotionNet model includes the backbone network that is used for camera motion prediction and gives pixel confidence mask and a branch network for camera intrinsic parameter prediction;

The model training module 13 is used to input the original continuous frame images into the constructed DepthNet model and the MotionNet model respectively after preprocessing, and perform the DepthNet model and the MotionNet model through the joint loss function. Unsupervised training and tuning of hyperparameters;

The model prediction module 14 is used to process the image to be measured by the trained described DepthNet model and the MotionNet model, and output the depth map of the described image to be measured in each frame, the motion of the camera, the internal parameters of the camera and include scene motion information The pixel confidence mask of .

In the above-mentioned camera parameter acquisition device provided by the embodiment of the present invention, the interaction of the above-mentioned four modules can eliminate the need to calibrate the camera and use the advantages of global information, only when the camera has translational and rotational motions After video recording, important data including camera internal parameters and depth map can be obtained, which can be used as a pre-algorithm for other computer vision applications, with small usage scene restrictions and convenient application.

For more specific working processes of the above-mentioned modules, reference may be made to the corresponding contents disclosed in the foregoing embodiments, which will not be repeated here.

Correspondingly, an embodiment of the present invention also discloses a device for acquiring camera parameters, including a processor and a memory; wherein the processor implements the method for acquiring camera parameters disclosed in the foregoing embodiments when the processor executes the computer program stored in the memory.

For a more specific process of the above method, reference may be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

Further, the present invention also discloses a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the method for obtaining the camera parameters disclosed above is implemented.

For a more specific process of the above method, reference may be made to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For the apparatuses, devices, and storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and reference may be made to the descriptions of the methods for related parts.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of functionality. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

In summary, a method, device, device, and storage medium for acquiring camera parameters provided by the embodiments of the present invention include: collecting original continuous frame images captured by a monocular camera; constructing a DepthNet model and a MotionNet model; A single-channel depth map network; the MotionNet model includes a backbone network for camera motion prediction and pixel confidence masking and a branch network for camera intrinsic parameter prediction; the original continuous frame images are preprocessed and input to the constructed In the DepthNet model and MotionNet model, unsupervised training is performed on the DepthNet model and MotionNet model through the joint loss function, and hyperparameter tuning is performed; the images to be tested are processed through the trained DepthNet model and MotionNet model, and each frame is output to be tested. The depth map of the measured image, the motion of the camera, the intrinsic parameters of the camera and the pixel confidence mask containing scene motion information. The present invention does not need to calibrate the camera, and has no additional restrictions on the use scene. Any video shot by the monocular camera can be directly input to obtain the camera motion track, the depth map of each frame and the internal parameters of the camera during shooting. When the camera internal parameters are unknown, using the joint loss function for unsupervised learning can ensure normal training; in addition, it provides a less constrained front-end solution for computer vision applications that require camera internal parameters, camera motion, and depth maps. Has good application value.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The method, device, device and storage medium for obtaining camera parameters provided by the present invention have been introduced in detail above. The principles and implementations of the present invention are described with specific examples in this paper. The descriptions of the above embodiments are only for help. Understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification does not It should be understood as a limitation of the present invention.

Claims (8)

1. A method for acquiring camera parameters is characterized by comprising the following steps:

collecting original continuous frame images shot by a monocular camera;

constructing a DepthNet model and a MotionNet model; the DepthNet model comprises a network for outputting a single-channel depth map; the MotionNet model comprises a main network used for camera motion prediction and giving a pixel confidence mask and a branch network used for camera internal parameter prediction;

respectively inputting the original continuous frame images into the constructed DepthNet model and the MotionNet model after preprocessing, and performing unsupervised training and super-parameter tuning on the DepthNet model and the MotionNet model through a joint loss function;

processing images to be detected through the trained DepthNet model and the trained MotionNet model, and outputting a depth map of each frame of the images to be detected, the motion of the camera, the internal reference of the camera and a pixel confidence mask containing scene motion information;

the DepthNet model is composed of a first encoder and a first decoder;

preprocessing the original continuous frame images and inputting the preprocessed original continuous frame images into the DepthNet model for training, wherein the method specifically comprises the following steps:

acquiring a preprocessed three-channel image through the first encoder, and successively encoding the three-channel image into features of multiple granularities;

decoding using the first decoder in conjunction with features of different granularity;

outputting, by the first decoder, a single-channel depth map of the same size as the input three-channel image;

the backbone network is composed of a second encoder and a second decoder;

inputting the original continuous frame images into the MotionNet model for training after preprocessing, specifically comprising:

acquiring two adjacent preprocessed frame images through the second encoder;

in the second encoder, 7 cascaded 3 × 3 2D convolutional layers are used, one 1 × 1 convolutional layer is connected to the bottleneck portion, the number of output channels is compressed to six, the first three channels output the translation of the camera, and the last three channels output the rotation of the camera;

in the second decoder, two parallel convolution paths are adopted and short-cut connection is used, the convolution output and the output of bilinear interpolation are combined to form the output of a Refine module, a pixel-level confidence mask is output and used for determining whether each pixel participates in calculation when a joint loss function is calculated, and meanwhile, a penalty function is added to the pixel confidence mask and used for preventing training degradation;

and outputting the internal reference matrix of the camera through the branch network connected to the lowest encoder of the backbone network.

2. The method according to claim 1, wherein the three-channel image is successively encoded into features of multiple granularities by the first encoder, and the first decoder is used to perform decoding in conjunction with the features of different granularities, specifically comprising:

in the first encoder, a 2D convolution with convolution kernel size of 7 multiplied by 7 is carried out, and after a batch standardization and linear rectification unit, a first-stage feature code is formed;

connecting a maximum pooling layer and two first residual modules to form a second-level feature code;

alternately connecting a second residual error module and the first residual error module to form a third-level feature code, a fourth-level feature code and a fifth-level feature code respectively;

inputting the first level feature encoding, the second level feature encoding, the third level feature encoding, the fourth level feature encoding, and the fifth level feature encoding to the first decoder;

in the first decoder, 2D transposition convolution and 2D convolution are alternately used, five levels of feature codes are combined step by step, and softplus activating functions are adopted for output at an output layer.

3. The method for acquiring camera parameters according to claim 1, wherein outputting the internal reference matrix of the camera specifically includes:

in the branch network, multiplying the network predicted value by the width and height of the image to obtain the actual focal length;

adding 0.5 to the network predicted value, and multiplying by the width and height of the image to obtain the pixel coordinate of the principal point;

and (3) the focal length is diagonal to form a diagonal matrix of 2 multiplied by 2, column vectors formed by connecting principal point coordinates are connected, and row vectors are added to form a 3 multiplied by 3 internal reference matrix.

4. The method for acquiring camera parameters according to claim 1, wherein the preprocessing of the original continuous frame images comprises:

adjusting the resolution of the original continuous frame images, and arranging and splicing the original continuous frame images to be spliced into a plurality of triple frame images;

when each triple frame image is input into the DepthNet model, outputting a depth map of each frame image;

and when each triple frame image is input into the MotionNet model, outputting four times of camera motion between every two adjacent frame images, and internal reference and pixel confidence mask of the camera.

5. The method of claim 1, wherein the joint loss function is calculated by using the following formula:

wherein L is _total As said joint loss function, L _R Is a reprojection error function, a is a weight of the reprojection error function,

and b is the depth smoothing loss, b is the weight of the depth smoothing loss, Λ is the regularization penalty function of the pixel confidence mask, and c is the weight of the penalty function.

6. An apparatus for acquiring camera parameters, comprising:

the image collection module is used for collecting original continuous frame images shot by the monocular camera;

the model building module is used for building a DepthNet model and a MotionNet model; the DepthNet model comprises a network for outputting a single-channel depth map; the MotionNet model comprises a main network used for camera motion prediction and giving a pixel confidence mask and a branch network used for camera internal parameter prediction;

the model training module is used for respectively inputting the original continuous frame images into the constructed DepthNet model and the MotionNet model after preprocessing, performing unsupervised training on the DepthNet model and the MotionNet model through a joint loss function, and performing super-parameter tuning;

the DepthNet model is composed of a first encoder and a first decoder;

preprocessing the original continuous frame images and inputting the preprocessed original continuous frame images into the DepthNet model for training, wherein the method specifically comprises the following steps:

acquiring a preprocessed three-channel image through the first encoder, and successively encoding the three-channel image into features of multiple granularities;

decoding using the first decoder in conjunction with features of different granularity;

outputting, by the first decoder, a single-channel depth map of the same size as the input three-channel image;

the backbone network is composed of a second encoder and a second decoder;

inputting the original continuous frame images into the MotionNet model for training after preprocessing, and specifically comprising the following steps:

acquiring two adjacent preprocessed frame images through the second encoder;

in the second encoder, 7 cascaded 3 × 3 2D convolutional layers are used, one 1 × 1 convolutional layer is connected to the bottleneck portion, the number of output channels is compressed to six, the first three channels output the translation of the camera, and the last three channels output the rotation of the camera;

in the second decoder, two parallel convolution paths are adopted and short-cut connection is used, the convolution output and the output of bilinear interpolation are combined to form the output of a Refine module, a pixel confidence mask is output and used for determining whether each pixel participates in calculation when a joint loss function is calculated, and meanwhile, a penalty function is added to the pixel confidence mask and used for preventing training degradation;

outputting an internal reference matrix of a camera through the branch network connected to a bottommost encoder of the backbone network; and the model prediction module is used for processing the image to be measured through the trained DepthNet model and the MotionNet model and outputting a depth map of each frame of the image to be measured, the motion of the camera, the internal parameters of the camera and a pixel confidence mask containing scene motion information.

7. An apparatus for acquiring camera parameters, comprising a processor and a memory, wherein the processor implements the method for acquiring camera parameters according to any one of claims 1 to 5 when executing a computer program stored in the memory.

8. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the method of acquiring camera parameters according to any one of claims 1 to 5.

Images