CN107133919A - Time dimension video super-resolution method based on deep learning - Google Patents

Abstract

The invention discloses a time-dimensional video super-resolution method based on deep learning, which mainly solves the problems of poor frame interpolation stability and low precision of reconstructed video images in the prior art. The key technology is to use neural network training to fit the nonlinear mapping relationship between the original video image and the downsampled video image, including: 1) obtaining the original video image set and the downsampled video image set as the training samples of the neural network; 2) Construct the neural network model and use the training samples to train the parameters of the neural network; 3) Input any given video as a test sample into the trained neural network model, and the output result of the neural network is the reconstructed video image. The invention reduces the computational complexity of frame interpolation and reconstruction of video images, improves the stability and accuracy of frame interpolation of reconstructed video images, and can be used for scene interpolation, animation production, and time-domain frame interpolation of low frame rate videos.

Description

Time-dimension video super-resolution method based on deep learning

technical field

The invention belongs to the field of image processing, and in particular relates to a time-dimensional video super-resolution method, which can be used for scene interpolation, animation production, and time domain frame interpolation of low frame rate videos.

Background technique

The video image not only contains the spatial information of the observed target, but also contains the motion information of the observed target in time, which has the property of "integration of space and time". Since the video image can maintain the spatial information and time information reflecting the nature of the object together, it greatly improves the ability of human beings to recognize the objective world, and has been proved to have great potential in remote sensing, military affairs, agriculture, medicine, biochemistry and other fields. application value.

The use of video imaging equipment to obtain precise video images is very costly, and is limited by the manufacturing process of sensors and optical devices. In order to improve the resolution of imaging video, it is usually necessary to compress the video at the expense of the temporal resolution of the video. Obviously, it is difficult to meet the needs of scientific research and large-scale practical applications. So using signal processing technology to reconstruct high-resolution video images from compressed video images has become an important way to obtain video images.

In "Dual Motion Estimation for Frame Rate Up-Conversion", Kang S J and others proposed an algorithm that uses motion estimation and motion compensation to realize frame interpolation and reconstruction of video images. The video image frame interpolation and reconstruction problem is an ill-conditioned inverse problem, which uses the time information of the video image and combines the spatial information of the video image to realize the video image frame interpolation reconstruction, but the algorithm does not make full use of the existing in the video image. The strong structural similarity between adjacent frames makes it difficult for the stability and accuracy of reconstructed video images to meet the requirements of scientific research and large-scale practical applications.

Contents of the invention

The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a time-dimensional video super-resolution method based on deep learning to improve the stability and accuracy of reconstructed video images and meet the requirements of large-scale practical applications.

Technical scheme of the present invention is realized like this:

The down-sampled video image set and the original video image set are respectively used as the input training samples and output training samples of the neural network, and the nonlinear mapping relationship between the down-sampled video images and the original video images is fitted through neural network training, and the This relationship is to guide the frame insertion and reconstruction of test samples, so as to achieve the purpose of using neural network for video time domain frame insertion. The specific steps include the following:

(1) Convert the color video image set S={S ₁ , S ₂ ,...,S _i ,...,S _N } into a grayscale video image set, that is, the original video image set X={X ₁ , X ₂ ,...,X _i ,...,X _N }, and use the downsampling matrix F to directly downsample the original video image set X to obtain the downsampled video image set Y={Y ₁ ,Y ₂ , ...,Y _i ,...,Y _N }, where, Indicates the i-th original video image sample, Represents the i-th downsampled video image sample, 1≤i≤N, N represents the number of image samples in the original video image set, M represents the size of the original video image block, L _h represents the size of the image block in each sample of the original video image set Quantity, L _l represents the number of image blocks in each sample of the downsampled video image set, and L _h = r × L _l , r represents the magnification factor of the original video image set to the downsampled video image set;

(2) Construct the neural network model, and utilize the downsampled video image set Y and the original video image set X to train the neural network parameters:

(2a) Determine the number of input layer nodes, the number of output layer nodes, the number of hidden layers and the number of hidden layer nodes of the neural network, randomly initialize the connection weight W ^(t) and bias b ^(t) of each layer, and give the learning rate η , the selected activation function is: Among them, g represents the input value of the neural network node, t=1,2,...,n, n represents the total number of layers of the neural network;

(2b) Randomly input a downsampled video image Y _i in the downsampled video image set as the input training sample, and at the same time input an original video image X _i in the corresponding original video image set as the output training sample, and use the selected activation function to calculate The activation value of each layer of the neural network is calculated as:

The activation value of the first layer, that is, the input layer: a ⁽¹⁾ = Y _i ,

The activation value of the t'=2,3,...,nth layer is: a ^(t') =f(W ^(t'-1) *a ^(t'-1) +b ^(t ′ ^-1) ), wherein, in the second layer of the network, the third layer, the fourth layer that is t'=2, t'=3, t'=4, in order to fully extract the correlation between video frames, three The three-dimensional filter is used to replace the traditional two-dimensional filter, f(g) represents the tanh(g) activation function, g=W ^(t′-1) *a ^(t′-1) +b ^(t′-1) , W ^(t' - ¹⁾ and b ^(t' - ¹⁾ represent the weight and bias of layer t'-1, respectively, and a ⁽ t'- ¹⁾ represents the activation value of layer t'-1;

(2c) Calculate the learning error of each layer of the neural network:

The error of the output layer, namely the nth layer, is: δ ⁽ⁿ⁾ =X _i -a ⁽ⁿ⁾ ,

The error of the t"=n-1, n-2,..., layer 2 is: δ ^(t") ＝((W ^(t") ) ^T δ ^(t"+1) ).*f'( W ^(t”-1) *a ^(t”-1) +b ^(t”-1) ), where W ^(t”) represents the weight of the t"th layer, δ ^(t"+1) represents the The error of the t"+1 layer, W ^(t"-1) and b ^(t"-1) respectively represent the weight and bias of the t"-1 layer, a ^(t"-1) represents the t"- The activation value of layer 1, f'(g') represents the derivative of the function f(g'), (g") ^T represents the transpose transformation, g'=W ^(t"-1) *a ^(t"-1) +b ^(t"-1) , g"=W ^(t") ;

(2d) Update the weights and biases of each layer of the neural network according to the error gradient descent method:

Update the weight to W ^(t) = W ^(t) -ηδ ^(t+1) (a ^(t) ) ^T , update the bias to b ^(t) = b ^(t) -ηδ ^(t+1) , where δ ^(t+1) represents the error of layer t+1, and a ^(t) represents the activation value of layer t;

(2e) Repeat steps (2b)-(2d) until the output layer error of the neural network reaches the preset accuracy requirement or the number of training times reaches the maximum number of iterations, end the training, save the network structure and parameters, and obtain the trained neural network model ;

(3) Given any video, input it into the trained neural network model, and the output of the neural network is the super-resolved video in the time dimension.

Compared with the prior art, the present invention has the following advantages:

1) Since the present invention uses a convolutional neural network to perform time-dimensional video super-resolution reconstruction, compared with the prior art, the computational complexity is reduced, and the stability of time-dimensional video image super-resolution reconstruction is improved;

2) The three-dimensional filter designed in the present invention improves the accuracy of temporal super-resolution reconstruction of time-dimensional video images due to fully considering the correlation between adjacent video frames.

Description of drawings

Fig. 1 is the realization flowchart of the present invention;

Fig. 2 is the neural network structural diagram that the present invention builds;

Fig. 3 is the original image of the used bus video of the simulation experiment of the present invention;

Fig. 4 is the reconstruction result figure of bus video image with existing Kang's method and Choi's method and the method of the present invention.

detailed description

Embodiments and effects of the present invention will be further described in detail below in conjunction with the accompanying drawings.

With reference to Fig. 1, the time dimension video super-resolution method based on deep learning of the present invention, its realization steps are as follows:

Step 1, get the color video image set S.

(1a) From a given database, select a color video image set S={S ₁ ,S ₂ ,...,S _i ,...,S ₄₆₄₈₁₄ } with a sample number of 464814, and convert S to a grayscale video Image set, that is, the original video image set X={X ₁ ,X ₂ ,...,X _i ,...,X ₄₆₄₈₁₄ }, wherein, Represent the i-th original video image sample, 1≤i≤464814, M represents the size of the original video image block, M=576, L _h represents the number of image blocks in each sample of the original video image set, L _h =6;

(1b) Use the downsampling matrix F to directly downsample the original video image set X to obtain the downsampled video image set Y=FX, which is equivalent to the original video image set X={X ₁ ,X ₂ ,..., Each sample in X _i ,...,X ₄₆₄₈₁₄ } is down-sampled to obtain a down-sampled video image set Y={Y ₁ ,Y ₂ ,...,Y _i ,...,Y ₄₆₄₈₁₄ }, where, Y _i represents the i-th downsampled video image sample, Y _i =FX _i , 1≤i≤464814, M represents the size of the downsampled video image block, M=576, L ₁ represents the number of image blocks in each sample of the downsampled video image set, L ₁ =3,

Step 2, constructing a neural network model, and using the downsampled video image set Y and the original video image set X to train the neural network parameters.

The specific implementation of this step is as follows:

(2a) Initialize the neural network parameters;

(2a1) the downsampled video image in the downsampled video image set is used as an input training sample, and the original video image in the original video image set is used as an output training sample;

(2a2) Determine the number of input layer nodes of the neural network according to the number of video frames of the input training samples. In this embodiment, according to the number of input layer nodes equal to the number L _l of image blocks in each sample of the downsampled video image set, set the input The number of layer nodes is 3;

(2a3) Determine the output layer node number of the neural network according to the video frame number of the output training sample. In this embodiment, the output layer node number is equal to the number L _h of image blocks in each sample of the original video image set, and the output layer is set. The number of nodes is 6;

(2a4) Determine the number of hidden layers and the number of hidden layer nodes:

Since the number of hidden layers of the neural network and the number of hidden layer nodes determine the scale of the neural network, the scale of the neural network should be as simple as possible under the premise of ensuring that the problem can be solved. In this embodiment, the number of hidden layers of the neural network is It is directly determined as 7 layers, and the number of nodes in the hidden layer is adjusted through experiments, that is, the number of nodes in the first layer is 64, the number of nodes in the second layer is 32, the number of nodes in the third layer is 24, and the number of nodes in the fourth layer The number of nodes is 12, the number of nodes in the fifth layer is 32, the number of nodes in the sixth layer is 32, and the number of nodes in the seventh layer is 6;

(2a5) Randomly initialize the connection weight W ^(t) and bias b ^(t) of each layer, t=1,2,3,4,5,6,7,8;

(2a6) given learning rate η=0.0005;

(2a7) The selected activation function is: Among them, g represents the input weighted sum of neural network nodes including bias;

(2b) Randomly input an input training sample Y _i , use the selected activation function to calculate the activation value of each layer of the neural network, and calculate:

The activation value of the first layer, that is, the input layer: a ⁽¹⁾ = Y _i ,

The activation value of layer t'=2,3,4,5,6,7 is: a ^(t') =f(W ^(t'-1) *a ^(t'-1) +b ^{(t'- 1)} ), wherein, in the second layer of the network, the third layer, the fourth layer that is t'=2, t'=3, t'=4, in order to fully extract the correlation between video frames, design Three three-dimensional filters are used to replace the traditional two-dimensional filter, f(g) represents the tanh(g) activation function, g=W ^(t'-1) *a ^(t'-1) +b ^{(t'-1 )} , W ^(t'-1) and b ^(t'-1) respectively represent the weight and bias of the t'-1th layer, a ^(t'-1) represents the activation value of the t'-1th layer;

(2c) Input a corresponding output training sample X _i , and calculate the learning error of each layer of the neural network:

The error of the output layer, that is, the fourth layer is: δ ⁽⁴⁾ ＝X _i -a ⁽⁴⁾ ,

t”=3, the error of layer 2 is: δ ^(t”) ＝((W ^(t”) ) ^T δ ^(t”+1) ).*f'(W ^(t”-1) *a ^{( t”-1)} +b ^(t”-1) ), where W ^(t”) represents the weight of the t”th layer, W ^(t”-1) and b ^(t”-1) respectively represent the weight of the t”th layer The weight and bias of the "-1 layer, a ^(t"-1) represents the activation value of the t"-1 layer, f'(g') represents the derivative of the function f(g'), (g") ^T Represent transpose transformation, g'=W ^{(t "-1)} *a ^{(t "-1)} +b ^{(t "-1)} , g "=W ^{(t ")} ;

(2d) Update the weights and biases of each layer of the neural network according to the error gradient descent method:

Update the weight as: W ^(t) = W ^(t) -ηδ ^(t+1) (a ^(t) ) ^T ,

Update the bias as: b ^(t) = b ^(t) -ηδ ^(t+1) , where δ ^(t+1) represents the error of the t+1th layer and a ^(t) represents the activation of the tth layer value;

(2e) Repeat steps (2b)-(2d) until the network output layer error reaches the preset accuracy requirement or the number of training times reaches the maximum number of iterations, then end the training, save the network structure and parameters, and obtain the trained neural network model. In an embodiment, the maximum number of iterations is 500;

The neural network constructed in step 2 is shown in Figure 2, which includes 1 input layer, 3 three-dimensional convolution layers, 3 two-dimensional convolution layers, and 1 output layer. The input layer has 3 nodes, 7 The number of nodes in each hidden layer is 64, 32, 24, 12, 32, 32, 6 respectively, and the output layer has 6 nodes.

Step 3, using the trained neural network model to perform time-dimensional super-resolution reconstruction of video images.

(3a) Take any given section of video as a test sample, and pull each video image sample in the video image into a column vector, and the size of each vector is ₁₇₂₈ ×1;

(3b) Using these column vectors as the input of the trained neural network model, for each input vector, the output result of the neural network is a vector with increased dimension, and the size of the vector is 3456×1;

(3c) Reconstruct and combine these vectors, that is, first reconstruct these vectors into single-frame images, and then combine these single-frame images into a video to obtain a time-dimensional super-resolution video.

Effect of the present invention can be specified by following simulation experiments:

1. Simulation conditions:

1) The direct downsampling transformation matrix F in the simulation experiment is obtained by the function imresize;

2) The programming platforms used in the simulation experiment are Matlab R2015a and Pycharm v2016;

3) The neural network structure constructed in the simulation experiment is shown in Figure 2;

4) The 14th frame image of the bus video sequence used in the simulation experiment is as shown in Figure 3;

5) The video images in the video image set used in the simulation experiment come from the Xiph database, with a total of 464814 training samples;

6) In the simulation experiment, the peak signal-to-noise ratio PSNR index is used to evaluate the experimental results, and the peak signal-to-noise ratio PSNR is defined as:

Among them, M represents the number of frames of the reconstructed video image, MAX _j represents the maximum pixel value of the reconstructed jth frame image, MSE _j represents the difference between the reconstructed video image frame j and the original video image frame j mean square error between.

2. Simulation content: adopt the inventive method to carry out time-dimension video super-resolution reconstruction to the bus video image shown in Figure 3, and its reconstruction result is as shown in Figure 4, wherein:

Figure 4(a) shows the 14th frame image reconstructed by Kang's method,

Figure 4(b) shows the 14th frame image reconstructed by Choi’s method,

Fig. 4 (c) represents the 14th frame image reconstructed by the method of the present invention,

As can be seen from the reconstruction results shown in Figure 4, the image reconstructed by the present invention is closer to the real image than the reconstructed image by Kang's method and Choi's method.

3. Peak Signal-to-Noise Ratio PSNR Comparison

Calculate the peak signal-to-noise ratio (PSNR) of the existing Tsai's method, Choi's method and the method of the present invention to carry out the video time super-resolution reconstruction of the bus video image, and the results are shown in Table 1.

Table 1 PSNR value of reconstructed video image (unit: dB)

As can be seen from Table 1, the peak signal-to-noise ratio PSNR of the video image reconstructed by the method of the present invention is 2.99dB higher than the existing Kang's method, and 2.38dB higher than the existing Choi's method.

Claims (5)

1. The time dimension video super-resolution method based on deep learning comprises the following steps:

(1) set S ═ S of color video images₁,S₂,...,S_i,...,S_NConverting into a gray scale video image set, i.e. an original video image set X ═ X₁,X₂,...,X_i,...,X_NDirectly down-sampling the original video image set X by using a down-sampling matrix F to obtain a down-sampling video image set Y ═ Y₁,Y₂,...,Y_i,...,Y_NAnd (c) the step of (c) in which,representing the ith original video image sample,representing the ith down-sampling video image sample, i is more than or equal to 1 and less than or equal to N, N represents the number of image samples in the original video image set, M represents the size of the original video image block, L_hRepresenting the number of image blocks, L, in each sample of the original video image set_lRepresenting the number of image blocks in each sample of a down-sampled video image set, and L_h＝r×L_lAnd r represents the magnification of the original video image set to the downsampled video image set;

(2) constructing a neural network model, and training neural network parameters by using a down-sampling video image set Y and an original video image set X:

(2a) determining the number of nodes of input layer, the number of nodes of output layer, the number of hidden layers and the number of nodes of hidden layer of neural network, and randomly initializing the connection weight W of each layer^(t)And bias b^(t)Given the learning rate η, the activation function is chosen to be:wherein g represents the input value of the neural network node, t is 1,2, …, n, n represents the total number of layers of the neural network;

(2b) randomly inputting a down-sampled video image Y in a down-sampled video image set_iAs an input training sample, simultaneously inputting an original video image X in a corresponding original video image set_iAs an output training sample, calculating an activation value of each layer of the neural network by using the selected activation function, and calculating to obtain:

the activation values of layer 1, the input layer, are: a is⁽¹⁾＝Y_i，

The activation values of the n-th layer are: a is^(t′)＝f(W^(t′-1)*a^(t′-1)+b^(t′-1)) Wherein, in the second layer, the third layer and the fourth layer of the network, i.e. t '2 and t' 3,when t' is 4, in order to sufficiently extract the correlation between video frames, three-dimensional filters are designed to replace the conventional two-dimensional filter, f (g) represents the tan h (g) activation function, and g is W^(t′-1)*a^(t′-1)+b^(t′-1)，W^(t′-1)And b^(t′-1)Respectively represent the weight and offset of the t' -1 th layer, a^(t′^-1)Represents the activation value of the t' -1 th layer;

(2c) calculating the learning error of each layer of the neural network:

the error of the output layer, i.e. the nth layer, is:⁽ⁿ⁾＝X_i-a⁽ⁿ⁾，

the t ″, n-1, n-2, 2-layer error is:^(t′)＝((W^(t”))^{T (t”+1)}).*f'(W^(t”-1)*a^(t”-1)+b^(t”-1)) Wherein W is^(t”)Represents the weight of the t-th layer,^(t″+1)denotes an error of t "+1 layer, W^(t”-1)And b^(t”-1)Respectively representing weight and offset of the t "-1 th layer, a^(t”^-1)Denotes the activation value of the t "-1 th layer, f ' (g ') denotes the derivative of the function f (g '), (g")^TRepresenting a transposed transform, g ═ W^(t”-1)*a^(t”-1)+b^(t”-1)，g”＝W^(t”)；

(2d) Updating the weight and the bias of each layer of the neural network according to an error gradient descent method:

update the weight value to W^(t)＝W^(t)-η^(t+1)(a^(t))^TUpdate the bias to b^(t)＝b^(t)-η^(t+1)Wherein^(t+1)error of t +1 th layer, a^(t)Represents an activation value of the t-th layer;

(2e) repeatedly executing the steps (2b) - (2d) until the error of the output layer of the neural network reaches the preset precision requirement or the training frequency reaches the maximum iteration frequency, finishing the training, and storing the network structure and parameters to obtain a trained neural network model;

(3) and inputting any section of video into the trained neural network model, wherein the output of the neural network is the video after time dimension super resolution.

2. The method of claim 1, wherein the step (1) of converting the original video image set X into the downsampled video image set Y using a downsampled matrix F is performed by multiplying the original video image by the downsampled matrix F:

Y＝FX，

wherein,m denotes the size of the original video image block, L_lRepresenting the number of image blocks in each sample of a down-sampled video image set, L_hRepresenting the number of image blocks in each sample of the original video image set, and L_h＝r×L_lAnd r represents the magnification of the downsampled video image set to the original video image set in the time dimension.

3. The method of claim 1, wherein the number of input layer nodes of the neural network determined in step (2a) is determined according to the number of video frames of the input training sample, i.e. the number of input layer nodes is equal to the number L of image blocks in each sample of the downsampled video image set_l。

4. The method of claim 1, wherein the number of output layer nodes of the neural network determined in step (2a) is determined according to the number of video frames of the output training samples, i.e. the number of output layer nodes is equal to the number L of image blocks in each sample of the original video image set_h。

5. The method of claim 1, wherein the number of hidden layer nodes of the neural network determined in step (2a) is determined by experimental adjustment.