CN110177282A - A kind of inter-frame prediction method based on SRCNN - Google Patents

Abstract

The invention discloses a kind of inter-frame prediction methods based on SRCNN, it is characterised in that carries out inter-prediction to image sequence using super-resolution convolutional neural networks；It takes exercises after estimation and operation of motion compensation to image sequence, trains characteristic model in conjunction with super-resolution convolutional neural networks；Super-resolution rebuilding is carried out to image using the parameter in model, while Motion estimation and compensation is carried out to image, obtains the consistent image of next frame image with present image.Deep learning is applied to the inter-prediction of Video coding by the present invention, using convolutional neural networks, carries out feature extraction to estimation, the operation of motion compensation image sequence and training learns.Meanwhile using super-resolution neural network, in image reconstruction, the image quality of image can be enhanced.

Description

An Inter-Frame Prediction Method Based on SRCNN

technical field

The invention belongs to inter-frame prediction in the field of video coding, mainly for improving video transmission efficiency, and specifically relates to an SRCNN-based inter-frame prediction method.

Background technique

Super-resolution (Super-Resolution) means converting a low-resolution (Low Resolution) image into a high-resolution (High Resolution) image, usually improving image quality and clarity. Super-Resolution Convolutional Neural Network (SRCNN) is a convolutional neural network applied to image super-resolution reconstruction. By extracting the features of image blocks and nonlinearly mapping the features, a high-resolution image can be reconstructed. image. Since this convolutional neural network was proposed, it has been widely used, and its accuracy and reliability have been well verified.

In today's information age, the research and statistical data of scientists show that about 75% of the information obtained by humans from the outside world is obtained through the eyes, and the information obtained by the eyes is converted into images through the visual system and transmitted to brain. With the rapid improvement of today's living standards, people have higher and higher requirements for image and video quality. The continuous improvement of image and video resolution also brings huge challenges to information transmission. Clearer images and videos mean larger data volumes and higher transmission rates. In order to ensure people's viewing comfort, the frame rate of movies and other videos is generally higher than 24 frames per second. If the image of each frame is saved and played frame by frame, not only the hard disk capacity is particularly high, but also the playback There are huge challenges in the transmission and display rate of the device. If the video is played in this way, due to the limitation of the transmission rate, there will be no high-definition videos such as 2K and 4K. Video coding technology has largely eliminated the redundancy between image sequences, greatly compressing the data volume of video, combined with existing hardware technology, allowing ultra-high-definition video to enter people's lives, to a great extent satisfying people's needs perception needs.

Inter-frame prediction is the most important part of video coding. It uses the correlation between video image frames, that is, time correlation, to achieve the purpose of image compression. It is widely used in ordinary TV, conference TV, video phone, high-definition Compression coding for television. In image transmission technology, moving images, especially TV images, are the main objects of concern. A moving image is a temporal image sequence composed of consecutive image frames temporally spaced by a frame period, and it has greater correlation in time than in space. The detail changes between adjacent frames of most TV images are very small, that is, there is a strong correlation between video image frames, and inter-frame coding can be obtained by using the correlation characteristics of frames, which is much higher than intra-frame coding. the compression ratio.

In inter-frame predictive coding, there is a certain correlation between the scenes in the adjacent frames of the moving image. Therefore, the active image can be divided into several blocks or macroblocks, and the position of each block or macroblock in the adjacent frame image can be searched, and the relative offset of the spatial position between the two can be obtained. The offset is usually referred to as a motion vector, and the process of obtaining the motion vector is called motion estimation. The motion vector and the prediction error obtained after motion matching are sent to the decoding end together. At the decoding end, according to the position indicated by the motion vector, the corresponding block or macroblock is found from the decoded adjacent reference frame image, and the prediction error is added. The position of the block or macroblock in the current frame is obtained. The inter-frame redundancy can be removed through motion estimation, which greatly reduces the number of bits in video transmission. Therefore, motion estimation is an important part of the video compression processing system. This section starts with the general method of motion estimation, focusing on three key issues of motion estimation: parameterizing the motion field, defining the optimal matching function, and how to find the optimal matching.

Contents of the invention

The purpose of the present invention is to propose an inter-frame prediction method based on SRCNN, which is different from the mainstream HEVC video encoding method. The present invention aims at inter-prediction of image sequences using super-resolution convolutional neural networks. After performing motion estimation and motion compensation operations on the image sequence, a feature model is trained with a super-resolution convolutional neural network. Using the parameters in the model, the super-resolution reconstruction of the image can be performed, and the motion estimation and motion compensation can be performed on the image at the same time, so as to obtain an image that is basically consistent with the next frame of the current image.

The technical solution adopted by the present invention to solve its technical problems comprises the steps:

Step 1: collect a large amount of video files of different scenes, and compress the video according to different quantization parameters (QP);

Step 2: Extract the image sequence from the video. When extracting the image sequence, the time interval between the two frames of images before and after is set to t, and t<0.1 second;

Step 3: Partition the image sequence into a validation set. Read the remaining images frame by frame. Except for the first frame of the read image sequence, each image uses the current frame and the previous frame, calculates the residual between the two frames, and combines the previous frame with the residual. Perform motion compensation on it to obtain the predicted frame of the previous frame image. Save the calculated predicted frame sequence, divide the predicted frame image sequence to obtain a training set and a test set, and the ratio of the two is 4:1.

Step 4: Input the training set and test set, set the appropriate hyperparameters, and use the super-resolution convolutional neural network (SRCNN) to train the parameter model;

Step 5: Calculate the peak signal-to-noise ratio (PSNR) of the i-th frame image and the i+1-th frame in each image sequence in the verification set, denoted as PSRN1; read the parameters in the parameter model to the obtained image sequence Process the i-th frame image to obtain a reconstructed image I; calculate the PSNR between the reconstructed image I and the i-th frame image of the image sequence in the verification set, denoted as PSNR2;

Compare the calculated two PSNR values, if PSNR2≥PSNR1, the model is considered valid;

If PSNR2<PSNR1, it is considered that the model effect is not good; record ERR=PSNR1-PSNR2; if ERR<5, it is considered that there is a problem with the training hyperparameter setting, return to step 4, adjust the hyperparameter of the learning rate, and then retrain the parameter model; If ERR ≥ 5, it is considered that the division strategy of the data set is problematic, return to step 3, expand the data set to include more scenes, re-divide the training set and test set, and then perform training and verification;

If the difference between the two images is large and the PSNR value exceeds the minimum preset threshold, adjust the training set and test set;

If the difference between the two images is small and the PSNR value is between the optimal preset threshold and the lowest preset threshold, return to step 4 to adjust the parameters of the super-resolution convolutional neural network and retrain the parameter model.

The specific implementation of the image reconstruction using the parameter model is as follows:

1. Convert the input low-resolution image to the YCbCr color space to take the grayscale image as the input i of the image reconstruction operation. Downsample the image i, and set the downsampling step size to k to obtain a low-dimensional image;

2. Use bicubic interpolation on the low-dimensional image to enlarge it to the target size, which is the size of the input low-resolution image;

3. Read the parameters in the parameter model, including the weight and bias of each network node. Perform nonlinear mapping on the interpolated image through a three-layer convolutional network, and obtain the reconstructed result, image I;

4. Convert the image I back to an RGB color map to obtain a reconstructed high-resolution image.

The beneficial effects of the present invention are as follows:

The innovation of the present invention lies in the application of deep learning to the inter-frame prediction of video coding, and the use of convolutional neural networks to perform feature extraction and training learning on motion estimation and motion compensation operations between image sequences. At the same time, using the super-resolution neural network, the quality of the image will be enhanced when the image is reconstructed.

Description of drawings

Figure 1 is a schematic diagram of a super-resolution convolutional neural network SRCNN;

Figure 2 is a flowchart of the implementation of the present invention.

Detailed ways

The present invention mainly focuses on the algorithm innovation of the inter-frame prediction method in video coding, and introduces the training process of the entire model in detail. The specific implementation steps of the present invention will be described in detail below in conjunction with the accompanying drawings. The purpose and effect of the present invention will change more obvious.

Figure 1 is a schematic diagram of the super-resolution convolutional neural network SRCNN. It can be clearly seen from the figure that the convolutional neural network has a simple structure and can enhance the image quality through nonlinear mapping and image reconstruction. Using this network, the resolution of the image can be improved while performing inter-frame prediction on the image sequence.

Fig. 2 is the implementation flowchart of the present invention, and wherein concrete operation comprises:

1. Collect a large number of video files in YUV format, including various scenes.

2. Use different quantization parameters to compress video files. The higher the quantization parameter, the higher the degree of compression. The main focus is on the compression ratio of the quantization parameter between 28 and 42.

3. Extract image sequences from video files, and extract different numbers of images according to videos of different durations to ensure consistent intervals between image sequences. In order to ensure that there is not much change between the two frames of images before and after, the time interval for extracting images should be set very small, specifically according to the length of the video.

4. Perform motion estimation and motion compensation for each extracted image. This operation is specifically to input the current frame and the next frame image, and perform motion estimation and motion compensation for the current frame by comparing the two frames of images.

5. Using the processed image sequence, organize the training set and test set. The verification set required to verify the model requires an image sequence that has not been motion estimated or compensated.

6. Input the training set and test set, set appropriate parameters, and use the super-resolution convolutional neural network SRCNN to train the model.

7. Verify whether the trained model is valid. By comparing the original extracted next frame image with the image reconstructed using the model parameters, if there is little difference between the two images, the model can be considered valid. If there are obvious differences between the two images, adjustments should be made according to different situations. If the difference between the two images is very large, you need to adjust the data set and retrain the model. If the difference between the two images is not very large, you need to improve the imaging effect, adjust the network parameters, and retrain the model with composite requirements.

When comparing the generated image with the next frame of the original image, it is necessary to combine visual subjective judgment with objective numerical analysis. Subjectively, by observing two frames of images with the naked eye, if there is little difference between the two images, the model can be considered to be valid subjectively. However, since there is not much difference between the front and rear frame images, it is necessary to use mathematical tools to compare the two images. It can be used to objectively evaluate the reconstruction effect, that is, the peak signal-to-noise ratio (PSNR). PSNR is an objective standard for evaluating images. The formula is as follows:

Among them, MSE is mean square error (Meansquarederror). Calculate the PSNR values between the original image and its next frame image, the original image and the reconstructed image respectively. If the two values are close, it means that the model works well, and basically reconstructs the same picture as the next frame of the original image. . If the PSNR value of the latter is higher, it can be considered that the program improves the image quality while performing inter-frame prediction on the image.

With the help of PSNR, the accuracy of the model can be verified again objectively, so as to reduce the workload and ensure the effective implementation of the scheme.

Claims (3)

1. An inter-frame prediction method based on SRCNN, characterized in that it uses a super-resolution convolutional neural network to perform inter-frame prediction on an image sequence; after performing motion estimation and motion compensation operations on the image sequence, it combines the super-resolution convolutional neural network The network trains a feature model; uses the parameters in the model to perform super-resolution reconstruction on the image, and at the same time performs motion estimation and motion compensation on the image to obtain an image that is consistent with the next frame of the current image. 2. a kind of inter-frame prediction method based on SRCNN according to claim 1, is characterized in that concrete realization comprises the steps: Step 1: Collect a large number of video files of different scenes, and compress the video according to different quantization parameters; Step 2: Extract the image sequence from the video. When extracting the image sequence, the time interval between the two frames of images before and after is set to t, and t<0.1 second; Step 3: Divide the part of the image sequence into a verification set; read the remaining image sequence frame by frame, except for the first frame of the read image sequence, each image uses the current frame and the previous frame to calculate the difference between the two frame images The residual error of the previous frame image is combined with this residual error, and motion compensation is performed on it to obtain the predicted frame of the previous frame image; the calculated predicted frame image sequence is saved, and the predicted frame image sequence is divided to obtain the training set and the test set, the ratio of the two is 4:1; Step 4: Input the training set and test set, set the hyperparameters, and use the super-resolution convolutional neural network to train the parameter model; Step 5: Calculate the peak signal-to-noise ratio (PSNR) of the i-th frame image and the i+1-th frame in each image sequence in the verification set, denoted as PSRN1; read the parameters in the parameter model to the obtained image sequence Process the i-th frame image to obtain a reconstructed image I; calculate the PSNR between the reconstructed image I and the i-th frame image of the image sequence in the verification set, denoted as PSNR2; Compare the calculated two PSNR values, if PSNR2≥PSNR1, the model is considered valid; If PSNR2<PSNR1, it is considered that the model effect is not good; record ERR=PSNR1-PSNR2; if ERR<5, it is considered that there is a problem with the training hyperparameter setting, return to step 4, adjust the hyperparameter of the learning rate, and then retrain the parameter model; If ERR ≥ 5, it is considered that the division strategy of the data set is problematic, return to step 3, expand the data set to make the data set include more scenes, and re-divide the training set and test set for training and verification; If the difference between the two images is large and the PSNR value exceeds the minimum preset threshold, adjust the training set and test set; If the difference between the two images is small and the PSNR value is between the optimal preset threshold and the lowest preset threshold, return to step 4 to adjust the parameters of the super-resolution convolutional neural network and retrain the parameter model. 3. a kind of interframe prediction method based on SRCNN according to claim 2, is characterized in that described use parametric model to reconstruct image concrete realization as follows: 1. Convert the input low-resolution image to the YCbCr color space to take the grayscale image as the input image i of the image reconstruction operation; down-sample the input image i, and set the step size of the down-sampling to k to obtain a low-dimensional image ; 2. Use bicubic interpolation on the low-dimensional image to enlarge it to the target size, which is the size of the input low-resolution image; 3. Read the parameters in the parameter model, including the weights and offsets of each network node; use a three-layer convolutional network to perform nonlinear mapping on the interpolated image to obtain the reconstructed image I; 4. Convert the image I back to an RGB color map to obtain a reconstructed high-resolution image.