CN110392264A - An Alignment and Extrapolation Frame Method Based on Neural Network - Google Patents

Abstract

The invention discloses frame interpolation methods outside a kind of alignment neural network based, it include: the M block that target frame is divided into specified size, select distance objective frame recently and continuous N number of past frame, for each of target frame piece, most like block is found from each past frame respectively to be aligned, and N number of alignment block is obtained；Wherein, N is the natural number of setting；N number of alignment block is input in multiple dimensioned and residual error network, the difference between learning objective frame and each past frame, and predicts to obtain 1 outer inserted block；All heterodyne blocks are spliced to obtain and are aligned outer interleave.This method handles past frame using alignment operation, reduces the diversity of extrapolation network inputs, can significantly reduce extrapolation difficulty by the difference between learning objective frame and past frame；Furthermore, it is possible to which outer interleave is applied in Video coding, code efficiency is promoted.

Description

An Alignment and Extrapolation Frame Method Based on Neural Network

technical field

The invention relates to the field of digital image processing technology and video coding technology, in particular to a method for aligning and extrapolating frames based on a neural network.

Background technique

As we all know, digital video is composed of a series of digital images (frames) in time sequence. In video, predicting a future frame from a past frame is called an extrapolation frame, that is, predicting the content of the future frame from the content and motion trend of the past frame. With the development of Convolutional Neural Network (CNN), the extrapolation frame method based on neural network has made remarkable progress. A common scenario for extrapolating frames is in video coding, because extrapolating frames can get more accurate predictions of future frames from past frames (reference frames), and when encoding future frames, accurate predictions can reduce the number of frames. Inter-redundancy to improve video coding efficiency.

The following are some works on the use of frame extrapolation techniques in frame extrapolation, inter-frame prediction and video coding:

Deep multi-scale video prediction beyond mean square error (M. Mathieu, C. Couprie, and Y. LeCun, "Deep multi-scale video prediction beyond mean square error," arXiv preprint arXiv:1511.05440, 2015.)

Generative adversarial network-based frame extrapolation for video coding (J.Lin, D.Liu, H.Li, and F.Wu, "Generative adversarial network-based frame extrapolation for video coding," inVCIP.IEEE, 2018.)

Disadvantages of the above method:

1. Due to the complexity and diversity of motion patterns in natural videos, it is still difficult to directly infer high-quality future frames from past frames without processing them.

2. Since the quality of the frames directly inferred from the past frames is not high enough, and it is difficult to deal with the complex motion in the video by the direct extrapolation, the coding efficiency improvement of applying the direct extrapolation method to the video coding is limited.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for aligning extrapolated frames based on a neural network, which improves the quality of extrapolated frames by aligning past frames.

The purpose of this invention is to realize through the following technical solutions:

A neural network-based method for aligning extrapolated frames, including:

Divide the target frame into M blocks of a specified size, select N past frames that are the closest and continuous to the target frame, and for each block in the target frame, find the most similar block from each past frame for alignment, and obtain N alignment blocks; wherein, N and M are both set natural numbers;

For each block in the target frame, input N aligned blocks into the multi-scale and residual network, learn the difference between the target frame and each past frame, and predict an extrapolated block;

All the extrapolation blocks are spliced according to the corresponding positional relationship to obtain the aligned extrapolation frame.

It can be seen from the above technical solutions provided by the present invention that the use of alignment operations to process past frames reduces the diversity of extrapolation network inputs, and the extrapolation difficulty can be significantly reduced by learning the difference between the target frame and the past frames; Apply extrapolation frames to video coding to improve coding efficiency.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is an overall framework diagram of a method for aligning and extrapolating frames based on a neural network according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

An embodiment of the present invention provides a method for aligning and extrapolating frames based on a neural network, including:

1. Divide the target frame into M blocks of a specified size, select N past frames that are the closest and continuous to the target frame, and for each block in the target frame, find the most similar block from each past frame for alignment , to obtain N aligned blocks. Through this step, the correlation between the target frame and the past frame can be increased, and at the same time, the block-level translation between the past frames is completely removed, which greatly reduces the diversity of the target frame and the past frame. The M is a natural number, the number of M is determined according to the resolution of the target frame, the number of blocks with different resolutions is different, and the target frame is also the frame to be predicted.

2. For each block in the target frame, input N aligned blocks into the multi-scale and residual network, learn the difference between the target frame and each past frame, and predict an extrapolated block. This step uses the residual learning method to let the network focus on learning the subtle differences between the target block and the aligned block in the target frame, including complex motion, texture changes, noise, etc., to reduce the difficulty of extrapolation. For each block in the target frame, through the aforementioned step 1, N aligned blocks can be found from N past frames, and these N aligned blocks are input into the network to predict an extrapolated block.

3. Splicing all the extrapolation blocks according to the corresponding positional relationship to obtain an aligned extrapolation frame. That is, if the coordinates of the starting position of the target block in the target frame It are ( _x , y), the coordinates of the starting position of the corresponding extrapolation block in the extrapolation frame are also (x, y).

In the embodiment of the present invention, the block division mode may be consistent with the coding tree unit (CTU) division mode of the coding standard HEVC, and the block size is consistent with the CTU (64×64).

For ease of understanding, the present invention will be further described below with reference to the framework shown in FIG. 1 .

1. Alignment method.

In the embodiment of the present invention, N past frames (Previous frames) are selected, the target frame is denoted as It, and the N past frames closest to the target frame and consecutive are _denoted as It _-1 , It _-2 , ... , I _tN ; wherein, N is a set natural number. Exemplarily, N=4 can be set, then the past frames are It _-1 , It _-2 , It _-3 , It _-4 .

Alignment consists of two steps:

1. The first step is to align the target frame It and the It _{-1 frame} to obtain the alignment block of the It _-1 frame.

As shown in Figure 1, the first step adopts either of the following two methods:

The first method (Scheme 1 in Fig. 1) is: _take the original value of the target block of the target frame It as the template of motion estimation (ME), and perform integer-pixel motion estimation in the It _-1 frame. The block with the smallest value error is used as the alignment block of the It _-1 frame.

Those skilled in the art can understand that a target block refers to a block that is divided in a target frame and is currently being processed, and the original value is a coding term, indicating the original image value of the corresponding target block.

The second method ( _Scheme2 in Figure 1) is: based on the assumption that the motion between the target frame It and the It _-1 frame is less than the set value (that is, the motion is small enough), a fixed position (Co-located) is used. way, the alignment block of the It _-1 frame is obtained, that is, if the starting position coordinates of the target block in the target frame It are (x, y), the alignment block is that the starting position coordinates in the It _-1 frame are ( _x , y) of the image block.

2. The second step is the alignment block based on the It _-1 frame, aligning the target frame It and N past frames It _-1 , It _-2 , ..., _ItN , to obtain the _{It -2} frame to Alignment block for _ItN frames.

No matter which method is used in the first step, the alignment method used in the second step is the same as the alignment method used in the first method in the first step, that is, using the motion estimation method, see also Figure 1:

No matter which method is used in the first step, for the target block in the target frame It, the alignment block of the It _{-1 frame} obtained in the first step is used as the template for motion estimation, and the whole pixel is performed in the It _-2 frame. For motion estimation, the block with the smallest mean absolute value error is used as the alignment block of the It _-2 frame; the execution is repeated in this way, and the alignment block of each past frame is updated to the template of the motion estimation of the previous frame, and finally, the The preliminary alignment block of the It- _(N-1) frame is used as the template for motion estimation, and the whole-pixel motion estimation is performed in the _ItN frame, and the block with the smallest mean absolute value error is used as the alignment block of the _ItN frame; thus, N Align blocks.

The above alignment operation keeps the objects in the target block always in the center of the network input block (ie, the alignment block), avoiding spending a lot of energy searching for the target object inside the network. In the embodiment of the present invention, both the first step and the second step are performed in a motion estimation manner (ie, the first method), which is abbreviated as MEA (ME Alignment). The first step adopts a fixed position method and the second step adopts a motion estimation method (ie, the second method), which is abbreviated as CoIMEA (Co-located MEAlignment).

In addition, in order to avoid the loss of block edge information caused by block segmentation, after N alignment blocks are obtained, the reconstructed pixels around each alignment block are supplemented around the corresponding alignment block, as the final N alignment blocks.

Exemplarily, the target frame is first divided into M 64x64 blocks, and for each divided block, 4 alignment blocks need to be found in 4 (ie N=4) past frames respectively, and finally M groups of alignment block groups are obtained, Each alignment block group contains 4 alignment blocks. For the size of the block, because the target block of the division is 64x64, the alignment is also performed according to the size of 64x64. Before entering the network, after 4 alignment blocks, the reconstructed pixels around each alignment block are filled around the corresponding alignment block. The top, bottom, left and right are supplemented with 32 pixels, and finally four 128x128 aligned blocks are obtained as network input, and the network outputs one 128x128 block, and the 64x64 block in the central part of the block is intercepted as the final extrapolation block, whose size is consistent with the size of the target block .

2. Multi-scale and residual network.

The multi-scale structure can fully capture the motion of different resolutions, and the residual network can learn the difference between the input image patch and the target image patch more efficiently, which is suitable for alignment and extrapolation tasks.

The multi-scale and residual network includes Q scales. In order to facilitate the drawing of the drawings, FIG. 1 takes Q=4 as an example, and schematically shows a network structure including four scales.

In the embodiment of the present invention, the resolution sizes of each scale are sequentially recorded as s ₁ , s ₂ , . . . , s _Q ; the resolution size of the current scale k is half of the next scale k+1, that is, k=1,2,...,Q. When k=Q, s _Q+1 is the set resolution size. Taking Q=4 as an example, the resolutions are set to s ₁ =16, s ₂ =32, s ₃ =64, and s ₄ =128.

The multi-scale and residual network makes a series of predictions starting from the lowest resolution s ₁ , and takes the prediction of size s _k as the starting point to make predictions of size s _k+1 , continuously upsampling, and in the next update. The learned residuals are added to the fine scale (ie, the k+2th scale) until returning to the full-resolution image, which is a 128x128 image in contrast to the previous example.

In the embodiment of the present invention, the network and the extrapolation block prediction made by it are defined recursively:

in, Refers to the extrapolation block predicted at the kth scale. In particular, the network only takes the down-sampled X ₁ as the input of the minimum size, that is, when k=1, with is X ₁ ; X represents N alignment blocks {X ^t-1 , X ^t-2 , ..., X ^tN }, which are the alignment blocks of It _-1 , It _-2 , ..., It _N in sequence , X _k represents the downsampled image of X at resolution _sk , G _k represents a value from X _k and learn residuals The network of , that is, the network of the kth scale in the network, _uk represents the upsampling operation towards the resolution _sk .

The total network loss is the sum of Q scale losses:

in:

||*|| ₁ means l ₁ loss, l ₁ loss is a technical term in deep learning, which means a loss function.

In the training phase of the multi-scale and residual network, the original uncompressed video is used for training. Each time the past N frames of the target frame are selected for block alignment, for each block, the N aligned blocks obtained are used as the network. Input, the original value of the target block is used as the label; in particular, in the alignment process, the alignment method of MEA is used, that is, the ME method is used in the first step of alignment to generate the most accurate alignment block. The way of downsampling the aligned block adopts the way of downsampling by bicubic interpolation. In the network training, only the Y component is used for training, and only one model is obtained. Since the network is a fully convolutional network, the trained model can be used for different resolutions.

Those skilled in the art can understand that YUV contains three channels: Y, U, and V. Looking at each channel separately, a frame of YUV image can be divided into three frames of images (Y image, U image, and V image). The block extrapolation operation is based on a certain channel. For example, perform block alignment and extrapolation on the Y channel, splicing it into a Y channel extrapolation frame, and also perform extrapolation on the U and V components to obtain a U, V extrapolation frame, and combine the Y, U, V channel extrapolation frames. Combined to get the extrapolated image in YUV format. In other words, YUV is three-dimensional, including channel, height, and width. Extrapolation of a component is based on the channel dimension, and the block is in the height and width dimensions.

3. Aligned extrapolation frames applied in video coding

In this embodiment of the present invention, applying the obtained aligned extrapolation frame to video coding includes at least two schemes:

The first is to use the aligned extrapolation frame directly as the result of motion compensation prediction in inter-frame prediction, which is referred to as MCP for short;

The second is to align the extrapolated frame into a new reference frame, and perform traditional motion compensation on the new reference frame. This scheme is referred to as REF for short.

For the first method, MCP, aligned frame extrapolation can be used as a new inter prediction mode, which is only applied to 32x32 and larger coding units (CUs), and this mode is rate-distortion optimized with other inter prediction modes. (RDO), select the optimal mode for encoding. In the code stream structure, for 32x32 and larger CUs, a flag needs to be transmitted to identify whether the CU uses the MCP mode.

For the second method REF, the alignment frame is extrapolated as a new reference frame. Following the traditional coding framework, motion compensation is performed in the existing and extrapolated reference frames so that the encoder can select the most accurate part for the divided prediction units (PUs), with applied PU sizes ranging from 4x4 to 64x64. In particular, set the size of the extrapolation block to be the same as the CTU, i.e. 64x64. Due to input padding, the network output block is twice the size of the CTU, i.e. 128x128, including the resulting padding pixels. To achieve HEVC reference, the entire network output block is used as the reference block for the corresponding CTU position, so that the boundary continuity between the extrapolated reference CTU and surrounding pixels is maintained during the process of motion estimation and motion compensation, and the reference block is repeated The policy then compresses the next CTU. Due to the special format of YUV420, the luminance and chrominance channels are processed separately. The alignment process is only performed on the Y component, and the Y component is first extrapolated by the network; in the extrapolation task, the Y component has the same characteristics as the U and V components, such as the same motion trend, so the U, V components corresponding to the Y component can be used. The V-aligned image is also input to the network model for extrapolation, and the extrapolated images of U and V are obtained respectively. The same method is used to generate extrapolated frames at the encoding end and the decoding end to ensure that the encoding and decoding ends match.

Fourth, the alignment method is combined with two application schemes.

In this embodiment, two alignment methods and two application schemes are cross-combined and integrated into the coding standard HEVC, and the combination schemes are as follows: MCP+ColMEA, MCP+MEA, REF+ColMEA and REF+MEA.

The basic implementation of the MCP scheme and the REF scheme has been introduced above. For the MCP+ColMEA scheme, since ColMEA does not need to transmit any additional information for prediction, and the inter-frame prediction value can be obtained directly from frame extrapolation, it can avoid transmitting the information required to obtain the prediction value in the traditional inter-frame mode, including AMVP mode The motion vector (MV) and reference frame index information (reference index) in the Merge mode and the Merge Flag and Merge Index in the Merge mode.

For the MCP+MEA scheme, it is necessary to transmit the alignment information, multiplex the traditional MV coding module, and use the integer component to represent the alignment information, and the fractional component to perform sub-pixel interpolation on the obtained extrapolation block, which can make full use of the MV structure to generate More accurate inter prediction.

For the REF+ColMEA scheme, since there is no change in any bitstream structure in ColMEA, it is only necessary to follow the traditional coding framework on the basis of adding new reference frames.

For the REF+MEA scheme, the better motion vector prediction (MVP) given by the surrounding PUs is used and is very close to the aligned MV to replace the aligned MV. The transmission of the MVP is more economical than the MV, and only the aligned MVP of the extrapolated blocks referenced by the PU is encoded to further save bits.

The above scheme of the embodiment of the present invention uses the alignment operation to process the past frames, which reduces the diversity of the extrapolation network input, and can significantly reduce the extrapolation difficulty by learning the difference between the target frame and the past frame; in addition, the extrapolation frame can be applied to In video coding, the coding efficiency is improved.

On the other hand, in order to illustrate the performance of the present invention, relevant tests are also carried out, and the extrapolation is mainly applicable to the low-latency configuration in encoding.

The test conditions include: 1) Inter-frame configuration: Low-delay B is Low-delay B, LDB; Low-delay P is Low-delay P, LDP. 2) The basic quantization step size (QP) is set to {27, 32, 37, 42}, the software based on it is HM12.0, and the test sequence is the HEVC standard test sequence.

Those skilled in the art can understand that the coding field needs to test the performance in the specified standard test sequences. The standard test sequences are divided into several categories (Class A to F). In the following test process, the results of the various test sequences are reported according to the standard test requirements.

1) Comparison of 4 combined coding performance between alignment method and extrapolation application

Table 1 shows the performance comparison of the combinations in 4 under the LDP setting. It can be seen from the table that the optimal solution is REF+ColMEA, which can be used as the preferred solution.

Table 1 Performance comparison results of 4 combinations under LDP setting

Table 2 shows the full performance of the preferred scheme. It can be seen from Table 2 that, compared with HM12.0, the above solution in the embodiment of the present invention can obtain code rate savings of 5.3% and 2.8% in LDP and LDB modes, respectively.

Table 2 Complete performance of the preferred scheme

2) Align extrapolation frame effect test

Table 3 shows the comparison of the coding performance between the traditional frame extrapolation without alignment and the aligned extrapolation frame proposed by the present invention under LDP conditions in the preferred solution, it can be seen that the extrapolation without alignment can only obtain a code rate saving of 2.2% , while aligned extrapolation yields 5.3% rate savings. Compared with the simple extrapolation frame, the alignment proposed by the present invention can significantly improve the coding performance. In addition, it is found through observation that the extrapolated frame generated by the alignment method proposed in the present invention is clearer and has better effect than the traditional method.

Table 3 Aligned extrapolation frame effects

From the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the above embodiments may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.), including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (9)

1. an alignment extrapolation frame method based on neural network, is characterized in that, comprises: Divide the target frame into M blocks of a specified size, select N past frames that are the closest and continuous to the target frame, and for each block in the target frame, find the most similar block from each past frame for alignment, and obtain N alignment blocks; wherein, N and M are both set natural numbers; For each block in the target frame, input N aligned blocks into the multi-scale and residual network, learn the difference between the target frame and each past frame, and predict an extrapolated block; All the extrapolation blocks are spliced according to the corresponding positional relationship to obtain the aligned extrapolation frame. 2. a kind of alignment extrapolation frame method based on neural network according to claim 1, it is characterised in that the selection is the nearest and continuous N past frames from the target frame, for each block in the target frame, Find the most similar blocks from each past frame for alignment, and the steps to obtain N aligned blocks include: The target frame is denoted as It, and the N nearest and consecutive past frames from the target frame are denoted as It _-1 , _{It -2} , ..., _ItN ; the alignment includes two steps: The first step is to align the target frame It and the It _{-1 frame} to obtain the alignment block of the It _-1 frame; The second step is the alignment block based on the It _-1 frame, aligning the target frame It and N past frames It _-1 , It _-2 , . . . , _ItN to obtain the _{It -2} frame to _ItN The alignment block of the frame. 3. a kind of alignment extrapolation frame method based on neural network according to claim 2 is characterized in that, the first step adopts any one of the following two modes: The first way is: take the original value of the target block of the target frame It as the template for motion estimation, perform whole-pixel motion estimation in the It _{-1 frame} , and take the block with the smallest average absolute value error as the It _-1 frame the alignment block; The second method is: based on the assumption that the motion between the target frame It and the It _{-1 frame} is smaller than the set value, the alignment block of the It _-1 frame is obtained by adopting a fixed position method. 4. a kind of alignment extrapolation frame method based on neural network according to claim 2 or 3, is characterized in that, the alignment mode used in the second step comprises: For the target block in the target frame It, the alignment block of the It _{-1 frame} obtained in the first step is used as the template for motion estimation, and the whole-pixel motion estimation is performed in the It _-2 frame, and the average absolute value error is the smallest. The block is used as the alignment block of the It _-2 frame; the execution is repeated in this way, and finally, the initial alignment block of the It- _(N-1) frame is used as the template for motion estimation, and the whole-pixel motion estimation is performed in the _ItN frame. , the block with the smallest average absolute value error is taken as the alignment block of the _ItN frame; thus N alignment blocks are obtained. 5. a kind of alignment extrapolation frame method based on neural network according to claim 4 is characterized in that, after obtaining N alignment blocks, the reconstruction pixels around each alignment block are respectively filled around the corresponding alignment blocks, as the final of N aligned blocks. 6. A neural network-based method for aligning and extrapolating frames according to claim 5, wherein the multi-scale and residual network includes Q scales, and the resolution is sequentially denoted as s ₁ , s ₂ , ..., s _Q ; the resolution of the current scale k is half of the next scale k+1, that is k=1,2,...,Q; The multi-scale and residual network makes a series of predictions starting from the lowest resolution s ₁ , and takes the prediction of size s _k as the starting point, makes predictions of size s _{k + 1} , continuously upsampling, and in the next update. The learned residuals are added at fine scales until back to the full resolution image, i.e. the same size as the final alignment block. 7. a kind of alignment extrapolation frame method based on neural network according to claim 6, is characterized in that, the extrapolation block prediction result of described multi-scale and residual network is: in, Refers to the extrapolation block predicted at the kth scale, and the network downsampled X ₁ as the input of the minimum size, that is, when k=1, with is X ₁ ; X represents N alignment blocks {X ^t-1 , X ^t-2 , ..., X ^tN }, which are the alignment blocks of It _-1 , It _-2 , ..., It _N in sequence , X _k represents the downsampled image of X at resolution _sk , G _k represents a value from X _k and learn residuals The network of , _uk represents the upsampling operation towards the resolution _sk ; The total network loss is the sum of Q scale losses: in: ||*|| ₁ means l ₁ loss. 8. A neural network-based alignment extrapolation frame method according to claim 4, wherein the method further comprises: applying the obtained alignment extrapolation frame in video coding, including at least two schemes: The first is to use the aligned extrapolation frame directly as the motion compensation prediction result of inter-frame prediction, which is referred to as MCP for short; The second is to align the extrapolated frame into a new reference frame, and perform traditional motion compensation on the new reference frame. This scheme is referred to as REF for short. 9. A neural network-based method for aligning and extrapolating frames according to claim 8, wherein the method further comprises: combining the alignment method with two application schemes; The alignment method includes: both the first step and the second step adopt a motion estimation method, referred to as MEA for short; the first step adopts a fixed position method and the second step adopts a motion estimation method, referred to as ColMEA for short; The two alignment methods and the two application schemes are cross-combined and integrated into the coding standard HEVC. The combination schemes are as follows: MCP+ColMEA, MCP+MEA, REF+ColMEA and REF+MEA.