CN110392264B - Alignment extrapolation frame method based on neural network - Google Patents

Abstract

The invention discloses an alignment extrapolation frame method based on a neural network, which comprises the following steps: dividing a target frame into M blocks with a specified size, selecting N continuous past frames closest to the target frame, and respectively finding out the most similar blocks from each past frame for alignment of each block in the target frame to obtain N aligned blocks; wherein N is a set natural number; inputting the N alignment blocks into a multi-scale and residual error network, learning the difference between a target frame and each past frame, and predicting to obtain 1 extrapolation block; and splicing all heterodyne blocks to obtain an aligned interpolation frame. The method uses the alignment operation to process the past frame, reduces the diversity of the extrapolation network input, and can obviously reduce the extrapolation difficulty by learning the difference between the target frame and the past frame; in addition, the extrapolation frame can be applied to video coding, and the coding efficiency is improved.

Description

A Neural Network-Based Method for Aligning and Extrapolating Frames

technical field

The invention relates to the fields of digital image processing technology and video coding technology, in particular to a neural network-based method for aligning and extrapolating frames.

Background technique

As we all know, digital video is composed of a series of digital images (frames) in time order. In video, predicting future frames from past frames is called extrapolation, which means predicting the content of future frames from the content and motion trend of past frames. With the development of Convolutional Neural Network (CNN), neural network-based extrapolation frame methods have made remarkable progress. A commonly used scenario for extrapolated frames is in video coding, because extrapolated frames can obtain more accurate predictions for future frames from past frames (reference frames), and when encoding future frames, accurate predictions can reduce frame Redundancy, improve video coding efficiency.

The following are some works investigating the use of frame extrapolation techniques in frame extrapolation, inter-frame prediction, and video coding:

Deep multi-scale video prediction beyond mean square error (M. Mathieu, C. Couprie, and Y. LeCun, "Deepmulti-scale video prediction beyond mean square error," arXiv preprint arXiv:1511.05440, 2015.)

Frame extrapolation based on generative adversarial network is applied to video coding (J.Lin, D.Liu, H.Li, and F.Wu, "Generative adversarial network-basedframe extrapolation for video coding," inVCIP.IEEE, 2018.)

Disadvantages of the above method:

1. Due to the complexity and diversity of motion patterns in natural videos, it is still difficult to directly infer high-quality future frames from past frames without processing past frames.

2. Since the quality of the frame directly inferred from the past frames is not high enough, and it is difficult to deal with the complex motion in the video through direct extrapolation, applying the direct extrapolation method to video coding has limited improvement in coding efficiency.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a neural network-based method for aligning extrapolated frames, which improves the quality of extrapolated frames by aligning past frames.

The purpose of the present invention is achieved through the following technical solutions:

A neural network-based method for aligning extrapolated frames, comprising:

Divide the target frame into M blocks of a specified size, select the nearest and continuous N past frames from the target frame, and for each block in the target frame, find the most similar block from each past frame for alignment, and obtain N alignment blocks; wherein, N and M are both set natural numbers;

For each block in the target frame, input N aligned blocks into the multi-scale and residual network, learn the difference between the target frame and each past frame, and predict an extrapolated block;

All the extrapolation blocks are spliced according to the corresponding positional relationship to obtain an aligned extrapolation frame.

It can be seen from the above-mentioned technical solution provided by the present invention that using the alignment operation to process past frames reduces the diversity of extrapolation network inputs, and the difficulty of extrapolation can be significantly reduced by learning the difference between the target frame and past frames; in addition, it can Apply extrapolated frames to video coding to improve coding efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative work.

FIG. 1 is an overall framework diagram of a neural network-based method for aligning and extrapolating frames provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

An embodiment of the present invention provides a method for aligning extrapolated frames based on a neural network, including:

1. Divide the target frame into M blocks of a specified size, select N past frames closest to the target frame and continuous, and for each block in the target frame, find the most similar block from each past frame for alignment , to obtain N aligned blocks. Through this step, the correlation between the target frame and the past frame can be increased, and at the same time, the block-level translation between the past frames is completely removed, greatly reducing the diversity of the target frame and the past frame. The M is a natural number, and the number of M is determined according to the resolution of the target frame. The number of blocks with different resolutions is different, and the target frame is also the frame to be predicted.

2. For each block in the target frame, input N aligned blocks into the multi-scale and residual network, learn the difference between the target frame and each past frame, and predict an extrapolated block. This step uses the method of residual learning to let the network focus on learning the subtle differences between the target block and the alignment block in the target frame, including complex motion, texture changes, noise, etc., to reduce the difficulty of extrapolation. For each block in the target frame, N alignment blocks can be found from N past frames through the aforementioned step 1, and these N alignment blocks are input into the network to predict an extrapolation block.

3. Splicing all the extrapolation blocks according to the corresponding positional relationship to obtain an aligned extrapolation frame. That is, if the starting position coordinates of the target block in the target frame I _t are (x, y), the starting position coordinates of the corresponding extrapolation block in the extrapolating frame are also (x, y).

In the embodiment of the present invention, the block division method may be consistent with the coding tree unit (CTU) division method of the coding standard HEVC, and the size of the block is consistent with the CTU (64x64).

For ease of understanding, the present invention will be further described below in conjunction with the framework shown in FIG. 1 .

1. Alignment method.

In the embodiment of the present invention, N past frames (Previous frames) are selected, the target frame is marked as I _t , and the N consecutive past frames closest to the target frame are marked as I _t-1 , It _-2 ,  … , I _tN ; wherein, N is a set natural number. Exemplarily, N=4 can be set, then the past frames are I _t-1 , It _-2 , It _-3 , It _-4 .

Alignment consists of two steps:

1. The first step is to align the target frame I _t with the I _t-1 frame to obtain the alignment block of the I _t-1 frame.

As shown in Figure 1, the first step adopts either of the following two methods:

The first way (Scheme 1 in Figure 1) is: take the original value of the target block in the target frame I _t as the template for motion estimation (ME), perform integer pixel motion estimation in the I _t-1 frame, and average absolute The block with the smallest value error is used as the alignment block of the I _t-1 frame.

Those skilled in the art can understand that the target block refers to the block that is separated from the target frame and is currently being processed, and the original value is a coding term, representing the original image value of the corresponding target block.

The second method (Scheme2 in Figure 1) is: based on the assumption that the motion between the target frame I _t and I _t-1 frame is smaller than the set value (that is, the motion is small enough), use a fixed position (Co-located) In this way, the alignment block of the I _t-1 frame is obtained, that is, if the starting position coordinates of the target block in the target frame I _t are (x, y), the alignment block is that the starting position coordinates in the I _t-1 frame are (x , y) image block.

2. The second step is to align the target frame I _t with N past frames I _t-1 , I _t-2 , ..., I _tN based on the alignment block of the I _t-1 frame, and obtain the I _t-2 frame to Alignment blocks for I _tN frames.

Regardless of which method is used in the first step, the alignment method used in the second step is the same as the alignment method used in the first method in the first step, that is, the motion estimation method is used for alignment, see also Figure 1:

Regardless of which method is used in the first step, for the target block in the target frame I _t , the aligned block of the I _t-1 frame obtained in the first step is used as a template for motion estimation, and the whole pixel is performed in the I _t-2 frame Motion estimation, the block with the smallest average absolute value error is used as the alignment block of the I _t-2 frame; in this way, the alignment block of each past frame is updated to the template of the motion estimation of the previous frame, and finally, the The preliminary alignment block of the It- _(N-1) frame is used as a template for motion estimation, and the whole pixel motion estimation is carried out in the _ItN frame, and the block with the smallest average absolute value error is used as the alignment block of the _ItN frame; thereby obtaining N Align blocks.

The above alignment operation keeps the object in the target block always at the center of the network input block (ie, the alignment block), avoiding spending a lot of energy searching for the target object inside the network. In the embodiment of the present invention, both the first step and the second step adopt motion estimation (that is, the first method), which is referred to as MEA (ME Alignment). The first step adopts the method of fixed position and the second step adopts the method of motion estimation (that is, the second method), which is referred to as CoIMEA (Co-locatedMEAlignment).

In addition, in order to avoid the loss of block edge information caused by block segmentation, after obtaining N aligned blocks, the reconstructed pixels around each aligned block are respectively supplemented around the corresponding aligned blocks, as the final N aligned blocks.

Exemplarily, the target frame is first divided into M blocks of 64x64, and for each divided block, it is necessary to find 4 aligned blocks in 4 (ie N=4) past frames, and finally obtain M groups of aligned blocks, Each alignment block group contains 4 alignment blocks. For the size of the block, because the target block is 64x64, the alignment is also carried out according to the size of 64x64. Before inputting the network, after the 4 alignment blocks, the reconstructed pixels around each alignment block are respectively supplemented around the corresponding alignment block. 32 pixels are added up, down, left, and right, and finally four 128x128 aligned blocks are obtained as network input, and the network outputs a 128x128 block, and the 64x64 block in the center of the block is intercepted as the final extrapolation block, whose size is consistent with the target block size .

2. Multi-scale and residual network.

The multi-scale structure can fully capture the motion of different resolutions, and the residual network can learn the difference between the input image block and the target image block more efficiently, which is suitable for the alignment extrapolation task.

The multi-scale and residual network includes Q scales. In order to facilitate the drawing of the drawings, FIG. 1 takes Q=4 as an example to schematically show a network structure including 4 scales.

k＝1，2，...，Q。当k＝Q时，s_Q+1为设定的分辨率大小。以Q＝4为例，分辨率设为s₁＝16，s₂＝32，s₃＝64，s₄＝128。In the embodiment of the present invention, the resolutions of each scale are sequentially recorded as s ₁ , s ₂ , ..., s _Q ; the resolution of the current scale k is half of the next scale k+1, namely

k=1, 2, . . . , Q. When k=Q, s _Q+1 is the set resolution size. Taking Q=4 as an example, the resolutions are set to s ₁ =16, s ₂ =32, s ₃ =64, and s ₄ =128.

The multi-scale and residual network starts from the lowest resolution s ₁ to make a series of predictions, and takes the prediction of s _k size as the starting point to make predictions of s _k+1 size, continuously upsampling, and in the next update Add the learned residuals on the fine scale (ie the k+2th scale) until returning to the full-resolution image. Compared with the previous example, the full-resolution image here is also a 128x128 image.

In the embodiment of the present invention, the network and its extrapolation block prediction are defined recursively:

是指第k个尺度预测到的外插块，特别的，网络仅以下采样的X₁作为最小尺寸的输入，即，k＝1时，以

为X₁；X表示N个对齐块{X^t-1，X^t-2，...，X^t-N}，依次为I_t-1，I_t-2，...，I_t-N的对齐块，X_k表示X在分辨率s_k的下采样图像，

G_k表示一个从X_k和

学习出残差

的网络，也即，网络中的第k个尺度的网络，u_k表示朝着分辨率s_k的上采样操作。in,

Refers to the extrapolation block predicted by the kth scale. In particular, the network only takes the downsampled X ₁ as the input of the minimum size, that is, when k=1, the

is X ₁ ; X represents N alignment blocks {X ^t-1 , X ^t-2 , ..., X ^tN }, which are the alignment blocks of I _t-1 , I _t-2 , ..., I _tN , X _k represents the downsampled image of X at resolution s _k ,

G _k represents a from X _k and

learn residuals

The network of , that is, the network of the k-th scale in the network, u _k denotes an upsampling operation towards resolution s _k .

The total network loss is the sum of Q scale losses:

in:

||*|| ₁ means l ₁ loss, l ₁ loss is a professional term in deep learning, which means a loss function.

In the training phase of the multi-scale and residual network, the original uncompressed video is used for training. Each time, the past N frames of the target frame are selected for block alignment. For each block, the obtained N alignment blocks are used as the network Input, the original value of the target block is used as the label; in particular, in the alignment process, the MEA alignment method is used, that is, the ME method is used in the first step of the alignment to generate the most accurate alignment block. The alignment block downsampling method adopts the bicubic interpolation (Bicubic) downsampling method. In the network training, only the Y component is used for training, and only one model is obtained. Since the network is a fully convolutional network, the trained model can be used for different resolutions.

Those skilled in the art can understand that YUV includes three channels of Y, U, and V. If each channel is viewed separately, a frame of YUV image can be divided into 3 frames of images (Y image, U image, V image), as described above The block extrapolation operation is based on a certain channel. For example, block-aligned extrapolation is performed on the Y channel, and the extrapolation frame of the Y channel is spliced into a Y channel extrapolation frame, and extrapolation is also performed on the U and V components to obtain U and V extrapolation frames, and the extrapolation frames of the Y, U, and V channels are Combine them to get an extrapolated image in YUV format. In other words, YUV is three-dimensional, including channels, heights, and widths. The extrapolation of a certain component is based on the channel dimension, and the block is in the height and width dimensions.

3. Aligned extrapolated frames are applied in video coding

In the embodiment of the present invention, applying the obtained aligned extrapolated frames to video coding includes at least two schemes:

The first is to use the aligned extrapolated frame directly as the motion compensation prediction result in inter-frame prediction, which is referred to as MCP for short;

The second is to align the extrapolated frame into a new reference frame, and perform traditional motion compensation on the new reference frame. This scheme is referred to as REF.

For the first method, MCP, aligned frame extrapolation can be used as a new inter-frame prediction mode, which is only applied to 32x32 and larger coding units (CUs), and this mode is optimized for rate distortion with other inter-frame prediction modes (RDO), select the optimal mode for encoding. In terms of code stream structure, for 32x32 and larger CUs, a flag needs to be transmitted to indicate whether the CU uses MCP mode.

For the second method, REF, the aligned frame is extrapolated as a new reference frame. Motion compensation is performed in existing reference frames and extrapolated reference frames following a traditional coding framework so that the encoder can select the most accurate part for a split prediction unit (PU), with applied PU sizes ranging from 4x4 to 64x64. In particular, set the size of the extrapolation block to be the same as the CTU, i.e. 64x64. Due to the input padding, the network output block is twice the size of the CTU, i.e. 128x128, including the generated padding pixels. In order to achieve HEVC reference, the entire network output block is used as the reference block of the corresponding CTU position, so that the boundary continuity between the reference CTU generated by extrapolation and surrounding pixels is maintained during the process of motion estimation and motion compensation, and the reference block is repeated The policy then compresses the next CTU. Due to the special format of YUV420, the luminance and chrominance channels are processed separately. The alignment process is only carried out on the Y component, and the Y component is first extrapolated by the network; in the extrapolation task, the Y component has the same characteristics as the U and V components, such as the same movement trend, so the U, V components corresponding to the Y component can be The V aligned image is also input to the network model for extrapolation, and the extrapolated images of U and V are obtained respectively. Use the same method to generate extrapolated frames at the encoding and decoding ends to ensure that the encoding and decoding ends match.

4. The alignment method is combined with two application schemes.

In this embodiment, two alignment methods and two application schemes are cross-combined and integrated into the encoding standard HEVC. The combination schemes are as follows: MCP+ColMEA, MCP+MEA, REF+ColMEA and REF+MEA.

The basic implementation of the MCP scheme and the REF scheme has been introduced above. For the MCP+ColMEA scheme, since the prediction of ColMEA does not need to transmit any additional information, and the inter-frame prediction value can be directly obtained from frame extrapolation, it can avoid the transmission of the information required to obtain the prediction value in the traditional inter-frame mode, including AMVP mode Motion vector (MV) and reference frame index information (reference index) under Merge mode and Merge Flag and Merge Index under Merge mode.

For the MCP+MEA scheme, the alignment information needs to be transmitted, the traditional MV coding module is reused, and the integer component is used to represent the alignment information, and the fractional component represents the sub-pixel interpolation of the obtained extrapolation block, which can make full use of the MV structure to generate More accurate inter prediction.

For the REF+ColMEA scheme, since there is no change in any bitstream structure in ColMEA, it only needs to follow the traditional coding framework on the basis of adding new reference frames.

For the REF+MEA scheme, better motion vector prediction (MVP) given by surrounding PUs is used and very close to aligned MVs to replace aligned MVs. MVPs are more economical to transmit than MVs, and only aligned MVPs with PU-referenced extrapolated blocks are coded for further bit savings.

In the above scheme of the embodiment of the present invention, the alignment operation is used to process the past frames, which reduces the diversity of extrapolation network input, and the difficulty of extrapolation can be significantly reduced by learning the difference between the target frame and the past frames; in addition, the extrapolation frame can be applied to In video coding, the coding efficiency is improved.

On the other hand, in order to illustrate the performance of the present invention, related tests are also carried out, and extrapolation is mainly suitable for low-latency configurations in encoding.

The test conditions include: 1) Inter-frame configuration: low delay B means Low-delay B, LDB; low delay P means Low-delayP, LDP. 2) The basic quantization step (QP) is set to {27,32,37,42}, the software based on it is HM12.0, and the test sequence is the HEVC standard test sequence.

Those skilled in the art can understand that the coding field needs to test the performance in the specified standard test sequence, and the standard test sequence is divided into several categories (ClassA-F), and the results of various test sequences are reported according to the standard test requirements in the following test process.

1) Comparison of the performance of the 4 combinations of the alignment method and the extrapolation application

Table 1 shows the performance comparison of the 4 combinations under the LDP setting. It can be seen from the table that the optimal solution is REF+ColMEA, which can be used as the preferred solution.

Table 1 The performance comparison results of the 4 combinations under the LDP setting

Table 2 is the complete performance of the preferred scheme. It can be seen from Table 2 that, compared with HM12.0, the above solution of the embodiment of the present invention can obtain 5.3% and 2.8% code rate savings in LDP and LDB modes, respectively.

The complete performance of table 2 preferred scheme

2) Alignment and extrapolation frame effect test

Table 3 shows the coding performance comparison between traditional frame extrapolation without alignment and the aligned extrapolation frame proposed by the present invention under LDP conditions in the preferred solution, it can be seen that extrapolation without alignment can only save 2.2% of the code rate , while aligned extrapolation yields a bitrate savings of 5.3%. Compared with simple extrapolated frames, the alignment proposed by the present invention can significantly improve the coding performance. In addition, it is found through observation that the extrapolation frame generated by the alignment method proposed by the present invention is clearer and has a better effect than the traditional method.

Table 3 Alignment and extrapolation frame effects

Through the above description of the implementation manners, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the above-mentioned embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments of the present invention.

The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (8)

1. A method for aligning extrapolated frames based on a neural network is characterized by comprising the following steps:

dividing a target frame into M blocks with a specified size, selecting N continuous past frames closest to the target frame, and respectively finding out the most similar blocks from each past frame for alignment of each block in the target frame to obtain N aligned blocks; wherein, N and M are set natural numbers;

for each block in a target frame, inputting N aligned blocks into a multi-scale and residual error network, learning the difference between the target frame and each past frame, and predicting to obtain 1 extrapolated block;

splicing all the extrapolation blocks according to the corresponding position relation to obtain an aligned extrapolation frame;

wherein, the step of selecting the N past frames which are nearest to the target frame and are continuous, and for each block in the target frame, respectively finding the most similar block from each past frame for alignment, and obtaining N aligned blocks comprises:

target frame is marked as I _t The N past frames nearest to the target frame and consecutive are marked as I _t-1 ,I _t-2 ,…,I _t-N (ii) a The alignment comprises two steps:

the first step is to align the target frame I _t And I _t-1 Frame, get I _t-1 An alignment block of a frame;

the second step is based on I _t-1 Alignment block of frame, alignment target frame I _t And N past frames I _t-1 ,I _t-2 ,…,I _t-N To obtain I _t-2 Frame to I _t-N An alignment block of a frame.

2. The neural network-based aligned extrapolation frame method of claim 1, wherein the first step employs either of the following two approaches:

the first mode is as follows: with a target frame I _t As a template for motion estimation, at I _t-1 Performing integer pixel motion estimation in frame, and correcting average absolute valueThe block with the smallest difference is taken as I _t-1 An alignment block of the frame;

the second way is: based on the target frame I _t And I _t-1 The assumption that the motion between frames is less than the set value adopts a fixed position mode to obtain I _t-1 An alignment block of a frame.

3. The neural network-based method for aligning extrapolated frames according to claim 1 or 2, wherein the alignment method used in the second step includes:

for target frame I _t Target block of (1), I obtained in the first step _t-1 Aligned blocks of frames as templates for motion estimation, at I _t-2 Performing integer pixel motion estimation in a frame, and taking the block with the minimum average absolute value error as I _t-2 An alignment block of a frame; repeatedly executing in such a way, and finally, I _t-(N-1) Preliminary aligned blocks of frames as templates for motion estimation, at I _t-N Performing integer pixel motion estimation in frame, and taking the block with minimum average absolute value error as I _t-N An alignment block of a frame; resulting in N aligned blocks.

4. The method of claim 3, wherein the N alignment blocks are obtained and then reconstructed pixels around each alignment block are respectively compensated around the corresponding alignment block to obtain the final N alignment blocks.

5. The method as claimed in claim 4, wherein the multi-scale and residual error network comprises Q scales, and the resolution is sequentially denoted as s ₁ ,s ₂ ,…,s _Q (ii) a The resolution size of the current scale k is half of the next scale k +1, i.e.

Multi-scale and residual networks, from lowest resolution s ₁ Starting a series of predictions and taking s _k Size prediction as a starting point, s _k+1 Prediction of size, upsampling is done continuously and the learned residual is added on the next finer scale until the full resolution picture is returned, i.e. the same size as the final aligned block.

6. The method of claim 5, wherein the extrapolation block prediction result of the multi-scale and residual error network is as follows:

wherein,

refers to the k-th scale predicted extrapolation block, X of network downsampling ₁ As input of minimum size, i.e., k =1, to

Is X ₁ (ii) a X denotes N aligned blocks { X ^t-1 ,X ^t-2 ,…,X ^t-N Are sequentially I _t-1 ,I _t-2 ,…,I _t-N Alignment block of (2), X _k Representing X at resolution s _k The down-sampled image of (a) is,

G _k represents a slave X _k And

learning out residual error

Network of u _k Representing towards resolution s _k The up-sampling operation of (2);

the total loss of the network is Q scale losses and:

wherein:

‖*‖ ₁ is represented by ₁ And (4) loss.

7. The neural network-based method for aligning extrapolated frames according to claim 3, further comprising: applying the obtained aligned interpolated frames in video coding includes at least two schemes:

the first is to use the aligned extrapolated frame as the motion compensation prediction result of inter-frame prediction, which is abbreviated as MCP;

second, conventional motion compensation is performed on a new reference frame by aligning the interpolated frame to the new reference frame, which is abbreviated as REF.

8. The neural network-based method for aligning extrapolated frames according to claim 7, further comprising: combining the alignment method with the two application schemes;

the alignment method comprises the following steps: the first step and the second step both adopt a motion estimation mode, which is abbreviated as MEA; the first step adopts a fixed position mode, and the second step adopts a motion estimation mode, which is called ColMEA for short;

the two alignment methods and two application schemes are combined in a cross mode and integrated into the coding standard HEVC, and the combination schemes are four as follows: MCP + ColMEA, MCP + MEA, REF + ColMEA and REF + MEA.

Images