CN111461043B - Video significance detection method based on deep network - Google Patents

Abstract

The present invention is a video saliency detection method based on a deep network, which relates to the field of image data processing. The method uses a ResNet50 deep network to obtain spatial features, and then extracts time and edge information to jointly obtain a saliency prediction result map, and completes a depth-based The video saliency detection of the network, the steps are, input video frame I, carry out preprocessing; Extract the initial spatial feature map S of video frame I ^' ; Obtain the spatial feature map S _{final of} five scales; Obtain feature map F; Obtain rough The spatiotemporal saliency map Y _ST and the edge contour map E _t of salient objects; obtain the final saliency prediction result map Y _final ; calculate the loss for the input video frame I, and complete the video saliency detection based on the deep network. The invention overcomes the defects of incomplete salient object detection and inaccurate algorithm detection when the foreground and background colors are similar in the prior art video salient detection.

Description

Video saliency detection method based on deep network

Technical Field

The technical solution of the present invention relates to the field of image data processing, and specifically to a video saliency detection method based on a deep network.

Background Art

Video saliency detection aims to extract the areas of greatest interest to the human eye in continuous video frames. Specifically, it uses computers to simulate the visual attention mechanism of the human eye and extract the areas of interest to the human eye from video frames. It is one of the key technologies in the field of computer vision.

Most of the traditional video saliency detection methods are based on low-level manual features (such as color, texture, etc.). These methods are typical heuristic methods with the disadvantages of slow speed (due to time-consuming optical flow calculation) and low prediction accuracy (due to the limited representability of low-level features). In recent years, deep neural networks have begun to be applied to the field of video saliency detection. Deep learning methods refer to the use of convolutional neural networks to extract high-level semantic features of images and calculate the saliency value of images. However, the use of deep convolutional networks will lose the location information and detail information of the target, which may introduce misleading information when detecting salient targets, resulting in incomplete detected targets.

In 2016, Liu et al. proposed the SGSP algorithm in the paper "Saliency detection for unconstrained videos using superpixel-level graph and spatiotemporal propagation". The algorithm uses superpixel-level graph models and spatiotemporal propagation to detect video saliency. First, superpixel-level motion and color histograms and global motion histograms are extracted to construct a graph. Then, based on the graph model, motion saliency is iteratively calculated through the shortest path on the graph using background priors. Then, it propagates forward and backward in time, and propagates locally and globally in space, and finally combines the two results to form the final saliency map. The algorithm is very computationally intensive, but the saliency map obtained still has the problem of incomplete detection of salient objects. The deep learning model aims to use convolutional neural networks to obtain richer deep features, thereby obtaining more accurate detection results. Wang et al. proposed a video saliency detection method based on a fully convolutional network in the article "Videosalient object detection via fully convolutional networks" in 2017. This is the first time that a fully convolutional network based on deep learning has been used in the field of video saliency detection. However, since the time information between frames is not taken into account, the edges of the obtained saliency map are not fine enough and the edge noise is relatively large. CN106372636A discloses a video saliency detection method based on HOG_TOP. The method uses the original video to calculate the HOG_TOP features on three orthogonal planes XY, XT, and YT, respectively calculates the spatial domain saliency map on the XY plane and the temporal domain saliency map on the XT and YT planes, and finally obtains the final saliency map through adaptive fusion. This method needs to calculate the optical flow of each pixel when calculating the temporal domain saliency map, which is very computationally intensive and slow. CN109784183A discloses a video salient target detection method based on cascade convolutional network and optical flow. The method uses a cascade network structure to perform pixel-level saliency prediction on the image of the current frame at three scales: high, medium and low. The cascade network structure is trained using the MSAR10K image data set, and the saliency annotation map is used as the supervisory information for training. The loss function is the cross entropy loss function. After the training is terminated, the trained cascade network is used to perform static saliency prediction on each frame of the video, and the Locus-Kanada algorithm is used to extract the optical flow field. Then a three-layer convolutional network structure is used to construct a dynamic optimization network structure. The static detection results of each frame of the image and the optical flow field detection results are spliced to obtain the input data of the optimization network. This method is time-consuming, and the optical flow information extracted by the Locus-Kanada algorithm is not accurate and has poor robustness in some complex scenes. CN109118469A discloses a method for predicting video saliency, which first quantizes the image to obtain a sparse matrix response, then obtains a decomposition matrix based on local coordinate constraints, and finally calculates a saliency map for each frame in the video and performs quality prediction. This method loses some detailed information of the salient target, so that the prediction result will have the problem of incomplete detection of salient targets. CN105913456B discloses a method for detecting video saliency by region segmentation, which first uses nonlinear clustering to obtain superpixel blocks to extract static features, then uses the optical flow method to obtain dynamic features, and finally uses a linear regression model to predict the saliency map after the fusion of the two features. The method has a large amount of calculation and low efficiency. CN109034001A discloses a cross-modal video saliency detection method based on spatiotemporal cues, which uses the initial saliency map and the weights of the two modalities of visible light and thermal infrared to construct a saliency map. The method is difficult to find a suitable weight value, resulting in poor robustness. CN108241854A discloses a deep video saliency detection method based on motion and memory information. The method first extracts local information and global information based on the human eye gaze map of the current frame, and then inputs this information into the deep network model together with the original image as prior information to predict the final saliency map. When the salient target touches the image boundary, the method will have false detection, and the salient target will be mistakenly detected as the background. CN110598537A discloses a video saliency detection method based on a deep convolutional network. The method uses the current frame of the video and its corresponding optical flow image as the input of the feature extraction network to predict the final saliency map. The method needs to calculate the optical flow information of the current frame in advance, and the amount of calculation is large.

In summary, the existing technology of video salient object detection still has problems such as incomplete salient object detection and inaccurate algorithm detection when the foreground and background colors are similar.

Summary of the invention

The technical problem to be solved by the present invention is to provide a video saliency detection method based on a deep network. The method first uses a ResNet50 deep network to obtain spatial features, and then extracts time and edge information to jointly obtain a saliency prediction result map, thereby completing video saliency detection based on a deep network, overcoming the defects of incomplete salient target detection and inaccurate algorithm detection when the foreground and background colors are similar in the prior art video saliency detection.

The technical solution adopted by the present invention to solve the technical problem is: a video saliency detection method based on a deep network first uses a ResNet50 deep network to obtain spatial features, and then extracts time and edge information to jointly obtain a saliency prediction result map to complete the video saliency detection based on a deep network. The specific steps are as follows:

The first step is to input the video frame I and perform preprocessing:

Input video frame I, unify the size of the video frame to 473×473 pixels in width and height, and subtract the mean of the corresponding channel from each pixel value in video frame I, where the mean of the R channel of each video frame I is 104.00698793, the mean of the G channel of each video frame I is 116.66876762, and the mean of the B channel of each video frame I is 122.67891434. In this way, the shape of video frame I before inputting into the ResNet50 deep network is 473×473×3. The video frame after such preprocessing is recorded as I′, as shown in the following formula (1):

I′＝Resize(I-Mean(R,G,B)) (1),

In formula (1), Mean(R,G,B) is the mean of the three color channels of red, green, and blue, and Resize(·) is the function for adjusting the size of the video frame I′;

The second step is to extract the initial spatial feature map S of the video frame I′:

The video frame I′ after the first step of preprocessing is sent to the ResNet50 deep network to extract the initial spatial feature map S, as shown in the following formula (2):

S＝ResNet50(I′) (2),

In formula (2), ResNet50(·) is the ResNet50 deep network.

The ResNet50 deep network contains convolutional layers, pooling layers, non-linear activation function Relu layers and residual connections;

The third step is to obtain the spatial feature map S _final of five scales:

The initial spatial feature map S of the video frame I′ extracted in the second step is sent to four different dilated convolutions with dilation rates of 2, 4, 8, and 16 in the ResNet50 deep network, and the results T _k of four scales with dilation rates of 2, 4, 8, and 16 are obtained. Then, the result is connected in series with the output result initial spatial feature map S of the ResNet50 deep network to finally obtain the spatial feature map S _final of five scales.

The fourth step is to obtain the feature map F:

The spatial feature maps S _final of the five scales obtained in the third step above are subjected to a convolution operation with a convolution kernel of 3×3×32 to obtain a feature map F with a shape of 60×60×32, as shown in the following formula (3):

F＝BN(Relu(Conv(S _final ))) (3),

In formula (3), Conv(·) is the convolution operation, Relu(·) is the nonlinear activation function, and BN(·) is the normalization operation.

The fifth step is to obtain a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object:

The feature map F obtained in the fourth step is input into the spatiotemporal branch and the edge detection branch to obtain a spatiotemporal feature map F _ST and an edge contour map E _t of a salient object. The specific operation is as follows:

The feature map F obtained in the fourth step above is input into the ConvLSTM of the spatiotemporal branch to obtain a spatiotemporal feature map F _ST , as shown in the following formula (4):

F _ST =ConvLSTM(F,H _t-1 ) (4),

In formula (4), ConvLSTM(·) is the ConvLSTM operation, H _t-1 is the state of the ConvLSTM unit at the previous moment;

Then the obtained spatiotemporal feature map F _ST is sent to a convolution layer with a convolution kernel size of 1×1 to obtain a rough spatiotemporal saliency map Y _ST , the formula is as follows:

Y _ST =Conv(F _ST ) (5),

In formula (5), Conv(·) is the convolution operation;

The feature map F obtained in the fourth step is input into the edge detection branch to obtain the edge contour map E _t of the salient object. The specific operation is as follows:

其中X_t为第t帧的视频帧，给定X_t，X_t经过边缘检测分支后输出为边缘轮廓图E_t∈[0,1]^W×H，其中W和H分别为预测边缘图像的宽度和高度，是从边缘检测网络

中计算出来的，它将先前的视频帧考虑在内，具体如下公式(6)和公式(7)所示，Through the ResNet50 deep network and dilated convolution, the static image of the input video of frame T is obtained as follows:

Where _Xt is the video frame of the tth frame. Given _Xt , _Xt is output as an edge contour map _Et∈ [0,1] ^W×H after the edge detection branch, where W and H are the width and height of the predicted edge image, respectively, which is obtained from the edge detection network.

It is calculated in , which takes the previous video frame into account, as shown in the following formulas (6) and (7),

H _t =ConvLSTM(X _t ,H _t-1 ) (6),

为3D张量隐藏状态，M为通道数，E_t′为未加权的边缘轮廓图，H_t为当前ConvLSTM单元的状态，H_t-1为上一时刻ConvLSTM单元的状态，X₁为第一帧的视频帧，In formula (6) and formula (7),

is the 3D tensor hidden state, M is the number of channels, E _t ′ is the unweighted edge contour map, H _t is the state of the current ConvLSTM unit, H _t-1 is the state of the ConvLSTM unit at the previous moment, X ₁ is the video frame of the first frame,

如下公式(8)所示，The key component of embedding ConvLSTM in ConvLSTM to obtain the edge contour map E _t is the edge detection network

As shown in the following formula (8),

进行加权，得到显著性物体的边缘轮廓图E_t，如下公式(9)所示，Then use the above edge detection network

After weighting, the edge contour map E _t of the salient object is obtained, as shown in the following formula (9):

为一个1×1的卷积核，用来映射边缘检测网络

得到一个权重矩阵，sigmoid函数σ为把这个矩阵归一化到[0,1]；In formula (9),

is a 1×1 convolution kernel used to map the edge detection network

Get a weight matrix, and the sigmoid function σ normalizes this matrix to [0,1];

Thus, a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object are obtained;

Step 6: Get the final significance prediction result graph Y _final :

The rough spatiotemporal saliency map Y _ST obtained in the fifth step above is fused with the edge contour map E _t of the salient object to obtain the final saliency prediction result map Y _final , as shown in the following formula (10):

为矩阵相乘，σ为sigmoid函数，Resize(·)为调整视频帧大小的函数，In formula (10),

is matrix multiplication, σ is the sigmoid function, Resize(·) is the function for adjusting the video frame size,

The obtained video frame is restored to the size of the original input video frame 473×473;

Step 7: Calculate the loss for the input video frame I:

如下公式(11)所示，After the first to sixth steps above, the saliency map for the input video frame I is calculated. In order to measure the difference between the final saliency prediction result map Y _final obtained in the sixth step above and the ground-truth, the binary cross entropy loss function is used during training

As shown in the following formula (11),

In formula (11), G(i,j)∈[0,1] is the true value of pixel (i,j), M(i,j)∈[0,1] is the predicted value of pixel (i,j), and N=473.

的大小进行网络的训练，采用随机梯度下降法优化二值交叉熵损失函数

By continuously shrinking

The size of the network is trained, and the stochastic gradient descent method is used to optimize the binary cross entropy loss function

This completes the video saliency detection based on deep network.

In the above-mentioned video saliency detection method based on deep network, the specific operation of obtaining the spatial feature map S _final of five scales is as follows:

其中K为扩张卷积层的个数，c×c为宽度和高度的相乘，C为通道数，

为扩张卷积的参数其步长设置为1，基于这些参数得到四个输出特征图

其中，W和H分别为宽度和高度，如下公式(12)所示，The dilated convolution kernel in the ResNet50 deep network is represented as

Where K is the number of dilated convolutional layers, c×c is the multiplication of width and height, and C is the number of channels.

The parameters of the dilated convolution are set to 1, and four output feature maps are obtained based on these parameters.

Where W and H are the width and height respectively, as shown in the following formula (12):

为扩张卷积操作，S为初始空间特征图，In formula (12), C _k is the dilated convolution kernel with a value of k, K is the number of dilated convolutions,

is the dilated convolution operation, S is the initial spatial feature map,

再将它们依次串联起来，由如下公式(13)所示，The shape of the initial spatial feature map S obtained after the ResNet50 deep network is 60×60×2048, the value is 4, the value range of k is [1, 2, 3, 4], the expansion rate r _k has four values, namely r _k = {2, 4, 8, 16}, and the shape of its expansion convolution kernel C _k is 3×3×512, thus finally obtaining feature maps of four different scales

Then connect them in series, as shown in the following formula (13):

S _final =[S,T ₁ ,T ₂ ,…,T _K ] (13),

In formula (13), S _final is the final multi-scale spatial feature map, S is the initial spatial feature map S extracted by the ResNet50 deep network, T _K is the feature map obtained after the dilated convolution, and the shape of the five-scale spatial feature map S _final is 60×60×4096.

The beneficial effects of the present invention are as follows: Compared with the prior art, the outstanding substantive features and significant improvements of the present invention are as follows:

(1) Compared with CN106372636A, the method of the present invention adopts a method based on deep learning, first using ResNet50 and dilated convolution to extract multi-scale spatial features, then using ConvLSTM to extract temporal information, and finally integrating it into spatiotemporal information. The outstanding substantive characteristics and significant progress of the present invention are that it does not need to calculate optical flow information, but uses ConvLSTM to extract temporal information. The detection accuracy of salient targets is better than the method of calculating optical flow, and the speed is faster.

(2) Compared with CN109784183A, the method of the present invention adopts a connection mode with a residual network, and multiple convolutional layers are connected with residual blocks. The outstanding substantial characteristics and significant progress of the present invention are that it can make the training network converge faster, the extracted features are more refined, and the prediction accuracy is higher.

(3) Compared with CN109118469A, the method of the present invention has outstanding substantive features and significant improvements in that it does not require cumbersome sparse matrix extraction, but uses a deep neural network to extract high-level features from video frames and predict each pixel point, so that the detection result is more accurate and has better robustness.

(4) Compared with CN105913456B, the method of the present invention has the outstanding substantive characteristics and significant progress that it does not require linear iteration and k-means clustering with large computational complexity, but directly adopts an end-to-end neural network method, and can obtain prediction results more quickly after training is completed.

(5) Compared with CN109034001A, the method of the present invention uses an edge detection branch based on a deep network to extract the edges of salient objects in the original image, and uses this to guide the generation of the following complete salient map. The outstanding substantive feature and significant progress of the present invention is that the salient objects in the obtained salient map are more complete.

(6) Compared with CN108241854A, although both methods use deep learning methods, the present invention uses dilated convolution to extract feature maps of four different scales. Compared with the above, the features extracted by the present invention are more comprehensive. Therefore, the outstanding substantial feature and significant improvement of the present invention is that the edges of the salient targets in the final salient map obtained are smoother.

(7) Compared with CN110598537A, the method of the present invention has an outstanding substantive feature and a significant improvement in that ConvLSTM is used to simulate the optical flow information between frames, and the extracted optical flow information is more accurate than that calculated by traditional methods.

(8) Compared with Video Salient Object Detection via Fully Convolutional Networks, the present invention has the outstanding substantial feature and significant improvement of utilizing the time information between frames, and the obtained prediction result map is more accurate.

(9) The method of the present invention proposes a video saliency detection method model based on a deep network. First, in the field of video saliency detection, a deep learning-based edge detection method of salient targets is used. This method is different from the traditional edge detection algorithm. It can accurately detect the contours of salient targets in each frame of the video sequence to guide the prediction of saliency maps.

(10) The present invention utilizes the deep salient target edge detection branch to generate a salient target contour map and fuses it with the spatiotemporal salient map of each frame in the video, making its contour smoother and being able to more accurately predict the salient targets in each frame in the video sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is further described below in conjunction with the accompanying drawings and embodiments.

FIG1 is a schematic flow chart of a method for detecting video saliency based on a deep network according to the present invention.

FIG. 2 is a saliency prediction result graph Y _final of a video frame I in which salient objects are a cat and a box in an embodiment of the present invention.

DETAILED DESCRIPTION

The embodiment shown in FIG1 shows that the process of the video saliency detection method based on deep network of the present invention is as follows:

Input video frame I, perform preprocessing → extract the initial spatial feature map S of video frame I′ → obtain the spatial feature map S _final at five scales → obtain the feature map F → obtain a rough spatiotemporal saliency map Y _ST and the edge contour map E _t of the salient object → obtain the final saliency prediction result map Y _final → calculate the loss for the input video frame I → complete the video saliency detection based on the deep network.

Example 1

In this embodiment, the salient objects are a cat and a box. The video saliency detection method based on a deep network described in this embodiment has the following specific steps:

The first step is to input the video frame I and perform preprocessing:

Input a video frame I with a cat and a box as salient objects. The size of the video frames is unified to 473×473 pixels in width and height, and the mean of the corresponding channel is subtracted from each pixel value in the video frame I. The mean of the R channel of each video frame I is 104.00698793, the mean of the G channel of each video frame I is 116.66876762, and the mean of the B channel of each video frame I is 122.67891434. In this way, the shape of the video frame I before inputting into the ResNet50 deep network is 473×473×3. The video frame after such preprocessing is recorded as I′, as shown in the following formula (1):

I′＝Resize(I-Mean(R,G,B)) (1),

In formula (1), Mean(R,G,B) is the mean of the three color channels of red, green, and blue, and Resize(·) is the function for adjusting the size of the video frame I′;

The second step is to extract the initial spatial feature map S of the video frame I′:

The video frame I′ after the first step of preprocessing is sent to the ResNet50 deep network to extract the initial spatial feature map S, as shown in the following formula (2):

S＝ResNet50(I′) (2),

In formula (2), ResNet50(·) is the ResNet50 deep network.

The ResNet50 deep network contains convolutional layers, pooling layers, non-linear activation function Relu layers and residual connections;

The third step is to obtain the spatial feature map S _final of five scales:

The initial spatial feature map S of the video frame I′ extracted in the second step is sent to four different dilated convolutions with dilation rates of 2, 4, 8, and 16 in the ResNet50 deep network, and the results T _k of four scales with dilation rates of 2, 4, 8, and 16 are obtained. Then, the result is connected in series with the output result initial spatial feature map S of the ResNet50 deep network to finally obtain the spatial feature map S _final of five scales.

The specific operations to obtain the spatial feature map S _final of five scales are as follows:

其中K为扩张卷积层的个数，c×c为宽度和高度的相乘，C为通道数，

为扩张卷积的参数其步长设置为1，基于这些参数得到四个输出特征图

其中，W和H分别为宽度和高度，如下公式(3)所示，The dilated convolution kernel in the ResNet50 deep network is represented as

Where K is the number of dilated convolutional layers, c×c is the multiplication of width and height, and C is the number of channels.

The parameters of the dilated convolution are set to 1, and four output feature maps are obtained based on these parameters.

Where W and H are the width and height respectively, as shown in the following formula (3):

为扩张卷积操作，S为初始空间特征图，In formula (3), C _k is the dilated convolution kernel with a value of k, K is the number of dilated convolutions,

is the dilated convolution operation, S is the initial spatial feature map,

再将它们依次串联起来，由如下公式(4)所示，The shape of the initial spatial feature map S obtained after the ResNet50 deep network is 60×60×2048, the value is 4, the value range of k is [1, 2, 3, 4], the expansion rate r _k has four values, namely r _k = {2, 4, 8, 16}, and the shape of its expansion convolution kernel C _k is 3×3×512, thus finally obtaining feature maps of four different scales

Then connect them in series, as shown in the following formula (4):

S _final =[S,T ₁ ,T ₂ ,…,T _K ] (4),

In formula (4), S _final is the final multi-scale spatial feature map, S is the initial spatial feature map S extracted by the ResNet50 deep network, T _K is the feature map obtained after the dilated convolution, and the shape of the five-scale spatial feature map S _final is 60×60×4096;

The fourth step is to obtain the feature map F:

The spatial feature maps S _final of the five scales obtained in the third step above are subjected to a convolution operation with a convolution kernel of 3×3×32 to obtain a feature map F with a shape of 60×60×32, as shown in the following formula (5):

F＝BN(Relu(Conv(S _final ))) (5),

In formula (5), Conv(·) is the convolution operation, Relu(·) is the nonlinear activation function, and BN(·) is the normalization operation.

The fifth step is to obtain a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object:

The feature map F obtained in the fourth step is input into the spatiotemporal branch and the edge detection branch to obtain a spatiotemporal feature map F _ST and an edge contour map E _t of a salient object. The specific operation is as follows:

The feature map F obtained in the fourth step above is input into the ConvLSTM of the spatiotemporal branch to obtain a spatiotemporal feature map F _ST , as shown in the following formula (6):

F _ST =ConvLSTM(F,H _t-1 ) (6),

In formula (6), ConvLSTM(·) is the ConvLSTM operation, H _t-1 is the state of the ConvLSTM unit at the previous moment;

Then the obtained spatiotemporal feature map F _ST is sent to a convolution layer with a convolution kernel size of 1×1 to obtain a rough spatiotemporal saliency map Y _ST , the formula is as follows:

Y _ST =Conv(F _ST ) (7),

In formula (7), Conv(·) is the convolution operation;

The feature map F obtained in the fourth step is input into the edge detection branch to obtain the edge contour map E _t of the salient object. The specific operation is as follows:

其中X_t为第t帧的视频帧，给定X_t，X_t经过边缘检测分支后输出为边缘轮廓图E_t∈[0,1]^W×H，其中W和H分别为预测边缘图像的宽度和高度，是从边缘检测网络

中计算出来的，它将先前的视频帧考虑在内，具体如下公式(8)和公式(9)所示，The edge detection branch contains a two-layer ConvLSTM, which is a powerful recurrent model that not only captures temporal information, but also depicts the contour edges of salient objects based on temporal information, distinguishing salient objects from non-salient objects in the image. More specifically, through the ResNet50 deep network and dilated convolution, the static image of the input video of frame T is obtained as

Where _Xt is the video frame of the tth frame. Given _Xt , _Xt is output as an edge contour map _Et∈ [0,1] ^W×H after the edge detection branch, where W and H are the width and height of the predicted edge image, respectively, which is obtained from the edge detection network.

It is calculated in , which takes the previous video frame into account, as shown in the following formulas (8) and (9),

H _t =ConvLSTM(X _t ,H _t-1 ) (8),

为3D张量隐藏状态，M为通道数，E_t′为未加权的边缘轮廓图，H_t为当前ConvLSTM单元的状态，H_t-1为上一时刻ConvLSTM单元的状态，X₁为第一帧的视频帧，In formula (8) and formula (9),

is the 3D tensor hidden state, M is the number of channels, E _t ′ is the unweighted edge contour map, H _t is the state of the current ConvLSTM unit, H _t-1 is the state of the ConvLSTM unit at the previous moment, X ₁ is the video frame of the first frame,

如下公式(10)所示，The key component of embedding ConvLSTM in ConvLSTM to obtain the edge contour map E _t is the edge detection network

As shown in the following formula (10),

进行加权，得到显著性物体的边缘轮廓图E_t，如下公式(11)所示，Then use the above edge detection network

After weighting, the edge contour map E _t of the salient object is obtained, as shown in the following formula (11):

为一个1×1的卷积核，用来映射边缘检测网络

得到一个权重矩阵，sigmoid函数σ为把这个矩阵归一化到[0,1]；In formula (11),

is a 1×1 convolution kernel used to map the edge detection network

Get a weight matrix, and the sigmoid function σ normalizes this matrix to [0,1];

Thus, a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object are obtained;

Step 6: Get the final significance prediction result graph Y _final :

The rough spatiotemporal saliency map Y _ST obtained in the fifth step above is fused with the edge contour map E _t of the salient object to obtain the final saliency prediction result map Y _final , as shown in the following formula (12):

In formula (12), ‘ο’ is matrix multiplication, σ is the sigmoid function, and Resize(·) is the function for adjusting the video frame size.

The obtained video frame is restored to the size of the original input video frame 473×473;

FIG. 2 is a final saliency prediction result graph Y _final of the video frame I in this embodiment, in which there are two salient objects, a cat and a box.

Step 7: Calculate the loss for the input video frame I:

如下公式(13)所示，After the first to sixth steps above, the saliency map for the input video frame I is calculated. In order to measure the difference between the final saliency prediction result map Y _final obtained in the sixth step above and the ground-truth, the binary cross entropy loss function is used during training

As shown in the following formula (13),

In formula (13), G(i,j)∈[0,1] is the true value of pixel (i,j), M(i,j)∈[0,1] is the predicted value of pixel (i,j), and N=473.

的大小进行网络的训练，采用随机梯度下降法优化二值交叉熵损失函数

By continuously shrinking

The size of the network is trained, and the stochastic gradient descent method is used to optimize the binary cross entropy loss function

This completes the video saliency detection based on deep network.

In the above embodiments, the ResNet50 deep network, ConvLSTM, ground-truth, and stochastic gradient descent method are all well known in the technical field.

Claims (2)

1. A video saliency detection method based on a deep network is characterized in that: the spatial features are first obtained by using a ResNet50 deep network, and then the time and edge information are extracted to jointly obtain a saliency prediction result map, thereby completing the video saliency detection based on a deep network. The specific steps are as follows: The first step is to input the video frame I and perform preprocessing: Input video frame I, unify the size of the video frame to 473×473 pixels in width and height, and subtract the mean of the corresponding channel from each pixel value in video frame I, where the mean of the R channel of each video frame I is 104.00698793, the mean of the G channel of each video frame I is 116.66876762, and the mean of the B channel of each video frame I is 122.67891434. In this way, the shape of video frame I before inputting into the ResNet50 deep network is 473×473×3. The video frame after such preprocessing is recorded as I′, as shown in the following formula (1): I′＝Resize(I-Mean(R,G,B)) (1), In formula (1), Mean(R,G,B) is the mean of the three color channels of red, green, and blue, and Resize(·) is the function for adjusting the size of the video frame I′; The second step is to extract the initial spatial feature map S of the video frame I′: The video frame I′ after the first step of preprocessing is sent to the ResNet50 deep network to extract the initial spatial feature map S, as shown in the following formula (2): S＝ResNet50(I′) (2), In formula (2), ResNet50(·) is the ResNet50 deep network. The ResNet50 deep network contains convolutional layers, pooling layers, non-linear activation function Relu layers and residual connections; The third step is to obtain the spatial feature map S _final of five scales: The initial spatial feature map S of the video frame I′ extracted in the second step is sent to four different dilated convolutions with dilation rates of 2, 4, 8, and 16 in the ResNet50 deep network, and the results T _k of four scales with dilation rates of 2, 4, 8, and 16 are obtained. Then, the result is connected in series with the output result initial spatial feature map S of the ResNet50 deep network to finally obtain the spatial feature map S _final of five scales. The fourth step is to obtain the feature map F: The spatial feature maps S _final of the five scales obtained in the third step above are subjected to a convolution operation with a convolution kernel of 3×3×32 to obtain a feature map F with a shape of 60×60×32, as shown in the following formula (3): F＝BN(Relu(Conv(S _final ))) (3), In formula (3), Conv(·) is the convolution operation, Relu(·) is the nonlinear activation function, and BN(·) is the normalization operation. The fifth step is to obtain a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object: The feature map F obtained in the fourth step is input into the spatiotemporal branch and the edge detection branch to obtain a spatiotemporal feature map F _ST and an edge contour map E _t of a salient object. The specific operation is as follows: The feature map F obtained in the fourth step above is input into the ConvLSTM of the spatiotemporal branch to obtain a spatiotemporal feature map F _ST , as shown in the following formula (4): F _ST =ConvLSTM(F,H _t-1 ) (4), In formula (4), ConvLSTM(·) is the ConvLSTM operation, H _t-1 is the state of the ConvLSTM unit at the previous moment; Then the obtained spatiotemporal feature map F _ST is sent to a convolution layer with a convolution kernel size of 1×1 to obtain a rough spatiotemporal saliency map Y _ST , the formula is as follows: Y _ST =Conv(F _ST ) (5), In formula (5), Conv(·) is the convolution operation; The feature map F obtained in the fourth step is input into the edge detection branch to obtain the edge contour map E _t of the salient object. The specific operation is as follows: Through the ResNet50 deep network and dilated convolution, the static image of the input video of frame T is obtained as follows:

Where _Xt is the video frame of the tth frame. Given _Xt , _Xt is output as an edge contour map _Et∈ [0,1] ^W×H after the edge detection branch, where W and H are the width and height of the predicted edge image, respectively, which is obtained from the edge detection network.

It is calculated in , which takes the previous video frame into account, as shown in the following formulas (6) and (7), H _t =ConvLSTM(X _t ,H _t-1 ) (6),

In formula (6) and formula (7),

is the 3D tensor hidden state, M is the number of channels, E _t ′ is the unweighted edge contour map, H _t is the state of the current ConvLSTM unit, H _t-1 is the state of the ConvLSTM unit at the previous moment, X ₁ is the video frame of the first frame, The key component of embedding ConvLSTM in ConvLSTM to obtain the edge contour map E _t is the edge detection network

As shown in the following formula (8),

Then use the above edge detection network

After weighting, the edge contour map E _t of the salient object is obtained, as shown in the following formula (9):

In formula (9),

is a 1×1 convolution kernel used to map the edge detection network

Get a weight matrix, and the sigmoid function σ normalizes this matrix to [0,1]; Thus, a rough spatiotemporal saliency map Y _ST and an edge contour map E _t of a salient object are obtained; Step 6: Get the final significance prediction result graph Y _final : The rough spatiotemporal saliency map Y _ST obtained in the fifth step above is fused with the edge contour map E _t of the salient object to obtain the final saliency prediction result map Y _final , as shown in the following formula (10):

In formula (10),

is matrix multiplication, σ is the sigmoid function, Resize(·) is the function for adjusting the video frame size, The obtained video frame is restored to the size of the original input video frame 473×473; Step 7: Calculate the loss for the input video frame I: After the first to sixth steps above, the saliency map for the input video frame I is calculated. In order to measure the difference between the final saliency prediction result map Y _final obtained in the sixth step above and the ground-truth, the binary cross entropy loss function is used during training

As shown in the following formula (11),

In formula (11), G(i,j)∈[0,1] is the true value of pixel (i,j), M(i,j)∈[0,1] is the predicted value of pixel (i,j), and N=473. By continuously shrinking

The size of the network is trained, and the stochastic gradient descent method is used to optimize the binary cross entropy loss function

This completes the video saliency detection based on deep network. 2. According to the deep network-based video saliency detection method of claim 1, it is characterized in that: the specific operation of obtaining the spatial feature map S _final of five scales is as follows: The dilated convolution kernel in the ResNet50 deep network is represented as

Where K is the number of dilated convolutional layers, c×c is the multiplication of width and height, and C is the number of channels.

The parameters of the dilated convolution are set to 1, and four output feature maps are obtained based on these parameters.

Where W and H are the width and height respectively, as shown in the following formula (12):

In formula (12), C _k is the dilated convolution kernel with a value of k, K is the number of dilated convolutions,

is the dilated convolution operation, S is the initial spatial feature map, The shape of the initial spatial feature map S obtained after the ResNet50 deep network is 60×60×2048, the value is 4, the value range of k is [1, 2, 3, 4], the expansion rate r _k has four values, namely r _k = {2, 4, 8, 16}, and the shape of its expansion convolution kernel C _k is 3×3×512, thus finally obtaining feature maps of four different scales

Then connect them in series, as shown in the following formula (13): S _final =[S,T ₁ ,T ₂ ,…,T _K ] (13), In formula (13), S _final is the final multi-scale spatial feature map, S is the initial spatial feature map S extracted by the ResNet50 deep network, T _K is the feature map obtained after the dilated convolution, and the shape of the five-scale spatial feature map S _final is 60×60×4096.

Images