CN110738128A - repeated video detection method based on deep learning - Google Patents

Abstract

The invention discloses a method for detecting repeated videos based on deep learning. A neural network is used to extract features from existing videos, a video feature library is established, and then features are extracted from the videos to be detected, and the Euclidean distance between the video features and the features in the library is calculated as similarity. Sex metric, when the distance is less than a set threshold, the video is marked as a duplicate. The method of the invention is different from the traditional method of extracting multiple frames from the video, and establishes a feature library by selecting a frame sequence. Instead, a single feature description file is generated for each video by combining the intermediate layer outputs of the neural network. The solution provided by the example of the present invention uses the method of deep learning to detect repeated videos, and the features extracted by the deep neural network can better represent the videos. Compared with using traditional image feature extraction operators, the accuracy of detecting repeated videos is higher. high.

Description

A Deep Learning-Based Duplicate Video Detection Method

technical field

The invention belongs to the technical fields of computer vision, digital image processing, and deep learning, and particularly relates to a method for detecting repeated videos based on deep learning technology.

Background technique

With the advent of the Internet era, the production and dissemination of videos has become more and more convenient, and the large-scale growth of video data, such as the widespread use of short video applications, has gradually become a way for many people to share life content. But at the same time, a lot of repetitive videos are inevitably generated. Video content has certain economic value. Nowadays, there are many behaviors such as using pirated videos for profit. Some copyright-protected videos are modified and uploaded to video websites without the authorization of the video producers, resulting in copyright issues and damage to video production. At the same time, the video website also faces certain legal risks. Duplicate videos also increase bandwidth and storage costs for video sites. Nowadays, video websites recommend videos according to user preferences, and recommending repeated videos will affect the video viewing experience for users. Therefore, the existence of repeated videos poses challenges to video copyright protection and content recommendation. Today's video data is huge, and it is impossible to screen duplicate videos manually, and computer technology is needed to identify them. Therefore, repetitive video retrieval technology has great practical application significance.

Common duplicate videos mainly include format conversion of the video; adding subtitles and watermarks to the video; video compression, rotation, clipping, etc. Traditional video file hash detection will generate the same hash value for video files with the same content, and judge whether they belong to the same video by comparing whether the hash values are consistent. However, this method can only detect videos with exactly the same content. The hash value generated by the algorithm changes greatly for the modified video files, which cannot be used for repeated video detection. Therefore, digital image processing technology is required to automatically detect the video content and determine the similarity of the video. Humans can easily tell if a video is repetitive, but computers struggle to perform that task. Unlike images, videos have temporal features, which are more difficult to extract than images and require corresponding image processing algorithms. At present, the detection methods of repeated videos mainly use traditional image feature descriptors, such as sift ^[1] , SURF ^[2] and other features, but this method has low robustness.

In recent years, convolutional neural networks have been widely used in computer vision tasks, promoting the accuracy of a series of vision tasks. Visual features extracted by convolutional neural networks are generally more robust, but for repetitive video detection, there are fewer ways to use neural networks to solve them. It still relies on extracting the features of several frames in the middle of the video to represent the whole video, and saves the features of multiple video frames in the retrieval system.

[references]

[1]Liu H,Lu H,Xue X.A Segmentation and Graph-Based Video SequenceMatching Method for Video Copy Detection[J].IEEE Transactions on Knowledge and Data Engineering,2013,25(8):1706-1718.

[2] Yang G, Chen N, Jiang Q. A robust hashing algorithm based on SURF for video copy detection [J]. Computers & Security, 2012, 31(1): 33-39.

[3] Simonyan K, Zisserman A.Very Deep Convolutional Networks for Large-Scale ImageRecognition[J].Computer Science, 2014.

SUMMARY OF THE INVENTION

Aiming at the scene that needs repeated video detection, the invention proposes a repeated video detection method based on deep learning. First, the neural network is used to extract the feature of the video frame, the output of the intermediate layer of the neural network is used as the feature representation of the image, and the features of all video frames are fused as the feature representation of the video. Finally, the distance between the video features is used to measure the similarity of different videos. Spend.

In order to solve the above technical problems, the present invention proposes a method for detecting repetitive videos based on deep learning, which uses neural networks to extract features from existing videos, establishes a video feature library, then extracts features from the video to be detected, and calculates the difference between the video features and the library. The Euclidean distance of the features is used as a similarity measure, and when the distance is less than a set threshold, the video is marked as a duplicate.

The duplicate video detection method includes the following steps:

Step 1: Obtain video frames from an existing video set to obtain a set of all video frames;

Step 2: Extract features from the video frame by using the middle layer of the convolutional neural network; the convolutional neural network is the network structure of vgg16;

First, for a video, obtain a video frame set S, each frame in the set is scaled to a 3-channel image of 224×224 size, as the input of the neural network; the output of the middle layer of the neural network is used as the video feature, and the network structure of vgg16 is Take the feature maps of the conv2_1, conv2_2, conv3_1, conv3_2, conv3_3, conv4_1, conv4_2, conv4_3, conv5_1, conv5_2, conv5_3 layers, a total of 11 layers, all of which are the output of the convolution layer; the size of the convolution kernel of these layers is the same as 3 × 3 , all 0 padding, the convolution kernel moves 1 pixel each time when sliding;

Combined with the feature output of the intermediate layer, a unique 4096 feature vector is finally obtained for each video;

Step 3: Extract features from the video library V to obtain the video feature library Fv;

Step 4: In the retrieval stage, features are extracted for the video v to be retrieved;

Step 5: Use the features of the video v to be retrieved to compare the features of the video feature library. If the conditions are met, it is a repeated video. The comparison method and conditions are set as follows: During retrieval, calculate the distance formula d between different video features, video i, j The distance between them is d, and the threshold t is set. After the distance d is obtained, when d is less than t, it is determined as a similar video, otherwise it is not a similar video.

Further, in step 1, video frames are obtained from the existing video set, and the set of all video frames is obtained as follows:

V=(S ₍₁₎ , S ₍₂₎ ,...S _(n) ), S=(P ₍₁₎ , P ₍₂₎ ,...P _(n) ); V, S represent a set of frames of a single video, S _(n) represents the frame set of the nth video, and P _(n) represents the nth video frame of the video.

In step 2, the process of finally obtaining a unique 4096 feature vector for each video by combining the feature output of the intermediate layer is: the dimension of the feature map output by each layer is:

F _(k) = d(W _(k) ×W _(k) ×C _(k) ), k=1, 2, ..., 11 (1)

Equation (1) indicates that the output dimension of the feature map of the kth layer is W _(k) ×W _(k) ×C _(k) , W _(k) ×W _(k) is the dimension of the feature map of the kth layer, C _{( k)} is the number of channels of the feature map of the kth layer;

The dimension of the compressed feature map:

FM _(k) , =max(F _(k) ), k=1, 2, ..., 11 (2)

Equation (2) represents taking the maximum value for each channel of the k-th layer feature map F _(k) , and obtaining a C _(k) -dimensional vector representation FM _(k) , the dimension is one-dimensional, and the vector length is C _(k) ;

Connect the feature representations of all layers to get the feature representation of the entire video frame: FP _n , the number of output channels of each convolutional layer are: 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 ; The final feature dimension is the sum of the dimensions of the corresponding layer:

128+128+256+256+256+512+512+512+512+512+512=4094 (3)

That is, for a video frame P _(n) , the size of the extracted vector is 4096, and then for all video frames of each video S=(P ₍₁₎ , P ₍₂₎ ,...P _(n) ) a total of n 4096 dimensional vector, take the mean of n vectors to obtain a 4096-dimensional vector T, and then normalize to obtain the feature representation F(V _(n) ) of the entire video V _(n );

The normalization formula is:

In formula (4), μ is the mean of the vector T, σ is the variance, Tv is the final video feature vector, and finally a unique 4096-dimensional vector table Tv _n is obtained for each video, representing the feature of the nth video.

In step 3, the method of step 2 is used to extract features from the video library V, and the video feature library Fv is obtained; in step 4, the method of step 2 is used to extract features of the video v to be retrieved.

In step 5, the calculation formula of the distance between videos i and j is:

Compared with the prior art, the beneficial effects of the present invention are:

In the repeated video detection method of the present invention, it is not necessary to manually determine whether the video is repeated with the existing video library to obtain the characteristics of the video to be detected. The above method is different from the traditional method of extracting multiple frames from a video, and establishing a feature library by selecting a sequence of frames. Instead, a single feature description file is generated for each video by combining the intermediate layer outputs of the neural network. The solution provided by the example of the present invention uses the method of deep learning to detect repeated videos, and the features extracted by the deep neural network can better represent the videos. Compared with using traditional image feature extraction operators, the accuracy of detecting repeated videos is higher. high.

Description of drawings

Fig. 1 is the flow chart of the repetitive video detection method based on deep learning of the present invention;

FIG. 2 is a schematic diagram of a feature extraction principle of a video frame in the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments do not limit the present invention by any means.

The main steps of the deep learning-based repetitive video detection method proposed by the present invention are: using neural network to extract features from existing videos, establishing a video feature library, then extracting features from the video to be detected, and calculating the Euclidean distance between the video features and the features in the library As a similarity measure, when the distance is less than a set threshold, the video is marked as a duplicate. The present invention obtains the feature representation of the low-level to high-level semantic features of the video by extracting the feature outputs of different levels of the neural network, and can obtain more accurate video feature representation by combining the features of different levels.

As shown in Figure 1, the repeated video detection method includes the following steps:

Step 1: Obtain video frames from an existing video set, and obtain a set of all video frames: V, S represent the frame set of a single video.

V=(S ₍₁₎ ,S ₍₂₎ ,...S _(n) ), S _(n) represents the frame set of the nth video.

S=(P ₍₁₎ ,P ₍₂₎ ,...P _(n) ), P _(n) denotes the nth video frame of the video.

Step 2: Use the middle layer of convolutional neural network vgg16 ^[3] to extract features from video frames. The network structure of vgg16 is shown in the following table:

Layer represents the different layers of the neural network, Output Shape represents the output vector dimension of each layer, and Param represents the number of parameters of each layer.

First, for a video, a set S of video frames is obtained, and each frame in the set is scaled to a 3-channel image of size 224×224, which is used as the input of the neural network. In this paper, the output of the intermediate layer of the neural network is used as the video feature, and the feature maps of the conv2_1, conv2_2, conv3_1, conv3_2, conv3_3, conv4_1, conv4_2, conv4_3, conv5_1, conv5_2, conv5_3 layers are taken for vgg16, with a total of 11 layers, all of which are convolutional layer outputs. . The size of the convolution kernels of these layers is uniformly 3×3, filled with all 0s, and the convolution kernels are moved by 1 pixel each time they are slid.

The dimension of the feature map output by each layer is:

F _(k) = d(W _(k) ×W _(k) ×C _(k) ), k=1, 2, ..., 11 (1)

The above formula indicates that the dimension of the feature map output of the kth layer is W _(k) ×W _(k) ×C _(k) , W _(k) ×W _(k) is the dimension of the feature map of the kth layer, C _(k) is the number of channels of the k-th layer feature map.

Step 3: Compress the dimension of the feature map:

FM _(k) , =max(F _(k) ), k=1, 2, ..., 11 (2)

The above formula means taking the maximum value for each channel of the feature map F _(k) of the kth layer, and obtaining a C _(k) -dimensional vector representation FM _(k) , the dimension is one-dimensional, and the vector length is C _(k) .

Then connect the feature representations of all layers, as shown in Figure 2, to obtain the feature representation of the entire video frame: FP _n , it can be seen from the table in step 2 that the number of output channels of each convolutional layer are: 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512. Therefore, the final feature dimension is the sum of the dimensions of the corresponding layers:

128+128+256+256+256+512+512+512+512+512+512=4094 (3)

That is, for a video frame P _(n) , the size of the extracted vector is 4096, and then for all video frames of each video S=(P ₍₁₎ , P ₍₂₎ ,...P _(n) ) a total of n 4096 Dimension vector, take the mean of n vectors to get a 4096-dimensional vector T, and then normalize it to get the feature representation of the entire video V _(n) F(V _(n) )

The normalization formula is:

μ is the mean of the vector T, σ is the variance, and Tv is the final video feature vector with a dimension of 4096. For the video frame set, a unique 4096-dimensional vector table Tv _n is finally obtained for each video, representing the nth Features of the video.

Step 3: Extract features from the video library V in the manner of step 2 to obtain a video feature library Fv.

Step 4: In the retrieval stage, for the video v to be retrieved, the feature Tv is extracted in the manner of step 2.

Step 5: Use TV to compare the features of the video feature library. If the conditions are met, it is a repeated video. The comparison method and conditions are set as follows:

During retrieval, calculate the distance formula d between different video features, the distance between videos i, j is d(Tv _i , Tv _j )), the calculation formula is:

Set the threshold t, and after the distance d is obtained, when d is less than t, it is determined to be a repeated video, otherwise it is not a repeated video.

To sum up, the deep learning-based repetitive video detection method of the present invention can be summarized as including a feature library establishment stage and a discrimination stage.

1. Feature library establishment stage: extract features from existing videos through neural network, and combine the output of the middle layer of neural network as the feature representation of the video, each video corresponds to a 4096-dimensional feature file (Figure 2 shows the features of a single video frame. The principle of extraction) is stored in the database to obtain the video feature library.

2. Discrimination stage: For the discriminated video, use the same method to extract features, and calculate the Euclidean distance for the existing video features in the library, if the distance is less than the threshold t set in advance (generally set the threshold to 0.3). Then it is determined to be a repeated video, otherwise it is not a repeated video. Specifically, the repeated video formats that can be identified in this example include: adding subtitles to the video, zooming in and out, picture-in-picture, format change, video length change, watermarking, mirror flipping, brightness change, and picture clipping.

The above network uses the tensorflow deep learning framework, obtains the video frame sequence through opencv, and scales the video frame sequence to meet the size of the neural network input. In this example, the feature representation of the entire video is obtained by merging the feature maps of the vgg16 intermediate layer.

Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of the present invention, many modifications can be made without departing from the spirit of the present invention, which all belong to the protection of the present invention.

Claims (6)

1. a kind of repetitive video detection method based on deep learning, it is characterized in that, use neural network to extract feature to existing video, set up video feature library, then treat to detect video extraction feature, calculate the Euclidean distance of feature in this video feature and library As a similarity measure, when the distance is less than a set threshold, the video is marked as a duplicate. 2. the repetitive video detection method based on deep learning according to claim 1, is characterized in that, comprises the following steps: Step 1: Obtain video frames from an existing video set to obtain a set of all video frames; Step 2: Extract features from the video frame by using the middle layer of the convolutional neural network; the convolutional neural network is the network structure of vgg16; First, for a video, obtain a video frame set S, each frame in the set is scaled to a 3-channel image of 224×224 size, as the input of the neural network; the output of the middle layer of the neural network is used as the video feature, and the network structure of vgg16 is Take the feature maps of the conv2_1, conv2_2, conv3_1, conv3_2, conv3_3, conv4_1, conv4_2, conv4_3, conv5_1, conv5_2, conv5_3 layers, a total of 11 layers, all of which are the output of the convolution layer; the size of the convolution kernel of these layers is the same as 3 × 3 , all 0 padding, the convolution kernel moves 1 pixel each time when sliding; Combined with the feature output of the intermediate layer, a unique 4096 feature vector is finally obtained for each video; Step 3: Extract features from the video library V to obtain the video feature library Fv; Step 4: In the retrieval stage, features are extracted for the video v to be retrieved; Step 5: Use the features of the video v to be retrieved to compare the features of the video feature library. If the conditions are met, it is a repeated video. The comparison method and conditions are set as follows: During retrieval, calculate the distance formula d between different video features, video i, j The distance between them is d, and the threshold t is set. After the distance d is obtained, when d is less than t, it is determined as a similar video, otherwise it is not a similar video. 3. the repeated video detection method based on deep learning according to claim 2, is characterized in that, obtains video frame from existing video collection, obtains the set of all video frames as follows: V=(S ₍₁₎ ,S ₍₂₎ ,...S _(n) ), S=(P ₍₁₎ ,P ₍₂₎ ,...P _(n) ); V, S denotes the frame set of a single video, S _(n) denotes the frame set of the nth video, and P _(n) denotes the nth video frame of the video. 4. the repetitive video detection method based on deep learning according to claim 2, is characterized in that, in step 2, the process that finally obtains a unique 4096 feature vector for each video in conjunction with the feature output of the middle layer is: The dimension of the feature map output by each layer is: F _(k) = d(W _(k) ×W _(k) ×C _(k) ), k=1,2,...,11 (1) Equation (1) indicates that the output dimension of the feature map of the kth layer is W _(k) ×W _(k) ×C _(k) , W _(k) ×W _(k) is the dimension of the feature map of the kth layer, C _{( k)} is the number of channels of the feature map of the kth layer; The dimension of the compressed feature map: FM _(k) ,=max(F _(k) ), k=1,2,...,11 (2) Equation (2) represents taking the maximum value for each channel of the k-th layer feature map F _(k) , and obtaining a C _(k) -dimensional vector representation FM _(k) , the dimension is one-dimensional, and the vector length is C _(k) ; Connect the feature representations of all layers to get the feature representation of the entire video frame: FP _n , the number of output channels of each convolutional layer are: 128, 128, 256, 256, 256, 512, 512, 512, 512, 512, 512 ; The final feature dimension is the sum of the dimensions of the corresponding layer: 128+128+256+256+256+512+512+512+512+512+512=4094 (3) That is, for a video frame P _(n) , the size of the extracted vector is 4096, and then for all video frames of each video S=(P ₍₁₎ , P ₍₂₎ ,...P _(n) ) a total of n 4096 dimensional vector, take the mean of n vectors to obtain a 4096-dimensional vector T, and then normalize to obtain the feature representation F(V _(n) ) of the entire video V _(n ); The normalization formula is: In formula (4), μ is the mean of the vector T, σ is the variance, Tv is the final video feature vector, and finally a unique 4096-dimensional vector table Tv _n is obtained for each video, representing the feature of the nth video. 5. the repeated video detection method based on deep learning according to claim 4, is characterized in that, in step 3, adopts the mode of step 2 to extract feature to video library V, obtains video feature library Fv; In step 4, adopts step 2 way to treat the retrieved video v to extract features. 6. the repeated video detection method based on deep learning according to claim 5, is characterized in that, in step 5, the calculation formula of the distance between video i, j is:

Images