CN108805036B - Unsupervised video semantic extraction method - Google Patents

Abstract

The invention discloses an unsupervised video semantic extraction method, which comprises the steps of constructing a three-dimensional convolutional neural network model, and training the three-dimensional convolutional neural network model by using labeled video data in a video database; processing video data without labels in a video database into data which is in accordance with the input of a three-dimensional convolution neural network by using a sliding window; using the generated data as input data of a three-dimensional convolution neural network model, and taking output data of a full connection layer of the three-dimensional convolution neural network model as semantic features of a video segment; and using the generated video segment semantic feature sequence as the input of a video semantic self-encoder, and integrating by a self-encoder to obtain the overall semantic features of the video. The embodiment of the invention solves the problem of unsupervised video semantic analysis and extraction by combining the scheme of the three-dimensional convolutional neural network and the cyclic automatic encoder, and improves the accuracy of video semantic extraction.

Description

An Unsupervised Video Semantic Extraction Method

technical field

The invention relates to the technical fields of artificial intelligence and pattern recognition, in particular to a method for extracting semantics from unsupervised video based on a deep learning model.

Background technique

The concept of "semantics" originated at the end of the 19th century. It is the expression of the meanings represented by things in the real world corresponding to virtual data, and the relationship between these meanings. It is the interpretation and logical representation of virtual data in a certain field. . Moreover, "video semantics" is aimed at human thinking. When we want to use computers to understand the "semantics" in videos, computers can only recognize low-level features such as color and shape. Therefore, we need to use some methods to connect these low-level features to form some higher-level meanings, so as to better express the information to be displayed in the video.

Video data is usually unstructured, so the semantic extraction of video needs to be considered from many aspects. In terms of content, it is necessary to consider the spatial and temporal attributes contained in the video. Semantically, it is necessary to consider image features, subtitle text features, voice features, and video description text features included in video information. Video is divided into four structural levels in physical structure: frame, shot, scene and video. The content of the video frame records the characteristics of the object in the video, such as color, texture and shape, etc.; the shot is composed of several consecutive frames, and its content records the motion characteristics of the object in the continuous frames, showing the temporal characteristics of the object. In reality, a shot is the basic unit of video generation, that is, the smallest unit obtained by a camera shooting once; a scene is composed of a series of shots with related semantic content and continuous time, and its content records relatively complex semantic information. Several scenes form a video file, and its content records the semantic information of the entire video.

(1) Video semantic extraction based on key frames, the usual technical process of key frame semantic extraction is: screenshot of video frames; key frame recognition of frame screenshots, semantic analysis of the obtained key frames; voice contained in the video The data is converted into text through speech recognition; the speech text is semantically recognized; the above key frame semantics and speech semantics are combined to obtain the semantics of the video; that is, the image features and sound mfcc features of the video are converted into semantic features , and then combined with the identification of subtitles, the subtitles are processed through Neuro-Linguistic Programming to obtain word vectors and document similarity. The advantage of this method is that it has a better extraction effect on videos with more text content on the video, such as some educational videos. The disadvantage is that for other types of videos with less text, because the subtitle information in the key frame is less, it is difficult to obtain useful text information from it.

(2) Keyword extraction based on video text information, this method is the extraction of plain text, and this method has relatively high requirements for the importance of the word itself and the position of the word, the former word is more important than the latter word, word frequency, The overall order of appearance of words also needs to be integrated. That is to say, the content of the title needs to be very consistent with the semantics of the video, otherwise the accuracy of this method will be very low. The advantage of this method is that the calculation complexity is low, there are mature text processing algorithms in the industry, and various open source packages of algorithms are very convenient. Disadvantages: There are some Internet terms whose meaning is quite different from the literal meaning, which will greatly interfere with the extraction of video semantics.

For the semantic analysis of sports videos, the current methods rarely consider the semantic extraction of unlabeled data, so when the test data does not belong to one of the training data types, domain drift will occur, which will affect the accuracy of video semantic extraction.

Contents of the invention

The purpose of the present invention is to overcome the existing technical deficiencies, and provide a method for video semantic extraction using a combination of a three-dimensional convolutional neural network model and a cyclic autoencoder, which can solve the problem of unsupervised video semantic analysis and extraction, and improve video quality. Semantic extraction accuracy.

Specifically, a method for unsupervised video semantic extraction, is characterized in that, comprises the following steps:

S1: Construct a three-dimensional convolutional neural network model, and use the labeled UCF-101 video set in the video database to train the three-dimensional convolutional neural network model;

S2: Use the sliding window to process the unlabeled video data in the video database into data that conforms to the input of the 3D convolutional neural network;

S3: Use the data generated in step S2 as the input data of the three-dimensional convolutional neural network model, and take the output data of the fully connected layer of the three-dimensional convolutional neural network model as the semantic feature of the video segment;

S4: Use the video segment semantic feature sequence generated in step S3 as the input of the video semantic autoencoder, and obtain the overall semantic feature of the video through the integration of the autoencoder.

Preferably, step S1 includes the following sub-steps:

S11: Construct a three-dimensional convolutional neural network model including five convolutional layers, pooling layers, two fully connected layers and one SOFTMAX layer;

S12: Before using the labeled UCF-101 video set in the video database to train the 3D convolutional neural network, the video data set video preprocessing is required: the original video in the UCF-101 video set needs to be converted into a video frame according to a certain FPS Image collection, image preprocessing of image size adjustment and noise filtering, converting the image into a unified specification of 112*112;

S13: The data format corresponding to the preprocessed UCF-101 video set training video is (X _n , L _n ): n is the number of training videos, where X _n =[x _n(1) ,x _n(2) ,x _n(3) ,...,x _n(m) ] is the preprocessed video picture collection of video X _n , m is the number of video frames converted into pictures, this method uses ffmpeg to convert the video at 20 frames per second Converted to a picture sequence, L _n is the label type corresponding to the video X _n ;

S14: Based on the three-dimensional convolutional neural network model and learning algorithm, use the preprocessed UCF-101 video dataset to train a video category recognition model with a high recognition rate.

Preferably, step S2 includes the following sub-steps:

S21: Supplementary processing is performed on the video frame picture set whose number m of video frame pictures in the test data does not satisfy m=kw, wherein k is any integer, w is the size of the sliding window, and the picture of the last frame of the video is copied until Satisfied that m is a multiple of w;

w代表一次滑动窗口取得的图片集合，其中

代表窗口滑动第k次滑动获得视频图片集。S22: Use the sliding window to slide and read the frame pictures of the video frame sequence. The sliding step size is half of the sliding window. Every time you slide, the acquired frame picture is an input of the three-dimensional convolutional neural network; take the sliding window size w=16 , so the test data form is processed into

w represents a collection of pictures obtained by a sliding window, where

Represents the kth sliding of the window to obtain the video picture set.

Preferably, step S3 includes the following sub-steps:

S31: Using the three-dimensional convolutional neural network model trained in S1 using the UCF-101 video set to identify the processed test video data in S2

S32: fixing the output of the fully connected layer of the three-dimensional convolutional neural network as the number of sub-action types;

输出为第一层全连接层的输出F_k＝[f₁,f₂,f₃,...,f₄₀₉₆]，其中F_k的维度4096为三维卷积神经网络第一层全连接层的输出维度；S33: The input of the three-dimensional convolutional neural network is defined in S22

The output is the output of the first fully connected layer F _k = [f ₁ , f ₂ , f ₃ ,..., f ₄₀₉₆ ], where the dimension of F _k is 4096 for the first fully connected layer of the three-dimensional convolutional neural network. output dimension;

对应三维卷积神经网络输出为[F₁,F₂,F₃,...,F_k]其维度为4096*k维。S34: Test video data

The output corresponding to the three-dimensional convolutional neural network is [F ₁ , F ₂ , F ₃ ,...,F _k ] and its dimension is 4096*k dimensions.

Preferably, step S4 includes the following sub-steps:

语义特征提取结果[F₁,F₂,F₃,...,F_k]作为视频语义自编码器的输入提取视频整体语义特征；S41: Use the three-dimensional convolutional neural network model in S3 to test the video data

The semantic feature extraction results [F ₁ , F ₂ , F ₃ ,...,F _k ] are used as the input of the video semantic autoencoder to extract the overall semantic features of the video;

循环自编码器的目标函数为：

其中A(x)表示输入序列[F₁,F₂,F₃,...,F_k]对应的语义树的所有可能，T(y)表示所有可能的特征对，循环自编码的一次编码过程是选出所有编码对中重构误差最小的一个特征对，将这对特征从特征序列中移除并将其父特征作为这一个特征对的代表组成一个新的特征序列；S42: The cyclic autoencoder converts the input feature sequence [F ₁ ,F ₂ ,F ₃ ,...,F _k ] into a feature pair sequence [[F ₁ ,F ₂ ],[F ₂ ,F ₃ ],[ F ₃ ,F ₄ ],...,[F _k-1 ,F _k ]] adopts the idea of greedy algorithm, and its process is to select each pair of features in the sequence of feature pairs in turn and integrate them into a parent feature, expressing It is: F _1,2 ＝f(W ⁽¹⁾ [F ₁ ,F ₂ ]+b ⁽¹⁾ ), where W ⁽¹⁾ represents the matrix parameter of n*n, b ⁽¹⁾ is a bias item, W ⁽¹⁾ and b ⁽¹⁾ are obtained by learning feature sequence pairs; the reconstruction process of F _1,2 is: [F ₁ ', F ₂ ']=W ⁽²⁾ F _1,2 +b ^{(2 )} where W ⁽²⁾ represents the matrix parameter of n*n, b ⁽²⁾ is a bias item different from b ⁽¹⁾ , and similarly W ⁽²⁾ and b ⁽²⁾ are obtained by learning reconstruction errors; self-encoding The reconstruction error of the device is:

The objective function of the recurrent autoencoder is:

Where A(x) represents all possibilities of the semantic tree corresponding to the input sequence [F ₁ , F ₂ , F ₃ ,...,F _k ], T(y) represents all possible feature pairs, one-time encoding of cyclic autoencoder The process is to select a feature pair with the smallest reconstruction error among all coding pairs, remove this pair of features from the feature sequence and use its parent feature as a representative of this feature pair to form a new feature sequence;

S43: Repeat the self-encoding process of S42 until the number of feature vectors in the feature sequence is 1;

作为视频X_n的语义特征向量。S44: Loop self-encoder to output the final feature vector

as the semantic feature vector of video X _n .

The beneficial effects of the present invention are:

The invention solves the problem of unsupervised video semantic analysis and extraction by combining the scheme of the three-dimensional convolutional neural network and the cyclic automatic encoder, and improves the accuracy of video semantic extraction.

Description of drawings

Fig. 1 is a flow chart of a method for extracting semantic meaning from unsupervised video proposed by the present invention.

Fig. 2 is a structural diagram of a three-dimensional convolutional neural network model constructed by the present invention.

Fig. 3 is a schematic flow chart of training a three-dimensional convolutional neural network model in the method of the present invention.

Fig. 4 is a schematic flow chart of extracting video semantic features in the method of the present invention.

Fig. 5 is an architecture diagram of the present invention based on a three-dimensional convolutional neural network and a cyclic autoencoder model.

Detailed ways

In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation manners of the present invention will now be described with reference to the accompanying drawings.

A flow chart of an embodiment of an unsupervised video semantic extraction method proposed by the present invention is shown in Figure 1, comprising the following steps:

S1: Construct a three-dimensional convolutional neural network model, and use the labeled UCF-101 video set in the video database to train the three-dimensional convolutional neural network model;

S2: Use the sliding window to process the unlabeled video data in the video database into data that conforms to the input of the 3D convolutional neural network;

S3: Use the data generated in step S2 as the input data of the three-dimensional convolutional neural network model, and take the output data of the fully connected layer of the three-dimensional convolutional neural network model as the semantic feature of the video segment;

S4: Use the video segment semantic feature sequence generated in step S3 as the input of the video semantic autoencoder, and obtain the overall semantic feature of the video through the integration of the autoencoder.

As a preferred embodiment, step S1 includes the following sub-steps:

S11: Construct a three-dimensional convolutional neural network model including five convolutional layers, a pooling layer, two fully connected layers and one layer of SOFTMAX. The structure of the constructed three-dimensional convolutional neural network model is shown in Figure 2;

S12: Before using the labeled UCF-101 video set in the video database to train the 3D convolutional neural network, the video data set video preprocessing is required: the original video in the UCF-101 video set needs to be converted into a video frame according to a certain FPS Image collection, image preprocessing of image size adjustment and noise filtering, converting the image into a unified specification of 112*112; image preprocessing is due to various conditions and random interference, these image collections often cannot are used directly, thus requiring image preprocessing such as resizing, noise filtering, etc., to be performed on them at an early stage of image processing;

S13: The data format corresponding to the preprocessed UCF-101 video set training video is (X _n , L _n ): n is the number of training videos, where X _n =[x _n(1) ,x _n(2) ,x _n(3) ,...,x _n(m) ] is the preprocessed video picture collection of video X _n , m is the number of video frames converted into pictures, this method uses ffmpeg to convert the video at 20 frames per second Converted to a picture sequence, L _n is the label type corresponding to the video X _n ;

S14: Based on the three-dimensional convolutional neural network model and learning algorithm, use the preprocessed UCF-101 video dataset to train a video category recognition model with a high recognition rate.

Among them, the flow chart of training the three-dimensional convolutional neural network model is shown in Fig. 3 . The parameters of the three-dimensional convolutional neural network are randomly initialized, and the UCF-101 video dataset is preprocessed, and then the BP algorithm is used to train the model to obtain the optimal video action category recognition model.

As a preferred embodiment, step S2 includes the following sub-steps:

S21: Supplementary processing is performed on the video frame picture set whose number m of video frame pictures in the test data does not satisfy m=kw, wherein k is any integer, w is the size of the sliding window, and the picture of the last frame of the video is copied until Satisfied that m is a multiple of w;

w代表一次滑动窗口取得的图片集合，其中

代表窗口滑动第k次滑动获得视频图片集。S22: Use the sliding window to slide and read the frame pictures of the video frame sequence. The sliding step size is half of the sliding window. Every time you slide, the acquired frame picture is an input of the three-dimensional convolutional neural network; take the sliding window size w=16 , so the test data form is processed into

w represents a collection of pictures obtained by a sliding window, where

Represents the kth sliding of the window to obtain the video picture set.

As a preferred embodiment, step S3 includes the following sub-steps:

S31: Using the three-dimensional convolutional neural network model trained in S1 using the UCF-101 video set to identify the processed test video data in S2

S32: fixing the output of the fully connected layer of the three-dimensional convolutional neural network as the number of sub-action types;

输出为第一层全连接层的输出F_k＝[f₁,f₂,f₃,...,f₄₀₉₆]，其中F_k的维度4096为三维卷积神经网络第一层全连接层的输出维度；S33: The input of the three-dimensional convolutional neural network is defined in S22

The output is the output of the first fully connected layer F _k = [f ₁ , f ₂ , f ₃ ,..., f ₄₀₉₆ ], where the dimension of F _k is 4096 for the first fully connected layer of the three-dimensional convolutional neural network. output dimension;

对应三维卷积神经网络输出为[F₁,F₂,F₃,...,F_k]其维度为4096*k维。S34: Test video data

The output corresponding to the three-dimensional convolutional neural network is [F ₁ , F ₂ , F ₃ ,...,F _k ] and its dimension is 4096*k dimensions.

As a preferred embodiment, step S4 includes the following sub-steps:

语义特征提取结果[F₁,F₂,F₃,...,F_k]作为视频语义自编码器的输入提取视频整体语义特征；S41: Use the three-dimensional convolutional neural network model in S3 to test the video data

The semantic feature extraction results [F ₁ , F ₂ , F ₃ ,...,F _k ] are used as the input of the video semantic autoencoder to extract the overall semantic features of the video;

循环自编码器的目标函数为：

其中A(x)表示输入序列[F₁,F₂,F₃,...,F_k]对应的语义树的所有可能，T(y)表示所有可能的特征对，循环自编码的一次编码过程是选出所有编码对中重构误差最小的一个特征对，将这对特征从特征序列中移除并将其父特征作为这一个特征对的代表组成一个新的特征序列；S42: The cyclic autoencoder converts the input feature sequence [F ₁ ,F ₂ ,F ₃ ,...,F _k ] into a feature pair sequence [[F ₁ ,F ₂ ],[F ₂ ,F ₃ ],[ F ₃ ,F ₄ ],...,[F _k-1 ,F _k ]] adopts the idea of greedy algorithm, and its process is to select each pair of features in the sequence of feature pairs in turn and integrate them into a parent feature, expressing It is: F _1,2 ＝f(W ⁽¹⁾ [F ₁ ,F ₂ ]+b ⁽¹⁾ ), where W ⁽¹⁾ represents the matrix parameter of n*n, b ⁽¹⁾ is a bias item, W ⁽¹⁾ and b ⁽¹⁾ are obtained by learning feature sequence pairs; the reconstruction process of F _1,2 is: [F ₁ ', F ₂ ']=W ⁽²⁾ F _1,2 +b ^{(2 )} where W ⁽²⁾ represents the matrix parameter of n*n, b ⁽²⁾ is a bias item different from b ⁽¹⁾ , and similarly W ⁽²⁾ and b ⁽²⁾ are obtained by learning reconstruction errors; self-encoding The reconstruction error of the device is:

The objective function of the recurrent autoencoder is:

Where A(x) represents all possibilities of the semantic tree corresponding to the input sequence [F ₁ , F ₂ , F ₃ ,...,F _k ], T(y) represents all possible feature pairs, one-time encoding of cyclic autoencoder The process is to select a feature pair with the smallest reconstruction error among all coding pairs, remove this pair of features from the feature sequence and use its parent feature as a representative of this feature pair to form a new feature sequence;

S43: Repeat the self-encoding process of S42 until the number of feature vectors in the feature sequence is 1;

作为视频X_n的语义特征向量。S44: Loop self-encoder to output the final feature vector

as the semantic feature vector of video X _n .

Fig. 4 is a schematic flow diagram of extracting video semantic features in the method of the embodiment of the present invention. The video data set is preprocessed through data, and then processed through a sliding window, and the trained three-dimensional convolutional neural network is used to extract features to obtain a feature sequence, and finally through a loop The autoencoder integrates feature sequences to obtain semantic features.

Fig. 5 is an architecture diagram based on a three-dimensional convolutional neural network and a cyclic autoencoder model according to an embodiment of the present invention. It can be seen that the video is processed to obtain a video frame sequence, and the processed video frame sequence extracts frame features through a three-dimensional convolutional neural network to form The video frame feature sequence is converted into a coded feature sequence to obtain video semantic features through a loop self-encoder.

The embodiment of the present invention solves the problem of unsupervised video semantic analysis and extraction by combining a three-dimensional convolutional neural network and a cyclic autoencoder, and improves the accuracy of video semantic extraction.

It should be noted that, for the sake of simple description, all the aforementioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because according to the application, certain steps may be performed in other order or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification belong to preferred embodiments, and the actions and units involved are not necessarily required by this application.

In the foregoing embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented through computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. During execution, it may include the processes of the embodiments of the above-mentioned methods. Wherein, the storage medium may be a magnetic disk, an optical disk, a ROM, a RAM or the like.

The above disclosures are only preferred embodiments of the present invention, and certainly cannot limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention still fall within the scope of the present invention.

Claims (2)

1. An unsupervised video semantic extraction method is characterized by comprising the following steps:

s1: constructing a three-dimensional convolutional neural network model, and training the three-dimensional convolutional neural network model by using a UCF-101 video set with a label in a video database;

s2: processing video data without labels in a video database into data which is in accordance with the input of a three-dimensional convolution neural network by using a sliding window;

s3: using the data generated in the step S2 as input data of the three-dimensional convolutional neural network model, and taking output data of a full connection layer of the three-dimensional convolutional neural network model as semantic features of the video segment;

s4: using the video segment semantic feature sequence generated in the step S3 as the input of a video semantic self-encoder, and integrating through the self-encoder to obtain the whole video semantic features;

step S1 comprises the following substeps:

s11: constructing a three-dimensional convolutional neural network model comprising five convolutional layers, a pooling layer, two full-connection layers and one SOFTMAX layer;

s12: before training the three-dimensional convolutional neural network by using the labeled UCF-101 video set in the video database, video preprocessing needs to be carried out on the video data set: converting an original video in a UCF-101 video set into a video frame picture set according to a certain FPS, carrying out image preprocessing of size adjustment and noise filtration on the picture, and converting the picture into a unified specification of 112 x 112;

s13: the corresponding data form of the training video of the UCF-101 video set after preprocessing is (X) _n ,L _n ): n is the number of training videos, wherein X _n ＝[x _n(1) ,x _n(2) ,x _n(3) ,...,x _n(m) ]Is a video X _n The method comprises the steps that a preprocessed video picture set is formed, m is the number of frames of a video converted into pictures, ffmpeg is used for converting the video into a picture sequence according to 20 frames per second, and L is _n As video X _n Corresponding to the label type;

s14: training a video type identification model with high identification rate by using a preprocessed UCF-101 video data set based on a three-dimensional convolution neural network model and a learning algorithm;

step S2 comprises the following substeps:

s21: performing supplementary processing on a video frame picture set of which the number m of video frame pictures in the test data does not satisfy m = kw, wherein k is any integer and w is the size of a sliding window, and copying the picture of the last frame of the video until m is a multiple of w;

s22: sliding the video frame sequence by using a sliding window to read a frame picture, wherein the sliding step length is half of that of the sliding window, and the obtained frame picture is input once for the three-dimensional convolutional neural network when the frame picture slides once; taking the sliding window size w =16, the test dataform is processed to become

w represents a set of pictures taken in a sliding window, wherein

Sliding the representative window for the kth time to obtain a video picture set;

s31: identifying the processed test video data in S2 by using a three-dimensional convolution neural network model obtained by training the UCF-101 video set in S1

S32: fixing the output of the full connection layer of the three-dimensional convolutional neural network as the number of the sub-action types;

s33: three-dimensional convolutional neural network input as defined in S22

The output is the output F of the first layer full connection layer _k ＝[f ₁ ,f ₂ ,f ₃ ,...,f ₄₀₉₆ ]In which F is _k The dimension 4096 is the output dimension of the first full link layer of the three-dimensional convolutional neural network;

s34: testing video data

Corresponding to a three-dimensional convolutional neural network output of [ F ₁ ,F ₂ ,F ₃ ,...,F _k ]Its dimension is 4096 × k dimensions.

2. The unsupervised video semantic extraction method according to claim 1, wherein the step S4 comprises the following sub-steps:

s41: testing video data by using three-dimensional convolution neural network model in S3

Semantic feature extraction result [ F ₁ ,F ₂ ,F ₃ ,...,F _k ]Extracting the integral semantic features of the video as the input of a video semantic self-encoder;

s42: cyclic self-encoder inputs characteristic sequence F ₁ ,F ₂ ,F ₃ ,...,F _k ]Conversion into a characteristic pair sequence [ [ F ] ₁ ,F ₂ ],[F ₂ ,F ₃ ],[F ₃ ,F ₄ ],...,[F _k-1 ,F _k ]]The method adopts the greedy algorithm idea, and the process is to sequentially select each pair of features in the feature pair sequence and integrate the features into a father feature, which is expressed as: f _1,2 ＝f(W ⁽¹⁾ [F ₁ ,F ₂ ]+b ⁽¹⁾ ) Wherein W is ⁽¹⁾ Parameters of a matrix representing n x n, b ⁽¹⁾ Is an offset term, W ⁽¹⁾ And b ⁽¹⁾ Is obtained by learning the feature sequence pair; f _1,2 The reconstruction process of (2) is as follows: [ F ] ₁ ',F ₂ ']＝W ⁽²⁾ F _1,2 +b ⁽²⁾ Wherein W ⁽²⁾ Parameters of a matrix representing n x n, b ⁽²⁾ Is different from b ⁽¹⁾ Bias term of (1), likewise W ⁽²⁾ And b ⁽²⁾ Is obtained by learning reconstruction errors; the reconstruction error from the encoder is:

the objective function of the cyclic auto-encoder is:

wherein A (x) represents the input sequence [ F ] ₁ ,F ₂ ,F ₃ ,...,F _k ]All the possibilities of the corresponding semantic tree, T (y) represents all the possible feature pairs, one coding process of the cyclic self-coding is to select a feature pair with the minimum reconstruction error in all the coding pairs, remove the feature pair from the feature sequence and take the parent feature of the feature pair as the representation of the feature pair to form a new feature sequence;

s43: repeating the self-coding process of S42 until the number of the feature vectors in the feature sequence is 1;

s44: outputting final feature vector from encoder circularly

As video X _n The semantic feature vector of (1).

Images