CN117994820B - Hierarchical graph convolution gesture recognition method and device based on time-frequency data fusion - Google Patents

Abstract

The invention discloses a hierarchical graph convolution gesture recognition method and device based on time-frequency data fusion, comprising a network structure consisting of a basic block and a full-connection layer; the network structure performs the following steps: the layered hand image convolution module splits the gesture image to be identified into joint layered images with preset numbers, and extracts characteristics of the joint layered images; splicing all joint layered graph features to obtain a first image feature; the time convolution module sequentially carries out convolution, activation and residual error connection operation according to the input first image characteristics to obtain second image characteristics; the spatial attention module performs feature extraction according to the input second image features to obtain third image features; the time convolution module sequentially carries out convolution, activation and residual error connection operation according to the input third image characteristic to obtain a fourth image characteristic; and outputting a gesture prediction result by the full-connection layer according to the fourth image characteristic. Thereby improving the effect of gesture recognition.

Description

Layered graph convolution gesture recognition method and device based on time-frequency data fusion

Technical Field

The present invention relates to the field of computer vision technology, and in particular to a layered graph convolution gesture recognition method and device based on time-frequency data fusion.

Background technique

With the development of artificial intelligence technology, the use of deep learning models for gesture recognition has become a trend. For example, prior art 1 (Devineau G, Moutarde F, Xi W, et al. "Deep learning for hand gesture recognition on skeletal data") first proposed to extract features along the time dimension of the skeleton sequence through a convolutional neural network (CNN). Prior art 2 (Lai K, Yanushkevich SN. "CNN+RNN depth and skeleton based dynamic hand gesture recognition") proposed to input the skeleton as a time series into a recurrent neural network (RNN) for feature extraction. However, prior art 1 and 2 cannot naturally represent non-Euclidean skeleton data.

ST-GCN (Spatial temporal graph convolutional networks) proposed in prior art 3 (Yan S, Xiong Y, Lin D. "Spatial temporal graph convolutional networks for skeleton-based action recognition") is a typical representative of spatial domain-based methods. ST-GCN is composed of a stack of several spatial blocks and temporal blocks, in which the spatial block is a submodule that aggregates the spatial information of a single frame through GCN (Graph Convolutional Network), while the temporal block is a submodule that uses one-dimensional convolution to associate the information between frames in the temporal dimension. ST-GCN is composed of a stack of several spatial blocks and temporal blocks, in which the spatial block is a submodule that aggregates the spatial information of a single frame through GCN, while the temporal block is a submodule that uses one-dimensional convolution to associate the information between frames in the temporal dimension. However, the graph convolution kernel cannot change during the training process, so it is difficult to associate the information of distant joints.

The 2s-AGCN (Two-stream adaptive graph convolutional networks, two-stream adaptive graph convolutional networks) proposed in the prior art 4 (Shi L, Zhang Y, Cheng J, et al. "Two-stream adaptive graph convolutional networks for skeleton-based action recognition") converts the input The linear transformation is and/> , then /> With/> Perform matrix multiplication to get a size of/> The matrix of , where/> represents the set of real numbers, /> is the number of input channels, /> is the number of frames and /> is the number of nodes. Finally/> Add it to the adjacency matrix of the skeleton graph to get the final skeleton graph topology matrix, where/> is the number of output channels. The purpose of this is to learn the potential relationship between nodes through network training. The CTR-GCN (Channel-wise topology refinement-graph convolutional networks) proposed in the prior art 5 (Chen Y, Zhang Z, Yuan C, et al. "Channel-wise topology refinement graph convolution for skeleton-based action recognition") uses a learnable matrix to find the potential relationship between distant skeleton nodes. Through data-driven learning of a graph topology structure for each channel, a graph adjacency matrix for the input instance is obtained, so that potential node relationships can be discovered.

Existing technologies 4 and 5 rely too much on the initial skeleton graph topology. Although the models can learn images, they are still affected by the initialized skeleton graph topology, thus paying more attention to the correlation between the original nodes.

Prior art 6 (Lee J, Lee M, Lee D, et al. "Hierarchically decomposed graph convolutional networks for skeleton-based action recognition") proposed that the nodes that are more important for identifying actions should be considered. To this end, the skeleton graph is divided into multiple layers, and the joint point groups are divided by the number of hops from the center point. The features extracted from the graph adjacency matrix of different joint point groups are averaged and pooled to obtain more representative action features. However, if the network model of the graph convolutional neural network series needs to take global information into account, deeper GCN layers must be stacked to expand its aggregation radius to the entire graph, which will cause the problem of excessive computation.

Summary of the invention

The technical problem to be solved by the present invention is to provide a method and device for gesture recognition based on layered graph convolution of time-frequency data fusion to improve the effect of gesture recognition.

In order to solve the above technical problems, the technical solution adopted by the present invention is:

A hierarchical graph convolution gesture recognition method based on time-frequency data fusion includes a network structure composed of a basic block and a fully connected layer; the basic block is composed of a hierarchical hand graph convolution module, a temporal convolution module, a spatial attention module, and a temporal convolution module sequentially connected by residual connection; the network structure performs the following steps:

The layered hand graph convolution module splits the gesture image to be recognized into a preset number of joint layered graphs, and extracts joint layered graph features; splices all the joint layered graph features to obtain a first image feature;

The temporal convolution module sequentially performs convolution, activation and residual connection operations according to the first image feature input to obtain a second image feature;

The spatial attention module performs feature extraction based on the input second image feature to obtain a third image feature;

The temporal convolution module sequentially performs convolution, activation and residual connection operations according to the input third image feature to obtain a fourth image feature;

The fully connected layer outputs a gesture prediction result according to the fourth image feature.

In order to solve the above technical problems, another technical solution adopted by the present invention is:

A device for layered graph convolution gesture recognition based on time-frequency data fusion includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned layered graph convolution gesture recognition method based on time-frequency data fusion is implemented.

The beneficial effects of the present invention are as follows: by using a graph convolutional neural network and a graph convolution kernel, it is possible to naturally express non-Euclidean structures and learn the correlation between distant joints through training samples; by adopting a layered hand graph structure, the hand skeleton is divided into multiple layers and the correlation between the nodes of each layer is learned through training, thereby getting rid of the dependence on the initial skeleton graph topological structure, so that the model can extract more diverse skeleton sequence features; by replacing redundant graph convolution blocks with spatial attention blocks to extract global features, and using spatial attention blocks and graph convolution blocks to be stacked alternately, it is possible to avoid the computational burden brought by stacking too many graph convolution layers, reduce system complexity, and ensure the global feature extraction capability of the model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a network structure of a layered graph convolution gesture recognition method based on time-frequency data fusion in an embodiment of the present invention;

FIG2 is a flowchart of a method for gesture recognition based on layered graph convolution and time-frequency data fusion according to an embodiment of the present invention;

FIG3 is a structural diagram of a hand hierarchical decomposition graph convolution block in a layered graph convolution gesture recognition method based on time-frequency data fusion in an embodiment of the present invention;

FIG4 is a visualized hand layer diagram in a layered graph convolution gesture recognition method based on time-frequency data fusion in an embodiment of the present invention;

FIG5 is a schematic diagram of hand node numbering in a layered graph convolution gesture recognition method based on time-frequency data fusion in an embodiment of the present invention;

FIG6 is a diagram of a multi-stream fusion prediction structure in a layered graph convolution gesture recognition method based on time-frequency data fusion in an embodiment of the present invention;

FIG. 7 is a schematic diagram of the structure of a layered graph convolution gesture recognition device based on time-frequency data fusion in an embodiment of the present invention.

Detailed ways

In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in conjunction with the implementation modes and the accompanying drawings.

Please refer to Figure 1, a layered graph convolution gesture recognition method based on time-frequency data fusion includes a network structure consisting of a basic block and a fully connected layer; the basic block is composed of a layered hand graph convolution module, a temporal convolution module, a spatial attention module, and a temporal convolution module, which are sequentially connected by residual connections; the network structure performs the following steps:

The layered hand graph convolution module splits the gesture image to be recognized into a preset number of joint layered graphs, and extracts joint layered graph features; splices all the joint layered graph features to obtain a first image feature;

The temporal convolution module sequentially performs convolution, activation and residual connection operations according to the first image feature input to obtain a second image feature;

The spatial attention module performs feature extraction based on the input second image feature to obtain a third image feature;

The temporal convolution module sequentially performs convolution, activation and residual connection operations according to the input third image feature to obtain a fourth image feature;

The fully connected layer outputs a gesture prediction result according to the fourth image feature.

From the above description, it can be seen that the beneficial effects of the present invention are: by using graph convolutional neural networks and graph convolution kernels, non-Euclidean structures can be naturally expressed and the correlation between distant joints can be learned through training samples; a layered hand graph structure is adopted, the hand skeleton is divided into multiple layers and the correlation between the nodes of each layer is learned through training, thereby getting rid of the dependence on the initial skeleton graph topological structure, so that the model can extract more diverse skeleton sequence features; global features are extracted by replacing redundant graph convolution blocks with spatial attention blocks, and spatial attention blocks and graph convolution blocks are used to stack alternately, which can not only avoid the computational burden brought by stacking too many graph convolution layers and reduce the system complexity, but also ensure the global feature extraction capability of the model.

Furthermore, the layered hand graph convolution module splits the gesture image to be recognized into a preset number of joint layered graphs including:

Converting the gesture image to be recognized into a node gesture graph with a preset number of joint points, and determining the central joint point of the node gesture graph;

Grouping all the joint points according to the distance from the joint point to the central joint point to obtain a hierarchical joint point set;

Two adjacent groups of the hierarchical joint point sets are sequentially connected to form edge sets, and according to the number of the edge sets, a corresponding number of the hierarchical graphs are obtained.

From the above description, it can be seen that by converting the gesture image to be recognized into a node gesture graph with a preset number of joint points, and taking the number of hops from the central node as the basis for grouping, a data set and a layered graph are obtained, so that the layered graph has different characteristics, and more diverse graph features can be extracted through the layered graph.

Furthermore, the layered hand graph convolution module includes three graph convolution flows and one graph edge convolution flow;

The extracted joint layered graph features include:

The layered graph is passed through four convolution kernels to change the number of channels to form a four-stream output, and feature extraction is performed through the graph convolution stream and the graph edge convolution stream respectively to obtain three groups of graph convolution features and one group of graph edge convolution features, wherein feature extraction through the graph convolution stream includes:

;

;

Among them, L represents the graph convolution feature; k represents the number of layers; S=0, 1, 2, represents the number of channels; and/> represents a linear layer; X represents input; /> Represents a graph adjacency matrix; /> Represents the intermediate variable, through the linear layer/> After processing the input X, we get:

Feature extraction through the graph edge convolution flow includes:

;

in, represents the k-th layer edge convolution feature,/> is the number of output channels, /> is the number of frames, /> is the number of key points of the hand; /> represents the set of real numbers;

The joint layered graph features are obtained by concatenating the graph convolution features and the graph edge convolution features. Specifically: ;

Among them, || represents the channel splicing operation; Represents the joint layered graph features of the kth layer; /> They respectively represent the k-th layer graph convolution features obtained based on the zeroth channel convolution flow (S=0), the k-th layer graph convolution features obtained based on the first channel convolution flow (S=1), and the k-th layer graph convolution features obtained based on the second channel convolution flow (S=2).

From the above description, we can see that features are extracted in parallel using graph convolution operations and graph edge convolution operations. The graph convolution operation can extract topological structure information based on the node position, while the graph edge convolution operation can find nodes with similar semantics for feature extraction. The combination of the two operations can extract different features, making the model more generalizable. And by using three graph convolution operations, different features can be learned based on each convolution operation channel, thereby extracting more diverse features.

Furthermore, the splicing of all the joint layered graph features to obtain the first image feature includes:

;

;

in, ; Z is the attention function, used to calculate the output of each layer/> The coefficients are then weighted and added to obtain / > ; /> Respectively represent the joint layered graph features of the first layer, the joint layered graph features of the second layer, the joint layered graph features of the third layer, and the joint layered graph features of the fourth layer;/> Represents the first image feature.

It can be seen from the above description that the first image feature is obtained by calculating the weight coefficient of each joint layered graph feature, thereby realizing the aggregation of different joint layered graph features.

Furthermore, the temporal convolution module sequentially performs convolution, activation and residual connection according to the first image feature of the input, including:

;

in, ,/> is the output of the temporal convolution module, representing the second image feature; /> is the number of output channels, /> is the number of frames output, /> is the number of key points of the hand; /> is the ReLU activation function, /> It is a convolution operation; /> For input.

From the above description, it can be seen that by performing convolution, activation and residual connection operations on the input features through the temporal convolution module, it is possible to simultaneously extract and aggregate multi-frame features and extract the global skeleton features within a single frame.

Furthermore, the spatial attention module performs feature extraction based on the input second image feature, and obtains the third image feature including:

Inputting the second image feature into a preset number of spatial attention blocks in series for processing;

The data processed by the spatial attention block is passed through a linear layer to change the number of channels and output through a residual connection;

Pass the output of the residual connection through the activation function, linear layer and batch normalization in sequence;

After being activated again by the activation function, a residual connection is performed and then output, so as to obtain the third image feature;

The specific formula is as follows:

;

;

;

;

in, represents the output of the spatial attention block, /> ; /> is the number of key points of the hand, /> is the number of heads of the multi-head attention mechanism, /> is the number of output channels of the self-attention block,/> The number of frames to be output; /> and/> Represents an embedded function; /> Represents the residual connection output, /> ,/> is the number of output channels; /> 、/> Represents the weights of different linear layers; /> Represents the output after batch normalization, /> , B is the batch standardization operation; /> Indicates the final output, /> ,/> is the activation function; /> Represents a set of real numbers; the softmax function is used to output the predicted probabilities of each category; /> Represents the output of the T module, T represents the number of frames, /> Indicates/> The features of the τth frame in .

From the above description, it can be seen that by processing the input features through the temporal convolution module and the spatial attention module, it is possible to simultaneously extract and aggregate multi-frame features and extract the global skeleton features within a single frame.

Furthermore, it also includes:

The joint flow data is converted from the time domain to the frequency domain through Fourier transform to obtain phase characteristics and amplitude characteristics;

The phase feature and amplitude feature are used for model learning.

From the above description, it can be seen that by expanding the feature data stream through Fourier transform, more diverse data features can be taken into account during fusion prediction, thereby improving the accuracy and robustness of the model.

Another embodiment of the present invention provides a layered graph convolution gesture recognition device based on time-frequency data fusion, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned layered graph convolution gesture recognition method based on time-frequency data fusion is implemented.

The layered graph convolution gesture recognition method and device based on time-frequency data fusion provided by the present invention can be applied to the scene of gesture recognition, which is described below through specific implementation methods:

Embodiment 1

Please refer to Figure 1, a hierarchical graph convolution gesture recognition method based on time-frequency data fusion, as shown in part (a), includes a network structure consisting of a basic block (HHTSAT) and a fully connected layer (Fully Connection, FC); as shown in part (b), the basic block is composed of a hierarchical hand graph convolution module (HH), a temporal convolution module (Temporal convolution module, T), a spatial attention module (SpatialAttention module, SA), and a temporal convolution module, which are residually connected in sequence, that is, HH, T, SA, and T are residually connected in sequence.

Referring to FIG. 2 , the network structure performs the following steps:

S1. The layered hand graph convolution module splits the gesture image to be recognized into a preset number of joint layered graphs, and extracts the joint layered graph features; splices all the joint layered graph features to obtain the first image feature; as shown in part (c) of Figure 1, the layered hand graph convolution module consists of a hand hierarchically decomposed graph convolution block (Hierarchically Decomposed Hand Graph Convolution, HDH-GC) and a layer attention fusion module (Attention-Guided Hierarchy Aggregation, A-HA), where HDH-GC can be further split into 4 graph convolution blocks as shown in Figure 3, which represents the structure of the k-th layer graph convolution block, which can extract more diverse graph features, specifically:

S11, converting the gesture image to be recognized into a node gesture graph with a preset number of joint points, and determining the central joint point of the node gesture graph; for example, determining the palm node as the central joint point;

S12, grouping all the joint points according to the distance from each joint point to the central joint point to obtain a hierarchical joint point set;

S13, connecting two adjacent groups of the layered joint point sets in sequence to form edge sets, and obtaining a corresponding number of the layered graphs according to the number of the edge sets; as shown in FIG4, it is a visualization result after the hand is layered; wherein, the hollow point set in each layer graph does not initially belong to the layer, while the solid point set and the oblique stripe point set are two adjacent groups of point sets, and the nodes of the two adjacent groups are connected in sequence to form an edge set, and the defined formed edge set is represented by an adjacency matrix; for example, the joint points of the hand are divided into K+1 groups (K=4, for the two data sets currently used, K can only be 4, if other data sets are stored and different numbers of key points are used to represent gestures, it is necessary to modify the K value and re-divide the groups), wherein each grouped node set is specifically:

;

;

;

;

;

As shown in Figure 5, 1 to 22 represent the hand node numbers; then define the grouped graph adjacency matrix:

;

;

;

Where k is the number of layers, In the centroid partitioning strategy, a set is formed by three edge subsets: the self-loop edge set, the outward edge set and the inward edge set;/> is the number of key points of the hand; /> It is the first after stratification/> （/> =1, 2, 3, 4) layer edge set; /> Indicates that two groups of nodes are connected only to themselves to form a self-loop edge set; /> Indicates that from the /> Each point in the joint point group points to the first/> The outward edge set of the one-way edges adjacent to each point in the joint point group; /> Indicates that from the /> Each point in the joint point group points to the first/> The inward edge set of unidirectional edges formed by each point in the joint point group;

As shown in Figure 3, it is the structure of the k-th layer graph convolution block, which includes three graph convolution flows and one graph edge convolution flow. The layered graph is passed through four CNN convolution kernels of size 1×1 to change the number of channels to form a four-stream output, and feature extraction is performed through the graph convolution flow and the graph edge convolution flow (Edge Convolution, Edge-Conv) respectively to obtain three groups of graph convolution features and one group of graph edge convolution features; the graph convolution features and the graph edge convolution features are concatenated to obtain the joint layered graph features, which is used to extract more diverse features, because the graph convolution operation can well extract topological structure information according to the node position; and the graph edge convolution operation can find nodes with similar semantics for feature extraction, so the combination of the two operations can extract different features to make the model more generalized; the purpose of using three graph convolution operations at the same time is to simulate the multi-channel of the CNN convolution kernel, and each channel can learn different features, specifically:

The feature extraction through the graph convolution flow includes, for example, dynamically aggregating the features of the first K other nodes related to each node through the K-nearest neighbor algorithm and aggregating the features by addition, and the formula is as follows:

;

;

Where L represents the graph convolution feature; k represents the number of layers; S=0, 1, 2 represents the number of channels, that is, S=0, 1, 2 represent the zeroth channel convolution flow, the first channel convolution flow and the second channel convolution flow respectively; Represents a linear layer that maps x from low-dimensional space to high-dimensional space; /> The weight of /> ,/> is the number of input channels, /> is the number of output channels; /> represents a linear layer, /> The weight of /> ; /> ;X indicates input;/> Represents a graph adjacency matrix; /> Represents the intermediate variable, through the linear layer/> After processing the input X, we get:

Feature extraction through the graph edge convolution flow includes:

;

in, represents the k-th layer edge convolution feature,/> is the number of frames, and is processed by the average pooling layer (avgpool) and the edge convolution flow;

The step of concatenating the graph convolution features and the graph edge convolution features to obtain the joint layered graph features includes:

;

Among them, || represents the channel splicing operation; Represents the joint layered graph features of the kth layer, /> ; They respectively represent the k-th layer graph convolution features obtained based on the zero-th channel convolution flow (S=0), the k-th layer graph convolution features obtained based on the first channel convolution flow (S=1), and the k-th layer graph convolution features obtained based on the second channel convolution flow (S=2);

The splicing of all the joint layered graph features to obtain the first image feature includes:

;

;

in, ; Z is the attention function, used to calculate the output of each layer/> The coefficients are then weighted and added to obtain / > ; /> Respectively represent the joint layered graph features of the first layer, the joint layered graph features of the second layer, the joint layered graph features of the third layer, and the joint layered graph features of the fourth layer; /> Represents the first image feature.

S2, the temporal convolution module sequentially performs convolution, activation and residual connection operations on each of the first image features of the input to obtain a second image feature; as shown in part (d) of FIG1, the input is a The convolution kernel is then activated by the activation function, and finally the residual connection is performed. The specific formula is as follows:

;

in, ,/> is the output of the temporal convolution module, representing the second image feature; /> is the number of output channels, /> is the number of frames output, /> is the number of key points of the hand; /> is the ReLU activation function, /> It is a convolution operation; /> The input may be the output of the HH module or the output of the SA module. In the detailed diagram of the T module in this embodiment, the output of the HH module is used for explanation.

S3, the spatial attention module extracts features according to the input second image features to obtain third image features; as shown in part (e) of FIG1 , the specific steps are as follows:

S31, inputting the second image feature into a preset number (U) of serial spatial attention blocks (SpatialAttentionBlock) for processing;

S32, passing the data processed by the spatial attention block through a linear layer to change the number of channels and perform residual connection output;

S33, pass the output of the residual connection through the activation function LeakyReLU, the linear layer (Linear) and batch normalization (BatchNormalization, BN) in sequence;

S34, after being activated again by the activation function LeakyReLU and performing residual connection, the third image feature is obtained. The specific formula is as follows:

;

;

;

;

in, Represents the output of the spatial attention block, through/> Transformed, /> ; /> is the number of key points of the hand, /> is the number of heads of the multi-head attention mechanism, /> is the number of output channels of the self-attention block,/> The number of frames to be output; /> and/> Represents an embedded function; /> Represents the residual connection output, /> ,/> is the number of output channels; /> , is a linear layer with weights/> ,/> For another linear layer its weights/> ; /> Represents the output after batch normalization, /> , B is the batch standardization operation; /> Indicates the final output, /> ,/> is the LeakyReLU activation function; the softmax function is used to output the final prediction probability of each category; /> Represents the output of the T module, T represents the number of frames, /> Indicates/> The features of the τth frame in .

S4, the temporal convolution module sequentially performs convolution, activation and residual connection operations on the input third image feature to obtain a fourth image feature; this part of the operation is the same as the operation in step S2.

The T module and SA module are used to simultaneously extract and aggregate multi-frame features and extract skeleton global features within a single frame.

S5. The fully connected layer outputs a gesture prediction result according to the fourth image feature; that is, the fully connected layer outputs various prediction scores.

Among them, in an optional implementation, in order to allow the model to learn more diverse data, the joint flow data is converted from the time domain to the frequency domain using Fast Fourier Transform (FFT) in this embodiment to obtain phase and amplitude features for model learning; the joint data stream refers to the original data of the data set, that is, the three-dimensional coordinates of the 25 joint points in each frame, which is the data before entering the model; thereafter, each data stream uses the corresponding weight parameters for model training in this embodiment.

As shown in Figure 6, the structure diagram of multi-stream fusion prediction is shown. The weighted sum of the scores predicted by each data stream is used to obtain a The prediction vector of is the number of predicted categories, /> Calculated by the following formula:

in, To correspond/> Prediction vector of streaming data; /> is a value between 0 and 1; /> ,/> is the joint flow data, /> is the joint motion flow data, /> This paper proposes the amplitude feature obtained by FFT of joint flow data; specifically, for/> Applying FFT can be expressed as:

in, is the real part of the result, /> is the imaginary part of the calculation result; in this embodiment, / > .

Embodiment 2

Please refer to Figure 7, a layered graph convolution gesture recognition device based on time-frequency data fusion includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the layered graph convolution gesture recognition method based on time-frequency data fusion as described in Example 1 is implemented; specific experimental results are shown on the DHG-14/28 and SHREC17 data sets:

Table 1. Performance comparison of the model in this example with other models on DHG-14

Wherein, * is the baseline model of this embodiment.

Table 2. Performance comparison of the model in this example with other models on DHG-28

Table 3. Performance comparison of the model in this embodiment with other models on the 14-category gesture classification task of SHREC’17

Table 4. Performance comparison of the model in this embodiment with other models on the 28-category gesture classification task of SHREC’17

It can be seen from Table 1, Table 2, Table 3 and Table 4 that the network model in this embodiment has a high accuracy on the DHG-14/28 and SHREC17 datasets.

In summary, the present invention provides a hierarchical graph convolution gesture recognition method and device based on time-frequency data fusion. By using a graph convolutional neural network and a graph convolution kernel, it is possible to naturally express non-Euclidean structures and learn the correlation between distant joints through training samples; a hierarchical hand graph structure is adopted to divide the hand skeleton into multiple layers and learn the correlation between the nodes of each layer through training, thereby getting rid of the dependence on the initial skeleton graph topological structure, so that the model can extract more diverse skeleton sequence features; global features are extracted by replacing redundant graph convolution blocks with spatial attention blocks, and spatial attention blocks and graph convolution blocks are alternately stacked, which can avoid the computational burden brought by stacking too many graph convolution layers, reduce system complexity, and ensure the global feature extraction capability of the model; and the feature data stream is expanded by Fourier transform, so that more diverse data features can be considered in fusion prediction, thereby improving the accuracy and robustness of the model.

That is, regarding problem 1 mentioned in the background technology: since the present invention uses a graph convolutional neural network, non-Euclidean structures can be expressed very naturally.

For question 2: The present invention uses a learnable graph convolution kernel, which can learn the correlation between distant joint points through training samples.

Regarding Question 3: Our invention uses a hand graph convolution module to divide the hand skeleton into multiple layers and learns the correlation between the nodes of each layer through training, thereby getting rid of the dependence on the initial skeleton graph topological structure.

Regarding Question 4: The present invention uses spatial attention blocks to replace redundant graph convolution blocks to extract global features, and uses spatial attention blocks and graph convolution blocks to be stacked alternately, which can not only reduce the high computational overhead mentioned in Disadvantage 4, but also retain the local and global feature information in the skeleton data.

The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims (9)

1. A hierarchical graph convolution gesture recognition method based on time-frequency data fusion is characterized by comprising a network structure formed by a basic block and a full-connection layer; the basic block is formed by sequentially connecting a layered hand graph convolution module, a time convolution module, a space attention module and a time convolution module in a residual way; the network structure performs the steps of:

the layered hand graph rolling module splits the gesture image to be recognized into joint layered graphs with preset numbers, and extracts characteristics of the joint layered graphs; splicing all the joint layered graph features to obtain a first image feature;

the time convolution module carries out convolution, activation and residual error connection operation on each input first image feature in sequence to obtain a second image feature;

The spatial attention module performs feature extraction according to the input second image features to obtain third image features;

The time convolution module carries out convolution, activation and residual error connection operation on the input third image feature in sequence to obtain a fourth image feature;

The full-connection layer outputs a gesture prediction result according to the fourth image characteristic;

the layering hand graph rolling module splits a gesture image to be recognized into joint layering graphs with preset numbers, and the joint layering graph rolling module comprises:

Converting the gesture image to be recognized into a node gesture image by a preset number of joint points, and determining a central joint point of the node gesture image;

grouping all the joint points according to the distance from each joint point to the central joint point to obtain a layered joint point set;

and connecting two adjacent layered joint point sets in sequence to form an edge set, and obtaining the layered graph with corresponding numbers according to the number of the edge sets.

2. The hierarchical graph convolution gesture recognition method based on time-frequency data fusion according to claim 1, wherein the hierarchical hand graph convolution module comprises three graph convolution streams and one graph edge convolution stream;

the extracting the joint layered graph features comprises the following steps:

Changing the channel number of the layered graph through four convolution kernels to form four-stream output, and extracting features through the graph convolution stream and the graph edge convolution stream respectively to obtain three groups of graph convolution features and one group of graph edge convolution features;

and splicing the graph convolution characteristic and the graph edge convolution characteristic to obtain the joint layered graph characteristic.

3. The hierarchical graph convolution gesture recognition method based on time-frequency data fusion according to claim 2, wherein feature extraction by the graph convolution stream comprises:

；

；

wherein L represents a graph convolution feature; k represents the number of layers; s=0, 1,2, representing the number of channels; /> Representing a linear layer; x represents an input; /(I)Representing a graph adjacency matrix; /(I)Representing intermediate variables by linear layer/>The input X is processed to obtain;

the feature extraction through the graph edge convolution flow comprises the following steps:

；

Wherein, Representing the edge convolution characteristics of the k-th layer diagram/>For outputting channel number,/>For the number of frames,/>The number of key points of the hand; Representing a real set;

The step of splicing the graph convolution feature and the graph edge convolution feature to obtain the joint layered graph feature comprises the following steps:

；

Wherein, the I represents a channel splicing operation; Features of the joint hierarchy map representing a kth layer; /(I) The k-th layer convolution feature obtained based on the zeroth channel convolution stream (s=0), the k-th layer convolution feature obtained based on the first channel convolution stream (s=1), and the k-th layer convolution feature obtained based on the second channel convolution stream (s=2) are respectively represented.

4. The method for recognizing a hierarchical graph convolution gesture based on time-frequency data fusion according to claim 3, wherein the stitching all the joint hierarchical graph features to obtain a first image feature comprises:

；

；

Wherein, ; Z is the attention function for calculating the output/>, of each layerAfter the coefficients of (2) are weighted and added to obtain/>；/>Respectively representing the characteristics of the joint hierarchy map of the 1 st layer, the characteristics of the joint hierarchy map of the 2 nd layer, the characteristics of the joint hierarchy map of the 3 rd layer and the characteristics of the joint hierarchy map of the 4 th layer; /(I)Representing a first image feature.

5. The method for recognizing a hierarchical convolution gesture based on time-frequency data fusion according to claim 1, wherein the time convolution module sequentially convolves, activates and residual connects each of the inputted first image features, comprising:

；

Wherein, ，/>Representing a second image feature as an output of the temporal convolution module; /(I)In order to output the number of channels,For the number of frames output,/>The number of key points of the hand; /(I)For ReLU activation function,/>Is a convolution operation; /(I)Is input.

6. The method for recognizing a hierarchical convolution gesture based on time-frequency data fusion according to claim 1, wherein the spatial attention module performs feature extraction according to the input second image feature, and obtaining a third image feature comprises:

Inputting the second image characteristic into a serial space attention block with a preset number for processing;

changing the channel number of the data processed by the spatial attention block through a linear layer, and carrying out residual connection output;

The output of the residual connection input sequentially passes through an activation function, a linear layer and batch standardization;

and performing residual connection after activating the activation function again, and outputting to obtain the third image feature.

7. The method for recognizing a hierarchical convolution gesture based on time-frequency data fusion according to claim 6, wherein the obtaining the third image feature comprises:

；

；

；

；

Wherein, Representing spatial attention block output,/>；/>For the number of key points of hands,/>The number of heads for the multi-head attention mechanism,/>Is the number of self-attention block output channels,/>The number of frames to be output; /(I)And/>Representing an embedding function; /(I)Representing residual connection output,/>，/>The number of output channels; /(I)、/>Representing different linear layer weights; /(I)Representing the output after batch normalization,/>B is batch standardization operation; /(I)The final output is indicated as such,，/>Is an activation function; the softmax function is used for outputting the prediction probability of the last category; /(I)Representing the output of the T module, T representing the number of frames,/>Representation/>Is characterized by the tau frame.

8. The hierarchical graph convolution gesture recognition method based on time-frequency data fusion according to claim 1, further comprising:

Converting the throttle data from a time domain to a frequency domain through Fourier change to obtain a phase characteristic and an amplitude characteristic;

The phase and amplitude characteristics are used to optimize the network structure.

9. A time-frequency data fusion-based hierarchical graph convolution gesture recognition device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein execution of the computer program by the processor implements the steps of a time-frequency data fusion-based hierarchical graph convolution gesture recognition method according to any one of claims 1-8.