CN110378372A - Diagram data recognition methods, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a graph data recognition method, device, computer equipment and storage medium. The method includes: obtaining an input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map generated according to image data, and obtaining a first bias matrix of the current convolutional layer, wherein the first A bias matrix is the matrix generated when the trained convolutional neural network is generated, a second bias matrix is generated according to the input feature map, a reference adjacency matrix is obtained, and a reference adjacency matrix, the first bias matrix and the second bias matrix are calculated The sum of the target adjacency matrix is obtained, the convolution kernel of the current convolution layer is obtained, the target output feature map is generated according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and the image is identified according to the target output feature map Data recognition results. An adjustable bias matrix is added to the existing fixed adjacency matrix to improve the recognition accuracy of the trained convolutional neural network.

Description

Image data recognition method, device, computer equipment and storage medium

technical field

The present application relates to the field of computer technology, and in particular to a method, device, computer equipment and storage medium for identifying image data.

Background technique

In the skeleton point data, the human body is represented by the coordinates of several predefined key joint points in the camera coordinate system. It can be easily obtained by depth cameras (such as Kinect) and various pose estimation algorithms (such as OpenPose). Figure 1 shows the key joint points of the human body defined by the Kinect depth camera. It defines the human body as the three-dimensional coordinates of 25 key joint points. Since behaviors often exist in the form of video, a behavior with a length of T frames can be represented by a Tx25x3 tensor.

Referring to FIG. 2, FIG. 2 is a space-time diagram in one embodiment. Each joint point is defined as a node of the graph, and the physical connection between the joint points is defined as the edge of the graph, and the edge of the time dimension is added between the same nodes of adjacent frames to obtain a space-time graph that can describe human behavior .

At present, the common behavior recognition method based on skeleton points is graph convolution. Graph convolution is different from ordinary convolution operations. When convolution is performed on a graph, the number of neighbor nodes of each node is not fixed, and the parameters of the convolution operation are fixed. In order to combine a fixed number of parameters with an indeterminate number To correspond to the number of adjacent nodes, it is necessary to define a mapping function, and realize the correspondence between parameters and nodes through the mapping function. If the size of the convolution kernel is defined as three, as shown in Figure 3, the three parameters correspond to the point 001 away from the center of the human body, the point 002 close to the center of the human body 000, and the convolution point itself 003. Then the convolution operation can be expressed by formula (1):

Among them, f is the input and output feature tensor, w is the convolution parameter, v is the node in the graph, l represents the mapping function between the node and the parameter, and Z is the normalization function. In the specific implementation, the mapping function can be realized through the adjacency matrix of the graph, and the convolution operation represented by the adjacency matrix is shown in formula (2):

Where A represents the adjacency matrix of the graph, K is the size of the convolution kernel, and Λ is used to normalize A. By multiplying with the adjacency matrix A, the required nodes are "filtered" from the feature tensor and multiplied with the corresponding parameters.

When the above convolution operation is represented by the adjacency matrix, the adjacency matrix defines the topology of the human body graph used in the graph convolutional network. There are various postures of the human body, and the fixed topological structure cannot accurately describe each posture of the human body, resulting in low recognition accuracy.

Contents of the invention

In order to solve the above-mentioned technical problems, the present application provides a graph data recognition method, device, computer equipment and storage medium.

In the first aspect, the present application provides a graph data recognition method, including:

Obtain the input feature map of the current convolution layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data;

Obtain the first bias matrix of the current convolutional layer, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated;

Generate a second bias matrix based on the input feature map;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

Generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map;

According to the target output feature map, the recognition result of the map data is recognized.

In a second aspect, the present application provides a special map generation device, including:

The data acquisition module is used to obtain the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data, and obtain the first bias matrix of the current convolutional layer, wherein The first bias matrix is a matrix generated when the trained convolutional neural network is generated;

The second offset matrix generation module is used to generate the second offset matrix according to the input feature map;

The target adjacency matrix generation module is used to obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix, the first offset matrix and the second offset matrix, and obtain the target adjacency matrix;

The target output feature map generation module is used to obtain the convolution kernel of the current convolution layer, and generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map.

The recognition module is configured to output a feature map according to the target, and recognize a recognition result of the map data.

A computer device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the following steps when executing the computer program:

Obtain the input feature map of the current convolution layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data;

Obtain the first bias matrix of the current convolutional layer, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated;

Generate a second bias matrix based on the input feature map;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

Generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map;

According to the target output feature map, the recognition result of the map data is recognized.

A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Obtain the input feature map of the current convolution layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data;

Obtain the first bias matrix of the current convolutional layer, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated;

Generate a second bias matrix based on the input feature map;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

Generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map;

According to the target output feature map, the recognition result of the map data is recognized.

The above image data recognition method, device, computer equipment and storage medium, the method includes: obtaining the input feature map of the current convolution layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data, Obtain the first bias matrix of the current convolutional layer, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated, generate the second bias matrix according to the input feature map, obtain the reference adjacency matrix, and calculate the reference The sum of the adjacency matrix, the first bias matrix and the second bias matrix is obtained to obtain the target adjacency matrix, and the convolution kernel of the current convolution layer is obtained, which is generated according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map The target output feature map is used to identify the recognition result of the map data according to the target output feature map. Add a bias matrix to the adjacency matrix in each convolutional layer in the trained convolutional neural network, the first bias matrix in the bias matrix is the matrix determined according to the requirements, and the second bias matrix generated according to the input feature map The offset matrix is a matrix generated according to the input data. Adding the offset matrix can represent the required features and the sample features of the input data, improve the accuracy of the generated feature map, and then improve the recognition accuracy of the trained convolutional neural network.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

Fig. 1 is the schematic diagram of the key joint point of the human body defined by Kinect depth camera in an embodiment;

Fig. 2 is a space-time diagram describing human behavior in one embodiment;

Fig. 3 is a schematic diagram of nodes defined in graph convolution in an embodiment;

Fig. 4 is an application environment diagram of a graph data identification method in an embodiment in an embodiment;

Fig. 5 is a schematic flow chart of a method for generating a feature map in an embodiment;

FIG. 6 is a schematic diagram of a data processing flow of a convolutional layer in an embodiment;

Fig. 7 is a structural block diagram of a feature map generation device in an embodiment;

Figure 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

Fig. 4 is an application environment diagram of the image data identification method in an embodiment. Referring to Fig. 4, the graph data recognition method is applied to a feature graph generation system. The feature map generation system includes a terminal 110 and a server 120 . Terminal 110 and server 120 are connected via a network. The terminal or server obtains the input feature map of the current convolutional layer of the trained convolutional neural network. The input feature map is a feature map generated based on image data, and the input feature map is a feature map obtained by extracting image data. Obtain the current The first bias matrix of the convolutional layer, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated, the second bias matrix is generated according to the input feature map, the reference adjacency matrix is obtained, and the reference adjacency matrix is calculated , the sum of the first bias matrix and the second bias matrix to obtain the target adjacency matrix, obtain the convolution kernel of the current convolution layer, and generate the target output according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map The feature map, according to the target output feature map, recognizes the recognition result of the map data. The terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

As shown in FIG. 5 , in one embodiment, a method for identifying graph data is provided. This embodiment is mainly described by taking the method applied to the terminal 110 (or server 120) in FIG. 4 above as an example. Referring to Fig. 5, the data identification method in this figure specifically includes the following steps:

Step S201, obtaining the input feature map of the current convolutional layer of the trained convolutional neural network.

In this specific embodiment, the input feature map is a feature map generated according to image data.

Specifically, the trained convolutional neural network is obtained by training a large amount of labeled graph data, where the graph data is a spatio-temporal graph of human behavior, and the spatio-temporal graph is shown in Figure 2. The labels carried by the graph data include human behaviors, such as individual or multi-person behaviors such as clapping, jumping, pulling hands, and fighting. The trained convolutional neural network contains multiple convolutional layers, and the current convolutional layer can be any convolutional layer in the convolutional neural network. The output data of the previous convolutional layer is the input data of the current convolutional layer, and the output data of the previous convolutional layer is obtained as the input data of the current convolutional layer, and the input data is the input feature map.

Step S202, acquiring the first bias matrix of the current convolutional layer.

Specifically, the first bias matrix is a matrix generated when the trained convolutional neural network is generated. The first offset matrix is the offset matrix obtained according to the training requirements. Different training requirements refer to the requirements for what to do after the convolutional neural network is trained. A bias matrix is not the same.

Step S203, generating a second bias matrix according to the input feature map.

Specifically, the second bias matrix is a matrix generated according to the input feature map. The second bias matrix is obtained by performing operations such as dimensionality reduction and normalization on the input matrix. The second bias matrix is related to the input feature map. The input feature map is mapped according to the function in the current convolution layer to obtain the mapped matrix. The mapped matrix will be multiplied and the result of the multiplication operation will be normalized. and so on to obtain the second bias matrix.

In one embodiment, step S203 includes: using a dimensionality reduction function in the trained convolutional neural network to perform dimensionality reduction on the input feature map to obtain a dimensionality reduction matrix, and normalize the dimensionality reduction matrix to obtain a normalized matrix, The normalization matrix is the second bias matrix.

Specifically, the dimensionality reduction function includes a first dimensionality reduction function and a second dimensionality reduction function. According to the first dimensionality reduction function, the input feature map pair is dimensionally reduced to obtain the first feature map, and according to the second dimensionality reduction function, the input feature map is Dimensionality reduction obtains the second feature map. Calculate the product of the first feature map and the second feature map to obtain a first product matrix. Different dimensionality reduction functions are used to reduce the dimensionality of the input feature map, and different dimensionality reduction matrices are obtained. The parameters of the dimensionality reduction functions are obtained by training according to the requirements. Calculate the product of the two dimension-reduced matrices, that is, the first product matrix, and each point in the first product matrix represents the feature similarity between the point corresponding to the abscissa and the point corresponding to the ordinate. For example, 1 represents joint point 1, and 2 represents joint point 2, then the matrix elements in the coordinates (1, 2) in the first product matrix represent the feature similarity between joint point 1 and joint point 2. Performing a normalization operation on the first product matrix to obtain a normalized matrix, the normalization operation, and performing the normalization operation on the data can improve the accuracy of data calculation and speed up the convergence speed of the data. The normalized first product matrix is used as the second bias matrix.

Step S204, obtaining the reference adjacency matrix, calculating the sum of the reference adjacency matrix, the first offset matrix and the second offset matrix, to obtain the target adjacency matrix.

Specifically, the first offset matrix, the second offset matrix, and the reference adjacency matrix have the same dimension information, and the sum of the reference adjacency matrix, the first offset matrix, and the second offset matrix is calculated, that is, for the matrix elements at the same position Add directly to get the target adjacency matrix.

Step S205, obtaining the convolution kernel of the current convolution layer.

Specifically, each convolution layer includes multiple convolution kernels, the number of convolution kernels corresponding to each convolution layer may be the same or different, and each convolution kernel may be the same or different. Convolution kernels are used to perform convolution operations on images, and different convolution kernels can extract different image features. The target adjacency matrix is used to perform feature extraction on the input feature map to obtain the feature map.

Step S205, generating a target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map.

Specifically, the target adjacency matrix is used to perform feature extraction corresponding to the input feature map, and the corresponding feature map is obtained, and the convolution kernel is used to perform convolution operation on the feature map extracted according to the target path matrix, and the feature map obtained by the convolution operation is used as the target output feature picture. Using the target adjacency matrix to extract features from the input feature map, more accurate features can be extracted.

In one embodiment, the input feature map includes at least three dimensions, and the target adjacency matrix includes at least three dimensions. Step S205 includes:

Step S2051, reshaping the input feature map to obtain the reshaping feature map.

In this specific embodiment, the first dimension of the reshaped feature map is the product of the first dimension and the second dimension of the input feature map.

Specifically, reshaping refers to adjusting the input feature map so that the product of the first dimension and the second dimension is the first dimension of the reshaping feature map, such as adjusting the input feature map containing three dimensions into two dimensions. Reshape the feature map, assuming that the input feature map is C×M×N, where the first dimension is C, the second dimension is M, and the third dimension is N, then the reshaping map can be C×M×N, the first dimension It is the product CM of C and M, the second dimension is the same as the third dimension of the input feature map, keeping the elements of the entire input feature map unchanged, and the total element is the product CMN of C, M and N. The first dimension C is the channel pixel, the second dimension M is the number of rows of the input feature map, and the third dimension N is the number of columns of the input feature map. Among them, N represents the number of joint points of the human body, and N is defined as 25 in Kinect. The reshaping of the matrix is for convenience of operation.

Step S2052, calculating the product of the reshaped feature map and the matrix of each channel of the target adjacency matrix to obtain a second product matrix of each channel.

Specifically, the second dimension of the reshaped feature map is the same as the first dimension of the matrix of each channel of the target adjacency matrix. For example, the reshaped feature map is C×M×N, the target adjacency matrix is C×N×N, and each channel The matrix is N×N, then the product matrix of the reshaped feature map and the matrix of each channel is CMN.

Step S2053, dereshaping the second product matrix of each channel to obtain the deremodeling feature map of each channel.

Specifically, anti-reshaping is the inverse process of reshaping. For example, reshaping is to convert a three-dimensional matrix into a two-dimensional matrix, then anti-reshaping is to convert a two-dimensional matrix into a three-dimensional matrix. For example, the product matrix of each channel is CM×N , then the anti-reshaping feature map obtained after anti-reshaping is a C×M×N matrix.

Step S2054, according to the convolution kernel of each channel, perform convolution operation on the anti-reshaping feature map according to the convolution kernel of each channel, and obtain the target feature map of each channel of the current convolution layer.

In step S2055, the target feature map of each channel is summed to obtain the output feature map of the current convolution layer, and the output feature map of the current convolution layer is used as the target output feature map.

Specifically, through the convolution kernels corresponding to each channel, feature extraction is performed on the anti-reshaping feature map to obtain the features corresponding to each convolution kernel, and the features extracted by each convolution kernel form the target feature map of each channel. Calculate the sum of the target feature maps of each channel, that is, add the matrix elements at the corresponding positions to obtain the output feature map, which is the output feature map of the current convolutional layer.

In one embodiment, step S205 further includes:

Step S2056, judging whether the number of channels of the output feature map is consistent with the number of channels of the input feature map.

Step S2057, when consistent, use the sum of the input feature map and the output feature map as the target output feature map of the current convolutional layer.

Step S2058, if they are inconsistent, perform a convolution operation on the input feature map to obtain a convolution feature map with the same number of channels as the output feature map, and use the sum of the convolution feature map and the output feature map as the target output feature map.

Specifically, according to each channel matrix of the output feature map generated by the convolution kernel of each channel and the corresponding frontal inverse reshaping matrix, it is judged whether the channel of the output feature map is consistent with the number of channels of the input feature map. When they are consistent, the input The feature map is added to the element at the position corresponding to the output feature to obtain the target output feature map. When inconsistent, perform convolution operation on the input feature map to obtain a convolution feature map with the same channel as the output feature map, calculate the sum of the elements at the same position in the convolution feature map and the output feature map, and obtain the target output feature map. By judging the channel numbers of the input feature map and the output feature map, the target output feature map is determined according to the channel number, which improves the accuracy of the target output feature map.

Step S206, according to the target output feature map, identify the recognition result of the map data.

Specifically, the target output feature map is input into the recognition layer in the trained convolutional neural network, the candidate recognition results corresponding to the target output feature map are identified through the recognition layer, and the candidate recognition result with the highest recognition probability is selected from the candidate recognition results as Target recognition result, the target recognition result is used as the recognition result corresponding to the graph data. For example, when the recognition types include clapping, jumping, and holding hands, the recognition probability corresponding to clapping is 0.89, the recognition probability corresponding to jumping is 0.01, and the recognition probability corresponding to holding hands is 0.1, then the recognition result corresponding to the graph data is clapping.

In one embodiment, when the trained convolutional neural network includes the current convolutional layer and the next convolutional layer of the current convolutional layer, the target output feature map is used as the input feature map of the next convolutional layer, Use the next convolutional layer as the current convolutional layer, and enter to obtain the input feature map of the current convolutional layer of the trained convolutional neural network until all convolutional layers in the trained convolutional neural network are completed. Output the target output feature map of the last convolutional layer, input the target output feature map of the last convolutional layer into the recognition layer, and obtain the corresponding recognition result of the book. The data processing flow of convolutional layers with the same network structure is the same.

The above image data recognition method obtains the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data, and obtains the first bias matrix of the current convolutional layer, where The first bias matrix is the matrix generated when the trained convolutional neural network is generated, the second bias matrix is generated according to the input feature map, the reference adjacency matrix is obtained, and the reference adjacency matrix, the first bias matrix and the second bias are calculated The sum of the matrices to obtain the target adjacency matrix, obtain the convolution kernel of the current convolution layer, generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and identify the The recognition result of graph data. An adjustable offset matrix is added to the existing fixed adjacency matrix. The first offset matrix in the offset matrix is obtained according to the training requirements, so it can represent the features corresponding to the requirements well. The second offset matrix The matrix determines the parameters according to the task requirements, and is generated according to the input graph data, which can represent the characteristics of each graph data. The offset matrix determined from the graph data itself and the requirements can better standardize the characteristics of each graph data itself, so that Improve the recognition accuracy of the trained convolutional neural network. In one embodiment, step S206 includes:

Step S2061, when the current convolutional layer is the last convolutional layer in the trained convolutional neural network, determine whether there is a target feature map that needs to be merged in multiple target output feature maps.

Step S2062, if it exists, merge the target output feature maps that need to be merged to obtain the merged feature map.

Step S2063, when the merged feature map contains all target output feature maps, identify the merged feature map to obtain a recognition result corresponding to the merged feature map.

Step S2064, when the merged feature map contains all target output feature maps, identify the merged feature map to obtain the recognition result corresponding to the merged feature map, and identify the unmerged target output feature map to obtain the unmerged target output feature map corresponding recognition results.

Step S2065, if it does not exist, identify each target output feature map, and obtain the recognition result corresponding to each target output feature map.

Specifically, when the current convolutional layer is the last convolutional layer in the trained convolutional neural network, it means that the convolution operation has been explained, and the final target output feature map of the graph data has been extracted, and the final target output feature Before the map is recognized, it is judged whether each final target output feature map needs to be merged. The need for merging is because some behaviors require multiple people to complete, such as holding hands, fighting and other behaviors require at least two people to complete. For behaviors that individuals can complete, there is no need to combine the target output feature map, and the target output feature map is directly recognized, and the recognition result of the target output feature map is used as the recognition result of the graph data. The recognition result of graph data can be one or more sub-recognition results. When the input of graph data includes a behavior, the obtained recognition result includes multiple behaviors, and each behavior is regarded as a sub-recognition result.

When there are target output feature maps that need to be merged, determine which target output feature maps need to be merged and how to merge them, merge the target output feature maps that need to be merged, and directly merge the feature maps, where the merged feature map can be one or more, the specific merging result is related to the human behavior in the input data, as shown in the figure, when the data includes multiple types of behaviors such as clapping, holding hands, fighting, etc., holding hands and fighting need to be merged, and the merged feature map includes two. The merged feature map is directly recognized, and the recognition result of the merged feature map is obtained. When there is a situation of partly merging and partly not merging, the non-merging target output feature map is directly recognized, and the recognition result corresponding to the non-merging target output feature map is obtained.

In a specific embodiment, refer to FIG. 6 , which is a schematic diagram of a data processing flow of a convolutional layer in an embodiment. _In the figure, fin is the input feature map of the current convolutional layer, and f _out is the output feature map of the current convolutional layer. The output feature map adopts the specific representation of the input feature map as shown in formula (3):

In the formula, A _k is the kth adjacency matrix in the reference adjacency matrix, B _k is the kth adjacency matrix in the bias matrix, C _k is the kth adjacency matrix in the bias matrix, softmax(S) means Perform normalization calculation on the matrix S, W _k is the kth parameter of the convolution kernel, Kv is the size of the convolution kernel, the size of the convolution kernel can be customized, for example, K _v =3 or 5 can be set. is the first dimensionality reduction function, is the second dimensionality reduction function. Assume that the size information of the input feature map is C _in × T × N, where C _in represents the number of input channels, T represents the number of frames of graph data, and N represents the number of joint nodes defined by kinect, where N=25. Reshape the input feature map to obtain the reshaped feature map of C _in T×N. The bias matrix B _k is the matrix obtained after training the convolutional neural network. B _k and A _k have the same size information, which is N× N, calculate and calculate the sum of the first bias matrix B _k , the second value matrix C _K and the reference adjacency matrix A _k to obtain the matrix of each channel of the target adjacency matrix, and calculate the matrix and reshape features of each channel of the target adjacency matrix The product of the graphs is obtained to obtain the second product matrix of each channel, and the second product matrix of each channel is reversely reshaped to obtain the reverse remodeling matrix, and a convolution kernel W _k is obtained, which, through the convolution kernel of each channel The anti-reshaping matrix performs convolution operation to obtain the output feature map corresponding to each channel. Determine whether the output feature map is consistent with the number of channels of the input feature map. If not, pass the residual network res, where the convolution kernel size in the residual network is 1×1. Adjust the input feature map to a matrix with the same number of channels as the output feature map, calculate the sum of the adjusted input feature map and output feature map, and obtain the target output feature map. When consistent, calculate the sum of the input feature map and the output feature map to obtain the target output feature map. According to the target output feature map, the behavior of each graph data is recognized, and the corresponding recognition result is obtained.

When the generation process of the above feature map is the data processing process of training the convolutional neural network, and when the identification corresponding to each graph data is inconsistent with the category in the label of the graph data, according to the loss value corresponding to each graph data according to the preset loss function, Return the loss value according to the gradient return algorithm to obtain the return value of the convolution layer, update the parameters of the convolution kernel w of each channel of the corresponding convolution layer according to the return value, the first bias matrix B and the calculation Parameters of the mapping function for the second bias matrix C.

When the above feature map generation process is the data processing process of the trained convolutional neural network, the recognition result of the feature map is output according to the target as the recognition result corresponding to the map data. Input the graph data into the trained convolutional neural network. The convolutional neural network includes multiple convolutional layers and recognition layers. Each convolutional layer includes a convolution kernel and a target adjacency matrix. The target adjacency matrix of each convolutional layer is used to Feature extraction is performed on the image data to obtain the corresponding image feature map set, and the convolution operation is performed on the image feature set through the convolution kernel to obtain the target output feature map of each convolution layer, and the target output feature map of the previous convolution layer of the recognition layer As the input data of the identification layer, according to the target output feature map of each graph data, the corresponding human behavior type is identified.

Fig. 5 is a schematic flowchart of a method for identifying image data in an embodiment. It should be understood that although the various steps in the flow chart of FIG. 5 are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Fig. 5 may include multiple sub-steps or multiple stages, these sub-steps or stages are not necessarily executed at the same moment, but may be executed at different moments, the execution of these sub-steps or stages The order is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 7 , a special map generation device 200 is provided, including:

The data acquisition module 201 is used to obtain the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map generated according to the image data, and obtain the first bias matrix of the current convolutional layer, Wherein the first bias matrix is a matrix generated when the trained convolutional neural network is generated.

The second offset matrix generation module 202 is configured to generate a second offset matrix according to the input feature map.

The target adjacency matrix generating module 203 is configured to obtain a reference adjacency matrix, calculate the sum of the reference adjacency matrix, the first offset matrix and the second offset matrix, and obtain the target adjacency matrix.

The target output feature map generating module 204 is configured to obtain the convolution kernel of the current convolution layer, and generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map.

The recognition module 205 is configured to recognize the recognition result of the map data according to the target output feature map.

In one embodiment, the second offset matrix generation module is specifically used to reduce the dimensionality of the input feature map by using the dimensionality reduction function in the trained convolutional neural network to obtain the dimensionality reduction matrix, and normalize the dimensionality reduction matrix to obtain Normalization matrix, the normalization matrix is the second bias matrix.

In one embodiment, the second offset matrix generation module is specifically configured to perform dimensionality reduction on the matrix of each channel of the input feature map according to the first dimensionality reduction function in the dimensionality reduction function, to obtain the first dimensionality reduction matrix of each channel, According to the second dimensionality reduction function in the dimensionality reduction function, the matrix of each channel of the input feature map is reduced in dimension to obtain the second dimensionality reduction matrix of each channel, wherein the dimensionality reduction function includes two, and the input feature map includes at least three dimension, the first dimension is the number of channels, calculate the product of the first dimensionality reduction matrix and the second dimensionality reduction matrix of each channel, obtain the first product matrix of each channel, normalize the first product matrix of each channel, and obtain normalization The matrix of channels corresponding to the transformation matrix.

In one embodiment, the above image data identification device further includes:

Network generation module for generating trained convolutional neural networks. Among them, the network generation module includes:

The data acquisition unit is configured to acquire a training set including a plurality of training graph data, and the training graph data carries label information.

The feature extraction unit is used to input the training graph data and label information into the initial convolutional neural network, and extract the features of each training graph data through the initial convolutional neural network.

The recognition unit is configured to recognize the recognition result corresponding to each training graph data according to the characteristics of each training graph data.

The loss value calculation unit is used to calculate the recognition result of each training image data and the loss value of the label according to a preset loss function.

The network determination unit is used to obtain the trained convolutional neural network when the loss value is less than or equal to the preset loss value.

In one embodiment, the network determining unit further includes:

The parameter update subunit is used to update the network parameters of the initial convolutional neural network through the gradient backpropagation algorithm according to the loss value when the loss value is greater than the preset loss value.

The network determination subunit is used to adopt the initial convolutional neural network with updated network parameters as the initial convolutional neural network, enter the training graph data and label information into the initial convolutional neural network, and calculate each training graph according to the preset loss function When the recognition result of the data and the loss value of the label are less than or equal to the preset loss value, a trained convolutional neural network is obtained.

In one embodiment, when the network determination subunit specifically returns the loss value to any convolution layer through the gradient return algorithm, it obtains the return value of each convolution layer, and updates the volume according to the return value of each convolution layer. The network parameters of the product layer, wherein the initial convolutional neural network model includes at least one convolutional layer, and the network parameters include the parameters of the initial dimensionality reduction function and the parameters of the initial bias matrix.

Figure 8 shows a diagram of the internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 (or server 120) in FIG. 4 . As shown in FIG. 8 , the computer equipment includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and may also store a computer program, and when the computer program is executed by the processor, the processor may realize the image data identification method. A computer program may also be stored in the internal memory, and when the computer program is executed by the processor, the processor may execute the image data identification method. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer equipment, or It can be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In an embodiment, the device for generating a special map provided in the present application may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in FIG. 8 . The memory of the computer equipment can store the various program modules that make up the special map generation device, such as the data acquisition module 201 shown in Figure 7, the second bias matrix generation module 202, the target adjacency matrix generation module 203, and the target output feature map A generation module 204 and an identification module 205 . The computer program constituted by each program module enables the processor to execute the steps in the image data identification method of each embodiment of the application described in this specification.

For example, the computer equipment shown in FIG. 8 can perform the acquisition of the input feature map of the current convolutional layer of the trained convolutional neural network through the data acquisition module 201 in the special map generation device as shown in FIG. Obtain the first bias matrix of the current convolutional layer for the feature map generated according to the image data, where the first bias matrix is a matrix generated when the trained convolutional neural network is generated. The computer device can generate the second offset matrix according to the input feature map through the second offset matrix generation module 202 . The computer device can obtain the reference adjacency matrix through the target adjacency matrix generation module 203, calculate the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, and obtain the target adjacency matrix. The computer device can obtain the convolution kernel of the current convolution layer through the target output feature map generation module 204, and generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map. The computer device can use the recognition module 205 to execute the recognition result of identifying the graph data according to the target output feature graph.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, the following steps are implemented: obtaining the input trained volume The input feature map of the current convolutional layer of the product neural network, the input feature map is the feature map generated according to the image data, and the first bias matrix of the current convolutional layer is obtained, wherein the first bias matrix is the generated convolution of the trained The matrix generated by the neural network generates the second bias matrix according to the input feature map, obtains the reference adjacency matrix, calculates the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, obtains the target adjacency matrix, and obtains the current volume The convolution kernel of the product layer generates the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and recognizes the recognition result of the graph data according to the target output feature map.

In one embodiment, the second bias matrix generated according to the input feature map includes: using the dimensionality reduction function in the trained convolutional neural network to perform dimensionality reduction on the input feature map to obtain a dimensionality reduction matrix, normalized reduction dimension matrix to obtain a normalized matrix, and the normalized matrix is the second bias matrix.

In one embodiment, the dimension reduction function includes two, and the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, including: according to the first dimension reduction function in the dimension reduction function, each of the input feature maps The matrix of the channel is dimensionally reduced to obtain the first dimensionality reduction matrix of each channel, and the matrix of each channel of the input feature map is reduced according to the second dimensionality reduction function in the dimensionality reduction function to obtain the second dimensionality reduction matrix of each channel , calculate the product of the first dimensionality reduction matrix and the second dimensionality reduction matrix of each channel, obtain the first product matrix of each channel, normalize the first product matrix of each channel, and obtain the matrix of the channel corresponding to the normalization matrix.

In one embodiment, the following steps are also implemented when the processor executes the computer program: the step of generating a trained convolutional neural network includes: obtaining a training set comprising a plurality of training graph data, the training graph data carrying label information, and training The graph data and label information are input into the initial convolutional neural network, and the characteristics of each training graph data are extracted through the initial convolutional neural network. According to the characteristics of each training graph data, the recognition results corresponding to each training graph data are identified, and the preset loss function is used. Calculate the recognition result of each training image data and the loss value of the label. When the loss value is less than or equal to the preset loss value, a trained convolutional neural network is obtained.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the loss value is greater than the preset loss value, update the network parameters of the initial convolutional neural network through the gradient return algorithm according to the loss value, and use the updated network parameters The initial convolutional neural network is used as the initial convolutional neural network, and the training image data and label information are input into the initial convolutional neural network until the recognition result of each training image data and the loss value of the label are calculated according to the preset loss function, which is less than or When equal to the preset loss value, the trained convolutional neural network is obtained.

In one embodiment, the initial convolutional neural network model includes at least one convolutional layer, the convolutional layer includes an initial bias matrix and an initial dimensionality reduction function, and the network of the initial convolutional neural network is updated through a gradient backpropagation algorithm according to the loss value Parameters, including: when the loss value is returned to any convolution layer through the gradient return algorithm, the return value of each convolution layer is obtained, and the parameters of the initial dimensionality reduction function and the initial dimensionality reduction function are updated according to the return value of each convolution layer. Parameters for the bias matrix.

In one embodiment, according to the target output feature map, identifying the recognition result corresponding to the graph data includes: when the current convolutional layer is the last convolutional layer in the trained convolutional neural network, judging multiple target output Whether there is a target feature map that needs to be merged in the feature map, if it exists, merge the target output feature map that needs to be merged to obtain the merged feature map, when the merged feature map contains all target output feature maps, identify the merged feature map to get The recognition result corresponding to the merged feature map. When the merged feature map contains all target output feature maps, the merged feature map is recognized to obtain the recognition result corresponding to the merged feature map. The unmerged target output feature map is recognized to obtain the unmerged feature map. If there is no recognition result corresponding to the target output feature map, each target output feature map is recognized, and the recognition result corresponding to each target output feature map is obtained.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining the input input to the current convolutional layer of the trained convolutional neural network Feature map, the input feature map is the feature map generated according to the image data, and the first bias matrix of the current convolutional layer is obtained, where the first bias matrix is the matrix generated when the trained convolutional neural network is generated, according to the input feature Figure generates the second bias matrix, obtains the reference adjacency matrix, calculates the sum of the reference adjacency matrix, the first bias matrix and the second bias matrix, obtains the target adjacency matrix, obtains the convolution kernel of the current convolution layer, and calculates the The convolution kernel of the product layer, the target adjacency matrix and the input feature map generate the target output feature map, and the recognition result of the graph data is recognized according to the target output feature map.

In one embodiment, the second bias matrix generated according to the input feature map includes: using the dimensionality reduction function in the trained convolutional neural network to perform dimensionality reduction on the input feature map to obtain a dimensionality reduction matrix, normalized reduction dimension matrix to obtain a normalized matrix, and the normalized matrix is the second bias matrix.

In one embodiment, the dimension reduction function includes two, and the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, including: according to the first dimension reduction function in the dimension reduction function, each of the input feature maps The matrix of the channel is dimensionally reduced to obtain the first dimensionality reduction matrix of each channel, and the matrix of each channel of the input feature map is reduced according to the second dimensionality reduction function in the dimensionality reduction function to obtain the second dimensionality reduction matrix of each channel , calculate the product of the first dimensionality reduction matrix and the second dimensionality reduction matrix of each channel, obtain the first product matrix of each channel, normalize the first product matrix of each channel, and obtain the matrix of the channel corresponding to the normalization matrix.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the step of generating a trained convolutional neural network includes: obtaining a training set comprising a plurality of training graph data, the training graph data carrying label information, and The training image data and label information are input into the initial convolutional neural network, and the characteristics of each training image data are extracted through the initial convolutional neural network. According to the characteristics of each training image data, the recognition results corresponding to each training image data are identified. According to the preset loss The function calculates the recognition result of each training image data and the loss value of the label. When the loss value is less than or equal to the preset loss value, a trained convolutional neural network is obtained.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the loss value is greater than the preset loss value, update the network parameters of the initial convolutional neural network through the gradient return algorithm according to the loss value, and use the updated network The initial convolutional neural network with parameters is used as the initial convolutional neural network, and the training image data and label information are input into the initial convolutional neural network until the recognition result of each training image data and the loss value of the label are calculated according to the preset loss function, which is less than Or when it is equal to the preset loss value, the trained convolutional neural network is obtained.

In one embodiment, the initial convolutional neural network model includes at least one convolutional layer, the convolutional layer includes an initial bias matrix and an initial dimensionality reduction function, and the network of the initial convolutional neural network is updated through a gradient backpropagation algorithm according to the loss value Parameters, including: when the loss value is returned to any convolution layer through the gradient return algorithm, the return value of each convolution layer is obtained, and the parameters of the initial dimensionality reduction function and the initial dimensionality reduction function are updated according to the return value of each convolution layer. Parameters for the bias matrix.

In one embodiment, according to the target output feature map, identifying the recognition result corresponding to the graph data includes: when the current convolutional layer is the last convolutional layer in the trained convolutional neural network, judging multiple target output Whether there is a target feature map that needs to be merged in the feature map, if it exists, merge the target output feature map that needs to be merged to obtain the merged feature map, when the merged feature map contains all target output feature maps, identify the merged feature map to get The recognition result corresponding to the merged feature map. When the merged feature map contains all target output feature maps, the merged feature map is recognized to obtain the recognition result corresponding to the merged feature map. The unmerged target output feature map is recognized to obtain the unmerged feature map. If there is no recognition result corresponding to the target output feature map, each target output feature map is recognized, and the recognition result corresponding to each target output feature map is obtained.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be realized through computer programs to instruct related hardware, and the programs can be stored in a non-volatile computer-readable storage medium When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above descriptions are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Accordingly, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. a kind of diagram data recognition methods, which is characterized in that the described method includes:

The input feature vector figure for the current convolutional layer of convolutional neural networks that input has been trained is obtained, according to the input feature vector figure The characteristic pattern that diagram data generates；

The first bias matrix of the current convolutional layer is obtained, wherein first bias matrix is to generate the volume trained The matrix generated when product neural network；

The second bias matrix is generated according to the input feature vector figure；

It obtains and refers to adjacency matrix, calculate described with reference to adjacency matrix, first bias matrix and second bias matrix Sum, obtain target adjacency matrix；

Obtain the convolution kernel of the current convolutional layer；

It is special that target output is generated according to the convolution kernel of the current convolutional layer, the target adjacency matrix and the input feature vector figure Sign figure；

Characteristic pattern is exported according to the target, identifies the corresponding recognition result of the diagram data.

2. the method stated when institute according to claim 1, which is characterized in that it is described according to the input feature vector figure generate second partially Set matrix, comprising:

Dimensionality reduction is carried out to the input feature vector figure using the dimensionality reduction function in the convolutional neural networks trained, obtains dimensionality reduction Matrix；

The dimensionality reduction matrix is normalized, normalization matrix is obtained, the normalization matrix is second bias matrix.

3. according to the method described in claim 2, it is characterized in that, the dimensionality reduction function include two, the input feature vector figure Including at least three dimensions, wherein the first dimension is port number, which comprises

Dimensionality reduction is carried out according to matrix of the first dimensionality reduction function in the dimensionality reduction function to each channel of the input feature vector figure, Obtain the first dimensionality reduction matrix in each channel；

Dimensionality reduction is carried out according to matrix of the second dimensionality reduction function in the dimensionality reduction function to each channel of the input feature vector figure, Obtain the second dimensionality reduction matrix in each channel；

The first dimensionality reduction matrix in each channel and the product of the second dimensionality reduction matrix are calculated, first product in each channel is obtained Matrix；

The first product matrix for normalizing each channel, obtains the matrix in the corresponding channel of the normalization matrix.

4. the method according to claim 1, wherein the step of generating the convolutional neural networks trained, Include:

The training set comprising multiple trained diagram datas is obtained, the trained diagram data carries label information；

The trained diagram data and the label information are inputted into initial convolutional neural networks, pass through the initial convolution nerve net Network extracts the feature of each trained diagram data；

According to the feature of each trained diagram data, the corresponding recognition result of each trained diagram data is identified；

The recognition result of each trained diagram data and the penalty values of the label are calculated according to default loss function；

When the penalty values are less than or equal to default penalty values, the convolutional neural networks trained are obtained.

5. according to the method described in claim 4, it is characterized in that, the method also includes:

It is described just by gradient passback algorithm update according to the penalty values when the penalty values are greater than the default penalty values The network parameter of beginning convolutional neural networks；

Using having updated the initial convolutional neural networks of network parameter as the initial convolutional neural networks, by the instruction Practice diagram data and the label information inputs initial convolutional neural networks, until described calculate each institute according to default loss function The recognition result of trained diagram data and the penalty values of the label are stated, when being less than or equal to the default penalty values, is obtained described The convolutional neural networks trained.

6. according to the method described in claim 5, it is characterized in that, the initial convolution neural network model includes at least one Convolutional layer, includes initial bias matrix and initial dimensionality reduction function in the convolutional layer, described to pass through gradient according to the penalty values Passback algorithm updates the network parameter of the initial convolutional neural networks, comprising:

When the penalty values being passed back to any one of convolutional layer by gradient passback algorithm, each volume is obtained The return value of lamination；

The network parameter of the convolutional layer is updated according to the return value of each convolutional layer, the network parameter includes initial drop Tie up the parameter of function and the parameter of the initial bias matrix.

7. method according to any one of claim 1 to 6, which is characterized in that described to export feature according to the target Figure, identifies the corresponding recognition result of the diagram data, comprising:

When the current convolutional layer is the last one convolutional layer in the convolutional neural networks trained, multiple institutes are judged State target signature combined with the presence or absence of needs in target output characteristic pattern；

When it is present, merge the target output characteristic pattern for needing to merge, obtain merging characteristic pattern；

When the merging characteristic pattern includes all target output characteristic patterns, the merging characteristic pattern is identified, is obtained To the corresponding recognition result of the merging characteristic pattern；

When the merging characteristic pattern includes all target output characteristic patterns, the merging characteristic pattern is identified, is obtained To the corresponding recognition result of the merging characteristic pattern, the target output characteristic pattern not merged is identified, is obtained described The corresponding recognition result of target output characteristic pattern not merged；

When it be not present, each target output characteristic pattern is identified, obtains each target output characteristic pattern pair The recognition result answered.

8. a kind of diagram data identification device, which is characterized in that described device includes:

Data acquisition module, the input feature vector figure of the current convolutional layer for obtaining the convolutional neural networks that input has been trained, institute Stating input feature vector figure is the characteristic pattern generated according to image data, obtains the first bias matrix of the current convolutional layer, wherein First bias matrix is the matrix generated when generating the convolutional neural networks trained；

Second bias matrix generation module, for generating the second bias matrix according to the input feature vector figure；

Target adjacency matrix life module, refers to adjacency matrix for obtaining, and calculates the reference adjacency matrix, first biasing The sum of matrix and second bias matrix, obtains target adjacency matrix；

Target exports characteristic pattern generation module, for obtaining the convolution kernel of the current convolutional layer, according to the current convolutional layer Convolution kernel, the target adjacency matrix and the input feature vector figure generate target and export characteristic pattern；

Identification module identifies the recognition result of the diagram data for exporting characteristic pattern according to the target.

9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor Calculation machine program, which is characterized in that the processor realizes any one of claims 1 to 7 institute when executing the computer program The step of stating method.

10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 7 is realized when being executed by processor.