CN110363086A - Image data recognition method, device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a graph data recognition method, device, computer equipment and storage medium. The method includes: obtaining the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map obtained by extracting map data, and obtaining the bias matrix of the current convolutional layer, wherein the bias The matrix is the matrix generated when the trained convolutional neural network is generated, obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the bias matrix, obtain the target adjacency matrix, and obtain the convolution kernel of the current convolution layer, according to the current convolution The convolution kernel of the layer, the target adjacency matrix and the input feature map generate the target output feature map. According to the target output feature map, the recognition result corresponding to the graph data is recognized, and the recognition result corresponding to the graph data is recognized according to the target output feature map. By adding a matrix generated according to task requirements to the fixed adjacency matrix in the convolutional layer, the recognition accuracy of the trained convolutional neural network is improved.

Description

Image data recognition method, device, computer equipment and storage medium

technical field

The present application relates to the field of computer technology, and in particular to a method, device, computer equipment and storage medium for identifying image data.

Background technique

In the skeleton point data, the human body is represented by the coordinates of several predefined key joint points in the camera coordinate system. It can be easily obtained by depth cameras (such as Kinect) and various pose estimation algorithms (such as OpenPose). Figure 1 shows the key joint points of the human body defined by the Kinect depth camera. It defines the human body as the three-dimensional coordinates of 25 key joint points. Since behaviors often exist in the form of video, a behavior with a length of T frames can be represented by a Tx25x3 tensor.

Referring to FIG. 2, FIG. 2 is a space-time diagram in one embodiment. Each joint point is defined as a node of the graph, and the physical connection between the joint points is defined as the edge of the graph, and the edge of the time dimension is added between the same nodes of adjacent frames to obtain a space-time graph that can describe human behavior .

At present, the common behavior recognition method based on skeleton points is graph convolution. Graph convolution is different from ordinary convolution operations. When convolution is performed on a graph, the number of neighbor nodes of each node is not fixed, and the parameters of the convolution operation are fixed. In order to combine a fixed number of parameters with an indeterminate number To correspond to the number of adjacent nodes, it is necessary to define a mapping function, and realize the correspondence between parameters and nodes through the mapping function. If the size of the convolution kernel is defined as three, as shown in Figure 3, the three parameters correspond to the point 001 far away from the center of the human body, the point 002 close to the center of the human body 000, and the convolution point itself 003. Then the convolution operation can be expressed by formula (1):

Among them, f is the input and output feature tensor, w is the convolution parameter, v is the node in the graph, l represents the mapping function between the node and the parameter, and Z is the normalization function. In the specific implementation, the mapping function can be realized through the adjacency matrix of the graph, and the convolution operation represented by the adjacency matrix is shown in formula (2):

Where A represents the adjacency matrix of the graph, K is the size of the convolution kernel, and Λ is used to normalize A. By multiplying with the adjacency matrix A, the required nodes are "filtered" from the feature tensor and multiplied with the corresponding parameters.

When the above convolution operation is represented by the adjacency matrix, the adjacency matrix defines the topology of the human body graph used in the graph convolutional network. There are various postures of the human body, and the fixed topological structure cannot accurately describe each posture of the human body, resulting in low recognition accuracy.

Contents of the invention

In order to solve the above-mentioned technical problems, the present application provides a graph data recognition method, device, computer equipment and storage medium.

In the first aspect, the present application provides a graph data recognition method, including:

Obtain the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is the feature map obtained by extracting the map data;

Obtain the bias matrix of the current convolutional layer, where the bias matrix is the matrix generated when the trained convolutional neural network is generated;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

According to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, the target output feature map is generated, and the recognition result corresponding to the map data is identified according to the target output feature map;

According to the target output feature map, identify the recognition result corresponding to the map data.

In a second aspect, the present application provides a graph data identification device, including:

The input feature map acquisition module is used to obtain the input feature map of the current convolutional layer of the trained convolutional neural network, and the input feature map is a feature map obtained by extracting map data;

The bias matrix acquisition module is used to obtain the bias matrix of the current convolutional layer, wherein the bias matrix is a matrix generated when the trained convolutional neural network is generated;

The target adjacency matrix calculation module is used to obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix;

The convolution kernel acquisition module is used to obtain the convolution kernel of the current convolution layer;

The feature map generation module is used to generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and identify the recognition result corresponding to the map data according to the target output feature map;

According to the target output feature map, identify the recognition result corresponding to the map data.

A computer device, comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the following steps when executing the computer program:

Obtain the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is the feature map obtained by extracting the map data;

Obtain the bias matrix of the current convolutional layer, where the bias matrix is the matrix generated when the trained convolutional neural network is generated;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

According to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, the target output feature map is generated, and the recognition result corresponding to the map data is identified according to the target output feature map;

According to the target output feature map, identify the recognition result corresponding to the map data.

A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Obtain the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is the feature map obtained by extracting the map data;

Obtain the bias matrix of the current convolutional layer, where the bias matrix is the matrix generated when the trained convolutional neural network is generated;

Obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix;

Get the convolution kernel of the current convolution layer;

According to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, the target output feature map is generated, and the recognition result corresponding to the map data is identified according to the target output feature map;

According to the target output feature map, identify the recognition result corresponding to the map data.

The above graph data identification method, device, computer equipment and storage medium, said method comprising: obtaining the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map obtained by extracting the graph data , to obtain the bias matrix of the current convolutional layer, where the bias matrix is the matrix generated when the trained convolutional neural network is generated, obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the bias matrix, and obtain the target adjacency matrix, Obtain the convolution kernel of the current convolution layer, generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and identify the recognition result corresponding to the graph data according to the target output feature map. Add a bias matrix to the adjacency matrix in each convolutional layer in the trained convolutional neural network. The bias matrix is the matrix generated by the trained convolutional neural network. Adding the bias matrix can better express the human body posture , improve the accuracy of the generated feature map, and then improve the recognition accuracy of the trained convolutional neural network.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, In other words, other drawings can also be obtained from these drawings without paying creative labor.

Fig. 1 is the schematic diagram of the key joint point of the human body defined by Kinect depth camera in an embodiment;

Fig. 2 is a space-time diagram describing human behavior in one embodiment;

Fig. 3 is a schematic diagram of nodes defined in graph convolution in an embodiment;

Fig. 4 is an application environment diagram of the image data identification method in one embodiment;

Fig. 5 is a schematic flow chart of a method for identifying image data in an embodiment;

Fig. 6 is a structural block diagram of an image data identification device in an embodiment;

Figure 7 is an internal block diagram of a computer device in one embodiment.

Figure 8 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

Fig. 4 is an application environment diagram of the image data identification method in an embodiment. Referring to Fig. 4, the data recognition method in this graph is applied to a behavior recognition system. The behavior recognition system includes a terminal 110 and a server 120 . Terminal 110 and server 120 are connected via a network. The terminal or server obtains the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is the feature map obtained by extracting the image data, and obtains the bias matrix of the current convolutional layer, where the bias matrix is Generate the matrix generated when the trained convolutional neural network is generated, obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the bias matrix, obtain the target adjacency matrix, obtain the convolution kernel of the current convolution layer, according to the current convolution layer The convolution kernel, the target adjacency matrix and the input feature map generate the target output feature map, and identify the recognition result corresponding to the graph data according to the target output feature map. The terminal 110 may specifically be a desktop terminal or a mobile terminal, and the mobile terminal may specifically be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers.

As shown in FIG. 5 , in one embodiment, a method for identifying graph data is provided. This embodiment is mainly described by taking the method applied to the terminal 110 (or server 120) in FIG. 4 above as an example. Referring to Fig. 2, the data identification method in this figure specifically includes the following steps:

Step S201, obtaining the input feature map of the current convolutional layer of the trained convolutional neural network.

In this specific embodiment, the input feature map is a feature map obtained by extracting map data.

Specifically, the trained convolutional neural network is obtained by training a large amount of labeled graph data, where the graph data is a spatio-temporal graph of human behavior, and the spatio-temporal graph is shown in Figure 2. The labels carried by the graph data include human behaviors, such as individual or multi-person behaviors such as clapping, jumping, pulling hands, and fighting. The trained convolutional neural network contains multiple convolutional layers, and the current convolutional layer can be any convolutional layer in the convolutional neural network. The output data of the previous convolutional layer is the input data of the current convolutional layer, and the output data of the previous convolutional layer is obtained as the input data of the current convolutional layer, and the input data is the input feature map.

Step S202, obtaining the bias matrix of the current convolutional layer.

In this specific embodiment, the bias matrix is a matrix generated when the trained convolutional neural network is generated.

Step S203, obtaining the reference adjacency matrix, calculating the sum of the reference adjacency matrix and the offset matrix, and obtaining the target adjacency matrix.

Specifically, the offset matrix is a matrix used to adjust the reference adjacency matrix, and the offset matrix and the reference adjacency matrix specifically have the same dimension information. The offset matrix is the offset matrix obtained according to the training requirements. Different training requirements refer to the requirements for what to do after the convolutional neural network is trained, such as the offset matrix obtained when it is used to identify clapping hands and for identifying fights. Are not the same. The reference adjacency matrix is used for the topology of human body graphs in graph convolutional networks, as a matrix containing fixed matrix elements. The bias matrix is a matrix generated according to the generated trained convolutional neural network, and the matrix elements at corresponding positions in the bias matrix of each convolution layer pair may be different. The matrix elements of the bias matrix are obtained by generating the trained convolutional neural network. That is, the bias matrix is to generate the network parameters in the trained convolutional neural network, and in the process of training the convolutional neural network, the update of the network parameters includes the update of the bias matrix. Calculate the sum of the matrix elements in the corresponding positions of the reference adjacency matrix and the offset matrix to obtain the target adjacency matrix. The reference adjacency matrix is adjusted by the bias matrix to obtain the target adjacency matrix, and the target adjacency matrix is used to describe human behavior, and a more accurate description of human behavior is obtained.

In one embodiment, generating a trained convolutional neural network includes: obtaining a training set that includes a plurality of training graph data, each training graph data includes corresponding label information, and performing an initial convolutional neural network on each training graph data Recognition, obtain the corresponding recognition results, calculate the recognition results of each training image data and the loss value of the label according to the preset loss function, and when the loss value is less than or equal to the preset difference loss value, the trained convolutional neural network is obtained.

Specifically, the training graph data is the collected spatio-temporal graph. Label information is used to identify the training graph data. Label information includes human behavior labels, figure numbers, and so on. Input the data of each training graph into the initial convolutional neural network, extract the features of the training graph data through the initial convolutional neural network, and obtain the corresponding recognition results according to the extracted feature recognition, and calculate the recognition results of each graph data according to the preset loss function. The loss value of the label information, according to the loss value to determine whether the initial convolutional neural network converges. The preset loss function is a preset function used to calculate the loss value of the network, and the loss function may adopt a common network loss function. When the loss value is less than or equal to the preset loss value, the initial convolutional neural network converges to obtain a trained convolutional neural network.

In one embodiment, when the loss value is greater than the preset loss value, the loss value is returned through the gradient return algorithm, the network parameters of the initial convolutional neural network are updated, and the initial convolutional neural network with updated network parameters is used for training The graph data is recognized, and the corresponding recognition result is obtained. When the loss value between the recognition result and the label information is less than the preset loss value, the trained convolutional neural network is obtained.

Specifically, the gradient backpropagation algorithm is a method of updating network parameters by chasing layers using a gradient descent method. When the loss value is greater than the preset loss value, it means that the initial convolutional neural network has not converged, and the network parameters need to be updated. The update of network parameters is determined according to the difference between the output result of the last layer and the real output result, that is, according to the return loss value through the gradient return algorithm, the initial convolutional neural network is updated layer by layer according to the return loss value. Network parameters. Re-identify the training image data for the initial convolutional neural network with updated network parameters. According to the loss value between the recognition result and the label information, determine whether the network is converged again. If not, continue to update the network parameters until the network converges. Trained Convolutional Neural Network.

In one embodiment, the initial convolutional neural network model includes at least one convolutional layer, the convolutional layer includes an initial bias matrix, and returns the degree of difference through the gradient return algorithm, and updates the network parameters of the initial convolutional neural network, including : When the loss value is returned to any convolution layer through the gradient return algorithm, the return value of each convolution layer is obtained, and the parameters of the bias matrix are updated according to the return value of each convolution layer.

Specifically, the preset loss function is a pre-configured function used to calculate the loss value of the network, and the preset loss function may be a common loss function, such as a cross-entropy loss function, a square loss function, and the like. The gradient return algorithm refers to passing the difference data between the wrong recognition result and the real label from the final output layer of the convolutional neural network layer by layer, and each layer updates the network parameters according to the difference data, that is, the return value. When it is returned to any convolutional layer, the network parameters are updated according to the returned value of the convolutional layer, wherein the network parameters include the parameters of the bias matrix.

Step S204, acquiring the convolution kernel of the current convolution layer.

Specifically, each convolution layer includes multiple convolution kernels, the number of convolution kernels corresponding to each convolution layer may be the same or different, and each convolution kernel may be the same or different. Convolution kernels are used to perform convolution operations on images, and different convolution kernels can extract different image features.

Step S205, generating a target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map.

Specifically, the target adjacency matrix is used to perform feature extraction on the input feature map to obtain the feature map. The convolution kernel is used to perform convolution operation on the feature map to obtain the convolution feature map, and the convolution feature map is used as the target output feature map. The target adjacency matrix is used to extract features from the input feature map to obtain more accurate image features.

In one embodiment, step S205 includes:

Step S2051, reshaping the input feature map to obtain the reshaping feature map.

In this specific embodiment, the first dimension of the reshaped feature map is the product of the first dimension and the second dimension of the input feature map.

Specifically, reshaping refers to adjusting the input feature map so that the product of the first dimension and the second dimension is the first dimension of the reshaping feature map, such as adjusting the input feature map containing three dimensions into two dimensions. Reshape the feature map, assuming that the input feature map is C×M×N, where the first dimension is C, the second dimension is M, and the third dimension is N, then the reshaping map can be C×M×N, the first dimension It is the product CM of C and M, the second dimension is the same as the third dimension of the input feature map, keeping the elements of the entire input feature map unchanged, and the total element is the product CMN of C, M and N. The first dimension C is the channel pixel, the second dimension M is the number of rows of the input feature map, and the third dimension N is the number of columns of the input feature map. Among them, N represents the number of joint points of the human body, and N is defined as 25 in Kinect. The reshaping of the matrix is for convenience of operation.

Step S2052, calculating the product of the reshaped feature map and the matrix of each channel of the target adjacency matrix to obtain a second product matrix of each channel.

Specifically, the second dimension of the reshaped feature map is the same as the first dimension of the matrix of each channel of the target adjacency matrix. For example, the reshaped feature map is C×M×N, the target adjacency matrix is C×N×N, and each channel The matrix is N×N, then the product matrix of the reshaped feature map and the matrix of each channel is CMN.

Step S2053, dereshaping the second product matrix of each channel to obtain the deremodeling feature map of each channel.

Specifically, anti-reshaping is the inverse process of reshaping. For example, reshaping is to convert a three-dimensional matrix into a two-dimensional matrix, then anti-reshaping is to convert a two-dimensional matrix into a three-dimensional matrix. For example, the product matrix of each channel is CM×N , then the anti-reshaping feature map obtained after anti-reshaping is a C×M×N matrix.

Step S2054, according to the convolution kernel of each channel, perform convolution operation on the anti-reshaping feature map according to the convolution kernel of each channel, and obtain the target feature map of each channel of the current convolution layer.

In step S2055, the target feature map of each channel is summed to obtain the output feature map of the current convolution layer, and the output feature map of the current convolution layer is used as the target output feature map.

Specifically, through the convolution kernels corresponding to each channel, feature extraction is performed on the anti-reshaping feature map to obtain the features corresponding to each convolution kernel, and the features extracted by each convolution kernel form the target feature map of each channel. Calculate the sum of the target feature maps of each channel, that is, add the matrix elements at the corresponding positions to obtain the output feature map, which is the output feature map of the current convolutional layer.

In one embodiment, step S205 further includes:

Step S2056, judging whether the number of channels of the output feature map is consistent with the number of channels of the input feature map.

Step S2057, when consistent, use the sum of the input feature map and the output feature map as the target output feature map of the current convolutional layer.

Step S2058, if they are inconsistent, perform a convolution operation on the input feature map to obtain a convolution feature map with the same number of channels as the output feature map, and use the sum of the convolution feature map and the output feature map as the target output feature map.

Specifically, according to each channel matrix of the output feature map generated by the convolution kernel of each channel and the corresponding frontal inverse reshaping matrix, it is judged whether the channel of the output feature map is consistent with the number of channels of the input feature map. When they are consistent, the input The feature map is added to the element at the position corresponding to the output feature to obtain the target output feature map. When inconsistent, perform convolution operation on the input feature map to obtain a convolution feature map with the same channel as the output feature map, calculate the sum of the elements at the same position in the convolution feature map and the output feature map, and obtain the target output feature map. By judging the channel numbers of the input feature map and the output feature map, the target output feature map is determined according to the channel number, which improves the accuracy of the target output feature map.

Step S206, according to the target output feature map, identify the recognition result corresponding to the map data.

Specifically, the target output feature map is input into the recognition layer in the trained convolutional neural network, the candidate recognition results corresponding to the target output feature map are identified through the recognition layer, and the candidate recognition result with the highest recognition probability is selected from the candidate recognition results as Target recognition result, the target recognition result is used as the recognition result corresponding to the graph data. For example, when the recognition types include clapping, jumping, and holding hands, the recognition probability corresponding to clapping is 0.89, the recognition probability corresponding to jumping is 0.01, and the recognition probability corresponding to holding hands is 0.1, then the recognition result corresponding to the graph data is clapping.

In one embodiment, when the trained convolutional neural network includes the current convolutional layer and the next convolutional layer of the current convolutional layer, the target output feature map is used as the input feature map of the next convolutional layer, Use the next convolutional layer as the current convolutional layer, and enter to obtain the input feature map of the current convolutional layer of the trained convolutional neural network until all convolutional layers in the trained convolutional neural network are completed. Output the target output feature map of the last convolutional layer, input the target output feature map of the last convolutional layer into the recognition layer, and obtain the corresponding recognition result of the book. The data processing flow of convolutional layers with the same network structure is the same.

The above graph data recognition method includes: obtaining the input feature map of the current convolutional layer of the trained convolutional neural network, the input feature map is a feature map obtained by extracting the graph data, and obtaining the bias matrix of the current convolutional layer, The bias matrix is the matrix generated when the trained convolutional neural network is generated, obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the bias matrix, obtain the target adjacency matrix, and obtain the convolution kernel of the current convolution layer, according to The convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map generate the target output feature map, and identify the recognition result corresponding to the graph data according to the target output feature map. Add an offset matrix to the adjacency matrix in each convolutional layer in the trained convolutional neural network. The offset matrix is the matrix generated by the trained convolutional neural network. The offset matrix can better express the posture of the human body , improve the accuracy of the generated feature map, and then improve the recognition accuracy of the trained convolutional neural network.

In a specific embodiment, the method for feature matrix generation includes: Referring to FIG. 6, FIG. 6 is a schematic diagram of a data processing flow diagram of a convolutional layer in an embodiment, wherein f _in is the input feature map of the current convolutional layer, and f _out is the output feature map of the current convolutional layer. The output feature map uses the specific representation of the input feature map as shown in formula (3):

Among them, A _k is the kth adjacency matrix in the reference adjacency matrix, B _k is the kth adjacency matrix in the offset matrix, W _k is the kth parameter of the convolution kernel, Kv is the size of the convolution kernel, The size of the convolution kernel can be customized, for example, K _v =3 or 5 can be set. Assume that the size information of the input feature map is C _in ×T×N, where C represents the number of channels, T represents the number of frames of graph data, and N represents the number of joint nodes defined by kinect, where N=25. Reshape the input feature map to obtain the reshaped feature map of C _in T×N. The bias matrix B _k is the matrix obtained after training the convolutional neural network. B _k and A _k have the same size information, which is N× N, calculate the sum of the bias matrix B _k and the reference adjacency matrix A _k to obtain the matrix of each channel of the target adjacency matrix, calculate the product of the matrix of each channel of the target adjacency matrix and the reshaped feature map, and obtain the second product of each channel Matrix, and dereshape the second product matrix of each channel to obtain the deremodeling matrix, and obtain a convolution kernel W _k , which, through the convolution kernel of each channel, perform convolution operation on the deremodeling matrix to obtain each The output feature map corresponding to the channel. Determine whether the output feature map is consistent with the number of channels of the input feature map. If not, pass the residual network res, where the convolution kernel size in the residual network is 1×1. Adjust the input feature map to a matrix with the same number of channels as the output feature map, calculate the sum of the adjusted input feature map and output feature map, and obtain the target output feature map. When consistent, calculate the sum of the input feature map and the output feature map to obtain the target output feature map. According to the target output feature map, the behavior of each graph data is recognized, and the corresponding recognition result is obtained.

When the generation process of the above feature map is the data processing process of training the convolutional neural network, and when the identification corresponding to each graph data is inconsistent with the category in the label of the graph data, according to the loss value corresponding to each graph data according to the preset loss function, Return the loss value according to the gradient return algorithm to obtain the return value of the convolution layer, and update the parameters of the convolution kernel w and the parameters of the bias matrix B of each channel of the corresponding convolution layer according to the return value.

When the above feature map generation process is the data processing process of the trained convolutional neural network, the recognition result of the feature map is output according to the target as the recognition result corresponding to the map data. Input the graph data into the trained convolutional neural network. The convolutional neural network includes multiple convolutional layers and recognition layers. Each convolutional layer includes a convolution kernel and a target adjacency matrix. The target adjacency matrix of each convolutional layer is used to Feature extraction is performed on the image data to obtain the corresponding image feature map set, and the convolution operation is performed on the image feature set through the convolution kernel to obtain the target output feature map of each convolution layer, and the target output feature map of the previous convolution layer of the recognition layer As the input data of the identification layer, according to the target output feature map of each graph data, the corresponding human behavior type is identified.

In the above feature map generation process, the offset matrix is a graph adapted to the behavior recognition task based on the statistics of the database. The offset matrix will be used as a parameter of the network and continuously updated according to the classification loss during the training process. It is for each convolutional layer. different. Since the bias matrix is a matrix obtained according to the statistics of the recognition task, the target adjacency matrix obtained according to the bias matrix can better extract the features of the input data and obtain an accurate target output feature map. Therefore, according to the target output feature map When performing recognition, the recognition result is more accurate.

Fig. 5 is a schematic flowchart of a method for identifying image data in an embodiment. It should be understood that although the various steps in the flow chart of FIG. 5 are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in Fig. 5 may include multiple sub-steps or multiple stages, these sub-steps or stages are not necessarily executed at the same moment, but may be executed at different moments, the execution of these sub-steps or stages The order is not necessarily performed sequentially, but may be performed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

In one embodiment, as shown in FIG. 7 , a graph data identification device 200 is provided, including:

The input feature map acquisition module 201 is configured to acquire the input feature map of the current convolutional layer of the trained convolutional neural network, where the input feature map is a feature map obtained by extracting map data.

The offset matrix acquiring module 202 is configured to acquire the offset matrix of the current convolutional layer, wherein the offset matrix is a matrix generated when the trained convolutional neural network is generated.

The target adjacency matrix calculation module 203 is configured to obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix.

The convolution kernel acquisition module 204 is configured to acquire the convolution kernel of the current convolution layer.

The feature map generation module 205 is used to generate a target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, and identify the recognition result corresponding to the map data according to the target output feature map.

The identification module 206 is configured to identify the identification result corresponding to the image data according to the target output feature map.

In one embodiment, the target adjacency matrix calculation module is also used to reduce the dimensionality of the input feature map of the current convolutional layer to obtain a dimensionality reduction matrix, normalize the dimensionality reduction matrix, obtain a normalization matrix, and convert the normalization matrix Each element of is added to the corresponding element of the target adjacency matrix to obtain the updated target adjacency matrix, and the updated target adjacency matrix is used as the target adjacency matrix.

In one embodiment, the above-mentioned target adjacency matrix calculation module includes:

The dimensionality reduction unit is used to reduce the dimensionality of the matrix of each channel of the input feature map according to the first dimensionality reduction function to obtain the first feature map corresponding to each channel. The input feature map includes at least three dimensions, wherein the first dimension is the number of channels, respectively performing dimensionality reduction on the matrix of each channel of the feature map according to the second dimensionality reduction function, to obtain the second feature map corresponding to each channel;

The first matrix product calculation unit is configured to calculate the first product matrix of the first feature map of each channel and the second feature map, and use the first product matrix of each channel as a matrix corresponding to the channel of the dimensionality reduction matrix.

The normalization unit is used to normalize the matrix of each channel in the dimensionality reduction matrix to obtain the matrix of the channel corresponding to the normalization matrix.

In one embodiment, the above image data identification device further includes:

Network generation module for generating trained convolutional neural networks. The network generation module includes:

The training data acquiring unit is configured to acquire a training set including a plurality of training graph data, each training graph data including corresponding label information.

The identification unit is configured to identify each training image data through the initial convolutional neural network to obtain corresponding identification results.

The difference calculation unit is used to calculate the recognition result of each training image data and the loss value of the label according to a preset loss function.

The model generation unit is used to obtain the trained convolutional neural network when the loss value is less than or equal to the preset difference loss value.

In one implementation, the above-mentioned network generation module further includes:

The parameter update unit is used to return the loss value through the gradient return algorithm when the loss value is greater than the preset loss value, and update the network parameters of the initial convolutional neural network.

The model determination unit is also used to use the initial convolutional neural network with updated network parameters to identify the training image data to obtain the corresponding recognition results, until the loss value between the recognition results and the label information is less than the preset loss value, it is obtained A trained convolutional neural network.

In one embodiment, the feature map generation module includes:

The feature map reshaping unit is used to reshape the input feature map to obtain the reshaped feature map. The first dimension of the reshaped feature map is the product of the first dimension and the second dimension of the input feature map, wherein the input feature map includes at least Three dimensions, the first dimension is the number of channels, and the target adjacency matrix includes at least three dimensions.

The second matrix product calculation unit is used to calculate the product of the reshaped feature map and the matrix of each channel of the target adjacency matrix to obtain the second product matrix of each channel.

The anti-reshaping unit is used for anti-reshaping the second product matrix of each channel to obtain the anti-reshaping feature map of each channel.

The convolution unit is configured to perform a convolution operation on the anti-reshaping feature map according to the convolution kernel of each channel to obtain the target feature map of each channel of the current convolution layer.

The feature map generation unit is configured to sum the target feature maps of each channel to obtain the output feature map of the current convolution layer, and use the output feature map of the current convolution layer as the target output feature map.

In one implementation, the above image data identification device further includes:

The channel judging module is used to judge whether the number of channels of the output feature map is consistent with the number of channels of the input feature map.

The feature map generation module is also used to use the sum of the input feature map and the output feature map as the target output feature map of the current convolution layer when they are consistent, and perform convolution operations on the input feature map to obtain the output feature map when they are inconsistent The convolutional feature map with the same number of channels in the graph uses the sum of the convolutional feature map and the output feature map as the target output feature map.

Figure 8 shows a diagram of the internal structure of a computer device in one embodiment. The computer device may specifically be the terminal 110 (or server 120) in FIG. 4 . As shown in FIG. 8 , the computer equipment includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and may also store a computer program, and when the computer program is executed by the processor, the processor may realize the image data identification method. A computer program may also be stored in the internal memory, and when the computer program is executed by the processor, the processor may execute the image data identification method. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer equipment, or It can be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation on the computer equipment to which the solution of this application is applied. The specific computer equipment can be More or fewer components than shown in the figures may be included, or some components may be combined, or have a different arrangement of components.

In one embodiment, the image data identification device provided in this application can be implemented in the form of a computer program, and the computer program can be run on the computer device as shown in FIG. 8 . Each program module forming the graph data recognition device can be stored in the memory of the computer equipment, for example, the feature map acquisition module 201 shown in Figure 7, the bias matrix acquisition module 202, the standard adjacency matrix calculation module 203, and the convolution kernel acquisition module 204 , feature map generation module 205 and recognition module 206 . The computer program constituted by each program module enables the processor to execute the steps in the image data identification method of each embodiment of the application described in this specification.

For example, the computer equipment shown in FIG. 8 can obtain the input feature map of the current convolutional layer of the trained convolutional neural network through the input feature map acquisition module 201 in the graph data recognition device as shown in FIG. 7 , input A feature map is a feature map obtained by extracting graph data. The computer device can obtain the bias matrix of the current convolutional layer through the bias matrix obtaining module 202, wherein the bias matrix is a matrix generated when the trained convolutional neural network is generated. The computer device can obtain the reference adjacency matrix through the target adjacency matrix calculation module 203, calculate the sum of the reference adjacency matrix and the offset matrix, and obtain the target adjacency matrix. The computer device can acquire the convolution kernel of the current convolution layer through the convolution kernel acquisition module 204 . The computer device can generate the target output feature map according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map through the feature map generation module 205 . The computer device can output the characteristic map according to the target through the recognition module 206, and recognize the recognition result corresponding to the map data.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and operable on the processor. When the processor executes the computer program, the following steps are implemented: obtaining the input trained volume The input feature map of the current convolutional layer of the convolutional neural network, the input feature map is the feature map obtained by extracting the image data, and the bias matrix of the current convolutional layer is obtained, where the bias matrix is generated when the trained convolutional neural network is The generated matrix, obtain the reference adjacency matrix, calculate the sum of the reference adjacency matrix and the bias matrix, obtain the target adjacency matrix, obtain the convolution kernel of the current convolution layer, according to the convolution kernel of the current convolution layer, the target adjacency matrix Generate a target output feature map with the input feature map, and identify the recognition result corresponding to the map data according to the target output feature map.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing dimensionality reduction on the input feature map of the current convolutional layer to obtain a dimensionality reduction matrix, normalizing the dimensionality reduction matrix, obtaining a normalization matrix, and normalizing Each element of the transformation matrix is added to the corresponding element of the target adjacency matrix to obtain the updated target adjacency matrix, and the updated target adjacency matrix is used as the target adjacency matrix.

In one embodiment, the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, including: performing dimensionality reduction on the matrix of each channel of the input feature map according to the first dimensionality reduction function to obtain the corresponding The first feature map of the feature map, according to the second dimensionality reduction function, the matrix of each channel of the feature map is respectively reduced in dimension, and the second feature map corresponding to each channel is obtained, and the first feature map and the second feature map of each channel are calculated. A product matrix, using the first product matrix of each channel as a matrix corresponding to the channel of the dimension reduction matrix, and normalizing the matrix of each channel in the dimension reduction matrix to obtain a matrix corresponding to the channel of the normalized matrix.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: forming a trained convolutional neural network, including: obtaining a training set comprising a plurality of training graph data, each training graph data containing corresponding label information, through The initial convolutional neural network recognizes each training image data and obtains the corresponding recognition results, and calculates the recognition results of each training image data and the loss value of the label according to the preset loss function. When the loss value is less than or equal to the preset difference loss value , to get the trained convolutional neural network.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: when the loss value is greater than the preset loss value, return the loss value through the gradient return algorithm, update the network parameters of the initial convolutional neural network, and use the updated The initial convolutional neural network of the network parameters recognizes the training image data, and obtains the corresponding recognition results, until the loss value between the recognition results and the label information is less than the preset loss value, and the trained convolutional neural network is obtained.

In one embodiment, the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, and the target adjacency matrix includes at least three dimensions, according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map Generating the target output feature map, including: reshaping the input feature map to obtain the reshaping feature map, the first dimension of the reshaping feature map is the product of the first dimension and the second dimension of the input feature map, and calculating the reshaping feature map and the target The product of the matrices of each channel of the adjacency matrix is obtained to obtain the second product matrix of each channel, and the second product matrix of each channel is inversely reshaped to obtain the inverse reshaping feature map of each channel. According to the convolution kernel of each channel, according to each channel The convolution kernel performs convolution operation on the anti-reshaping feature map to obtain the target feature map of each channel of the current convolution layer, and sums the target feature maps of each channel to obtain the output feature map of the current convolution layer, and the current convolution layer The output feature map of the product layer is used as the target output feature map.

In one embodiment, when the processor executes the computer program, the following steps are also implemented: judging whether the channel number of the output feature map is consistent with the channel number of the input feature map, and when consistent, the sum of the input feature map to the output feature map, As the target output feature map of the current convolutional layer, when it is inconsistent, perform a convolution operation on the input feature map to obtain a convolution feature map with the same number of channels as the output feature map, and combine the convolution feature map with the output feature map and , as the target output feature map.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining the input input to the current convolutional layer of the trained convolutional neural network The feature map, the input feature map is the feature map obtained by extracting the image data, and the bias matrix of the current convolutional layer is obtained, where the bias matrix is the matrix generated when the trained convolutional neural network is generated, the reference adjacency matrix is obtained, and the calculation Referring to the sum of the adjacency matrix and the bias matrix, the target adjacency matrix is obtained, the convolution kernel of the current convolution layer is obtained, and the target output feature map is generated according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map, According to the target output feature map, identify the recognition result corresponding to the map data.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing dimensionality reduction on the input feature map of the current convolutional layer to obtain a dimensionality reduction matrix, normalizing the dimensionality reduction matrix, obtaining a normalization matrix, and normalizing the dimensionality reduction matrix. Each element of the normalization matrix is added to the corresponding element of the target adjacency matrix to obtain an updated target adjacency matrix, and the updated target adjacency matrix is used as the target adjacency matrix.

In one embodiment, the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, including: performing dimensionality reduction on the matrix of each channel of the input feature map according to the first dimensionality reduction function to obtain the corresponding The first feature map of the feature map, according to the second dimensionality reduction function, the matrix of each channel of the feature map is respectively reduced in dimension, and the second feature map corresponding to each channel is obtained, and the first feature map and the second feature map of each channel are calculated. A product matrix, using the first product matrix of each channel as a matrix corresponding to the channel of the dimension reduction matrix, and normalizing the matrix of each channel in the dimension reduction matrix to obtain a matrix corresponding to the channel of the normalized matrix.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: forming a trained convolutional neural network, comprising: obtaining a training set comprising a plurality of training graph data, each training graph data including corresponding label information, Identify the data of each training image through the initial convolutional neural network to obtain the corresponding recognition results, calculate the recognition results of each training image data and the loss value of the label according to the preset loss function, when the loss value is less than or equal to the preset difference loss value , the trained convolutional neural network is obtained.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: when the loss value is greater than the preset loss value, return the loss value through the gradient return algorithm, update the network parameters of the initial convolutional neural network, and use the updated The initial convolutional neural network with network parameters recognizes each training image data, and obtains the corresponding recognition results, until the loss value between the recognition results and the label information is less than the preset loss value, and the trained convolutional neural network is obtained.

In one embodiment, the input feature map includes at least three dimensions, wherein the first dimension is the number of channels, and the target adjacency matrix includes at least three dimensions, according to the convolution kernel of the current convolution layer, the target adjacency matrix and the input feature map Generating the target output feature map, including: reshaping the input feature map to obtain the reshaping feature map, the first dimension of the reshaping feature map is the product of the first dimension and the second dimension of the input feature map, and calculating the reshaping feature map and the target The product of the matrices of each channel of the adjacency matrix is obtained to obtain the second product matrix of each channel, and the second product matrix of each channel is inversely reshaped to obtain the inverse reshaping feature map of each channel. According to the convolution kernel of each channel, according to each channel The convolution kernel performs convolution operation on the anti-reshaping feature map to obtain the target feature map of each channel of the current convolution layer, and sums the target feature maps of each channel to obtain the output feature map of the current convolution layer, and the current convolution layer The output feature map of the product layer is used as the target output feature map.

In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: judging whether the channel number of the output feature map is consistent with the channel number of the input feature map, and when consistent, the sum of the input feature map to the output feature map , as the target output feature map of the current convolutional layer, when it is inconsistent, perform a convolution operation on the input feature map to obtain a convolutional feature map with the same number of channels as the output feature map, and combine the convolutional feature map with the output feature map and, as the target output feature map.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be realized through computer programs to instruct related hardware, and the programs can be stored in a non-volatile computer-readable storage medium When the program is executed, it may include the processes of the embodiments of the above-mentioned methods. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The above descriptions are only specific embodiments of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Accordingly, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims (10)

1. a kind of diagram data recognition methods, which is characterized in that the described method includes:

The input feature vector figure of the current convolutional layer for the convolutional neural networks that input has been trained is obtained, the input feature vector figure is to pass through Extract the characteristic pattern that diagram data obtains；

The bias matrix of the current convolutional layer is obtained, wherein the bias matrix is to generate the convolutional Neural net trained The matrix generated when network；

It obtains and refers to adjacency matrix, calculate the sum with reference to adjacency matrix and the bias matrix, obtain target adjacency matrix；

Obtain the convolution kernel of the current convolutional layer；

It is special that target output is generated according to the convolution kernel of the current convolutional layer, the target adjacency matrix and the input feature vector figure Sign figure；

Characteristic pattern is exported according to the target, identifies the corresponding recognition result of the diagram data.

2. the method according to claim 1, wherein generating the convolutional neural networks trained, comprising:

The training set comprising multiple trained diagram datas is obtained, each trained diagram data includes corresponding label information；

Each trained diagram data is identified by initial convolutional neural networks, obtains corresponding recognition result；

The recognition result of each trained diagram data and the penalty values of the label are calculated according to default loss function；

When the penalty values are less than or equal to default poor penalty values, the convolutional neural networks trained are obtained.

3. according to the method described in claim 2, it is characterized in that, the method also includes:

When the penalty values are greater than the default penalty values, algorithm is returned by the gradient and returns the diversity factor, is updated The network parameter of the initial convolutional neural networks；

A trained diagram data is identified using the initial convolutional neural networks for having updated network parameter, is obtained corresponding Recognition result, until being obtained when the penalty values between the recognition result and the label information are less than the default penalty values The convolutional neural networks trained.

4. according to the method described in claim 3, it is characterized in that, initial convolution neural network model includes at least one convolution Layer, it include initial bias matrix in the convolutional layer, the gradient passback algorithm that passes through returns the diversity factor, updates institute State the network parameter of initial convolutional neural networks, comprising:

When the penalty values being passed back to any one of convolutional layer by gradient passback algorithm, each volume is obtained The return value of lamination；

The network parameter of the convolutional layer is updated according to the return value of each convolutional layer, the network parameter includes described inclined Set the parameter of matrix.

5. method according to claim 1 to 4, which is characterized in that the input feature vector figure includes at least three A dimension, wherein the first dimension is port number, and the target adjacency matrix includes at least three dimensions, described to work as according to The convolution kernel of preceding convolutional layer, the target adjacency matrix and the input feature vector figure generate target and export characteristic pattern, comprising:

The input feature vector figure is remolded, obtains remodeling characteristic pattern, the first dimension of the remodeling characteristic pattern is the input feature vector First dimension of figure and the product of the second dimension；

The product for calculating the matrix in each channel of the remodeling characteristic pattern and the target adjacency matrix, obtains each channel Second product matrix；

Bob-weight moulds second product matrix in each channel, obtains the bob-weight modeling characteristic pattern in each channel；

According to the convolution kernel in each channel, convolution fortune is carried out to bob-weight modeling characteristic pattern according to the convolution kernel in each channel It calculates, obtains the target signature in each channel of current convolutional layer；

It sums to the target signature in each channel, obtains the output characteristic pattern of the current convolutional layer, it will be described The output characteristic pattern of current convolutional layer exports characteristic pattern as the target.

6. according to the method described in claim 5, it is characterized in that, the method also includes:

Judge whether the port number of the output characteristic pattern is consistent with the port number of the input feature vector figure；

When consistent, by the input feature vector figure to the sum of the output characteristic pattern, the target as the current convolutional layer is defeated Characteristic pattern out；

When there is inconsistency, convolution algorithm is carried out to the input feature vector figure, obtained consistent with the output port number of characteristic pattern Convolution characteristic pattern, by the convolution characteristic pattern and it is described output characteristic pattern and, as the target output characteristic pattern.

7. a kind of diagram data identification device, which is characterized in that described device includes:

Input feature vector figure obtains module, the input feature vector of the current convolutional layer for obtaining the convolutional neural networks that input has been trained Figure, the input feature vector figure are the characteristic patterns obtained by extracting diagram data；

Bias matrix obtains module, for obtaining the bias matrix of the current convolutional layer, wherein the bias matrix is to generate The matrix generated when the convolutional neural networks trained；

Target adjacency matrix computing module refers to adjacency matrix for obtaining, and calculates described with reference to adjacency matrix and the biasing The sum of matrix obtains target adjacency matrix；

Convolution kernel obtains module, for obtaining the convolution kernel of the current convolutional layer；

Characteristic pattern generation module, for the convolution kernel, the target adjacency matrix and the input according to the current convolutional layer Characteristic pattern generates target and exports characteristic pattern；

Identification module identifies the corresponding recognition result of the diagram data for exporting characteristic pattern according to the target.

8. device according to claim 7, which is characterized in that described device further include:

Network generation module, for generating the convolutional neural networks trained.Wherein network generation module includes:

Data capture unit includes that training for multiple trained diagram datas is gathered for obtaining, and each trained diagram data includes Corresponding label information；

Recognition unit obtains corresponding for being identified by initial convolutional neural networks to each trained diagram data Recognition result；

Calculate loss value cell, for according to preset loss function calculate the recognition result of each trained diagram data with it is described The penalty values of label；

Network determination unit, for obtaining the volume trained when the penalty values are less than or equal to default poor penalty values Product neural network.

9. a kind of computer equipment including memory, processor and stores the meter that can be run on a memory and on a processor Calculation machine program, which is characterized in that the processor realizes any one of claims 1 to 6 institute when executing the computer program The step of stating method.

10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 6 is realized when being executed by processor.