CN112949476B - Text relation detection method, device and storage medium based on graph convolution neural network - Google Patents

Abstract

This application relates to a text relationship detection method, device and storage medium based on graph convolutional neural network, which belongs to the field of computer technology. The method includes: acquiring multiple key information blocks of text information in the target image, and each key information block Each text block includes at least one string; input the string of each text block in each key information block into the node feature extraction model to obtain the node characteristics of the key information block; construct each text block in the key information block and other key information blocks The connectivity relationship between each text block in the information block; based on the connectivity relationships corresponding to each key information block and the location information corresponding to each connectivity relationship, determine the edge features of the key information block; input the node features and edge features in advance The trained graph convolutional neural network can obtain the edge types between each key information block; it is determined that key information blocks with the same edge type are associated; it can improve the accuracy and efficiency of identifying association relationships.

Description

Text relationship detection method, device and storage medium based on graph convolutional neural network

【Technical field】

The present application relates to a text relationship detection method, device and storage medium based on graph convolutional neural network, and belongs to the field of computer technology.

【Background technique】

Text relationship detection is a common requirement in the field of natural language processing. To put it simply, text relationship detection is to identify entities of interest contained in a document and then classify them according to relationship types. For example, extract key information and correlations between documents from documents such as bills and logistics orders. The certification information is structured to help improve the work efficiency of practitioners.

Starting from original documents such as bills and logistics orders, optical character recognition (Optical Character Recognition, OCR) is first used to identify text characters and character position information, and the characters are aggregated based on distance thresholds to obtain text block nodes. Then, the document information can be structured through two steps. The first step is to use the document information extraction model to extract key information blocks with the aggregated text block nodes as input. The second step is to use the key information blocks as input to detect the key information between files. relation.

After extracting the key information blocks, the most common method currently is to write logical rules based on the location information of the key information blocks, and use manually set distance thresholds to determine the correlation between the key information blocks in the horizontal and vertical directions. Another approach is to distinguish key information blocks from keys and values, and there is a matching relationship between keys and values. By building a deep learning model to detect the relationship between key-value pairs, we can obtain the relationship between key information blocks.

However, the method of using logical rules to determine the correlation between key information blocks based on location information is relatively rough. The selection of the distance threshold relies on manual experience and is strongly correlated with the sample. There are differences in the appropriate thresholds for different files, and some file relationship detection results are unreasonable. situations, such as files that do not conform to the normal page layout format.

Information in normal files appears in lines, and there are divisions between lines. Text in the same line can be considered to have the simplest association relationship, that is, line association. However, files that do not conform to normal page layout may have part of the text content that is too long or shifted in position, causing the text to appear in other lines. The text will be misclassified into other lines based on the distance threshold, causing association detection errors. In many cases in document documents, the format requirements for page layout are not strict enough. A certain proportion of documents have no table lines and no obvious separation between rows and columns. Wrong rows and columns are common.

Another method is to first distinguish the key and value of the key information block, and then detect the key-value pair relationship. Among the currently known relationship detection types, there are two types: matching and non-matching. In actual business usage scenarios, there are relationship types between texts. There are many types, and only the matching relationship detection is not flexible enough to meet actual use. Since this method increases the constraints on the relationship between key-value pairs, it cannot support the detection of textual relationships when all values in a row or column are present.

[Content of the invention]

This application provides a text relationship detection method, device and storage medium based on graph convolutional neural network, which can solve the problem of setting logical rules based on location information to determine association relationships, which may lead to unreasonable detection results and inflexible detection methods. question. This application provides the following technical solutions:

In a first aspect, a text relationship detection method based on graph convolutional neural network is provided. The method includes:

Obtain multiple key information blocks of text information in the target image, where the key information blocks include multiple text blocks, and each text block includes at least one character string;

Enter the string string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block;

For each key information block in the plurality of key information blocks, construct a connectivity relationship between each text block in the key information block and each text block in other key information blocks;

Based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship, determine the edge characteristics of the key information block;

Input the node features and the edge features into a pre-trained graph convolutional neural network to obtain the edge types between each key information block;

It is determined that key information blocks with the same edge type are associated.

Optionally, determining the edge characteristics of the key information block based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship includes:

For each connected relationship, determine the sub-edge characteristics corresponding to the connected relationship according to the relative position between the two text blocks connected by the connected relationship;

For each key information block, determine the sub-edge features connected to each text block in the key information block;

The edge features are generated based on sub-edge features corresponding to each key information block.

Optionally, for each connected relationship, determine the sub-edge characteristics corresponding to the connected relationship according to the relative position between the two text blocks connected by the connected relationship, including:

Discretize the relative position according to direction and distance to obtain direction encoding and distance encoding;

Input the direction code and the distance code into the embedding model to obtain the direction embedding code, the horizontal distance embedding code and the vertical distance embedding code;

After splicing the direction embedding code, the horizontal distance embedding code and the vertical distance embedding code, a fixed-length vector is projected to obtain the sub-edge feature.

Optionally, generating the edge features based on sub-edge features corresponding to each key information block includes:

Process each sub-edge feature into the same dimension;

Each processed sub-edge feature is processed into a first fixed dimension to obtain the edge feature.

Optionally, generating the edge features based on sub-edge features corresponding to each key information block includes:

For each key information block, obtain a connectivity relationship matching table composed of each connectivity relationship of the key information block;

According to the connectivity relationship matching table, the corresponding sub-edge feature vector set is searched from the vector table composed of the sub-edge features to obtain the edge features of the key information block.

Optionally, inputting the string string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block includes:

For each text block, input the string in the text block into the pre-trained recurrent neural network RNN to obtain the feature vector of each string;

The feature vector of each string is processed into a second fixed dimension to obtain the node features.

Optionally, inputting the node features and edge features into a pre-trained graph convolutional neural network to obtain edge types between each key information block includes:

For each key information block, calculate the target node information through the graph convolutional neural network using the node characteristics of the key information block and the node characteristics and edge characteristics that have a connectivity relationship with the key information block;

The information of each node associated with the edge is spliced, and the edge attributes are calculated through a multi-layer forward network to obtain the edge type.

In a second aspect, a text relationship detection device based on a graph convolutional neural network is provided. The device includes:

A key information acquisition module, used to obtain multiple key information blocks of text information in the target image, where the key information blocks include multiple text blocks, and each text block includes at least one character string;

The node feature extraction module is used to input the string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block;

A connectivity relationship building module configured to, for each key information block among the plurality of key information blocks, construct a connectivity relationship between each text block in the key information block and each text block in other key information blocks;

An edge feature extraction module, configured to determine the edge features of the key information block based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship;

An edge type calculation module, used to input the node characteristics and the edge characteristics into a pre-trained graph convolutional neural network to obtain the edge type between each key information block;

The association relationship determination module is used to determine that key information blocks with the same edge type have an association relationship.

In a third aspect, a text relationship detection device based on a graph convolutional neural network is provided. The device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement the first The text relationship detection method based on graph convolutional neural network described in one aspect.

In a fourth aspect, a computer-readable storage medium is provided. A program is stored in the storage medium. The program is loaded and executed by the processor to implement the text relationship based on the graph convolutional neural network described in the first aspect. Detection method.

The beneficial effects of this application are: by obtaining multiple key information blocks of text information in the target image, the key information blocks include multiple text blocks, each text block includes at least one character string; each text in each key information block is The string of the block is input into the node feature extraction model to obtain the node features of the key information block; for each key information block in multiple key information blocks, each text block in the key information block and each text in other key information blocks are constructed. Connectivity relationships between blocks; determine the edge features of key information blocks based on the connectivity relationships corresponding to each key information block and the position information corresponding to each connectivity relationship; input the node features and edge features into the pre-trained graph convolution Neural network can obtain the edge types between each key information block; determine that key information blocks with the same edge type are associated; it can solve the problem of setting logical rules based on location information to determine the association, which may lead to unreasonable detection results, and improve Accuracy of association relationship identification. At the same time, the relationship detection method provided by this application does not need to distinguish the key and value of the key information block, and directly detects the association relationship, which can improve the efficiency of association relationship detection.

The above description is only an overview of the technical solutions of the present application. In order to have a clearer understanding of the technical means of the present application and implement them according to the contents of the specification, the preferred embodiments of the present application are described in detail below with reference to the accompanying drawings.

[Picture description]

Figure 1 is a flow chart of a text relationship detection method based on a graph convolutional neural network provided by an embodiment of the present application;

Figure 2 is a flow chart of a text relationship detection method based on a graph convolutional neural network provided by another embodiment of the present application;

Figure 3 is a block diagram of a text relationship detection device based on a graph convolutional neural network provided by an embodiment of the present application;

Figure 4 is a block diagram of a text relationship detection device based on a graph convolutional neural network provided by an embodiment of the present application.

【Detailed ways】

Specific implementations of the present application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

First, some terms involved in this application will be introduced.

Optical Character Recognition (OCR): It is a recognition technology that converts information in images into text.

Text block node: a text block segmented by a certain threshold, including text content, text position and related image background.

Key information block: It consists of a set of text block nodes of existing type, which are valuable information in the document file, such as price, weight, etc.

Node characteristics: refers to the characteristic information of key information blocks, which is encoded and obtained from the text content and type of key information blocks.

Edge characteristics: refers to the characteristic information of the connected edges between key information blocks.

Sub-edge features: refer to the characteristic information of the edges connected between text block nodes. The characteristic information of edge connections is obtained by encoding from the position information of the nodes.

Information extraction model: A model that uses text block nodes as input to extract key information blocks in text files. The graph convolution network is its core component.

Recurrent Neural Network (RNN): It is a special neural network structure consisting of an input layer, a hidden layer and an output layer.

Summary model: An artificially designed neural network structure that takes a set of feature vectors as input and outputs a fixed-dimensional vector to represent the semantic information of this set of feature vectors.

Graph Convolutional Network (GCN): refers to a neural network that uses graph convolution and can be applied to graph embedding (Graph Embedding/NetworkEmbedding, GE).

Graph G = (V, E), V is a set of nodes, and E is a set of edges. For each node i, it has its characteristics x _i , which can be represented by the matrix X _N*D . Among them, N represents the number of nodes, and D represents the number of features of each node, or the dimension of the feature vector.

Graph convolution refers to the process of determining the feature representation of the current node through the surrounding nodes of the current node. The surrounding nodes may be neighbor nodes of the current node, that is, nodes (or nodes) that are associated with the current node, or neighbor nodes of the neighbor nodes of the current node, etc. This application does not limit the types of surrounding nodes.

Graph convolution can be represented by the following nonlinear function:

H ^l+1 =f(H ^l ,A)

Among them, H ⁰ =X is the input of the first layer ^, The functions f are the same or different.

Optionally, this application takes as an example that the execution subject of each embodiment is an electronic device with computing capabilities. The electronic device can be a terminal or a server. The terminal can be a computer, a notebook computer, a tablet computer, etc. In this embodiment The type of terminal and the type of electronic equipment are not limited.

The text relationship detection method provided in this embodiment is suitable for identifying associations between key information blocks in text files. Each key information block corresponds to a named entity. For example, the bill file includes four key information blocks, namely: product country of origin, product quantity, product unit price and product total price. The number of each key information block is multiple. This embodiment can detect key information blocks that have an associated relationship (eg, belong to the same product).

In practical applications, key information blocks can be key information blocks in various documents such as value-added tax invoices and insurance policies, or key information blocks in other types of documents, such as key information blocks in document images. This implementation The example does not limit the type of key information block.

Based on this, if the key information blocks contain multiple types of key information blocks, and the number of each type of key information blocks is multiple, how to accurately identify the key information blocks with associated relationships has become an urgent problem to be solved. Based on this, the text relationship detection scheme provided by this application takes key information blocks in text files (such as document files, certificate files, etc.) as input. The key information blocks are composed of a set of text block nodes of existing types; with key information blocks As nodes, the connectivity between key information blocks is used as edges to construct an undirected graph. Using this graph, graph neural network coding is used to learn features and predict the type of edge connection relationships; among them, the nodes and edges input to the graph neural network need to be drawn from key information blocks. Extract corresponding feature information, node features are extracted from the key information blocks themselves, and edge features are extracted from the edge connectivity relationships between key information blocks; then the node features and edge features are input into the constructed graph neural network architecture to encode and learn the edge relationship features vector; finally, the edge type is predicted based on the edge relationship feature vector and aggregated to obtain the association between key information blocks.

Since the association relationship can be automatically detected based on the characteristics and edge features of key information blocks, it can solve the problem of setting logical rules based on location information to determine the association relationship, which may lead to unreasonable detection results, and improve the accuracy of association relationship identification.

In addition, the relationship detection method provided by this application does not need to distinguish the key and value of the key information block, and directly detects the association relationship, which can improve the efficiency of association relationship detection.

The text relationship detection method based on graph convolutional neural network provided by this application is introduced in detail below.

Figure 1 is a flow chart of a text relationship detection method based on a graph convolutional neural network provided by an embodiment of the present application. This method includes at least the following steps:

Step 101: Obtain multiple key information blocks of text information in the target image. The key information blocks include multiple text blocks, and each text block includes at least one character string.

The key information block is the information that needs to be extracted from the text information of the target image, or the information that the user pays attention to.

Optionally, the key information block can be recognized by the electronic device, such as using an information extraction model stored inside the electronic device to extract the key information block in the text file using text block nodes as input; or it can be sent by other devices. The embodiment does not limit the acquisition method of the key information block.

In this embodiment, the key information block is composed of a group of text blocks of existing types; the text block contains two kinds of information, one is the text content of the text block, and the other is the position information of the text block.

Optionally, the position information is a text box identifier composed of the upper left corner and the lower right corner of the text block. Alternatively, the position information may also be each pixel position corresponding to the text block. This embodiment does not limit the implementation method of the position information.

Optionally, the electronic device can also perform data format conversion on the key information blocks so that the converted data format is suitable for subsequent steps. Data format conversion includes but is not limited to: association indexes between key information blocks and text block nodes, formatting of text block node position coordinates, etc. This embodiment does not limit the specific content of data format conversion.

Step 102: Input the string string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block.

In this embodiment, the node characteristics of the key information block are extracted using its type information and a set of corresponding text blocks.

In one example, the string of each text block in each key information block is input into the node feature extraction model to obtain the node features of the key information block, including: for each text block, input the string in the text block in advance The trained RNN obtains the feature vector of each string; the feature vector of each string is processed into the second fixed dimension to obtain the node features.

Specifically, for the string corresponding to a set of text blocks of the input node feature extraction model, the string is first divided into characters to obtain a vectorized representation of the character, and then the character vector is encoded by RNN, and then the Summary model is used to weight and superpose the character vector into a fixed A dimensional feature vector that represents the text features of text block nodes.

The above example takes the node feature extraction model including RNN and Summary model as an example. In actual implementation, the node feature extraction model can also be other types of neural networks, such as: calculating the feature vector of a string through a linear regression model, or Use word2vector to calculate feature vectors. This embodiment does not limit the implementation of the node feature extraction model.

Optionally, the electronic device randomly initializes a vector table of node types in advance, searches for a vector in the vector table through the index subscript corresponding to the node type of the key information block, and uses this vector to characterize the type characteristics of the key information block. After that, the type features are expanded into the same dimensions as the text features and spliced into a new set of feature vectors. The feature vectors are extracted using RNN and Summary model coding to obtain a fixed-dimensional feature vector. The extracted feature vectors represent key information blocks. node characteristics.

Step 103: For each key information block among the multiple key information blocks, construct a connectivity relationship between each text block in the key information block and each text block in other key information blocks.

The key information block consists of a group of text blocks, and there is multiple text position information. One strategy is to construct a location area large enough to cover all text block nodes, but the description of this location area is not precise enough and may overlap with the location areas of other key information nodes. Based on this, in this embodiment, the connectivity relationship between two key information blocks is defined as the sum of the pairwise connectivity relationships between the two groups of text block nodes. That is, the first key information block contains M text blocks, and the second key information block contains N text blocks, then there are M*N text block connectivity relationships between the two key information blocks. The sum of these connectivity relationships represents the two The connectivity relationship between key information nodes, where M and N are positive integers greater than or equal to 1.

Step 104: Determine the edge characteristics of the key information block based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship.

Optionally, steps 103 and 104 may be executed after step 102, or may be executed before step 102, or may be executed simultaneously with step 102. This embodiment does not change the execution order between steps 103 and 104 and step 102. limited.

In one example, based on each connected relationship corresponding to each key information block and the location information corresponding to each connected relationship, the edge characteristics of the key information block are determined, including: for each connected relationship, two connected according to the connected relationship The relative positions between text blocks determine the sub-edge characteristics corresponding to the connectivity relationship; for each key information block, determine the sub-edge characteristics corresponding to the connectivity of each text block in the key information block; based on the sub-edge characteristics corresponding to each key information block Features generate edge features.

Among them, for each connected relationship, determine the sub-edge characteristics corresponding to the connected relationship according to the relative position between the two text blocks connected by the connected relationship, including: discretizing the relative position according to the direction and distance to obtain the direction code and distance Coding; input the direction code and distance code into the embedding model to obtain the direction embedding code, horizontal distance embedding code and vertical distance embedding code; after splicing the direction embedding code, horizontal distance embedding code and vertical distance embedding code, project to obtain a fixed length vector , get the sub-edge features.

The embedding model can be a pre-trained embedding layer.

For example: for the relative position (center connection vector) of two connected text blocks, discretize according to direction and distance. Discretization refers to: the direction is divided into multiple directions according to angle (such as 360 directions, The difference between two adjacent directions is 1°). Divide the vertical and horizontal distance by the length and width of the target image to obtain the normalized vertical distance and horizontal distance, then multiply by 1000 and round. In this way, an integer encoding of direction and an integer encoding of distance are obtained. The integer codes of direction, horizontal and vertical are calculated through the embedding model to calculate the corresponding embedding codes, and three embedding codes of direction, horizontal and vertical are obtained. The embedding codes are concatenated and projected into fixed-length vectors as edge features of the directed graph.

Optionally, there are two steps in the edge feature extraction process: first, extract sub-edge features between text blocks, and then search and encode the edge features between key information blocks from the vector table of sub-edge features.

Among them, the edge features are obtained by searching and encoding the vector table of the sub-edge features, including: for each key information block, obtaining a connectivity relationship matching table composed of each connectivity relationship of the key information block; according to the connectivity relationship matching table, from Search the corresponding set of sub-edge feature vectors in the vector table composed of sub-edge features to obtain the edge features of the key information block.

Optionally, after obtaining the set of sub-edge feature vectors, the electronic device processes each sub-edge feature into the same dimension; processes each processed sub-edge feature into a first fixed dimension to obtain the edge feature.

The following is an example of determining the edge characteristics of key information blocks:

1) Calculate the sub-edge feature vectors between text blocks, and construct a vector table composed of sub-edge feature vectors from the sub-edge connections between text blocks.

2) Search the sub-edge connectivity matching table between key information blocks in the sub-edge feature vector table to obtain vectorized representation;

3) Expand the vectorized sub-edge feature set between key information blocks into the same dimension, and then process it into a first fixed-dimensional feature vector through the Summary model, and use this vector to represent the edge features of the key information block.

Step 105: Input node features and edge features into the pre-trained graph convolutional neural network to obtain edge types between each key information block.

The input of the graph convolutional neural network is node features and edge features. The process of calculating edge types includes: for each key information block, the node features of the key information block and the node features and edge features that have a connectivity relationship with the key information block are passed through The graph convolutional neural network calculates the target node information; the information of each node associated with the edge is spliced, and the edge attributes are calculated through a multi-layer forward network to obtain the edge type.

Among them, the process of calculating target node information and calculating edge attributes is repeated multiple times to expand the field of view and obtain high-level semantic features.

Step 106: Determine that key information blocks with the same edge type have an associated relationship.

Key information blocks are aggregated into a group by connections of the same edge type, and the group indicates that there is a certain correlation between the key information blocks. The correlation here is the detected edge type. The key information blocks and edge type prediction results are aggregated into a set of typed node relationships. This relationship set is a structured information extraction result, that is, the inter-text relationship that needs to be detected.

In order to more clearly understand the text relationship detection method based on the graph convolutional neural network provided by this application, refer to Figure 2. The text relationship detection method is illustrated below with an example. Figure 2 illustrates this by taking key information blocks in the document file as input as an example. Extract text features from the text content in the key information blocks, and encode the type features of the key information blocks; combine and encode the text features and type features of the key information blocks to obtain node features; dismantle all key information blocks to obtain candidate texts Block set, extract the sub-edge features between pairs of text blocks based on the position information of the text blocks and use them to construct a sub-edge feature vector table; regard the edge features between pairs of key information blocks as the two sets of corresponding text blocks. The aggregation of sub-edge features, that is, each edge feature is composed of a set of sub-edge features. The subscript index corresponding to this set of sub-edges can be searched and obtained in the sub-edge feature vector table. After obtaining a set of sub-edge features corresponding to After edge feature vectors are extracted, the edge features between key information blocks are obtained through coding and extraction; the key information block features and the edge features between key information blocks are encoded through the constructed graph neural network layer to learn the feature vectors of edge relationships; the feature vector detection of edge relationships is used Types of edge connections between key information blocks, and aggregation based on edge types to obtain a set of typed key information block sets.

Optionally, steps 101-105 can be implemented in the same network model. At this time, the input of the network model is the key information block and the output is the edge type.

To sum up, the text relationship detection method based on graph convolutional neural network provided by this embodiment obtains multiple key information blocks of text information in the target image. The key information blocks include multiple text blocks, and each text block includes At least one string; input the string of each text block in each key information block into the node feature extraction model to obtain the node characteristics of the key information block; for each key information block in multiple key information blocks, construct the key information The connectivity relationship between each text block in the block and each text block in other key information blocks; based on each connectivity relationship corresponding to each key information block and the position information corresponding to each connectivity relationship, determine the edges of the key information block Features; input node features and edge features into the pre-trained graph convolutional neural network to obtain the edge types between each key information block; determine that key information blocks with the same edge type are associated; can solve the problem of setting logical rules based on location information To determine the association relationship, it may lead to unreasonable detection results and improve the accuracy of association relationship identification. At the same time, the relationship detection method provided by this application does not need to distinguish the key and value of the key information block, and directly detects the association relationship, which can improve the efficiency of association relationship detection.

Figure 3 is a block diagram of a text relationship detection device based on a graph convolutional neural network provided by an embodiment of the present application. The device at least includes the following modules: key information acquisition module 310, node feature extraction module 320, connectivity relationship building module 330, edge feature extraction module 340, edge type calculation module 350 and association relationship determination module 360.

The key information acquisition module 310 is used to obtain multiple key information blocks of text information in the target image, where the key information blocks include multiple text blocks, and each text block includes at least one character string;

The node feature extraction module 320 is used to input the string string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block;

The connectivity relationship building module 330 is configured to, for each key information block in the plurality of key information blocks, build a connectivity relationship between each text block in the key information block and each text block in other key information blocks. ;

The edge feature extraction module 340 is used to determine the edge features of the key information block based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship;

The edge type calculation module 350 is used to input the node features and the edge features into a pre-trained graph convolutional neural network to obtain the edge types between each key information block;

The association relationship determination module 360 is used to determine that key information blocks with the same edge type have an association relationship.

For relevant details, refer to the above method embodiments.

It should be noted that when the text relationship detection device based on the graph convolutional neural network provided in the above embodiment performs the text relationship detection based on the graph convolutional neural network, only the division of the above functional modules is used as an example, and the actual application is , the above function allocation can be completed by different functional modules as needed, that is, the internal structure of the text relationship detection device based on the graph convolutional neural network is divided into different functional modules to complete all or part of the functions described above. In addition, the text relationship detection device based on graph convolutional neural network provided by the above embodiments and the text relationship detection method embodiment based on graph convolutional neural network belong to the same concept. The specific implementation process can be found in the method embodiments, and will not be described again here. .

Figure 4 is a block diagram of a text relationship detection device based on a graph convolutional neural network provided by an embodiment of the present application. The device includes at least a processor 401 and a memory 402.

The processor 401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 can adopt at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array). accomplish. The processor 401 may also include a main processor and a co-processor. The main processor is a processor used to process data in the wake-up state, also called CPU (Central Processing Unit, central processing unit); the co-processor is A low-power processor used to process data in standby mode. In some embodiments, the processor 401 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is responsible for rendering and drawing content that needs to be displayed on the display screen. In some embodiments, the processor 401 may also include an AI (Artificial Intelligence, artificial intelligence) processor, which is used to process computing operations related to machine learning.

Memory 402 may include one or more computer-readable storage media, which may be non-transitory. Memory 402 may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 401 to implement the graph-based method provided by the method embodiments in this application. Text relationship detection method using convolutional neural networks.

In some embodiments, the text relationship detection apparatus based on graph convolutional neural network optionally further includes: a peripheral device interface and at least one peripheral device. The processor 401, the memory 402 and the peripheral device interface may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface through a bus, a signal line or a circuit board. Illustratively, peripheral devices include but are not limited to: radio frequency circuits, touch display screens, audio circuits, power supplies, etc.

Of course, the text relationship detection device based on graph convolutional neural network may also include fewer or more components, which is not limited in this embodiment.

Optionally, this application also provides a computer-readable storage medium in which a program is stored, and the program is loaded and executed by a processor to implement the graph convolutional neural network based on the above method embodiment. Text relationship detection method.

Optionally, this application also provides a computer product. The computer product includes a computer-readable storage medium. A program is stored in the computer-readable storage medium. The program is loaded and executed by a processor to implement the above method embodiments. Text relationship detection method based on graph convolutional neural network.

The technical features of the above-described embodiments can be combined in any way. To simplify the description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, All should be considered to be within the scope of this manual.

The above-described embodiments only express several implementation modes of the present application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all fall within the protection scope of the present application. Therefore, the protection scope of this patent application should be determined by the appended claims.

The above is only a specific embodiment of the present application, and any other improvements made based on the concept of the present application shall be deemed to be within the protection scope of the present application.

Claims (8)

1. A text relationship detection method based on graph convolutional neural network, characterized in that the method includes: Obtain multiple key information blocks of text information in the target image, where the key information blocks include multiple text blocks, and each text block includes at least one character string; Enter the string string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block; For each key information block in the plurality of key information blocks, construct a connectivity relationship between each text block in the key information block and each text block in other key information blocks; Based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship, determine the edge characteristics of the key information block; specifically, it includes: for each connectivity relationship, determine the number of connections connected according to the connectivity relationship. The relative position between the two text blocks determines the sub-edge characteristics corresponding to the connectivity relationship; for each key information block, determines the sub-edge characteristics corresponding to the connectivity of each text block in the key information block; based on each key information block The sub-edge features corresponding to the information block generate the edge features; Wherein, for each connected relationship, determining the sub-edge characteristics corresponding to the connected relationship according to the relative position between the two text blocks connected by the connected relationship includes: determining the relative position according to the direction and distance. Discretize to obtain direction code and distance code; input the direction code and distance code into the embedding model to obtain direction embedding code, horizontal distance embedding code and vertical distance embedding code; embedding the direction code, the horizontal distance After the embedding code and the vertical distance embedding code are spliced, a fixed-length vector is obtained by projection, and the sub-edge feature is obtained; Input the node features and the edge features into a pre-trained graph convolutional neural network to obtain the edge types between each key information block; It is determined that key information blocks with the same edge type are associated. 2. The method according to claim 1, characterized in that generating the edge features based on sub-edge features corresponding to each key information block includes: Process each sub-edge feature into the same dimension; Each processed sub-edge feature is processed into a first fixed dimension to obtain the edge feature. 3. The method of claim 1, wherein generating the edge features based on sub-edge features corresponding to each key information block includes: For each key information block, obtain a connectivity relationship matching table composed of each connectivity relationship of the key information block; According to the connectivity relationship matching table, the corresponding sub-edge feature vector set is searched from the vector table composed of the sub-edge features to obtain the edge features of the key information block. 4. The method according to claim 1, characterized in that the string of each text block in each key information block is input into the node feature extraction model to obtain the node features of the key information block, including: For each text block, input the string in the text block into the pre-trained recurrent neural network RNN to obtain the feature vector of each string; The feature vector of each string is processed into a second fixed dimension to obtain the node features. 5. The method according to claim 1, characterized in that, inputting the node features and the edge features into a pre-trained graph convolutional neural network to obtain edge types between each key information block includes: For each key information block, calculate the target node information through the graph convolutional neural network using the node characteristics of the key information block and the node characteristics and edge characteristics that have a connectivity relationship with the key information block; The information of each node associated with the edge is spliced, and the edge attributes are calculated through a multi-layer forward network to obtain the edge type. 6. A text relationship detection device based on graph convolutional neural network, characterized in that the device includes: A key information acquisition module, used to obtain multiple key information blocks of text information in the target image, where the key information blocks include multiple text blocks, and each text block includes at least one character string; The node feature extraction module is used to input the string of each text block in each key information block into the node feature extraction model to obtain the node features of the key information block; A connectivity relationship building module configured to, for each key information block among the plurality of key information blocks, construct a connectivity relationship between each text block in the key information block and each text block in other key information blocks; The edge feature extraction module is used to determine the edge features of the key information block based on each connectivity relationship corresponding to each key information block and the location information corresponding to each connectivity relationship; specifically, it includes: for each connectivity relationship, According to the relative position between the two text blocks connected by the connectivity relationship, determine the sub-edge characteristics corresponding to the connectivity relationship; for each key information block, determine the connected sub-edge corresponding to each text block in the key information block. Edge features; generate the edge features based on the sub-edge features corresponding to each key information block; Wherein, for each connected relationship, determining the sub-edge characteristics corresponding to the connected relationship according to the relative position between the two text blocks connected by the connected relationship includes: determining the relative position according to the direction and distance. Discretize to obtain direction code and distance code; input the direction code and distance code into the embedding model to obtain direction embedding code, horizontal distance embedding code and vertical distance embedding code; embedding the direction code, the horizontal distance After the embedding code and the vertical distance embedding code are spliced, a fixed-length vector is obtained by projection, and the sub-edge feature is obtained; An edge type calculation module, used to input the node characteristics and the edge characteristics into a pre-trained graph convolutional neural network to obtain the edge type between each key information block; The association relationship determination module is used to determine that key information blocks with the same edge type have an association relationship. 7. A text relationship detection device based on graph convolutional neural network, characterized in that the device includes a processor and a memory; a program is stored in the memory, and the program is loaded and executed by the processor to implement The text relationship detection method based on graph convolutional neural network according to any one of claims 1 to 5. 8. A computer-readable storage medium, characterized in that a program is stored in the storage medium, and when the program is executed by a processor, the program is used to implement the graph-based convolution as claimed in any one of claims 1 to 5. Textual relationship detection method using neural networks.