CN114359086B - Molecular formula recognition method and related device, equipment and storage medium - Google Patents

Abstract

The present application discloses a molecular formula recognition method and related devices, equipment and storage medium, the method comprising: using a molecular formula recognition model to recognize an image to be recognized to obtain a symbol sequence; based on the symbol sequence, recovering the target molecular formula in the image to be recognized; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. The above scheme can improve the accuracy of molecular formula recognition and the generalization ability of molecular formula recognition.

Description

Molecular formula recognition method and related device, equipment and storage medium

Technical Field

The present application relates to the field of image recognition technology, and in particular to a molecular formula recognition method and related devices, equipment and storage media.

Background Art

With the development of deep learning, image and text recognition technology has become more mature and has begun to empower more and more industries. Especially in the education industry, the ability to recognize images of homework, answers, and test papers has become a very important part. On the one hand, it can be used for efficient electronicization of teaching knowledge, and on the other hand, it can serve automatic correction, realize learning situation analysis, and teach students in accordance with their aptitude. It has undoubtedly become an absolute necessity.

At present, the image and text recognition technology has basically matured in the recognition of Chinese and English data, and the recognition ability of structured data such as formulas has basically reached a usable state, which can meet most scenarios in Chinese and English mathematics, physics and chemistry. However, for some special scenarios, the image and text recognition technology has not yet reached a usable state, and organic chemistry is one of them. Because the appearance of organic chemistry questions and answer image data is very different from Chinese and English data and mathematical formulas, it does not yet have the ability to recognize organic chemistry data.

Summary of the invention

The main technical problem solved by the present application is to provide a molecular formula recognition method and related devices, equipment and storage media, which can improve the accuracy of molecular formula recognition and the generalization ability of molecular formula recognition.

In order to solve the above technical problems, the first aspect of the present application provides a molecular formula recognition method, including: using a molecular formula recognition model to recognize an image to be recognized to obtain a symbol sequence; based on the symbol sequence, recovering the target molecular formula in the image to be recognized; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.

In order to solve the above-mentioned technical problems, the second aspect of the present application provides a molecular formula recognition device, including: a sequence recognition module and a formula recovery module; the sequence recognition module is used to use a molecular formula recognition model to recognize an image to be recognized and obtain a symbol sequence; the formula recovery module is used to recover the target molecular formula in the image to be recognized based on the symbol sequence; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.

In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the molecular formula recognition method in the above first aspect.

In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the molecular formula recognition method in the above first aspect.

The above scheme uses a molecular formula recognition model to recognize the image to be recognized, obtains a symbol sequence, and based on the symbol sequence, recovers the target molecular formula in the image to be recognized; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. Therefore, the symbol sequence obtained by the formula recognition model can accurately reflect the graphic visual state of the target formula, that is, it can truthfully reflect the content of the molecular formula in the image to be recognized, and fully retain the original information of the molecular formula in the image to be recognized, thereby improving the accuracy and generalization ability of recognizing the target molecular formula.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of an embodiment of a molecular formula recognition method provided by the present application;

FIG2 is a schematic diagram of an embodiment of a SMILES recognition result provided by the present application;

FIG3 is a schematic diagram of an embodiment of a Chemfig annotation result provided by the present application;

FIG4 is a schematic structural diagram of an embodiment of a molecular formula recognition model provided by the present application;

FIG5 is a schematic diagram of an embodiment of sample data provided by the present application;

FIG6 is a schematic diagram of a flow chart of an embodiment of step S11 shown in FIG1 ;

FIG7 is a flow chart of an embodiment of processing an original tag sequence to obtain a sample symbol sequence provided by the present application;

FIG8 is a schematic diagram of a flow chart of an embodiment of determining whether an original tag sequence is correct provided by the present application;

FIG9 is a schematic diagram of an embodiment of a sample symbol sequence provided by the present application;

FIG10 is a schematic diagram of a flow chart of an embodiment of step S112 shown in FIG6 ;

FIG11 is a schematic diagram of another embodiment of a sample symbol sequence provided by the present application;

FIG12 is a schematic diagram of a flow chart of an embodiment of a training molecular formula recognition model provided by the present application;

FIG13 is a schematic diagram of an embodiment of a molecular formula recognition model decoding process provided by the present application;

FIG. 14 is a flow chart of an embodiment of determining the end of decoding provided by the present application.

FIG15 is a schematic diagram of a framework of an embodiment of a molecular formula recognition device provided by the present application;

FIG16 is a schematic diagram of a framework of an electronic device according to an embodiment of the present application;

FIG. 17 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided in the present application.

DETAILED DESCRIPTION

The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably in this article. The term "and/or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character "/" in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

Please refer to FIG. 1 , which is a schematic flow chart of an embodiment of a molecular formula recognition method provided in the present application. It should be noted that if there are substantially the same results, this embodiment is not limited to the process sequence shown in FIG. 1 . As shown in FIG. 1 , this embodiment includes:

Step S11: using the molecular formula recognition model to recognize the image to be recognized, and obtaining a symbol sequence.

The method of this embodiment is used to identify the image to be identified so as to recover the target molecular formula in the image to be identified. The image to be identified described herein may be an image including any molecular formula, which may be obtained from local storage or cloud storage. It is understandable that in other embodiments, the image to be identified may also be obtained by real-time acquisition by an image acquisition device, which is not specifically limited here.

In this embodiment, the molecular formula recognition model is used to recognize the image to be recognized, and a symbol sequence can be obtained. The molecular formula recognition model is trained using a sample image containing a sample molecular formula, and the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. In other words, the molecular formula recognition model is trained based on a sample image annotated with a sample symbol sequence of the sample molecular formula constructed from the graphic visual form of the sample molecular formula; in addition, the molecular formula recognition model completed by training can process the image to be recognized to obtain the target molecular formula.

For example, as shown in Figure 2, Figure 2 is a schematic diagram of an embodiment of the SMILES recognition result provided by the present application. Deep learning molecular formula recognition methods generally directly apply the existing Encoder-Decoder model for modeling. The model can directly learn the molecular formula image to the recognition result, and the recognition result is generally stored in the SMILES string format; however, as shown in Figure 2, the left input image and the right input image are handwritten molecular formulas of two students. The molecular formula in the left input image is written correctly, and the molecular formula in the right input image is written incorrectly (the dotted box in the right input image can only be 3, otherwise it violates the valence rule). At this time, using the SMILES specification, only the molecular formula on the left can be correctly expressed, and the molecular formula on the right cannot be correctly expressed, that is, using SMILES cannot truthfully express some handwritten answers that do not conform to chemical rules. In addition, the input image on the left side of Figure 2 has the atomic group "H3C" written on it, but in the SMILES recognition result, "H3C" is not reflected, but is simply omitted according to the rule of "carbon and hydrogen are omitted"; and the "N" atom is adjusted to "N+", and the "O" atom is adjusted to "O-", all of which are also based on chemical principles. Therefore, the SMILES recognition results are standardized, but they are all based on a lot of professional chemical knowledge, without considering the form of writing, and cannot truly reflect the content in the image, so they are not based on the visual form of the graphic, that is, the picture and text are not related.

As shown in FIG3 , FIG3 is a schematic diagram of an embodiment of the Chemfig annotation result provided by the present application. For the left input image and the right input image in FIG3 , the "H3C" atom group symbol in the left input image complies with the chemical valence rule, and the "H4C" atom group in the right input image does not comply with the chemical valence rule, which is an incorrect writing. The writing error can be reflected in the Chemfig annotation result, that is, the Chemfig annotation result corresponding to the right input image contains the words "H_{4}C". Since the Chemfig syntax relies less on abstract chemical knowledge, but is completely customized based on the molecular formula image itself, the sample symbol sequence obtained by using the Chemfig syntax is constructed from the graphic visual form of the sample molecular formula, that is, the Chemfig syntax is a "what you see is what you get" molecular formula image annotation method, which can fully retain the original information of the sample molecular formula, so that the trained molecular formula recognition model has the possibility of being corrected. It can be understood that in other embodiments, other regular grammars can also be used to construct a sample symbol sequence that can reflect the graphic visual form of the sample molecular formula, which is not specifically limited here. It should be noted that, for ease of description, the following will take the example of constructing a sample symbol sequence using Chemfig syntax as an example, but it can be understood that this does not limit the syntax rules for constructing the sample symbol sequence.

Among them, the network architecture of the molecular formula recognition model is not limited, and can be specifically set according to actual use needs. In one embodiment, as shown in Figure 4, Figure 4 is a structural schematic diagram of an embodiment of the molecular formula recognition model provided by the present application, and the molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model, and the image-text encoding model is trained using a sample image containing at least one of a sample text and a sample formula. For example, as shown in Figure 5, Figure 5 is a schematic diagram of an embodiment of sample data provided by the present application, and the sample text is the Chinese and English text "and passes through point A", and the sample formula is a mathematical formula Therefore, the image-text coding model is trained using sample images containing at least one of Chinese and English texts and mathematical formula images and texts. In other words, some parameters in the image-text coding model can directly reuse the trained parameters of at least one of the Chinese and English text recognition model and the mathematical formula image-text recognition model, thereby accelerating the convergence speed of the molecular recognition model and, to a certain extent, compensating for the problem of insufficient molecular formula training data caused by the high difficulty of labeling, thereby improving the recognition accuracy of the molecular formula recognition model.

In one embodiment, the sample symbol sequence includes a character string representing an atomic group in the sample molecular formula, and a character string representing a chemical bond in the sample molecular formula, and the character string representing the chemical bond at least includes the angle of the chemical bond. Specifically, as shown in FIG3 , the sample image input on the left is annotated using the Chemfig syntax to obtain a sample symbol sequence of "H-{3}C-N(＝[1]O)＝[-1]O", which includes the character string "H-{3}C" representing the atomic group in the sample molecular formula, and the character strings "-" and "＝" representing the chemical bonds in the sample molecular formula; the character string representing the chemical bond includes the angle of the chemical bond, for example, the "N" atom is connected to two double bonds, facing the upper right and lower right respectively, and the "angle codes" such as "[1]" and "[-1]" are used in the sample symbol sequence to represent the approximate direction of the chemical bond "＝". The angle information of the chemical bond can more accurately portray the graphic visual form of the sample molecular formula, providing richer visual annotation information.

In one embodiment, the sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents the direction of the branch. That is, the sample symbol sequence includes a string representing an atomic group in the sample molecular formula, a string representing a chemical bond in the sample molecular formula, a string representing a chemical bond including at least the angle of the chemical bond, and a branch symbol representing a branch in the sample molecular formula.

In one embodiment, when the sample symbol sequence includes branch symbols representing branches in the sample molecular formula, the sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each branch, the sample first subsequence includes branch symbols representing each branch respectively, and the branch symbols also represent the identification of the branch, and the sample second subsequence includes an order symbol, and the order symbol represents the identification of the branch.

Step S12: based on the symbol sequence, recover the target molecular formula in the image to be identified.

In this embodiment, the target molecular formula in the image to be recognized is restored based on the symbol sequence obtained in the above manner. Specifically, the characters in the second subsequence of the sample representing each branch are connected to the position of the branch symbol in the first subsequence through the chemical bond in the second subsequence, and the connection direction of the chemical bond and the first subsequence is adjusted according to the angle of the chemical bond to restore the target molecular formula.

In one embodiment, the symbol sequence can be restored to the DFS (Depth First Search) form of the target molecular formula. The specific meaning of the DFS form can be found in the following related descriptions and will not be repeated here. It can be understood that in other embodiments, the symbol sequence can also be restored to the graphical data structure form of the target molecular formula.

In the above implementation, the molecular formula recognition model is used to recognize the image to be recognized, and a symbol sequence is obtained, and based on the symbol sequence, the target molecular formula in the image to be recognized is restored; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula. Therefore, the symbol sequence obtained by the formula recognition model can accurately reflect the graphic visual state of the target formula, that is, it can truthfully reflect the content of the molecular formula in the image to be recognized, and fully retain the original information of the molecular formula in the image to be recognized, thereby improving the accuracy and generalization ability of recognizing the target molecular formula.

Please refer to FIG. 6 , which is a schematic diagram of a flow chart of an embodiment of step S11 shown in FIG. 1 . It should be noted that if there is substantially the same result, this embodiment is not limited to the flow chart sequence shown in FIG. 6 . As shown in FIG. 6 , this embodiment includes:

Step S111: Perform structural analysis based on the original tag sequence to obtain graphic data of the sample molecular formula.

The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language. The preset molecular formula markup language is not limited and can be specifically set according to actual use needs. For example, the preset molecular formula markup language is Chemfig language.

For the image of the same sample molecular formula, there will be multiple different but correct annotation results depending on the selected starting point and traversal order, that is, for the image of the same sample molecular formula, there will be multiple different original label sequences, and the annotator will only randomly select one of them during annotation, and it is impossible to make clear rules to restrict the annotation process. Therefore, the original label sequence obtained by Chemfig language annotation will be ambiguous, which will increase the learning difficulty of the molecular formula recognition model. In addition, in order to maintain the simplicity of the original label sequence obtained by Chemfig language annotation, a large number of prior rules will be set up in the grammar to help the annotator save the number of characters entered during annotation. For example, for the benzene ring, it can be simply represented by "*6(-----)", in which the 6 single bonds do not need to specify the direction, but the default direction is automatically inferred according to the rules of the benzene ring. It can be seen that these set prior rules are relatively complex, and the molecular formula recognition model is difficult to learn, which will cause the generalization performance to decline.

Therefore, in this embodiment, structural analysis is performed based on the original label sequence to obtain graphic data of the sample molecular formula, so as to eliminate the ambiguity and complexity of the original label sequence and unify the expression form of the same sample molecular formula as much as possible. Specifically, as shown in Figure 7, Figure 7 is a flow chart of an embodiment of processing the original label sequence to obtain a sample symbol sequence provided by the present application. The "Original Chemfig annotation string" column has three different Chemfig annotation strings, that is, there are three different original label sequences, but they actually express the same sample molecular formula; therefore, in this embodiment, the original label sequence will be structurally analyzed according to the original Chemfig grammatical rules to obtain graphic data of the sample molecular formula.

Among them, the graphic data is composed of several data elements, and the several data elements include nodes and edges connecting nodes. The nodes represent atomic groups, and the edges represent chemical bonds. Specifically, using the Chemfig syntax rules, all the data elements such as atomic groups and chemical bonds behind the original label sequence are parsed and restored by manual programming, and they are connected to obtain graphic data. As shown in Figure 7, the "Annotated Graph Representation" column is the graphic data of the sample molecular formula obtained after structural analysis, wherein the node, i.e., the rectangular box, refers to the atomic group. The atomic group can be a string, such as "HO", "COOH", and the atomic group can also have no content, such as the vertex on the benzene ring. For the benzene ring with a circle, the middle circle is also treated as a special atomic group; the edge other than the node, i.e., the line segment other than the atomic group, is a chemical bond. The chemical bond can be a single bond, a double bond, or a triple bond, etc. Figure 7 only contains single bonds. The two ends of a chemical bond must be connected to an atomic group, and an atomic group can be connected to multiple chemical bonds. Through this connection relationship, the atomic group is used as a node and the chemical bond is used as an edge, and the graphic data of the sample molecular formula can be obtained. As can be seen from Figure 7, even if the original label sequences are diverse, as long as they are the same atom, the graphic data behind them are the same, which eliminates the problem of ambiguity in the original label sequence obtained by annotating with the Chemfig language.

Among them, each data element in the graphic data is marked with a data attribute, that is, by performing structural analysis on the original label sequence, the attributes of each data element in the graphic data of the sample molecular formula can also be analyzed. The data attributes marked on each data element are not limited and can be specifically set according to actual use needs.

In one embodiment, the data attributes of the node include characters representing the atomic group, i.e., what atomic group the node represents. In one embodiment, the data attributes of the edge include at least the angle of the chemical bond, i.e., what the angle of the chemical bond is. It is understandable that in other embodiments, the data attributes of the edge may also include what chemical bond the edge represents, or may also include whether the chemical bond is a single bond, a double bond, or a triple bond, etc., i.e., the type of covalent bond of the chemical bond.

In order to ensure that the graphic data of the sample molecular formula obtained by performing structural analysis based on the original label sequence is correct, it is necessary to ensure that the original label sequence is correctly labeled. In one embodiment, as shown in FIG. 8 , FIG. 8 is a flow chart of an embodiment of determining whether the original label sequence is correct provided by the present application. Before performing structural analysis based on the original label sequence to obtain the graphic data of the sample molecular formula, it is determined whether the original label sequence is correctly labeled by comparing the rendered molecular formula with the sample molecular formula, which specifically includes the following sub-steps:

Step S81: Rendering the original tag sequence using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula.

In this embodiment, the original tag sequence is rendered using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula. The preset molecular formula markup language is not limited and can be specifically set according to actual use needs. For example, the preset molecular formula markup language is Chemfig language.

In a specific embodiment, as shown in FIG3 , the preset molecular formula markup language is the Chemfig language, and the Chemfig language has an existing rendering engine. The original label sequence is rendered by the rendering engine of the Chemfig language to obtain a rendered molecular formula, i.e., the rendering result marked in FIG3 .

Step S82: Based on the difference check result between the rendered molecular formula and the sample molecular formula, determine whether the original label sequence is correctly labeled.

In this embodiment, whether the original label sequence is correctly labeled is determined based on the difference detection result between the rendered molecular formula and the sample molecular formula. Specifically, as shown in FIG3 , the labeled rendering result in FIG3 is consistent with the corresponding input image, that is, there is no difference between the rendered molecular formula and the corresponding sample molecular formula, so it is determined that the labeling of the original label sequence is correct.

Step S112: Traverse based on the graphic data to obtain a sample symbol sequence.

Since the data structure in the graphic data of the sample molecular formula is relatively complex, it is also necessary to convert it into a form that can be trained by the molecular formula recognition model. Therefore, in this embodiment, based on traversing the graphic data, a sample symbol sequence is obtained. Among them, the method of traversing the graphic data is not limited and can be specifically set according to actual use needs. For example, the graphic data is traversed using depth-first traversal DFS to obtain a sample symbol sequence; or the graphic data is traversed using breadth-first traversal BFS (Breadth First Search) to obtain a sample symbol sequence.

Specifically, a point is selected according to a fixed rule, and the graph data is traversed according to the fixed rule, and the visited nodes and edges are appended to the end of the string, thereby completing the traversal of the graph data and obtaining a sample symbol sequence.

In a specific embodiment, as shown in FIG. 7 and FIG. 9, FIG. 9 is a schematic diagram of an embodiment of a sample symbol sequence provided by the present application, and the graphic data shown in FIG. 7 is traversed by depth-first traversal to obtain the sample symbol sequence shown in the upper right corner of FIG. 7 and FIG. 9, i.e., the regular label. From this sample symbol sequence, it can be seen that a pair of parentheses are used to indicate the occurrence of branches, and the sample symbol sequence contains four elements: "atomic group", "chemical bond", "angle of chemical bond" and "virtual body"; all chemical bonds are followed by the angle descriptor "[:<angle value>]", and the angle value has been calculated when constructing the graphic data and can be directly output in the format; "virtual body" is used to indicate the nesting of branches. Therefore, when the sample molecular formula has many branches and is deep, the sample symbol sequence obtained by traversing the graphic data by depth-first traversal will have more virtual body nesting, and the molecular formula recognition model will pay too much attention to the depth when learning, and ignore the global structure.

In other specific implementations, as shown in FIG7 , the graphic data shown in FIG7 is traversed by breadth-first traversal to obtain the sample symbol sequence shown in the lower right corner of FIG7 , that is, the regularized label. As shown in FIG10 , FIG10 is a flow chart of an embodiment of step S112 shown in FIG6 , and traversing the graphic data by breadth-first traversal to obtain the sample symbol sequence specifically includes the following sub-steps:

Step S1001: traverse the data elements on the sample molecular formula trunk in the graphic data to obtain the sample first subsequence, and traverse the data elements on the sample molecular formula branches in the graphic data to obtain the sample second subsequence.

In this embodiment, first, the data elements on the sample molecular formula trunk are traversed in the graphic data to obtain the first subsequence of the sample, and the data elements on the sample molecular formula branches are traversed in the graphic data to obtain the second subsequence of the sample. Among them, the character string representing the data element in the sample symbol sequence includes the data attribute of the data element, the branch is represented by the branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains the sequence symbol, and the sequence symbol represents the identification of the branch. In other words, in this embodiment, the strategy of traversal first as a whole and then as a part is followed. The data elements on the sample molecular formula trunk will be traversed first on the graphic data, that is, the whole traversal will be performed first, and then the data elements on the sample molecular formula branches will be traversed on the graphic data, that is, the local traversal will be performed. Through the principle of traversal first as a whole and then as a part, when the sample symbol sequence obtained later is used for the training of the molecular formula recognition model, the molecular formula recognition model can better grasp the overall structure of the sample molecular formula and not ignore the branch structure, that is, the detailed structure of the sample molecular formula.

There is no limitation on the manifestation of branching symbols and sequence symbols, which can be set according to actual use needs. For example, "\place{x}[：{θ}]" is represented as a branching symbol, that is, when traversing the data elements on the sample molecular formula trunk, when encountering a branch, the content of the branch is temporarily not output, and the branch is replaced by the branch symbol first; wherein {x} represents an integer serial number, and {θ} represents an angle and represents the direction of the branch. "\solveplace{x}" is represented as a sequence symbol, that is, it means starting to output the content of the corresponding branch symbol "\place{x}[：{θ}]", wherein {x} represents an integer serial number.

In one embodiment, after the content corresponding to a branch symbol is output, that is, after the graphic data traverses the data elements on any branch of the sample molecular formula, it ends with a preset end symbol, that is, the last symbol of the branch symbol is the preset end symbol. Among them, the embodiment of the preset end symbol is not limited, and it can be specifically set according to actual use needs. For example, the preset end symbol is "\eol" or "\eos".

For example, as shown in Figure 11, Figure 11 is a schematic diagram of another embodiment of the sample symbol sequence provided by the present application. The graphic data of the sample molecular formula is shown in Figure 11(a). Following the principle of traversal of the whole first and then the part, the data elements on the trunk of the sample molecular formula are first traversed in the graphic data to obtain the first subsequence of the sample, and the first subsequence of the sample is specifically shown in Figure 11(b); then the data elements on the branches of the sample molecular formula are traversed in the graphic data. Since there are three branch structures, three second subsequences are obtained; among which, the second subsequence corresponding to the branch symbol "\place1[：300]" is "\solveplace1\eol", the second subsequence corresponding to the branch symbol "\place2[：0]" is "\solveplace2-[：0]C O O H\eol", and the second subsequence corresponding to the branch symbol "\place1[：60]" is "\solveplace3--[：60]\circle\eol".

Specifically, in the breadth-first traversal process, the sample symbol sequence text can be first initialized to "\place[:0]\eol", and another queue Q = [(x,0)], where the element x is any element in the above-mentioned graphic data G, and the branch indicator ind is initialized to 1. On this basis, when the queue Q is not empty, the following steps can be executed cyclically: (1) Take out the element x and the current placeholder indicator ind' (initial 0, and then accumulate 1 each time this step is executed) from the head of the queue Q; (2) Extract the maximum connected subgraph G' containing the element x from the graphic data G; (3) Find the longest branch on the maximum connected subgraph G' as the main branch Bmain; (4) Delete all elements on Bmain from the graphic data G; (5) ) outputs "\solveplace{ind'}" to the sample symbol sequence text; (6) traverses each element y on Bmain in the connection order, and outputs y to the end of the sample symbol sequence text, and if y is an atomic group and y is connected to a chemical bond e that is not on Bmain, it can be determined that y is a branch, and the branch direction angle can be set to the angle of the chemical bond e, and outputs "\place{ind}[:{angle}]" to the sample symbol sequence text, and adds [(e,ind)] to the queue Q, and adds 1 to ind to update ind. After traversing Bmain, it can output "\eol" to the end of the sample symbol sequence text, thereby completing a loop execution operation. It should be noted that when the loop operation is executed for the first time, the first subsequence of the trunk can be obtained, and when the loop operation is continued to be executed subsequently, the second subsequence of the branch can be obtained.

Step S1002: Combine the first subsequence of samples and the second subsequence of samples to obtain a sample symbol sequence.

In this embodiment, the sample first subsequence and the sample second subsequence are combined to obtain a sample symbol sequence. That is, the first subsequence obtained by traversing the data elements on the sample molecular formula trunk and the second subsequence obtained by traversing the data elements on the sample molecular formula branches are combined to obtain a sequence corresponding to the data elements of the entire sample molecular formula, that is, to obtain a sample symbol sequence.

Please refer to Figure 12, which is a schematic diagram of a flow chart of an embodiment of training a molecular formula recognition model provided by the present application. It should be noted that if there is substantially the same result, the present embodiment is not limited to the flow chart shown in Figure 12.

As shown in FIG4 , before training the molecular formula recognition model, the molecular formula recognition model is first built. The molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network. Specifically, after the molecular formula encoding network E encodes the input image I, it outputs a sample feature map F; the molecular formula decoding network D adopts an autoregressive iterative modeling method. At each decoding time t, the sample feature map F and the decoding result y _t-1 of the previous decoding time are used as the input of the molecular formula decoding network D, and the decoding result y _t of the current decoding time is output. The decoding is iterated in this way until the end signal is encountered. Assuming that there are a total of T decoding moments, the final output decoding result y = [y ₁ ,y ₂ ,…,y _T ]. Among them, y _t is a basic modeling unit. The sample symbol sequence shown in FIG11 is separated by spaces to obtain a modeling unit sequence.

Among them, the decoding process of the traditional molecular formula decoding network D is as follows:

c _t = Attn(s _t-1 ,y _t-1 ,F)

o _t ,s _t =RNN(c _t ,y _t-1 ,s _t-1 )

p _t =MLP(o _t )

y _t =argmaxx _i p _t (i),i∈{0,1,...V-1}

Among them, c _t is the feature that needs to be paid attention to at the current decoding moment; Attn is the attention mechanism network; s _t-1 is the decoding state at the previous decoding moment; y _t-1 is the decoding result at the previous decoding moment; F is the sample feature map; s _t is the hidden state of the recurrent neural network RNN; o _t is the output feature of the recurrent neural network RNN; p _t is the output probability distribution, p _t ∈ ^RV , V is the number of modeling units; MLP is a neural network composed of a fully connected layer and a softmax layer; y _t is the decoding result corresponding to the current decoding moment. Specifically, the correlation is calculated according to the decoding state s _t-1 at the previous decoding moment, the decoding result y _t-1 at the previous decoding moment, and the features of all positions in the sample feature map F, and the features of all positions in the sample feature map F are weighted and summed through the correlation to obtain the feature c _t that needs to be paid attention to at the current decoding moment; then, the recurrent neural network RNN is used to associate the feature c _t that needs to be paid attention to at the current decoding moment, the decoding state s _t-1 at the previous decoding moment, and the decoding result y _t-1 at the previous decoding moment to obtain the decoding state o _t at the current decoding moment; then, symbol prediction is performed according to the decoding state o _t at the current decoding moment, and the probability distribution p _t of the preset symbol, that is, the probability prediction value of the preset symbol, can be obtained. Specifically, a decoding symbol dictionary will be provided, and the probability distribution includes the probability prediction value of each preset decoding symbol in the corresponding decoding symbol dictionary; the preset symbol corresponding to the maximum probability prediction value in the decoding symbol dictionary is used as the decoding result y _t at the current decoding moment.

Since the sample symbol sequence includes a large number of branch symbols (e.g., \place{x}[：{θ}]) and sequence symbols (e.g., \solveplace{x}), the decoding process from the sequence symbol to the next preset end symbol (e.g., \eol) belongs to the extended content of the branch symbol, so the decoding state when decoding to the corresponding branch symbol is used when decoding the branch to be decoded. However, in the sample symbol sequence, the distance between the sequence symbol and the preset end symbol is far, as shown in Figure 11(b), the distance between the sequence symbol "\solveplace2" and the next preset end symbol "\eol", that is, the distance between the sequence symbol "\solveplace2" and the placeholder "\place2[：0]" is 8 decoding moments, and the memory capacity of the RNN model in the traditional molecular formula decoding network D cannot associate two decoding moments with a long interval. Therefore, in order to make the molecular formula recognition model fully utilize the sample symbol sequence and learn the graphic visual morphological information of the sample molecular formula, in this embodiment, the "angle condition decoding" mechanism is added to the molecular formula decoding network D in the constructed molecular formula recognition model, so that the molecular formula recognition model can effectively associate the features of the placeholders and pseudo symbols that are far apart in time. Specifically, as shown in FIG12, this embodiment includes:

Step S1201: Randomly select a reference state that has not been selected.

In this embodiment, the "angle conditional decoding" mechanism added to the molecular formula decoding network D specifically adds a conditional input z _t ∈[0, t-1] to tell the molecular formula recognition model which branch symbol is currently being expanded.

In one embodiment, the sample molecular formula is composed of a trunk and branches, and the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branches. Before decoding each branch to be decoded, it is necessary to first decode the predicted subsequence of the trunk of the sample molecular formula. Specifically, decoding is performed according to a preset state and a sample feature map to obtain a predicted subsequence of the trunk. Since decoding the predicted subsequence of the trunk of the sample molecular formula is the first decoding stage, there is no reference decoding state, so in the first decoding stage, the preset state can be set to a 0 vector.

In this embodiment, after decoding the predicted subsequence of the trunk of the sample molecular formula, it is necessary to decode the predicted subsequence of the branch of the sample molecular formula. First, a reference state that has not been selected is randomly selected, wherein the reference state is the decoding state when the branch symbol is decoded, so that the unexpanded branch symbol can be decoded subsequently. The selection strategy of randomly selecting a reference state that has not been selected can make certain disturbances to the decoding order of the branch symbol before training, thereby avoiding the molecular formula recognition model from overfitting to a certain fixed order of traversing branches during the training process, thereby forcing the molecular formula recognition model to fully consider the input conditional information, and use the conditional information to correctly decode, thereby improving the generalization ability of the molecular formula recognition model.

For example, as shown in FIG13 , FIG13 is a schematic diagram of an embodiment of the molecular formula recognition model decoding process provided in the present application. After the molecular formula recognition model decodes an end symbol “eol” at decoding time t-1, a reference state that has not been selected will be randomly selected at the next decoding time.

Step S1202: taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded.

In this embodiment, the branch represented by the branch symbol corresponding to the reference state is used as the branch to be decoded. In other words, the branch symbol corresponding to the decoding state when the branch symbol is decoded is used as the branch to be decoded, so that the data content corresponding to the branch symbol is subsequently decoded.

Step S1203: Decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded.

In this embodiment, decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded. Specifically, as shown in FIG13 , the molecular formula recognition model will take the branch represented by the branch symbol corresponding to the randomly selected reference state as the branch to be decoded and decode its content. Among them, the specific formula for obtaining the predicted subsequence of the branch to be decoded is as follows:

c _t =Attn(s _t-1 ,s _zt ,y _t-1 ,F)

o _t ,s _t =RNN(c _t ,y _t-1 ,s _t-1 ,s _zt )

p _t =MLP(o _t )

y _t =argmaxx _i p _t (i),i∈{0,1,...V-1}

Among them, c _t is the feature that needs to be paid attention to at the current decoding moment; Attn is the attention mechanism network; s _t-1 is the decoding state at the previous decoding moment; s _zt is the decoding state when decoding to the branch symbol; y _t-1 is the decoding result at the previous decoding moment; F is the sample feature map; s _t is the hidden state of the recurrent neural network RNN; o _t is the output feature of the recurrent neural network RNN; p _t is the output probability distribution, p _t ∈ ^RV , V is the number of modeling units; MLP is a neural network composed of a fully connected layer and a softmax layer; y _t is the decoding result corresponding to the current decoding moment. Specifically, the correlation is calculated according to the decoding state s _t-1 at the last decoding moment, the decoding result y _t-1 at the last decoding moment, the decoding state s _zt when decoding to the branch symbol, and the features of all positions in the sample feature map F, and the features of all positions in the sample feature map F are weighted and summed by the correlation to obtain the feature c _t that needs to be paid attention to at the current decoding moment; then, the recurrent neural network RNN is used to associate the feature c _t that needs to be paid attention to at the current decoding moment, the decoding state s _t-1 at the last decoding moment, the decoding result y _t-1 at the last decoding moment, and the decoding state s _zt when decoding to the branch symbol to obtain the decoding state o _t at the current decoding moment; then, according to the decoding state o _t at the current decoding moment, symbol prediction is performed to obtain the probability distribution p _t of the preset symbol, that is, the probability prediction value of the preset symbol. Specifically, a decoding symbol dictionary is provided, and the probability distribution includes the probability prediction values of each preset decoding symbol in the corresponding decoding symbol dictionary; the preset symbol corresponding to the maximum probability prediction value in the decoding symbol dictionary is used as the decoding result y _t at the current decoding moment.

Step S1204: adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.

In this embodiment, according to the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted so that the molecular formula recognition network converges, thereby improving the recognition ability of the molecular formula recognition model and improving the generalization ability of the molecular formula recognition model.

In one embodiment, a sub-prediction loss value between each decoded character in a sample sub-sequence and each predicted character in a predicted sub-sequence belonging to the same branch is calculated by a loss function such as cross entropy, and then each sub-prediction loss value is counted to obtain a total loss value, so that the network parameters of the molecular formula recognition model can be adjusted based on the total loss value through optimization methods such as gradient descent, and the above process is repeated until the molecular formula recognition model is trained to convergence using a sample image containing the sample molecular formula.

In other embodiments, the sample molecular formula is composed of a trunk and branches, the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branches, and the network parameters of the molecular formula recognition model can be adjusted according to the difference between the predicted subsequence and the sample subsequence of the trunk and the difference between the sample subsequence and the predicted subsequence belonging to the same branch.

In one embodiment, as shown in FIG. 14 , FIG. 14 is a flow chart of an embodiment of determining the end of decoding provided by the present application. Before adjusting the network parameters of the molecular formula recognition model, it is necessary to confirm whether the decoding of the entire sample image is completed, which specifically includes the following sub-steps:

Step S1401: Check whether the end character is decoded and all reference states have been selected.

In this embodiment, it is determined whether the end character is decoded (e.g., the end character is eol, etc.) and all reference states have been selected. If the end character has not been decoded or the end character is decoded and there is a reference state that has not been selected for use, it indicates that the decoding of the sample image has not been completed. If the end character is decoded and there is a reference state that has not been selected, step 1402 is executed; if the end character is decoded and all reference states have been selected for use, it indicates that the decoding of the sample image has been completed, and step S1403 is executed.

Step S1402: In response to decoding to an end character and there being reference states that have not been selected, executing a step of randomly selecting a reference state that has not been selected and subsequent steps.

In this embodiment, the sample subsequence ends with an end symbol, and in response to decoding to the end symbol and there are still reference states that have not been selected for use, the step of randomly selecting a reference state that has not been selected and the subsequent steps are re-executed. That is, in the case where there are still reference states that have not been selected for use, the step of randomly selecting a reference state that has not been selected and the subsequent steps are re-executed until all reference states have been selected for use, indicating that the decoding is completed.

Step S1403: In response to decoding to the end symbol and all reference states being selected, a step of adjusting network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed.

In this embodiment, in response to decoding to the end symbol and all reference states being selected for use, it is determined that decoding is finished, and a step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed. That is, after decoding to the end symbol (for example, the end symbol is "eol") and all reference states are selected for use in the current decoding stage, it indicates that decoding is finished; in addition, a step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed.

Please refer to FIG. 15 , which is a schematic diagram of the framework of an embodiment of a molecular formula recognition device provided by the present application. The molecular formula recognition device 150 includes a sequence recognition module 151 and a formula recovery module 152; the sequence recognition module 151 is used to recognize the image to be recognized using the molecular formula recognition model to obtain a symbol sequence; the formula recovery module 152 is used to recover the target molecular formula in the image to be recognized based on the symbol sequence; wherein the molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from the graphic visual form of the sample molecular formula.

The sample symbol sequence includes a character string representing an atomic group in the sample molecular formula and a character string representing a chemical bond in the sample molecular formula, and the character string representing a chemical bond at least includes an angle of the chemical bond.

The sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents the direction of the branch.

The sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each branch. The sample first subsequence contains branch symbols representing each branch respectively, and the branch symbols also represent the identification of the branch. The sample second subsequence contains an order symbol, and the order symbol represents the identification of the branch.

Among them, the above-mentioned sample molecular formula is pre-annotated as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The molecular formula recognition device 150 also includes an acquisition module 153. The acquisition module 153 is used for the acquisition steps of the sample symbol sequence including: performing structural analysis based on the original label sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a number of data elements, and the several data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each data element in the graphic data is marked with data attributes; traversal is performed based on the graphic data to obtain the sample symbol sequence.

The data attributes of the nodes include characters representing atomic groups; and/or the data attributes of the edges include at least the angle of the chemical bond.

Among them, the acquisition module 153 is used to traverse based on the graphic data to obtain a sample symbol sequence, specifically including: traversing the data elements on the sample molecular formula trunk in the graphic data to obtain the sample first subsequence, and traversing the data elements on the sample molecular formula branches in the graphic data to obtain the sample second subsequence; combining the sample first subsequence and the sample second subsequence to obtain a sample symbol sequence; wherein the character string representing the data element in the sample symbol sequence includes the data attributes of the data element, the branch is represented by a branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains a sequence symbol, and the sequence symbol represents the identification of the branch.

Among them, before performing structural analysis based on the original label sequence to obtain graphic data of the sample molecular formula, the acquisition module 153 is also used to: render the original label sequence using a rendering engine of a preset molecular formula markup language to obtain a rendered molecular formula; and determine whether the original label sequence is correctly labeled based on the difference check result between the rendered molecular formula and the sample molecular formula.

The sample molecular formula is composed of a trunk and branches, and the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branch, and the branch is represented by a branch symbol in the sample subsequence of the trunk; the molecular formula recognition device 150 also includes a training module 154, and the training module 154 is used for the training steps of the molecular formula recognition model, including: randomly selecting a reference state that has not been selected; wherein the reference state is a decoding state when decoding to a branch symbol; taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded; decoding based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded; adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.

The sample subsequence ends with a terminator; the training module 154 is used to randomly select a reference state that has not been selected, specifically including: in response to decoding to the terminator and there is still a reference state that has not been selected, executing the step of randomly selecting a reference state that has not been selected and subsequent steps.

Wherein, before adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the training module 154 is also used to: check whether the end symbol is decoded and all reference states have been selected; in response to decoding to the end symbol and all reference states have been selected, perform the step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch.

Among them, before randomly selecting a reference state that has not been selected, the training module 154 is also used to: decode based on the preset state and the sample feature map to obtain a predicted subsequence of the trunk; the training module 154 is used to adjust the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, specifically including: adjusting the network parameters of the molecular formula recognition model based on the difference between the predicted subsequence and the sample subsequence of the trunk, and the difference between the sample subsequence and the predicted subsequence belonging to the same branch.

Among them, the above-mentioned molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model, and the image-text encoding model is trained using sample images containing at least one of sample text and sample formula.

Please refer to FIG. 16 , which is a schematic diagram of the framework of an embodiment of an electronic device provided by the present application. The electronic device 160 includes a memory 161 and a processor 162 coupled to each other, the memory 161 stores program instructions, and the processor 162 is used to execute the program instructions to implement the steps in any of the above-mentioned molecular formula recognition method embodiments. Specifically, the electronic device 160 may include, but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

Specifically, the processor 162 is used to control itself and the memory 161 to implement the steps in any of the above-mentioned molecular formula recognition method embodiments. The processor 162 can also be referred to as a CPU (Central Processing Unit). The processor 162 may be an integrated circuit chip with signal processing capabilities. The processor 162 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 162 may be implemented by an integrated circuit chip.

Please refer to Figure 17, which is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 170 stores program instructions 171 that can be executed by a processor, and the program instructions 171 are used to implement the steps in any of the above molecular formula recognition method embodiments.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims (15)

1. A molecular formula recognition method, comprising: The image to be identified is identified using the molecular formula recognition model to obtain a symbol sequence; Based on the symbol sequence, recovering the target molecular formula in the image to be identified; The molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from a graphic visual form of the sample molecular formula; The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The step of obtaining the sample symbol sequence includes: Structural analysis is performed based on the original tag sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a plurality of data elements, the plurality of data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each of the data elements in the graphic data is marked with a data attribute; The sample symbol sequence is obtained by traversing based on the graphic data. 2. The method according to claim 1 is characterized in that the sample symbol sequence includes a character string representing an atomic group in the sample molecular formula and a character string representing a chemical bond in the sample molecular formula, and the character string representing the chemical bond at least includes an angle of the chemical bond. 3. The method according to claim 2, characterized in that the sample symbol sequence further includes a branch symbol representing a branch in the sample molecular formula, and the branch symbol at least represents the direction of the branch. 4. The method according to claim 3 is characterized in that the sample symbol sequence is composed of a sample first subsequence of the sample molecular formula trunk and a sample second subsequence of each of the branches, the sample first subsequence contains branch symbols representing each of the branches respectively, and the branch symbols also represent the identifiers of the branches, and the sample second subsequence contains sequence symbols, and the sequence symbols represent the identifiers of the branches. 5. The method according to claim 1, characterized in that the data attribute of the node includes a character representing the atomic group; And/or, the data attributes of the edge include at least the angle of the chemical bond. 6. The method according to claim 1, characterized in that the traversal based on the graphic data to obtain the sample symbol sequence comprises: Traversing the data elements on the trunk of the sample molecular formula in the graphic data to obtain a first subsequence of samples, and traversing the data elements on the branches of the sample molecular formula in the graphic data to obtain a second subsequence of samples; Combining the first subsequence of samples with the second subsequence of samples to obtain the sample symbol sequence; Among them, the character string representing the data element in the sample symbol sequence includes the data attribute of the data element, the branch is represented by a branch symbol in the sample first subsequence, and the branch symbol represents the direction and identification of the branch, and the sample second subsequence contains an order symbol, and the order symbol represents the identification of the branch. 7. The method according to claim 1, characterized in that before performing structural analysis based on the original tag sequence to obtain graphic data of the sample molecular formula, the method further comprises: Rendering the original tag sequence using a rendering engine of the preset molecular formula markup language to obtain a rendered molecular formula; Based on the difference check result between the rendered molecular formula and the sample molecular formula, it is determined whether the original label sequence is correctly labeled. 8. The method according to claim 1, characterized in that the sample molecular formula consists of a trunk and branches, the sample symbol sequence includes a sample subsequence of the trunk and a sample subsequence of the branch, and the branch is represented by a branch symbol in the sample subsequence of the trunk; the training step of the molecular formula recognition model comprises: Randomly select a reference state that has not been selected; wherein the reference state is a decoding state when decoding to the branch symbol; Taking the branch represented by the branch symbol corresponding to the reference state as the branch to be decoded; Decoding is performed based on the reference state and the sample feature map extracted from the sample image to obtain a predicted subsequence of the branch to be decoded; Based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted. 9. The method according to claim 8, wherein the sample subsequence ends with a terminator; and the randomly selecting a reference state that has not been selected comprises: In response to decoding the end symbol and there being a reference state that has not been selected, the step of randomly selecting a reference state that has not been selected and subsequent steps are performed. 10. The method according to claim 9, characterized in that before adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the method further comprises: Check whether the terminator is decoded and all the reference states have been selected; In response to decoding the end symbol and all the reference states being selected, the step of adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch is performed. 11. The method according to claim 8, characterized in that before randomly selecting a reference state that has not been selected, the method further comprises: Decoding is performed based on a preset state and the sample feature map to obtain a predicted subsequence of the backbone; The adjusting the network parameters of the molecular formula recognition model based on the difference between the sample subsequence and the predicted subsequence belonging to the same branch comprises: Based on the difference between the predicted subsequence and the sample subsequence of the trunk, and the difference between the sample subsequence and the predicted subsequence belonging to the same branch, the network parameters of the molecular formula recognition model are adjusted. 12. The method according to claim 1 is characterized in that the molecular formula recognition model includes a molecular formula encoding network and a molecular formula decoding network, and the molecular formula encoding network is initialized by the image-text encoding network of the image-text recognition model, and the image-text encoding model is trained using a sample image containing at least one of a sample text and a sample formula. 13. A molecular formula recognition device, comprising: A sequence recognition module is used to recognize the image to be recognized by using a molecular formula recognition model to obtain a symbol sequence; A formula recovery module, used for recovering the target molecular formula in the image to be identified based on the symbol sequence; The molecular formula recognition model is trained using a sample image containing a sample molecular formula, the sample image is annotated with a sample symbol sequence of the sample molecular formula, and the sample symbol sequence is constructed from a graphic visual form of the sample molecular formula; The sample molecular formula is pre-marked as an original label sequence in a preset molecular formula markup language, and the grammatical rules of the preset molecular formula markup language follow the graphic visual form of the molecular formula. The step of obtaining the sample symbol sequence includes: Structural analysis is performed based on the original tag sequence to obtain graphic data of the sample molecular formula; wherein the graphic data is composed of a plurality of data elements, the plurality of data elements include nodes and edges connecting the nodes, the nodes represent atomic groups, the edges represent chemical bonds, and each of the data elements in the graphic data is marked with a data attribute; The sample symbol sequence is obtained by traversing based on the graphic data. 14. An electronic device, characterized in that it comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the molecular formula recognition method according to any one of claims 1 to 12. 15 . A computer-readable storage medium, characterized in that program instructions that can be executed by a processor are stored therein, wherein the program instructions are used to implement the molecular formula recognition method according to any one of claims 1 to 12.