CN113836878B - Table generation method, device, electronic device and storage medium combining RPA and AI - Google Patents

Abstract

The present disclosure provides a table generation method, device, electronic device and storage medium combining RPA and AI, and relates to the field of artificial intelligence. The scheme is: executed by an RPA system, the RPA system extracts the horizontal and vertical lines of the first table from the image based on artificial intelligence AI; the RPA system obtains a set of intersections of the horizontal and vertical lines, wherein the intersection set includes a first type of intersection formed by the intersection of the horizontal and vertical lines, and a second type of intersection formed by the intersection of the extension of the horizontal line and/or the extension of the vertical line; the RPA system generates a blank second table consistent with the first table based on the intersection set; the RPA system fills the text entries identified from the image based on OCR into the blank second table to obtain the target table. The present disclosure uses RPA technology to identify the table in the image and restore it to a table document with the same table structure, automatically converting offline data into online data, replacing the cumbersome manual processing process, and improving the efficiency of table generation.

Description

Table generation method, device, electronic device and storage medium combining RPA and AI

Technical Field

The present disclosure relates to the field of artificial intelligence, and in particular to a table generation method, device, electronic device and storage medium combining RPA and AI.

Background Art

Robotic Process Automation (RPA) uses specific "robot software" to simulate human operations on computers and automatically execute process tasks according to rules.

Artificial Intelligence (AI) is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

In the related technology, the staff typed, copied and pasted to convert the table in the picture into an online table document, which was repetitive and mechanical, wasted manpower and was inefficient. Therefore, how to improve the efficiency of table generation and free up manpower is an urgent issue that needs to be solved now.

Summary of the invention

The present disclosure provides a table generation method, device, electronic device and storage medium combining RPA and AI.

According to one aspect of the present disclosure, a table generation method combining RPA and AI is provided, including:

The RPA system extracts the horizontal and vertical lines of the first table from the image based on artificial intelligence AI;

The RPA system obtains a set of intersections of the horizontal line and the vertical line, wherein the set of intersections includes a first type of intersection formed by the intersection of the horizontal line and the vertical line, and a second type of intersection formed by the intersection of the extension line of the horizontal line and/or the extension line of the vertical line;

The RPA system generates a blank second table consistent with the first table based on the set of intersection points;

The RPA system fills the text entries recognized based on OCR from the image into the blank second table to obtain the target table.

The disclosed embodiment uses RPA technology to identify tables in images and restore them to table documents with the same table structure, automatically converting offline data into online data, replacing cumbersome manual processing procedures and improving the efficiency of table generation.

According to another aspect of the present disclosure, a table generation device combining RPA and AI is provided, including:

An extraction module, used for extracting horizontal lines and vertical lines of the first table from the image based on artificial intelligence AI;

An intersection acquisition module, used to acquire a set of intersections of horizontal lines and vertical lines, wherein the set of intersections includes first-type intersections formed by the intersection of horizontal lines and vertical lines, and second-type intersections formed by the intersection of extended lines of horizontal lines and/or extended lines of vertical lines;

A generating module, used for generating a blank second table consistent with the first table according to the intersection point set;

The filling module is used to fill the text entries recognized based on OCR in the image into the blank second table to obtain the target table.

According to another aspect of the present disclosure, there is provided an electronic device, including a memory and a processor;

Among them, the processor runs the program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the table generation method combining RPA and AI in the embodiment of the first aspect of the present disclosure.

According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the table generation method combining RPA and AI of the first aspect of the present disclosure is implemented.

According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the table generation method combining RPA and AI of the first aspect of the present disclosure.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG2 is a schematic diagram of a table to be identified and detected horizontal and vertical lines;

FIG3 is a schematic diagram of a set of intersection points formed by detected horizontal and vertical lines;

FIG4 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG5 is a schematic diagram of enumerating candidate cells;

FIG6 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG7 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG8 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG9 is a schematic diagram of a pair of intersection points composed of intersection points;

FIG10 is a schematic diagram of a basic unit cell composed of a pair of intersection points;

FIG11 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG12 is a schematic diagram of updating the values of matrix elements after determining the target cell;

FIG13 is a schematic diagram of a Boolean matrix split into two sub-targets after detecting the target row;

FIG14 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG15 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG16 is a flowchart of a table generation method combining RPA and AI according to an embodiment of the present disclosure;

FIG17 is a structural diagram of a table generation device combining RPA and AI according to an embodiment of the present disclosure;

FIG18 is a block diagram of an electronic device used to implement the table generation method combining RPA and AI according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

The following describes the table generation method, device, electronic device and storage medium combining RPA and AI in the present disclosure with reference to the accompanying drawings.

FIG. 1 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. As shown in FIG. 1 , the method includes the following steps:

S101, the RPA system extracts horizontal lines and vertical lines of the first table from the image based on artificial intelligence AI.

RPA is a technology that simulates human operation on a PC and has begun to be applied to corporate production and office. The core of RPA is to use automation technology to "replace people" to perform repetitive, low-value, fixed process operations that do not require manual decision-making, thereby effectively improving work efficiency and reducing errors.

The RPA system obtains an image with a table. The first table is the table to be identified in the image. The table model is used to detect the horizontal and vertical table lines in the image respectively. The table lines can be tilted and curved to a certain extent. The horizontal and vertical lines of the first table are extracted based on artificial intelligence AI. Optionally, the horizontal and vertical lines in the image can be extracted using traditional computer vision (CV) algorithms. Line segments with close endpoints are connected to prevent recognition errors and obtain the final horizontal and vertical lines. Taking Figure 2 as an example, from left to right are the original image, the detected horizontal lines, and the detected vertical lines.

S102: The RPA system obtains a set of intersections of horizontal lines and vertical lines, wherein the set of intersections includes first-type intersections formed by the intersection of horizontal lines and vertical lines, and second-type intersections formed by the intersection of extended lines of the horizontal lines and/or extended lines of the vertical lines.

The RPA system obtains the set of intersection points of horizontal and vertical lines. Assuming that x horizontal lines and y vertical lines are obtained in the above steps, the coordinates of the intersection points of each pair of horizontal and vertical lines are calculated. Regardless of whether the line segments intersect or not, a total of x*y points can be obtained, as shown in Figure 3.

The intersection point set includes first-class intersection points formed by the intersection of horizontal lines and vertical lines, and such intersection points are crossed by both horizontal and vertical lines.

The intersection set also includes second-type intersections formed by the intersection of the extension lines of the lines and/or the extension lines of the vertical lines. This type of intersection is only crossed by one horizontal line/one vertical line.

S103: The RPA system generates a blank second table consistent with the first table according to the intersection point set.

As shown in reference Figure 3, these intersections can form many tables. The RPA system identifies the table structure of the first table based on the set of first-type intersections and second-type intersections, and performs operations such as screening and deduplication on these tables to screen out blank tables that are consistent with the structure of the first table, namely, the second table.

S104, the RPA system fills the text entries recognized from the image based on OCR into the blank second table to obtain a target table.

Optical Character Recognition (ORC) technology converts the text of various bills, newspapers, books, manuscripts and other printed materials into image information through optical input methods such as scanning, and then uses text recognition technology to convert the image information into usable computer input technology.

The RPA system identifies the text entries of the first table in the image based on ORC technology, and fills them into the corresponding positions of the second table to obtain the target table. The table format and table content of the target table are the same as those of the first table. The first table is paper table data, while the target table is electronic table data.

In the disclosed embodiment, the RPA system extracts the horizontal and vertical lines of the first table from the image based on artificial intelligence AI, and the RPA system obtains a set of intersections of the horizontal and vertical lines, wherein the set of intersections includes a first type of intersection formed by the intersection of the horizontal and vertical lines, and a second type of intersection formed by the intersection of the extensions of the horizontal and/or vertical lines. The RPA system generates a blank second table consistent with the first table based on the set of intersections, and the RPA system fills the text entries identified from the image based on OCR into the blank second table to obtain the target table. In the disclosed embodiment, RPA technology is used to identify the table in the image and restore it to a table document with the same table structure, automatically converting offline data into online data, replacing the cumbersome manual processing flow and improving the efficiency of table generation.

FIG4 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, further combined with FIG4, a process of generating a blank second table consistent with the first table according to the intersection set by the RPA system is explained, including the following steps:

S401, the RPA system enumerates cells according to the intersection points in the intersection point set to obtain candidate cells and attribute information of the candidate cells.

Assuming that x horizontal lines and y vertical lines are obtained in the above steps, a total of x*y intersections can be obtained. For these x*y intersections, all possible cells are enumerated as candidate cells. Each cell consists of a starting row, an ending row, a starting column, an ending column, and four intersection coordinates.

The attribute information of the candidate cell includes the area of the cell, the coordinates of the four corners of the cell, the starting and ending points of the horizontal line and the starting and ending points of the vertical line corresponding to the cell.

Taking the three pairs of intersections in Figure 5 as an example, the process of unit enumeration is explained. For the convenience of description, the three pairs of intersections are numbered a, b, and c. As shown in Figure 5, a and b can form a candidate cell, and correspondingly, b and c can also form a candidate cell. Finally, a and c can also form a candidate cell.

S402: The RPA system identifies a target cell for generating a blank second table from the candidate cells according to the attribute information of the candidate cells.

The RPA system traverses all candidate cells according to the attribute information of the candidate cells, and determines whether all four edges of each candidate cell exist one by one. If all four edges exist, the candidate cell is used as the target cell.

S403, the RPA system generates a blank second table for the target cells according to the position arrangement.

The RPA system obtains all target cells and attribute information of the target cells, arranges the cells according to the four corner coordinates of the target cells, and generates a blank second table.

In the disclosed embodiment, the RPA system enumerates the cells according to the intersections in the intersection set, obtains the candidate cells and the attribute information of the candidate cells, and the RPA system identifies the target cells for generating the blank second table from the candidate cells according to the attribute information of the candidate cells, and the RPA system generates the blank second table according to the position arrangement of the target cells. In the disclosed embodiment, a blank second table with the same structure as the first table is generated according to the intersection set, and the detection and generation of the table structure is realized, which is the most important step in table generation and lays the foundation for the subsequent filling of text entries.

FIG6 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, FIG6 is further combined to explain the process of the RPA system identifying target cells for generating a blank second table from candidate cells according to attribute information of candidate cells, including the following steps:

S601, the RPA system sorts all the enumerated candidate cells in ascending order of cell area.

According to the coordinates of the four corners of the candidate cells, the area of the cells can be calculated, and all the enumerated candidate cells are sorted from small to large according to the cell area.

S602: The RPA system traverses the candidate cells in order and determines the existence of the traversed target candidate cells.

The candidate cells are traversed in order of area from small to large, and the existence of the traversed target candidate cells is judged.

The RPA system obtains the first starting and ending points of the horizontal line and the second starting and ending points of the vertical line corresponding to the target candidate cell, and determines whether all four sides of the target candidate cell exist based on the four corner coordinates, the first starting and ending points, and the second starting and ending points of the target candidate cell. When all four sides exist, it is determined that the target candidate cell exists.

S603, whenever the RPA system determines that a target candidate cell exists, the RPA system deletes cells that overlap with the target candidate cell from the candidate cells that have not been traversed, and determines the target candidate cell that is determined to exist as a target cell.

When it is determined that a target candidate cell exists, the candidate cell that overlaps with it must not exist. The RPA system deletes the cells that overlap with the target candidate cell from the candidate cells that have not been traversed, reducing the workload of subsequent traversal.

The RPA system determines the target candidate cell that exists as a target cell for generating a blank second table.

S604, the RPA system continues to traverse the candidate cells that have not been traversed after deletion in order until all target cells are obtained after the traversal.

The RPA system continues to traverse the candidate cells that have not been traversed after deletion in order, pauses the traversal after obtaining a new target cell, deletes the candidate cells that have not been traversed subsequently and overlap with the new target cell, and repeats this operation until the traversal is completed and all the target cells are obtained.

In the disclosed embodiment, the RPA system sorts all the enumerated candidate cells according to the cell area from small to large, and the RPA system traverses the candidate cells in order, and judges the existence of the traversed target candidate cells. Whenever the RPA system determines that the target candidate cell exists, it deletes the cells that overlap with the target candidate cells from the candidate cells that have not been traversed, and determines the target candidate cells that are determined to exist as a target cell. The RPA system continues to traverse the candidate cells that have not been traversed after the deletion in order until all the target cells are obtained after the traversal. In the disclosed embodiment, the target cells for generating the second table are obtained from the candidate cells, and the structural composition of the second table is preliminarily determined, laying the foundation for the generation of the second table.

FIG7 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. On the basis of the above embodiment, FIG7 is further combined to explain the process of the RPA system traversing the candidate cells in sequence and judging the existence of the traversed target candidate cells, including the following steps:

S701: The RPA system obtains the first starting and ending points of the horizontal line and the second starting and ending points of the vertical line corresponding to the target candidate cell.

The RPA system obtains the first starting and ending points of the horizontal line and the second starting and ending points of the vertical line corresponding to the target candidate cell, that is, obtains the spatial orientation and length of the corresponding horizontal and vertical lines.

S702: The RPA system determines whether all four sides of the target candidate cell exist based on the coordinates of the four corners, the first starting point, and the second starting point of the target candidate cell.

According to the coordinates of the four corners of the candidate cell, we can get the four edges formed by the four corners. Take the edge formed by the upper left corner and the lower left corner as an example:

Project this edge onto the Y axis to get its position and length on the Y axis, and project the scanned vertical line corresponding to this edge onto the Y axis to get its position and length on the Y axis. See if there is any overlap between the two. If there is, it proves that this edge exists, and if there is no overlap, it proves that this edge does not exist. Similarly, we can prove whether the edge formed by the two points in the upper right corner and the lower right corner exists.

Accordingly, take the edge formed by the upper left and upper right points as an example:

Project this edge onto the X-axis to get its position and length on the X-axis, and project the scanned horizontal line corresponding to this edge onto the X-axis to get its position and length on the X-axis. See if there is any overlap between the two. If there is, it proves that this edge exists, and if there is no overlap, it proves that this edge does not exist. Similarly, we can prove whether the edge formed by the two points in the lower left corner and the lower right corner exists.

S703, when the RPA system determines that all four edges exist, it determines that the target candidate cell exists.

If all four sides of the candidate cell exist, it is determined that the target candidate cell exists and can be used as the target cell.

In the disclosed embodiment, the RPA system obtains the first starting and ending point of the horizontal line and the second starting and ending point of the vertical line corresponding to the target candidate cell. The RPA system determines whether all four sides of the target candidate cell exist based on the four corner coordinates, the first starting and ending points, and the second starting and ending points of the target candidate cell. When the RPA system determines that all four sides exist, it determines that the target candidate cell exists. The disclosed embodiment provides a method for determining whether a target candidate cell exists, which lays the foundation for the RPA system to traverse the candidate cells to obtain all target cells.

FIG8 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, before the RPA system identifies a target cell for generating a blank second table from the candidate cells according to the attribute information of the candidate cells, as shown in FIG8, it further includes:

S801, the RPA system arranges the intersection points in the intersection point set in order, and groups the adjacent intersection points in the intersection point set into a plurality of intersection point pairs in the same direction.

The RPA system arranges the intersections of each row and column in the intersection set in spatial order, and groups adjacent intersections in the intersection set into multiple intersection pairs in the same direction, optionally horizontally or vertically.

As shown in FIG. 9 , when the intersection pairs are formed horizontally, an intersection will form an intersection pair with the intersection on its left, and another intersection pair with the intersection on its right. An intersection cannot exist in only one intersection pair.

Correspondingly, when forming intersection pairs in the vertical direction, an intersection will form an intersection pair with the intersection above it, and another intersection pair with the intersection below it.

S802, the RPA system sequentially obtains adjacent intersection pairs in the same direction, and forms basic cells from the adjacent intersection pairs.

The RPA system sequentially obtains adjacent intersection pairs in the horizontal or vertical direction, and the adjacent intersection pairs form basic cells. When adjacent intersection pairs are formed into intersection pairs in the horizontal direction, adjacent intersection pairs are obtained in the vertical direction. Correspondingly, when adjacent intersection pairs are formed into intersection pairs in the vertical direction, adjacent intersection pairs are obtained in the horizontal direction.

As shown in FIG10 , when adjacent intersection pairs are obtained horizontally to form basic cells, a group of intersection pairs b will form a basic cell with the intersection pair a on its left, and another basic cell with the intersection pair c on its right. A group of intersection pairs cannot only form one basic cell.

Correspondingly, when adjacent intersection pairs are obtained vertically to form basic cells, a group of intersection pairs e will form a basic cell with the intersection pair d above it, and form another basic cell with the intersection pair f below it.

S803, the RPA system uses the basic cells as matrix elements to construct a Boolean matrix.

The RPA system uses the basic cells as matrix elements to construct a two-dimensional Boolean matrix. Assuming that x horizontal lines and y vertical lines are extracted in the above steps, x*y intersections can be obtained. At this time, a (x-1)*(y-1) Boolean matrix can be constructed to identify whether the cell at each position exists.

In the embodiment of the present disclosure, the RPA system arranges the intersections in the intersection set in order, and forms multiple intersection pairs with adjacent intersections in the intersection set in the same direction. The RPA system sequentially obtains adjacent intersection pairs in the same direction, and forms basic cells with adjacent intersection pairs. The RPA system uses the basic cells as matrix elements to construct a Boolean matrix. In the implementation of the present disclosure, a Boolean matrix is constructed to identify whether a cell at each position exists, which can assist in recording the target cell determined by traversal and the position of the target cell, and facilitate subsequent operations.

FIG11 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, FIG11 is further combined to explain the process of the RPA system generating a blank second table for the target cells according to the position arrangement, including the following steps:

S1101, the RPA system assigns a value to each matrix element in the Boolean matrix according to the target cell to generate a target Boolean matrix.

The initial matrix elements in the Boolean matrix are all F, which is the first value. When a target cell is determined, the RPA system obtains the first basic cell included in the target cell based on the four corner coordinates of the target cell, and updates the matrix element corresponding to the first basic cell from the first value F to the second value T.

As shown in FIG. 12 , for example, the target cell in the second row includes three basic cells, and the corresponding matrix elements of the three basic cells need to be updated from the first value F to the second value T.

Each matrix element in the Boolean matrix is assigned a value according to the target cell to generate a target Boolean matrix, wherein the matrix element corresponding to the target cell is updated to the second value, and the remaining matrix elements remain unchanged at the initial first value.

S1102: The RPA system identifies whether the target Boolean matrix includes target rows and/or target columns that are all first values.

The RPA system detects the value of each matrix element in the target Boolean matrix row by row and column by column, and identifies whether the target Boolean matrix includes target rows and/or target columns that all have the first value.

S1103: When there are target rows and/or target columns, the RPA system splits the Boolean matrix according to the target rows and/or target columns to generate sub-target Boolean matrices.

If the existence of a target row and/or target column is detected, it proves that the row and/or column does not contain a target cell, indicating that there are two tables with up-down/left-right relationships in the figure. The Boolean matrix is split according to the target row and/or target column to generate a sub-target Boolean matrix.

As shown in FIG. 13 , if the target row is detected, it is proved that the row does not contain the target cell, indicating that there are two tables in a top-to-bottom relationship in the figure.

If the existence of a target column is detected, it proves that this column does not contain a target cell, indicating that there are two tables with a left-right relationship in the figure.

If the existence of the target row and target column is detected, it proves that the row and the column do not contain the target cell, indicating that there are four tables in the “田” character relationship in the figure.

S1104, the RPA system obtains the target cells corresponding to the sub-target Boolean matrix, and arranges them according to the positions to generate a blank second table corresponding to the sub-target Boolean matrix.

The process of generating a blank second table corresponding to the sub-target Boolean matrix in step S1104 can be referred to the process of generating a blank second table according to the position arrangement of the target cells in step S403, which will not be repeated here.

In the disclosed embodiment, the RPA system assigns a value to each matrix element in the Boolean matrix according to the target cell to generate a target Boolean matrix. The RPA system identifies whether the target Boolean matrix includes target rows and/or target columns that are all first values. If there are target rows and/or target columns, the RPA system splits the Boolean matrix according to the target rows and/or target columns to generate a sub-target Boolean matrix. The RPA system obtains the target cells corresponding to the sub-target Boolean matrix and arranges them according to the position to generate a blank second table corresponding to the sub-target Boolean matrix. In the disclosed embodiment, the RPA system determines whether there are two tables in the figure with an up-down/left-right relationship according to the value of each matrix element in the Boolean matrix, and further determines the structure of the table.

FIG14 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, when there are target rows and/or target columns, the RPA system splits the Boolean matrix according to the target rows and/or target columns to generate sub-target Boolean matrices, as shown in FIG14, further comprising:

S1401, the RPA system identifies whether there is a target matrix element with a first value in the sub-target Boolean matrix.

The RPA system traverses each matrix element in the sub-target Boolean matrix in order to determine whether the value of each matrix element is the first value F. If it is determined that the currently traversed matrix element is the first value, the matrix element is marked as the target matrix element. Continue to traverse the remaining matrix elements until the last matrix element is traversed to obtain all the target matrix elements included in the sub-target Boolean matrix.

S1402: The RPA system updates the value of the target matrix element from the first value to the second value.

In order to ensure the integrity of the table, the value of the target matrix element needs to be updated from the first value to the second value to obtain the final second table. However, the target matrix element whose value is the first value may exist in two situations, which may be an independent target cell that is missed, or may belong to a target cell that has been identified. In the embodiment of the present application, the target matrix element can be judged, as shown in Figure 15.

The RPA system obtains the target basic cell corresponding to the target matrix element. In the present disclosure, a matrix element in a Boolean matrix corresponds to a basic cell. After determining the target matrix element, the RPA system can obtain the basic cell corresponding to the target matrix element based on the position of the target matrix element in the Boolean matrix, which is referred to as the target basic cell.

The RPA system determines whether the target basic cell needs to be merged into the target cell corresponding to the adjacent matrix element adjacent to the target matrix element. The RPA system can obtain the four corner coordinates of the target basic cell, and determine whether the target basic cell needs to be merged into the existing target cell based on the four corner coordinates. If the coverage of the existing target cell includes the four corner coordinates of the target basic cell, it is determined that it needs to be merged into the existing target cell. If the coverage of the existing target cell does not include the four corner coordinates of the target basic cell, it is determined that it does not need to be merged into the existing target cell, and the target basic cell is an independent target cell. Among them, the existing target cell is the target cell corresponding to the adjacent matrix element adjacent to the target matrix element.

If the RPA system determines that a merge is required, the target base cell is merged into the target cell corresponding to the adjacent matrix element to form a larger target cell.

If the RPA system determines that there is no need to merge, the target basic cell is determined as a missing target cell, and the value of this place is filled with T as an independent target cell, and the missing target cell is filled when the target cells are arranged according to the position.

In the embodiment of the present disclosure, the RPA system identifies whether there is a target matrix element with a first value in the sub-target Boolean matrix, and the RPA system updates the value of the target matrix element from the first value to the second value. In the embodiment of the present disclosure, the target matrix element with a first value in the sub-target Boolean matrix is updated to the second value, the missing cells are completed, and the table structure is determined.

FIG16 is a flow chart of a table generation method combining RPA and AI according to an embodiment of the present disclosure. Based on the above embodiment, FIG16 is further combined to explain the process of the RPA system filling the text entries recognized by OCR from the image into the blank second table to obtain the target table, including the following steps:

S1601, the RPA system obtains the four corner coordinates of the text entry.

The RPA system obtains the text recognition box where the text entry is located, and uses the four corner coordinates of the text recognition box as the four corner coordinates of the text entry.

Alternatively, an algorithm can be used to extract text entries from a text image containing a table. The algorithm forms a rectangular frame surrounding the foreground pixels of the image through template scanning, thereby extracting the foreground pixels, and then combining the rectangular frames to form a pattern chain. The three statistical features of the maximum black run, length, and width of the pattern are used to classify the pattern and extract the text.

S1602: The RPA system matches the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain the target text entry corresponding to the target cell.

The RPA system matches the four corner coordinates of the text entry with the four corner coordinates of the target cell to determine whether there is a first text entry that occupies at least two target cells.

The four corner coordinates of a text entry can form an area. If the four corner coordinates of a cell all fall within the area, and two corner coordinates of another cell also fall within the area, then it occupies two cells.

Similarly, if the coordinates of the four corners of two cells fall inside it, and the coordinates of the third cell fall inside it, it means that three cells are occupied.

If the first text entry exists, the RPA system divides the first text entry according to at least two occupied cells to obtain a target text entry for each target cell occupied by the first text entry.

For example, if the first text entry occupies two adjacent cells, the adjacent cells are segmented using the common edges to obtain the text entries of the two occupied target cells.

If a text entry only occupies one target cell, it is directly used as the target text entry corresponding to the target cell.

S1603, the RPA system fills the target text entry into the corresponding target unit to obtain a target table.

For the specific implementation of step S1603, please refer to the relevant introduction in each embodiment of the present disclosure, which will not be repeated here.

In the disclosed embodiment, the RPA system obtains the four corner coordinates of the text entry, matches the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain the target text entry corresponding to the target cell, and fills the target text entry into the corresponding target cell to obtain the target table. In the disclosed embodiment, the text entry across cells is split, and the text entry can be accurately filled into the corresponding target cell to generate a complete table document.

FIG. 17 is a structural diagram of a table generation apparatus combining RPA and AI according to an embodiment of the present disclosure. As shown in FIG. 17 , a table generation apparatus 1700 combining RPA and AI includes:

An extraction module 1710 is used to extract horizontal lines and vertical lines of the first table from the image based on artificial intelligence AI;

The intersection acquisition module 1720 is used to acquire a set of intersections of horizontal lines and vertical lines, wherein the set of intersections includes first-type intersections formed by the intersection of horizontal lines and vertical lines, and second-type intersections formed by the intersection of extended lines of horizontal lines and/or extended lines of vertical lines;

A generating module 1730, configured to generate a blank second table consistent with the first table according to the intersection point set;

The filling module 1740 is used to fill the text entries recognized from the image based on OCR into the blank second table to obtain a target table.

In the disclosed embodiment, RPA and AI technologies are used to identify tables in images and restore them to table documents with the same table structure, automatically converting offline data into online data, replacing cumbersome manual processing procedures and improving the efficiency of table generation.

It should be noted that the above explanation of the embodiment of the table generation method combining RPA and AI is also applicable to the table generation device combining RPA and AI in this embodiment, and will not be repeated here.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: enumerate the cells according to the intersections in the intersection set to obtain candidate cells and attribute information of the candidate cells; identify target cells for generating a blank second table from the candidate cells according to the attribute information of the candidate cells; and generate a blank second table for the target cells according to their positional arrangements.

Furthermore, in a possible implementation of the disclosed embodiment, the generation module 1730 is also used to: sort all the enumerated candidate cells in ascending order according to the cell area; traverse the candidate cells in order, and judge the existence of the traversed target candidate cells; whenever it is judged that the target candidate cell exists, delete the cells that overlap with the target candidate cell from the candidate cells that have not been traversed, and determine the target candidate cell that is judged to exist as a target cell; continue to traverse the candidate cells that have not been traversed after the deletion in order until all the target cells are obtained after the traversal.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: obtain the first starting and ending points of the horizontal line and the second starting and ending points of the vertical line corresponding to the target candidate cell; determine whether all four sides of the target candidate cell exist based on the four corner coordinates, the first starting and ending points, and the second starting and ending points of the target candidate cell; and determine that the target candidate cell exists when it is determined that all four sides exist.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: arrange the intersections in the intersection set in order, and group the adjacent intersections in the intersection set into multiple intersection pairs in the same direction; obtain adjacent intersection pairs in sequence in the same direction, and group the adjacent intersection pairs into basic cells; and construct a Boolean matrix using the basic cells as matrix elements.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: assign values to each matrix element in the Boolean matrix according to the target cell to generate a target Boolean matrix; identify whether the target Boolean matrix includes target rows and/or target columns that all have the first value; if there are target rows and/or target columns, split the Boolean matrix according to the target rows and/or target columns to generate a sub-target Boolean matrix; obtain the target cells corresponding to the sub-target Boolean matrix, and arrange them according to the position to generate a blank second table corresponding to the sub-target Boolean matrix.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: obtain the first basic cell included in the target cell according to the four corner coordinates of the target cell, and update the matrix element corresponding to the first basic cell from the first value to the second value.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: identify whether there is a target matrix element with a first value in the sub-target Boolean matrix; and update the value of the target matrix element from the first value to the second value.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: obtain the target basic cell corresponding to the target matrix element; determine whether the target basic cell needs to be merged into the target cell corresponding to the adjacent matrix element adjacent to the target matrix element; and merge the target basic cell into the target cell corresponding to the adjacent matrix element if it is determined that merging is required.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the generation module 1730 is also used to: if it is determined that no merging is required, determine the target basic cell as a missing target cell, and complete the missing target cell when arranging the target cells according to the position.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the filling module 1740 is also used to: obtain the four corner coordinates of the text entry; match the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain the target text entry corresponding to the target cell; fill the target text entry into the corresponding target cell to obtain the target table.

Furthermore, in a possible implementation of the embodiment of the present disclosure, the filling module 1740 is also used to: match the four corner coordinates of the text entry with the four corner coordinates of the target cell to determine whether there is a first text entry that occupies at least two target cells; and divide the first text entry according to the at least two occupied cells to obtain a target text entry for each target cell occupied by the first text entry.

According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

FIG. 18 shows a schematic block diagram of an example electronic device 1800 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and/or required herein.

As shown in FIG. 18 , the system includes a memory 181 , a processor 182 , and a computer program stored in the memory 181 and executable on the processor 182 . When the processor 182 executes the program, the aforementioned table generating method is implemented.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims (16)

1. A table generation method combining RPA and AI, characterized in that it is executed by an RPA system, and the method comprises: The RPA system extracts horizontal lines and vertical lines of the first table from the image based on artificial intelligence AI; The RPA system acquires a set of intersections of the horizontal line and the vertical line, wherein the set of intersections includes a first type of intersection formed by the intersection of the horizontal line and the vertical line, and a second type of intersection formed by the intersection of an extension line of the horizontal line and/or an extension line of the vertical line; The RPA system generates a blank second table consistent with the first table according to the set of intersection points; The RPA system fills the text entries recognized from the image based on optical character recognition (ORC) into the blank second table to obtain a target table; The RPA system generates a blank second table consistent with the first table according to the intersection set, including: The RPA system enumerates cells according to the intersection points in the intersection point set to obtain candidate cells and attribute information of the candidate cells; The RPA system sorts all the enumerated candidate cells in ascending order of cell area; The RPA system traverses the candidate cells in order and determines the existence of the traversed target candidate cells; Whenever the RPA system determines that the target candidate cell exists, the RPA system deletes cells that overlap with the target candidate cell from the candidate cells that have not been traversed, and determines the target candidate cell that is determined to exist as a target cell; The RPA system continues to traverse the candidate cells that have not been traversed after deletion in order until all the target cells are obtained after the traversal is completed; The RPA system generates the blank second table for the target cells according to position arrangement. 2. The method according to claim 1, characterized in that the RPA system traverses the candidate cells in order and judges the existence of the traversed target candidate cells, including: The RPA system obtains a first starting point and a second starting point of the horizontal line and a second starting point of the vertical line corresponding to the target candidate cell; The RPA system determines whether all four sides of the target candidate cell exist according to the four corner coordinates of the target candidate cell, the first starting and ending points, and the second starting and ending points; When the RPA system determines that all four edges exist, it determines that the target candidate cell exists. 3. The method according to any one of claims 1 to 2, characterized in that before the RPA system identifies the target cell for generating the blank second table from the candidate cells according to the attribute information of the candidate cells, it further comprises: The RPA system arranges the intersection points in the intersection point set in order, and groups adjacent intersection points in the intersection point set into a plurality of intersection point pairs in the same direction; The RPA system sequentially acquires adjacent pairs of intersections in the same direction, and forms basic cells from the adjacent pairs of intersections; The RPA system uses the basic cells as matrix elements to construct a Boolean matrix. 4. The method according to claim 3, characterized in that the RPA system generates the blank second table for the target cells according to position arrangement, comprising: The RPA system assigns a value to each matrix element in the Boolean matrix according to the target cell to generate a target Boolean matrix; The RPA system identifies whether the target Boolean matrix includes target rows and/or target columns that all have first values; When the target row and/or the target column exist, the RPA system splits the Boolean matrix according to the target row and/or the target column to generate a sub-target Boolean matrix; The RPA system obtains the target cells corresponding to the sub-target Boolean matrix, and arranges them according to positions to generate the blank second table corresponding to the sub-target Boolean matrix. 5. The method according to claim 4, characterized in that the RPA system assigns a value to each matrix element in the Boolean matrix according to the target cell to generate a target Boolean matrix, comprising: The RPA system obtains a first basic cell included in the target cell based on the four corner coordinates of the target cell and the four corner coordinates, and updates a matrix element corresponding to the first basic cell from the first value to a second value. 6. The method according to claim 4, characterized in that, when the target row and/or the target column exist, the RPA system splits the Boolean matrix according to the target row and/or the target column to generate a sub-target Boolean matrix, further comprising: The RPA system identifies whether there is a target matrix element whose value is the first value in the sub-target Boolean matrix; The RPA system updates the value of the target matrix element from the first value to a second value. 7. The method according to claim 6, characterized in that before the RPA system updates the value of the matrix element from the first value to the second value, it also includes: The RPA system obtains a target basic cell corresponding to the target matrix element; The RPA system determines whether the target basic cell needs to be merged into the target cell corresponding to the adjacent matrix element adjacent to the target matrix element; When the RPA system determines that merging is required, it merges the target basic cell into the target cell corresponding to the adjacent matrix element. 8. The method according to claim 7, characterized in that the method further comprises: If the RPA system determines that merging is not necessary, the target basic cell is determined as a missing target cell, and the missing target cell is completed when arranging the target cells according to positions. 9. The method according to any one of claims 1 to 2, characterized in that the RPA system fills the text entries recognized based on OCR from the image into the blank second table to obtain a target table, comprising: The RPA system obtains the four corner coordinates of the text entry; The RPA system matches the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain a target text entry corresponding to the target cell; The RPA system fills the target text entry into the corresponding target unit to obtain the target table. 10. The method according to claim 9, characterized in that the RPA system matches the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain the target text entry corresponding to the target cell, comprising: The RPA system matches the four corner coordinates of the text entry with the four corner coordinates of the target cell, and determines whether the text entry includes a first text entry occupying at least two target cells; The RPA system divides the first text entry according to the at least two occupied cells to obtain a target text entry for each target cell occupied by the first text entry. 11. A table generation device combining RPA and AI, characterized by comprising: An extraction module, used for extracting horizontal lines and vertical lines of the first table from the image based on artificial intelligence AI; An intersection acquisition module, used for acquiring a set of intersections of the horizontal line and the vertical line, wherein the set of intersections includes first-type intersections formed by the intersection of the horizontal line and the vertical line, and second-type intersections formed by the intersection of the extension lines of the horizontal line and/or the extension lines of the vertical line; A generating module, configured to generate a blank second table consistent with the first table according to the intersection point set; A filling module, used for filling the text entries recognized based on OCR from the image into the blank second table to obtain a target table; The generating module is further used for: Enumerating cells according to the intersection points in the intersection point set to obtain candidate cells and attribute information of the candidate cells; For all the enumerated candidate cells, sort them from small to large according to the cell area; Traversing the candidate cells in order, and judging the existence of the traversed target candidate cells; Whenever it is determined that the target candidate cell exists, cells overlapping with the target candidate cell are deleted from the candidate cells that have not been traversed, and the target candidate cell that is determined to exist is determined as a target cell; Continue to traverse the candidate cells that have not been traversed after deletion in order until all the target cells are obtained after the traversal is completed; The blank second table is generated by arranging the target cells according to their positions. 12. The device according to claim 11, characterized in that the generating module is further used for: Obtaining the first starting and ending points of the horizontal line and the second starting and ending points of the vertical line corresponding to the target candidate cell; According to the coordinates of the four corners of the target candidate cell, the first starting and ending points, and the second starting and ending points, determining whether all four sides of the target candidate cell exist; When it is determined that the four edges all exist, it is determined that the target candidate cell exists. 13. The device according to any one of claims 11 to 12, characterized in that the filling module is further used for: Get the four corner coordinates of the text entry; Matching the four corner coordinates of the text entry with the four corner coordinates of the target cell to obtain a target text entry corresponding to the target cell; The target text entries are filled into the corresponding target cells to obtain the target table. 14. An electronic device, comprising a memory and a processor; The processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the method according to any one of claims 1 to 10. 15. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented. 16. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.