CN115994230A - A smart file construction method integrating artificial intelligence and knowledge graph technology - Google Patents

Abstract

The invention discloses an intelligent archive construction method integrating artificial intelligence and knowledge graph technology, which relates to the technical field of computers and comprises 6 modules, namely a data storage module, an AI learning engine, a basic AI model, a business model, a knowledge graph and a front-end functional interface, wherein the modules are used for receiving and processing 3 types of digital archive format files, such as documents, images and videos; based on the deep learning model construction capability provided by a plurality of AI learning engines, a plurality of basic AI models are constructed, support is provided for the identification and analysis of key information in different types of data of documents, images and videos respectively, a plurality of business models suitable for the field are constructed based on the plurality of basic AI models and combined with the specific requirements of the intelligent archive field, so that the full analysis capability of multi-source heterogeneous archive data is realized, and then a plurality of atlas capabilities are constructed from analysis results based on a knowledge atlas technology and provided for archive users as interactable knowledge services.

Description

A smart file construction method integrating artificial intelligence and knowledge graph technology

technical field

The present invention relates to the field of computer technology, in particular to a method for constructing smart archives that integrates artificial intelligence and knowledge graph technology.

Background technique

With the deepening development of the digital economy, archival data is increasingly characterized by massive, multi-source, and heterogeneous features. This poses a huge challenge to the existing archives construction model, but it also provides opportunities for the development of new archival information management and service methods. opportunity. Smart archives have gradually evolved from traditional archives and digital archives. At present, artificial intelligence can be divided into two fields: perceptual intelligence and cognitive intelligence. The former mainly focuses on the content analysis of image, text, voice and video data, while the latter involves cognitive abilities such as knowledge reasoning and causal analysis. It is considered to be an important component technology of cognitive intelligence. Therefore, the construction of a smart archive based on artificial intelligence and knowledge graph technology provides a new idea for the analysis and archiving of massive multi-source heterogeneous data and the discovery of knowledge services from archive data.

Contents of the invention

The technical problem to be solved by the present invention is to solve the problem that the development of digital intelligence in the existing archives development model is insufficient, and it is difficult to deal with the problem that the archival data is increasingly characterized by massive multi-source heterogeneity. This patent proposes a fusion of artificial intelligence and knowledge graph. A technology-based smart archive construction method. This method first uses deep learning technology as a support to construct a multi-source heterogeneous archive data structured analysis service, and then builds a variety of graph capabilities from the analysis results based on knowledge graph technology, and serves as an interactive Knowledge services are provided to users of archives.

The present invention adopts the following technical solutions for solving the problems of the technologies described above:

A smart file construction method that integrates artificial intelligence and knowledge graph technology, including 6 modules: data storage, AI learning engine, basic AI model, business model, knowledge graph and front-end functional interface, used to receive and process documents, images and videos 3 types of digital archive format files;

Specifically, relying on the deep learning model construction capabilities provided by multiple AI learning engines, a variety of basic AI models have been constructed to provide support for the identification and analysis of key information in different types of data such as documents, images, and videos. Based on the model, combined with the specific needs of the field of smart archives, a variety of business models suitable for this field have been constructed to achieve sufficient analysis capabilities for multi-source heterogeneous archive data;

Step 1. Based on deep learning technology, build a key information analysis model for documents, images, audio and video multi-source heterogeneous data, and realize the structured analysis of multi-source heterogeneous archive data;

Step 2, the result data obtained after parsing is presented in a lightweight data exchange format JSON and data table format;

Step 3: Based on the knowledge map and its visualization technology, the analysis results are further integrated to form a variety of map knowledge service capabilities for users, thereby effectively improving the efficiency of users in discovering and utilizing archive resources.

As a further preferred solution of the smart file construction method of the present invention that integrates artificial intelligence and knowledge graph technology, the structured analysis process of document data is as follows:

Step A1, after the document data is preprocessed, it first performs seal detection and line text detection, separates the seal graphics and locates areas such as red heads, titles, text paragraphs, etc., and then performs text recognition on the target area where line text is detected;

Step A2, after summarizing the text recognition results into paragraphs, the specific text content of the document is formed;

Step A3, extracting a series of key fields of the document from the recognition result, including the full case number, document number, receiving unit, and signature, to form structured document information, and complete subsequent retrieval and information mining.

As a further preferred solution of the smart file construction method of the present invention that integrates artificial intelligence and knowledge graph technology, the structured analysis process of video data is as follows:

Step B1, key frame extraction:

Step B11, detecting and extracting key frames from the video;

In step B12, the shot segmentation algorithm is implemented based on the hash-aware algorithm, and the video stream is divided into image data;

Step B13, the key frame extraction module is more specifically divided into three sub-modules such as subtitle detection, famous person face detection and voice activity detection, which are respectively used to extract subtitle frames, human face frames or voice activity time segments from the video stream;

In step B14, the extracted key frames are respectively sent to subsequent corresponding processing modules for further analysis;

Step B2, face recognition: by connecting several face image processing sub-modules in series, it realizes the recognition of relevant leaders in the photos of leaders attending events, and the ability to classify images such as certificates, bills, landscapes, buildings, etc.; based on target detection The algorithm YOLO model detects the face of the image, and uses an algorithm FaceNet tool that converts the face image into a feature vector to extract the face code from the target area where the face is detected, and then compares it with the leader's face database. By comparing the pre-stored data for similarity, it is possible to determine whether there is a leader that the user cares about in the image;

Step B3, subtitle OCR: including scene text detection CTPN and text recognition Densenet, used to realize the positioning, sorting, identification and arrangement of the subtitle area in the video screen, and extract specific news events or conference report information from the video; use based on A text classification model implemented by TextCNN, a convolutional neural network-based text classification algorithm, identifies the subject of the video content and forms hot topic tags;

Step B4, speech recognition:

Step B41, the video stream is converted into an audio stream, and the voice segment in the audio stream is extracted by a voice activity detection VAD algorithm;

Step B42, the audio is recognized as a corresponding phoneme sequence by the speech recognition model based on the chain-type time-delay neural network ChainTDNN network model,

In step B43, the phoneme sequence is decoded based on the language model of the trigram 3-gram, and a text result that can be read by the user is obtained.

As a further preferred solution of a smart file construction method that integrates artificial intelligence and knowledge graph technology in the present invention, the structured analysis process of image file data is as follows:

Step C1, after performing a series of preprocessing measures on the image data uploaded by the user, perform face detection on it; wherein, a series of preprocessing measures include unifying the file format, image size, and color coding;

Step C2, if a face is detected, vectorize the target area based on Facenet technology to obtain a face code, and perform similarity between the obtained face code and the pre-stored leader face standard code data in the leader database Degree comparison to determine whether there is a leader followed by the user in the picture;

Step C3, if the leader is successfully identified, output the list of leaders; and if the leader is not successfully identified, then go to the image object classification module, and perform the following five common image tags in archives: certificates, bills, buildings, landscapes and others for easy archiving of pictures.

As a further preferred solution of the smart file construction method of the present invention that integrates artificial intelligence and knowledge graph technology, the preprocessing of document data includes inclination detection and correction, seal detection and removal, grayscale, binarization, and noise removal;

Among them, (1) inclination detection and correction:

The Hough transform is used to correct the image data of the archives; the Hough transform mainly adopts the method of polar coordinate transformation, mapping the point (x, y) in the conventional Cartesian coordinate system to the point (ρ, θ) in the Hough space, in:

ρ＝x·cosθ+y·sinθ

The Hough transform will calculate all possible (ρ, θ) values for each edge point on the image, and connect these polar coordinate points into a curve. When enough curves intersect at a certain point (ρ, θ) in Hough space ), it can be considered that (ρ, θ) corresponds to a straight line in the image space position, and after position mapping, the position and inclination angle of the straight line in the original image can be obtained, and the corrected coordinate point (x' ,y');

(2) Seal detection and removal:

Using YOLO-based target detection technology, extract the center coordinates, radius, coordinate frame and page number of the stamp that may appear in the document, and then remove the stamp graphics in this area by means of channel threshold;

(3) Grayscale: convert the color image into a black and white image and then process it to effectively remove redundant information in the data, that is, grayscale. The image is converted from a conventional three-channel image to a single-channel image through a weighted average algorithm It is a grayscale image, and the calculation formula is as follows:

Gray(x,y)＝0.229*R(x,y)+0.587*G(x,y)+0.114*B(x,y)

Among them, Gray(x,y), R(x,y), G(x,y), B(x,y) are the gray value, red, green and blue value of the pixel respectively;

(6) Binarization: binarize the grayscale image, that is, divide the pixels of the image into two parts by setting a threshold, the grayscale value of some pixels is greater than the threshold value, and the grayscale value of the other part of pixels is less than the threshold;

The area composed of pixels judged to be 1 according to the gray value is considered to be a recognizable character graphic;

The grayscale image of the document to be processed is binarized by the method of iterative threshold segmentation. First, the initial value of the binarization threshold is set as the average value of the grayscale value of the image, and then the appropriate threshold is found through several rounds of iterations. The final value, and then binarize the grayscale image;

(7) Noise removal: The noise in the image to be processed is removed by mathematical morphology filtering, which can effectively improve the recognition accuracy of the document.

(8) As a further preferred solution of the smart archive construction method that integrates artificial intelligence and knowledge graph technology in the present invention, the OCR of archive files is divided into two steps: line text detection and line text recognition: using a method based on differentiable binarization DB text detection method and text recognition method based on CRNN network;

Text detection method: through the text detection module, the coordinates of the text area in the document are obtained, and marked in the form of a bounding box, which facilitates the subsequent text recognition steps; the DB algorithm based on segmentation is adopted, and the text line is used as the segmentation target , and give the segmentation result to build a text box, and then get a line text box; in the implementation, the text classification algorithm ResNet34_vd based on the convolutional neural network is used as the feature extraction layer of the backbone network, followed by the feature map pyramid FPN layer to ensure the capture of features The diversity and robustness of the model, and a 3-layer convolutional network as the feature prediction component of the model;

(3) Text recognition method: the function of recognizing the character graphics existing in the target word box as the corresponding text string: using the CRNN network based on the convolutional cyclic neural network as the backbone network of this module, and using a mobile terminal The lightweight neural network architecture MobileNetV3 network of the device performs preliminary feature extraction, and then uses several layers of bidirectional long-short-term memory BiLSTM network to output the label probability distribution output at each character position, and optimizes the output distribution through the connection time series classification CTC algorithm train.

As a further preferred solution of the smart file construction method that integrates artificial intelligence and knowledge graph technology in the present invention, through the text processing module and the document information extraction business model, the list of character strings is converted into natural paragraphs that are easy for users to read, and automatically extracted Key fields are extracted to form a structured text information analysis result;

Among them, (1) rule extraction: through the understanding of specific archives, the system needs to set several key fields in advance;

(2) Character calculation: In addition, there are also several fields in the archive document that can be judged by analyzing the layout features;

(3) Integration output: Finally, summarize the results given by the above system flow, and integrate the results of the same fields.

As a further preferred solution of the smart file construction method that integrates artificial intelligence and knowledge graph technology in the present invention, in step 3, the knowledge graph construction process is specifically divided into four sub-sections: entity recognition, relationship extraction, entity alignment, and knowledge reasoning. module;

Entity recognition layer: Entities are the basis of knowledge graphs, and entity recognition is the basic task of building knowledge graphs. Good entity recognition results can significantly improve the quality of knowledge graph construction results; entity recognition models based on deep learning networks can achieve using only a small amount of Construct a training set from manually labeled data, and use the BERT model to extract sentence vectors to train an entity recognition model with BiLSTM+CRF as the backbone network;

Relationship extraction layer: According to the identified entity results and full-text data, the relationship between entities is further extracted; the full-text data is word-segmented and sent to the CBoW model to extract word vectors, and then sent to the BiLSTM model to extract context-related semantic features Coding, the model introduces the attention mechanism, automatically adjusts the attention weight parameters between the sample nodes according to the sample data, and forms the relationship information between the entity nodes;

Entity alignment layer: The system makes full use of full-text data, abstracts, and attribute data such as topics, keywords, leaders, time, sources, types, etc., and sends them into the BERT model to convert them into vector representations; the full-text entities are calculated separately between the archive sample data Cosine similarity and entity summary cosine similarity, and by combining the two similarities to judge the semantic similarity between archive sample data; input multiple similarity indicators into the multi-feature fusion model built by the SENet layer to output entity alignment The result; all the archive data can form a set of aligned entities after being predicted by the entity alignment model; the entities and their relationships given by the relationship extraction layer are aligned using the aligned entity set;

Knowledge reasoning layer: After the processing of the previous entity recognition layer, relationship extraction layer and entity alignment layer, the data has formed a triplet of <head entity-relationship-tail entity> structure; the knowledge map construction system also needs to carry out knowledge reasoning to Enrich and enrich the relationship between entities, and solve the problem that the relationship between entities based on original data is relatively sparse.

Compared with the prior art, the present invention adopts the above technical scheme and has the following technical effects:

This patent proposes a new smart file construction method that integrates artificial intelligence and knowledge graph technology. Supported by deep learning technology and relying on the deep learning model construction capabilities provided by various AI learning engines, a variety of basic AI models are constructed. , image and video data to provide support for the identification and analysis of key information in different types of data, and based on these basic AI models, combined with the specific needs of the field of smart archives, a variety of business models suitable for this field have been constructed to achieve multi-to-many Sufficient analytical capabilities for source heterogeneous archival data; construct multi-source heterogeneous archival data structured analysis services, and then construct various graph capabilities from the analysis results based on knowledge graph technology, and provide them to archive users as interactive knowledge services .

Description of drawings

Fig. 1 is a flow chart of a method for constructing smart archives that integrates artificial intelligence and knowledge graph technology according to the present invention;

Fig. 2 is a flow chart of document structure parsing in the present invention;

Fig. 3 is a flow chart of video structured analysis in the present invention;

Fig. 4 is the image analysis flowchart of the present invention combining face recognition and object classification;

Fig. 5 is the knowledge graph and its visualization processing flow of the present invention;

Fig. 6 is the construction process of the knowledge map of the present invention;

Fig. 7 is the document analysis function user interface of the present invention;

Fig. 8 is the video analysis function user interface of the present invention;

Fig. 9 is the image analysis function user interface of the present invention;

Fig. 10 is the knowledge map display interface of the present invention.

Detailed ways

Below in conjunction with accompanying drawing, technical scheme of the present invention is described in further detail:

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

As shown in Figures 1 to 10, the purpose of this patent is to solve the problem of insufficient development of digital intelligence in the development of existing archives under the background of the deepening development of the digital economy, and it is difficult to deal with the problem of collation and utilization of massive multi-source heterogeneous archive data . This patent proposes a new smart file construction method that integrates artificial intelligence and knowledge graph technology. First, build a key information analysis model for multi-source heterogeneous data such as documents, images, audio and video based on deep learning technology, and realize the structured analysis capability for multi-source heterogeneous archive data; format, and then further integrate these analytical results based on knowledge graph technology to form a variety of graph knowledge service capabilities for users, which can effectively improve the efficiency of users in discovering and utilizing archive resources.

The overall structure of the smart file construction method that integrates artificial intelligence and knowledge graph technology is shown in Figure 1. The specific steps are as follows:

As can be seen from Figure 1, the smart archive construction method proposed by this patent that integrates artificial intelligence and knowledge graph technology mainly consists of six modules: data storage, AI learning engine, basic AI model, business model, knowledge graph, and front-end functional interface. It can receive and process three types of digital archive format files: document, image and video. Among them, the processing of documents is mainly aimed at the most common official documents and newspaper documents in the archival data. The processing of images combines the capabilities of face recognition and object classification, while the processing of video files is mainly aimed at news report videos. Structured analysis. Specifically, the method proposed in this patent first relies on the deep learning model construction capabilities provided by multiple AI learning engines to build a variety of basic AI models, which provide information for the identification and analysis of key information in different types of data such as documents, images, and videos. Support; further, based on the above-mentioned various basic AI models, combined with the specific needs of the field of smart archives, this patent designs and builds a variety of business models applicable to this field, in order to fully realize the multi-source heterogeneous archive data analytical ability.

The structured parsing process of document data is shown in Figure 2. After the document data is preprocessed, it first performs seal detection and line text detection, separates the seal graphics and locates areas such as red heads, titles, and text paragraphs, and then performs text recognition on the target area where line text is detected. After summarizing the text recognition results into paragraphs, the specific text content of the document is formed. In addition, a series of key fields of the document need to be extracted from the recognition results, such as the full case number, document number, receiving unit, signature, etc., to form structured document information to facilitate subsequent retrieval and information mining.

The archives can track the city's party and government dynamics and current hot news in a timely manner by collecting audio and video materials such as leaders' activity reports, conference news, and policy propaganda. After the system obtains audio and video materials, it can slice the data based on news events and quickly locate specific event segments in the video. In order to obtain the specific content of the news event, the system needs to extract the corresponding subtitle information from the video screen, and recognize the narration broadcast information from the video voice to form the broadcast content of the news. Then, the content classification label of the news can be obtained by classifying the recognized text. Finally, in order to obtain which leaders attended the meeting activities in the news event, in addition to extracting keywords from the recognition text results, this system also introduces face detection, coding and recognition technology, by comparing the leaders stored in the database in advance People coding can extract the leaders appearing in the video.

As can be seen from Figure 3, the workflow of the video parsing module is as follows:

(1) Key frame extraction: In order to facilitate the extraction of various key information in the video data, the system first needs to know the time information of these key information (that is, locate the "key frame") before further processing the information on the screen. Therefore, the first step of the video information structure module is to detect and extract key frames of the video. This system implements the lens segmentation algorithm based on the hash perception algorithm, and divides the video stream into image data. For different detection targets, the key frame extraction module can be more specifically divided into three sub-modules: subtitle detection, famous person face detection and voice activity detection, which are used to extract subtitle frames, face frames or voice from video streams respectively. Active time period. The extracted key frames are respectively sent to subsequent corresponding processing modules for further analysis.

(2) Face recognition: This module realizes the recognition of relevant leaders in the photos of leaders attending activities by connecting several face image processing sub-modules in series, and the ability to classify images such as certificates, bills, landscapes, and buildings. First, face detection is performed on the image based on the YOLO (YouOnlyLookOnce, a target detection algorithm) model, and an algorithm Facenet tool that converts the face image into a feature vector is used to extract the face code for the target area where the face is detected, and then By comparing the similarity with the pre-stored data in the leader's face database, it can be determined whether there is a leader that the user cares about in the image.

(3) Subtitle OCR: This module is composed of scene text detection CTPN, text recognition Densenet and other sub-modules, which can realize the positioning, sorting, identification and arrangement of the subtitle area in the video screen, and can extract specific news events from the video or conference coverage information. Further, the text classification model based on the convolutional neural network-based text classification algorithm TextCNN can be used to identify the subject of the video content and form hot topic tags.

(4) Speech recognition: This module first converts the video stream into an audio stream, extracts the speech segment in the audio stream through the voice activity detection VAD algorithm, and recognizes the audio as The corresponding phoneme sequence is then decoded using a 3-gram-based language model to obtain a text result that can be read by the user. In addition, some post-processing sub-modules can be added, such as the punctuation recovery model based on the BiLSTM network, to further improve the readability of the text results.

The process steps of analyzing the image file data are shown in Figure 4. After a series of preprocessing measures (unification of file format, image size, color coding, etc.) face detection. If a face is detected, the target area is vectorized based on Facenet technology to obtain a face code, and the obtained face code is compared with the pre-stored leader face standard code data in the leader database. To determine whether there is a leader followed by the user in the picture. If the leader is successfully identified, the list of leaders will be output; if the leader is not successfully identified, then go to the image object classification module to perform the following five common image tags in archives: certificates, bills, buildings, landscapes, and others. Easy to archive pictures.

As shown in Figure 5, based on the information extraction module for images, documents, and audio and video materials designed above, the archives design a knowledge map service platform for archives data that integrates the extracted structured information into a graphic network, solving the problem of archives Library users deal with the difficulties of multi-source heterogeneous data. The knowledge map construction system needs to preprocess a variety of data accordingly, and extract structured information such as text content, text summaries, hot topics, keywords, reporting time, and involved leaders from image, document, and video data. These processes have been introduced separately above. Then the system will perform named entity recognition on these structured data, further extract leaders, institutional units, place names, proper nouns and other entities, and establish the relationship between each entity. Entities with similar actual meanings may have large differences in names, so it is necessary to calculate the similarity between entities and perform fusion and normalization processing on the entities. Finally, infer, expand and complete the entities and the relationships between entities, and complete the construction process of the multi-source heterogeneous data knowledge map of the archives.

Due to the diversity of document styles and various problems that may arise during digital processing, before analyzing document data, preprocessing is required to standardize the format of the data to reduce the probability of system analysis errors. Specifically, the preprocessing module for document data consists of the following submodules: inclination detection and correction, seal detection and removal, grayscale, binarization, and noise removal, etc.

(1) Inclination detection and correction: The document data stored in the archives will inevitably have incorrect image inclination, which affects the quality of text content recognition to a certain extent. Therefore, the image preprocessing stage first needs to correct the angle of the image. This system adopts Hough transform to correct the image data of the archives; Hough transform mainly adopts the method of polar coordinate conversion, and maps the point (x, y) in the conventional Cartesian coordinate system to the point (ρ, θ) in the Hough space ),in:

ρ=x·cosθ+y·sinθ

Since the edge detection of the document image is required before the Hough transform, the Hough transform will calculate all possible (ρ, θ) values for each edge point on the image, and connect these polar coordinate points into a curve, when enough When many curves intersect at a certain point (ρ, θ) in Hough space, it can be considered that (ρ, θ) corresponds to a straight line in the image space position, and after position mapping, the position of the straight line in the original image can be obtained and the tilt angle, thus obtaining the corrected coordinate point (x', y');

(2) Seal detection and removal: The seal graphics that often appear in archives will cover part of the characters to form a block, which is not good for character recognition. Therefore, it is necessary to automatically detect and remove the stamp part in the system. The smart archive system uses YOLO-based target detection technology to extract the center coordinates, radius, coordinate frame and page number of the seal that may appear in the document, and then removes the seal graphics in this area based on the channel threshold. Taking the common red stamp as an example, extract the red channel in the target area to obtain the gray value of the red channel, and use the triangulation method to calculate the maximum distance of the statistical histogram of the gray image to determine the optimal threshold for image segmentation. Then pass the threshold to remove the red stamp.

(3) Grayscale: The background model of the system does not directly input color images, but converts color images into black and white images for processing, which can effectively remove redundant information in the data. This process is grayscale. The image is converted from a conventional three-channel image to a single-channel image (also called a grayscale image) through a weighted average algorithm. The calculation formula is as follows:

Gray(x,y)＝0.229*R(x,y)+0.587*G(x,y)+0.114*B(x,y)

Among them, Gray(x, y), R(x, y), G(x, y), B(x, y) are the gray value, red, green and blue value of the pixel, respectively.

(4) Binarization: In order to make the various features in the grayscale image more obvious, the system further performs binary processing on the grayscale image, that is, divides the pixels of the image into two parts by setting a threshold, and one part The gray value of the pixel is greater than the threshold, while the gray value of another part of the pixel is smaller than the threshold. The area formed by pixels whose gray value is determined to be 1 is considered to be a recognizable character graphic. This system adopts the iterative threshold segmentation method to binarize the gray image of the document to be processed. First, the average value of the image gray value is set as the initial value of the binarization threshold, and then through several rounds of iterations to find the appropriate threshold value. The final value, and then binarize the grayscale image.

(5) Noise removal: Due to the age of some archives, there are obvious creases or stains before digitization, and the overall or partial resolution of the scanned image is low, which is not conducive to subsequent OCR processing. This system uses mathematical morphology filtering to remove the noise in the image to be processed, which can effectively improve the accuracy of document recognition.

The OCR of archive files can be divided into two steps: line text detection and line text recognition. This system adopts the text detection method based on the differentiable binarization algorithm DB and the text recognition method based on the CRNN network.

OCR for document data is different from subtitle OCR. The biggest difference between the two is that documents generally have multiple lines of text, while subtitles usually only have a single line of text in most cases.

(1) Text detection: Through the text detection module, the system can obtain the coordinates of the text area in the document, and can mark it in the form of a bounding box, which facilitates the subsequent text recognition steps. This system adopts the DB algorithm based on segmentation, takes the text line as the segmentation target, and constructs a text box based on the segmentation result, and then obtains the text box of the line. In the implementation, the feature extraction layer based on ResNet34_vd is used as the backbone network, followed by the Feature Map Pyramid (FPN) layer to ensure the diversity and robustness of the captured features, and a 3-layer convolutional network is used as the feature prediction component of the model. (2) Text recognition: This module realizes the function of recognizing the character graphics existing in the target word box as corresponding text strings. Using a convolutional recurrent neural network (Convolutional Recurrent Neural Network, CRNN) network as the backbone network of this module, a lightweight neural network architecture MobileNetV3 network suitable for mobile devices is used for preliminary feature extraction, and then several layers of bidirectional length The time-memory BiLSTM network outputs the label probability distribution output at each character position, and optimizes the output distribution by connecting the time series classification CTC algorithm; this model naturally integrates the respective advantages of convolutional neural networks and cyclic neural networks, and fully analyzes The spatial graphic features and contextual semantic connections of the image to be recognized are obtained, and the problem of automatic label alignment between the line text image and the string sequence is solved. This model design naturally combines the respective advantages of convolutional neural networks and cyclic neural networks, fully analyzes the spatial graphic features and contextual semantic connections of images to be recognized, and solves the problem of automatic labeling between line text images and string sequences Alignment issues.

What the OCR module identified in the previous step is a list of several character strings. Next, some text processing modules and document information extraction business models need to be used to convert the list of strings into natural paragraphs that are easy for users to read, and automatically extract key fields from them to form a structured text information analysis result. (1) Rule extraction: Through the understanding of specific archives, the system must set up several key fields in advance, such as "issuer", "receiver", "document number", "signature", etc.; (2) Character calculation: In addition, There are also several fields in the archive document that can be judged by analyzing the layout characteristics, such as "red head", "headline", "text", etc.; (3) Integrated output: finally, summarize the results given by the above system process, and combine the same fields results are integrated.

The functional interface of the archives system for processing document data is shown in Figure 7. Click "Select File" to upload the document data to be processed. Currently, the system supports the processing of files in PDF, JPEG, PNG and TIF formats. After the upload of the document is completed, the text of the document will appear at the bottom of the page, where users can browse the document in real time. The document browsing function of this system is developed based on the PDF.js library. Compared with the native iFrame tag of the web page, PDF.js is more stable and beautiful, and can realize additional functions such as page zooming and real-time rendering. Click "Start Analysis" to call the background service to analyze the document in real time. The parsing result is displayed on the right side of the page, and the key field information extracted from the document is displayed in the form of a table. As shown in Figure 7, in the text display area, the system marks the different paragraphs of the document with boxes based on the coordinate results obtained by the background analysis based on the rendering function of PDF.js, so that users can clearly and intuitively distinguish different contents in the document , so that you can quickly locate the target area you want to browse.

The video analysis interface of the archives system is shown in Figure 8. Click the "Upload" button to upload the video file. After the file upload is displayed, click "Start Analysis" to analyze the file. As shown in Figure 8, this module has realized the structured extraction function of multiple key data in the video. The system automatically intercepts the screen most relevant to the video content as the cover, and generally selects screens with relevant leaders or news titles and subtitles to capture, so that users can quickly understand the main content of the video file. Below the screenshot, the interface also displays the key information of the video, such as classification of hot topics, content summary, and the names of recognized leaders. If there are multiple news reports in the video, the system will also automatically divide them, so that users can browse different reports separately. After the original video data is processed by the above three key modules, the extracted key information is saved in JSON format, including fields such as leader information, hot topic classification, and video content summary. In order to make full use of the analysis results, the smart archive system will further build a knowledge map based on these JSON data, integrate the original isolated data into an interrelated information map and present it to users, so that users can understand the existing data from a holistic perspective.

The construction process of the knowledge map can be subdivided into four sub-modules: entity recognition, relationship extraction, entity alignment and knowledge reasoning.

(1) Entity recognition layer: Entities are the basis of knowledge graphs, and entity recognition is the basic task of building knowledge graphs. Good entity recognition results can significantly improve the quality of knowledge graph construction results. This module implements an entity recognition model based on a deep learning network, which can use only a small amount of manually labeled data to construct a training set, and use the BERT model to extract sentence vectors and then train entities with bidirectional long-short-term memory BiLSTM+ conditional random field CRF as the backbone network recognition model; .

(2) Relationship extraction layer: According to the identified entity results and full-text data, the system will further extract the relationship between entities. Segment the full-text data and send it to the continuous bag-of-words model CBoW model to extract word vectors, and then send it to the BiLSTM model to extract context-related semantic feature codes. The model introduces an attention mechanism to automatically adjust the mutual attention between sample nodes according to the sample data. The force weight parameter forms the relationship information between entity nodes.

(3) Entity alignment layer: the system makes full use of full-text data, abstracts, and attribute data such as topics, keywords, leaders, time, sources, types, etc., and sends them to a transformer-based bidirectional encoder to represent BERT models and convert them into vector representations . The entity full-text cosine similarity and the entity abstract cosine similarity are calculated respectively between the archive sample data, and the semantic similarity between the archive sample data is judged by combining the two similarities. Then, multiple similarity indicators are input into a multi-feature fusion model built by a neural network structure SENet (Squeeze-and-Excitation Networks) layer originally used for image recognition to output the result of entity alignment. After all the archive data are predicted by the entity alignment model, a set of aligned entities can be formed. Finally, the entities and their relationships given by the relationship extraction layer are aligned using the aligned entity set.

(4) Knowledge reasoning layer: After the processing of the previous entity recognition layer, relationship extraction layer and entity alignment layer, the data has formed a triplet of <head entity-relationship-tail entity> structure. At this time, the knowledge graph construction system also needs to perform knowledge reasoning to enrich and enrich the relationship between entities, and solve the problem that the relationship between entities based on the original data is relatively sparse.

Based on the parsed document, video, and image data, the archives system constructs a knowledge map, and draws it into an information map and presents it on the user interface. The processing flow of the knowledge map module is shown in Figure 5. Among them, for the three types of archival knowledge graphs constructed by the system: single leader, multi-leader and hot topic, the front end of the system visualizes them and transforms them into "leader event relationship axis" and "leader accompanying event relationship axis" respectively. There are 3 kinds of information graphs in "Event Graph". The user interface style is shown in Figure 10. Among them, the first two are in the form of time axis, showing the basic information of the target leader and the recent dynamic compilation, or the joint attendance of the leaders in the meeting activities. The latter is a compilation of leaders' dynamics, conference activities, policy promotion and other events under a specific hot topic. These infographics are all interactive, users can click to browse event details, and can also jump between events, which is convenient for users to have a more detailed understanding of related matters.

(1) Leader event relationship axis: It mainly presents the relevant information of a single leader. Enter the leader's name as a keyword to retrieve the basic information of the leader, including his recent attendance at a series of meetings and various activities. The information is displayed in the order of the time axis, which allows the user to browse the information he wants to know very conveniently and quickly. The map here shows a compilation of recent major activities of a leader of City S according to the timeline, and also presents important information such as the time, place, and related people of each activity.

(2) Leader-accompanied-event relationship axis: It mainly presents the compilation of materials on meetings and activities attended by multiple leaders. The information is also displayed in the order of the time axis.

(3) Event graph: The system can retrieve policy documents or event reports under hot topics based on hot topic tags. Not only a series of detailed information about this event is listed here, but related events can also be displayed below the event details Atlas.

Claims (8)

1. A smart file construction method integrating artificial intelligence and knowledge graph technology, characterized in that it includes 6 modules: data storage, AI learning engine, basic AI model, business model, knowledge graph and front-end function interface, used to receive and Handle 3 types of digital archive format files: document, image and video; Specifically, relying on the deep learning model construction capabilities provided by multiple AI learning engines, a variety of basic AI models have been constructed to provide support for the identification and analysis of key information in different types of data such as documents, images, and videos. Based on the model, combined with the specific needs of the field of smart archives, a variety of business models suitable for this field have been constructed to achieve sufficient analysis capabilities for multi-source heterogeneous archive data; Step 1. Based on deep learning technology, build a key information analysis model for documents, images, audio and video multi-source heterogeneous data, and realize the structured analysis of multi-source heterogeneous archive data; Step 2, the result data obtained after parsing is presented in the format of data exchange format JSON and data table; Step 3: Based on the knowledge map and its visualization technology, the analysis results are further integrated to form a variety of map knowledge service capabilities for users, thereby effectively improving the efficiency of users in discovering and utilizing archive resources. 2. A smart file construction method that integrates artificial intelligence and knowledge graph technology according to claim 1, characterized in that: the structured analysis process of document data is as follows: Step A1, after the document data is preprocessed, it first performs seal detection and line text detection, separates the seal graphics and locates areas such as red heads, titles, text paragraphs, etc., and then performs text recognition on the target area where line text is detected; Step A2, after summarizing the text recognition results into paragraphs, the specific text content of the document is formed; Step A3, extracting a series of key fields of the document from the recognition result, including the full case number, document number, receiving unit, and signature, to form structured document information, and complete subsequent retrieval and information mining. 3. A method for constructing smart archives that integrates artificial intelligence and knowledge graph technology according to claim 1, characterized in that: the structured analysis process of video data is as follows: Step B1, key frame extraction: Step B11, detecting and extracting key frames from the video; In step B12, the shot segmentation algorithm is implemented based on the hash-aware algorithm, and the video stream is divided into image data; Step B13, the key frame extraction module is more specifically divided into three sub-modules such as subtitle detection, famous person face detection and voice activity detection, which are respectively used to extract subtitle frames, human face frames or voice activity time segments from the video stream; In step B14, the extracted key frames are respectively sent to subsequent corresponding processing modules for further analysis; Step B2, face recognition: by connecting several face image processing sub-modules in series, it realizes the recognition of relevant leaders in the photos of leaders attending events, and the ability to classify images such as certificates, bills, landscapes, buildings, etc.; based on target detection The algorithm YOLO model detects the face of the image, and uses an algorithm FaceNet tool that converts the face image into a feature vector to extract the face code from the target area where the face is detected, and then compares it with the leader's face database. By comparing the pre-stored data for similarity, it is possible to determine whether there is a leader that the user cares about in the image; Step B3, subtitle OCR: including scene text detection CTPN and text recognition Densenet, used to realize the positioning, sorting, identification and arrangement of the subtitle area in the video screen, and extract specific news events or conference report information from the video; use based on The text classification model implemented by the convolutional neural network text classification algorithm TextCNN identifies the subject of the video content and forms hot topic tags; Step B4, speech recognition: Step B41, the video stream is converted into an audio stream, and the voice segment in the audio stream is extracted by a voice activity detection VAD algorithm; Step B42, the audio is recognized as a corresponding phoneme sequence by a speech recognition model based on a chained time-delay neural network Chain TDNN network model, Step B43, decode the phoneme sequence based on the 3-gram language model of the trigram model, and obtain a text result that can be read by the user. 4. A method for constructing smart archives that integrates artificial intelligence and knowledge graph technology according to claim 1, characterized in that: the structured analysis process of image archive data is as follows: Step C1, after performing a series of preprocessing measures on the image data uploaded by the user, perform face detection on it; wherein, a series of preprocessing measures include unifying the file format, image size, and color coding; Step C2, if a face is detected, the target area is vectorized based on Facenet technology to obtain a face code, and the obtained face code is similar to the pre-stored leader face standard code data in the leader database Degree comparison to determine whether there is a leader followed by the user in the picture; Step C3, if the leader is successfully identified, output the list of leaders; and if the leader is not successfully identified, go to the image object classification module to identify the following five common image tags in archives: certificates, bills, buildings , landscape, and others for easy archiving of pictures. 5. A smart file construction method that integrates artificial intelligence and knowledge graph technology according to claim 2, characterized in that: the preprocessing of document data includes inclination detection and correction, seal detection and removal, grayscale, binary and noise removal; Among them, (1) inclination detection and correction: The Hough transform is used to correct the image data of the archives; the Hough transform mainly adopts the method of polar coordinate transformation, mapping the point (x, y) in the conventional Cartesian coordinate system to the point (ρ, θ) in the Hough space, in: ρ=x·cosθ+y·sinθ The Hough transform will calculate all possible (ρ, θ) values for each edge point on the image, and connect these polar coordinate points into a curve. When enough curves intersect at a certain point (ρ, θ) in Hough space ), it can be considered that (ρ, θ) corresponds to a straight line in the image space position, and after position mapping, the position and inclination angle of the straight line in the original image can be obtained, and the corrected coordinate point (x' ,y'); (2) Seal detection and removal: Using YOLO-based target detection technology, extract the center coordinates, radius, coordinate frame and page number of the stamp that may appear in the document, and then remove the stamp graphics in this area by means of channel threshold; (3) Grayscale: convert the color image into a black and white image and then process it to effectively remove redundant information in the data, that is, grayscale. The image is converted from a conventional three-channel image to a single-channel image through a weighted average algorithm It is a grayscale image, and the calculation formula is as follows: Gray(x,y)＝0.229*R(x,y)+0.587*G(x,y)+0.114*B(x,y) Among them, Gray(x,y), R(x,y), G(x,y), B(x,y) are the gray value, red, green and blue value of the pixel respectively; (4) Binarization: binarize the grayscale image, that is, divide the pixels of the image into two parts by setting a threshold, the grayscale value of some pixels is greater than the threshold value, and the grayscale value of the other part of pixels is less than the threshold; The area composed of pixels whose gray value is judged to be 1 is considered to be a recognizable character graphic; The grayscale image of the document to be processed is binarized by the method of iterative threshold segmentation. First, the initial value of the binarization threshold is set to the average value of the grayscale value of the image, and then the appropriate final value is found through several rounds of iterations. value, and then binarize the grayscale image; (5) Noise removal: The mathematical morphology filter is used to remove the noise in the image to be processed, which can effectively improve the recognition accuracy of the document. 6. A smart file construction method that integrates artificial intelligence and knowledge graph technology according to claim 1, characterized in that: the OCR of file files is divided into two steps: line text detection and line text recognition: a differentiable based The text detection method of binary DB and the text recognition method based on CRNN network; Text detection method: through the text detection module, the coordinates of the text area in the document are obtained, and marked in the form of a bounding box, which facilitates the subsequent text recognition steps; the DB algorithm based on segmentation is adopted, and the text line is used as the segmentation target , and give the segmentation result to construct the text box, and then get the line text box; in the implementation, an improved residual network model ResNet34_vd is used as the feature extraction layer of the backbone network, followed by the feature map pyramid FPN layer to ensure the capture of features The diversity and robustness of the model, and a 3-layer convolutional network as the feature prediction component of the model; (2) Text recognition method: the function of recognizing the character graphics existing in the target word box as the corresponding text string: the convolutional cyclic neural network CRNN is used as the backbone network of this module to realize, and a mobile terminal device is used. The lightweight neural network architecture MobileNetV3 network performs preliminary feature extraction, and then uses several layers of bidirectional long-short-term memory BiLSTM network to determine the output label probability distribution at each character position, and at the same time optimizes the output distribution through the connection sequence classification CTC algorithm; The above approach naturally combines the respective advantages of convolutional neural networks and cyclic neural networks, fully analyzes the spatial graphic features and contextual semantic connections of images to be recognized, and solves the problem of automatic label alignment between line text images and string sequences . 7. A method for constructing smart archives that integrates artificial intelligence and knowledge graph technology according to claim 6, characterized in that: through the text processing module and the document information extraction business model, the list of character strings is converted into a natural file that is convenient for users to read. Paragraphs, and key fields are automatically extracted to form structured text information analysis results; Among them, (1) rule extraction: through the understanding of specific archives, the system should set several key fields in advance; (2) Character calculation: In addition, there are also several fields in the archive document that can be judged by analyzing the layout features; (3) Integration output: Finally, summarize the results given by the above system flow, and integrate the results of the same fields. 8. A smart file construction method that integrates artificial intelligence and knowledge graph technology according to claim 6, characterized in that: in step 3, the construction process of the knowledge graph is specifically divided into entity recognition, relationship extraction, and entity alignment There are 4 sub-modules in total and knowledge reasoning; Entity recognition layer: Entities are the basis of knowledge graphs, and entity recognition is the basic task of building knowledge graphs. Good entity recognition results can significantly improve the quality of knowledge graph construction results; entity recognition models based on deep learning networks can achieve using only a small amount of Construct a training set from manually labeled data, and use the BERT model to extract sentence vectors to train an entity recognition model with bidirectional long-short-term memory BiLSTM+ conditional random field CRF as the backbone network; Relationship extraction layer: According to the identified entity results and full-text data, further extract the relationship between entities; after word segmentation, the full-text data is sent to the continuous bag-of-words model CBoW to extract word vectors, and then sent to the BiLSTM model to extract context Correlation semantic feature coding, the model introduces the attention mechanism, automatically adjusts the mutual attention weight parameters between sample nodes according to the sample data, and forms the relationship information between entity nodes; Entity alignment layer: The system makes full use of full-text data, abstracts, and attribute data such as topics, keywords, leaders, time, sources, types, etc., and sends them to the transformer-based bidirectional encoder to represent BERT models into vector representations; The cosine similarity of the entity full text and the cosine similarity of the entity summary are calculated separately between the sample data, and the semantic similarity between the archive sample data is judged by combining the two similarities; multiple similarity indicators are input into the The multi-feature fusion model built by the neural network structure SENet layer outputs the result of entity alignment; after all the archive data is predicted by the entity alignment model, a set of aligned entities can be formed; the entities and their relationships given by the relationship extraction layer are used Align the entity collection to complete the alignment; Knowledge reasoning layer: After the processing of the previous entity recognition layer, relationship extraction layer and entity alignment layer, the data has formed a triplet of <head entity-relationship-tail entity> structure; the knowledge map construction system also needs to carry out knowledge reasoning to Enrich and enrich the relationship between entities, and solve the problem that the relationship between entities based on original data is relatively sparse.

Images