CN117112872A - Government affair text archiving method and system based on semi-supervised learning - Google Patents

Abstract

The invention discloses a government affair text archiving method and system based on semi-supervised learning, which belong to the technical field of machine learning and intelligent government affairs, and comprise a manual input stage and an automatic archiving stage, wherein the manual input stage is used for scanning materials and inputting or selecting labels by a transaction staff during business transaction, and a background program is used for storing the scanned materials under a specified path according to the labels to finish text archiving; in the automatic filing stage, a clerk scans the material, the label to which the material belongs is judged through an automatic verification module, and a background program stores the scanned material under a specified path according to the label; the automatic archiving phase starts a self-learning mechanism. The invention improves the accuracy of government text identification and classification, reduces the identification and archiving uncertainty caused by excessive manual intervention, and further improves the business service level; the method is beneficial to the electronization of information materials and greatly facilitates the management of government affair data.

Description

A method and system for archiving government affairs texts based on semi-supervised learning

Technical field

The invention relates to the technical fields of machine learning and smart government affairs, specifically a method and system for archiving government affairs texts based on semi-supervised learning.

Background technique

With the development of e-government, automated processes are more commonly used in government systems, and paperless office has become an inevitable trend. However, when handling business, there are still a large number of paper materials that need to be electronically filed and processed; currently, manual entry and filing are still used more often, and too much manual intervention can easily lead to uncertainty in filing identification, and manual filing low efficiency.

Contents of the invention

The technical task of the present invention is to address the above shortcomings and provide a government affairs text archiving method and system based on semi-supervised learning to improve the accuracy of identification and classification of government affairs texts and reduce uncertainty in identification and archiving caused by excessive manual intervention. nature, thereby improving the level of business services; it contributes to the electronicization of information data and greatly facilitates the management of government data.

The technical solutions adopted by the present invention to solve the technical problems are:

A government affairs text archiving method based on semi-supervised learning, including a manual entry stage and an automatic archiving stage. In the manual entry stage, during business processing, the handler scans the materials and inputs or selects labels, and the background program stores the scanned materials according to the labels. Go to the designated path and complete the text archiving; in the automatic archiving stage, the handler scans the materials and determines the label to which the material belongs through the automatic verification module. The background program stores the scanned materials in the designated path according to the label;

The automatic archiving phase starts the self-learning mechanism. First, the feature vectors of the text materials under each tag are extracted, then the relationship between the tags and the feature vectors of the text materials is established, and finally the relationship table is updated to the automatic verification module, and the cycle is repeated to realize the dynamic management of tags. ;

Among them, the text material comes from manual calibration text and automatically calibrated text. The manual calibration text has the exact correspondence between the text material and the label. The automatically calibrated text has uncertainty. It is necessary to add a penalty mechanism to the automatically calibrated text to control the entry of noisy text information into the training machine. ; After training, obtain the model, update it to the automatic verification module, and repeat the cycle to achieve uninterrupted model optimization.

This method changes manual entry and archiving into automatic archiving, providing great convenience for government services; in application scenarios, it improves the accuracy of government text recognition and classification, and reduces the uncertainty in identification and archiving caused by excessive manual intervention. This will further improve business service levels; a complete automatic archiving method can help digitize information data and greatly facilitate the management of government data.

Preferably, the automatic verification module processes the obtained real-time material information, and extracts strings from text materials as one of the basis for text classification; and then converts this set of strings describing text characteristics into features. vector.

Furthermore, since the extracted string contains many worthless characters, it is necessary to design strategies to extract multiple characters that can describe the characteristics of the text.

Further, a set of tag sets is created in advance, which corresponds to the directory to be archived; at the same time, the feature vector corresponding to each tag is pre-constructed to form a set of feature vectors of the tags;

The feature vectors generated from the newly entered materials are analyzed one by one in the feature vector set of the tags; thus a group of tags with strong correlation is obtained, and in order to achieve a relatively high accuracy, the correlation can be obtained by controlling the correlation degree The strongest label. If there is no label that meets the conditions, you can choose to create this label, or define it as something else;

Query the archived information resource library through the acquired tags to obtain archived information;

Finally, based on the archived information, the backend program performs the archiving operation.

Preferably, the implementation business of this method mainly includes data collection, information entry, archiving management and data storage management, and archiving of texts including licenses, contracts, power of attorney, policies and regulations, and certification materials; wherein,

Data collection methods include using high-definition cameras, scanners, electronic materials, and portable devices to capture government documents;

Information entry includes manual review entry and automatic review entry mode;

Archive management includes new archive directory and existing archive directory management based on whether there is an archive directory in advance;

Data management includes data storage, ER index, data query, and data deletion, so that the data can be fully utilized in other businesses.

Preferably, the implementation of this method includes a task scheduling module, a business processing module, a data management module and an AI service module, where,

The task scheduling module serves as the Controller (main control) to coordinate the operation between various modules, including turning on or off the automatic archiving mode and turning on or off the self-learning mode;

The business processing module is responsible for matters during business processing, including entering/selecting labels, scanning materials, and storage operations;

The data management module is responsible for adding, deleting, modifying and checking data and coordinating data resources;

The AI service module is responsible for intelligent computing services, including providing text recognition and strategic judgment.

Preferably, the specific implementation of this method includes a manual recruitment stage, a semi-supervised learning stage and an unsupervised learning stage,

In the manual entry stage, material images are entered according to conventional methods, and a large number of valid government affairs text images with labels are gradually accumulated;

In the semi-supervised learning stage, on the one hand, manual input is continued, and on the other hand, the self-learning model is turned on, making full use of the accumulated sample pictures for classification learning, and gradually optimizing the recognition accuracy, that is, manual input and self-learning are carried out at the same time; in After reaching a certain level of accumulation, the self-learning function is turned on to assist manual entry;

In the unsupervised learning stage, the system has the ability to learn independently and has a high accuracy. There is no need for manual input and archiving. The clerk only needs to submit materials to realize automatic archiving of text and image materials.

The present invention also claims a government document archiving system based on semi-supervised learning, which implements the above-mentioned government document archiving method based on semi-supervised learning; the system includes an interactive client, an application server cluster, an AI server cluster, and various data and database systems and components used to improve functionality;

The interactive client includes a business lobby, a mobile client, a Web client and an administrator client, providing functions such as user information entry and review, and providing administrator user operation and maintenance functions;

The application server cluster is used to implement the basic functions of the system, and the AI server cluster is used to provide AI computing services for the system; through the configuration center, customized archiving tasks and control of system operating parameters are realized;

Deploy database services to provide data storage, addition, deletion, modification and query operations. At the same time, deploy message queue services and cache services to enhance the stability of the system.

Preferably, all types of clients connect to the gateway cluster through Nginx+firewall mode to ensure system information security;

Personal customers' mobile devices and Web clients access the system from the public cloud through the firewall to complete business processing; the business hall, internal private equipment and operation and maintenance clients access the system from the private cloud.

Preferably, the back-end server includes an application server (App Server), an AI server (AI Server) and a database server (DB Server). Depending on the task type, the back-end server runs the application program interface service (API Service) and the AI service (AI Service) respectively. ) and database (DB) operation tasks.

Compared with the existing technology, the government text archiving method and system based on semi-supervised learning of the present invention has the following beneficial effects:

1. Make full use of existing service processes to implement a semi-supervised learning government document archiving solution;

2. The self-learning process based on labeled materials improves the accuracy and efficiency of the algorithm;

3. Add a penalty mechanism and threshold control (correlation control) to enhance the robustness and stability of the algorithm;

4. The implementation of this technical solution greatly saves human and material costs and improves business processing efficiency;

5. The system adopts modular design and development, takes up little computing resources, is simple to deploy, and is easy to apply.

Description of drawings

Figure 1 is a flowchart showing the implementation of a government text archiving method based on semi-supervised learning provided by an embodiment of the present invention;

Figure 2 is a business logic diagram of the government affairs text archiving method based on semi-supervised learning provided by the embodiment of the present invention;

Figure 3 is a business implementation flow diagram of the government text archiving method based on semi-supervised learning provided by the embodiment of the present invention;

Figure 4 is a self-learning flow diagram provided by an embodiment of the present invention;

Figure 5 is a diagram of the system functional module composition provided by the embodiment of the present invention;

Figure 6 is a deployment diagram of a government text archiving system based on semi-supervised learning provided by an embodiment of the present invention;

Figure 7 is a network architecture diagram of a government text archiving system based on semi-supervised learning provided by an embodiment of the present invention.

Detailed ways

The embodiment of the present invention provides a method for archiving government documents based on semi-supervised learning, which includes a manual entry stage and an automatic archiving stage. As shown in Figure 3, in the manual entry stage, during business processing, the handler scans the materials and inputs or selects labels. The background program stores the scanned materials in the designated path according to the labels to complete text archiving. This is also the current business processing. Solution: In the automatic archiving stage, the handler scans the materials and determines the label to which the material belongs through the automatic verification module. The background program stores the scanned materials in the designated path according to the label.

The automatic archiving phase starts the self-learning mechanism. First, the feature vectors of the text materials under each tag are extracted, then the relationship between the tags and the feature vectors of the text materials is established, and finally the relationship table is updated to the automatic verification module, and the cycle is repeated to realize the dynamic management of tags. , to achieve the purpose of self-learning.

Among them, the text material comes from manual calibration text and automatically calibrated text. The manual calibration text has the exact correspondence between the text material and the label. The automatically calibrated text has uncertainty. It is necessary to add a penalty mechanism to the automatically calibrated text to control the entry of noisy text information into the training machine. ; After training, obtain the model, update it to the automatic verification module, and repeat the cycle to achieve uninterrupted model optimization. Referring to Figure 4,

The penalty mechanism refers to a mechanism commonly used in machine learning to control the regularization process to adjust the error. It is usually implemented by a penalty function and its coefficients. The detailed penalty mechanism is not limited in this article, but the purpose to be achieved is consistent All of them are to enhance the fitting ability of the machine learning model and make more accurate inferences.

The automatic verification module is the key to realizing semi-supervised learning, aiming to obtain archivable information of entered materials with less human intervention. The specific implementation process is: process the obtained real-time material information. For text materials, extract strings as one of the basis for text classification; because the extracted strings contain many worthless characters, it is necessary to use design strategies to extract many characters. characters that can describe the characteristics of the text; then convert this set of character strings that describe the characteristics of the text into a feature vector.

A set of tag sets can be created in advance, which corresponds to the directory to be archived; at the same time, the feature vector corresponding to each tag can be pre-constructed to form a set of tag feature vectors;

At this time, the feature vectors generated from the newly entered materials can be analyzed one by one in the feature vector set of the tags; thus a group of tags with strong correlation can be obtained, and in order to achieve a relatively high accuracy, the control Relevance to get the most relevant tag. If there is no tag that meets the conditions, you can choose to create the tag, or define it as something else;

Query the archived information resource library through the acquired tags to obtain archived information;

Finally, based on the archived information, the backend program performs the archiving operation.

The process is shown in Figure 1.

As shown in Figure 2, the implementation business of this method mainly consists of four parts: data collection, information entry, archiving management and data storage management. It can mainly archive documents such as licenses, contracts, letters of attorney, policies and regulations, and certification materials. in,

Data collection can be done in a variety of ways, such as using high-definition cameras, scanners, electronic materials, portable devices, etc. to obtain government documents;

The information entry stage is mainly divided into two modes: manual review and entry and automatic review and entry;

Archive management depends on whether there is an archive directory in advance, including new archive directory and existing archive directory management;

The data management stage mainly includes data storage, ER index, data query, and data deletion, so that the data can be fully utilized in other businesses.

ER index: ER stands for Entity Relationship, which is translated as entity relationship. It is often expressed in the form of a diagram, that is, the entity relationship diagram, which is a method that provides entities, attributes and relationships; using this method, establish the relationship between the entity and each entity of the transaction. Class materials and the relationship between materials provide an index that describes this complex relationship, which is called an ER index.

As shown in Figure 5, the implementation of this method includes a task scheduling module, a business processing module, a data management module and an AI service module, where,

The task scheduling module serves as the Controller (main control) to coordinate the operation between various modules, including turning on or off the automatic archiving mode, turning on or off the self-learning mode, etc.;

The business processing module is responsible for matters during business processing, including entering/selecting labels, scanning materials, storage operations, etc.;

The data management module is responsible for adding, deleting, modifying and checking data and coordinating data resources;

The AI service module mainly provides intelligent computing services such as text recognition and strategic judgment.

The application of this method is specifically described as follows based on the implementation process of automatic archiving of government documents in a city's smart approval system:

This project requires the archiving of texts provided by people handling business. Common text materials include ID cards, business licenses, bank cards, contracts, power of attorney, policies and regulations, certification materials, etc. The types of materials are generally divided into photos, scanned copies, and electronic materials. wait. Archived materials can be used for resource sharing within the government system, reducing work processes in other links and improving work efficiency.

Generally, the submitted materials are reviewed and archived one by one using manual review and entry. With the growth of business volume, a large amount of material review work has seriously affected the progress of matter processing, and even brought the risk of filing errors. An intelligent method is needed to realize the automatic review and filing of submitted materials.

Use this method to make full use of existing systems for optimization and upgrades. Specifically, the implementation based on this method is divided into three stages, namely the manual recruitment stage, the semi-supervised learning stage and the unsupervised learning stage.

In the manual entry stage, material images are entered according to conventional methods, and a large number of valid government affairs text images with labels are gradually accumulated.

In the semi-supervised learning stage, on the one hand, manual input is continued, and on the other hand, the self-learning model is turned on, making full use of the accumulated sample pictures for classification learning, and gradually optimizing the recognition accuracy, that is, manual input and self-learning are carried out at the same time, and even After reaching a certain amount of accumulation, the self-learning function is turned on to assist manual entry.

In the unsupervised learning stage, the system has the ability to learn independently and has a high accuracy. There is no need for manual input and archiving. The clerk only needs to submit materials to realize automatic archiving of text and image materials.

When developing a system using this method, it can be divided into four major modules: task scheduling, business processing, data management, and AI services. This architecture fully considers the decoupling design. The development tasks of the first three are developed by front-end and back-end engineers. The Java language, which is the mainstream of the government system, can be compatible with other systems. At the same time, the AI service is developed by algorithm engineers and uses industry-leading Java language. The mainstream Python language gives full play to its algorithm advantages, provides web services for the entire system, and quickly optimizes and updates iteratively.

Text recognition technology is widely used in the digital construction of government affairs systems. It can improve administrative efficiency and service levels. It is widely used in application scenarios such as digital processing of government documents, digital entry of form information, textualization and automatic classification of smart city data. Text recognition The application of technology speeds up the processing speed of government affairs systems and reduces the processing costs of government affairs systems. Automatic text archiving is a file organization and management method based on artificial intelligence technology that helps users quickly and accurately identify and archive various documents, pictures, audio and video files. Automatic text archiving mainly uses natural language processing, machine learning, deep learning and other technologies to carry out document classification, labeling, archiving and other processes.

Commonly used methods of machine learning are mainly divided into supervised learning and unsupervised learning. A simple summary is that whether there is supervision (supervised) depends on whether the input data has a label (label). If the input data has labels, it is supervised learning; if there are no labels, it is unsupervised learning. In addition, a learning algorithm included between supervised and unsupervised is semi-supervised learning. For semi-supervised learning, part of the training data is labeled, and the other part is unlabeled, and the number of unlabeled data is often greater than the number of labeled data (this is also consistent with reality). The basic rule hidden under semi-supervised learning is that the distribution of data is not completely random. Through some local features of labeled data and more overall distribution of unlabeled data, acceptable or even very good classification results can be obtained. .

Embodiments of the present invention also provide a government text archiving system based on semi-supervised learning, which implements the semi-supervised learning-based government text archiving method described in the above embodiment; as shown in Figure 6, the system includes an interactive client, Application server clusters, AI server clusters, various data and database systems, and components used to improve functions, etc.

The interactive client includes a business lobby, a mobile client, a Web client, an administrator client, etc., and provides functions such as user information entry and review, as well as administrator user operation and maintenance functions.

Various external clients connect to the gateway cluster through the "Nginx+firewall" mode to ensure system information security.

The system provides an application server cluster and an AI server cluster. The application server cluster is used to implement the basic functions of the system, and the AI server cluster is used to provide AI computing services for the system. Through the configuration center, customized archiving tasks and system operating parameters can be controlled;

PHP application server cluster: PHP's full name is Hypertext Preprocessor, and its Chinese name is "Hypertext Preprocessor". It is a general open source scripting language; based on this language, application server clusters can be developed, with high concurrency and distributed characteristics. K8s: Its full name is kubernetes. Because the name is too long, "8" is used to replace the middle 8 letters; it is an open source and well-known container-based cluster management platform. In this article, it is used to build an AI server cluster and provides Docker (container) management and load balancing. In addition, there are many types of management platforms. In the technology selection stage, you can choose a service management platform that suits you based on actual conditions.

Deploy database services to provide data storage, addition, deletion, modification and query operations. At the same time, deploy message queue services and cache services to enhance the stability of the system. Kafka is an open source distributed streaming platform owned by Apache. It is a high-throughput, durable, distributed publish-subscribe message queue system. Redis is open source and free, abides by the BSD protocol, and is a high-performance key-value non-relational database. It is used in this article to implement caching services.

As shown in Figure 7, the network architecture diagram of the system developed based on this method is introduced. Personal customers' mobile devices and Web clients access the system from the public cloud through the firewall to complete business processing; the business hall, internal private equipment and operation and maintenance clients access the system from the private cloud.

The back-end server includes application server (App Server), AI server (AI Server) and database server (DB Server). Depending on the task type, it runs the application program interface service (API Service), AI service (AI Service) and database respectively. (DB) operations and other tasks.

Through the above specific embodiments, those skilled in the technical field can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. On the basis of the disclosed embodiments, those skilled in the technical field can arbitrarily combine different technical features to achieve different technical solutions.

Except for the technical features described in the specification, they are all technologies known to those skilled in the art.

Claims (10)

1. A government affairs text archiving method based on semi-supervised learning, which is characterized by including a manual entry stage and an automatic archiving stage. In the manual entry stage, during business processing, the handler scans the materials and inputs or selects tags, and the background program will The scanned materials are stored in the designated path according to the label to complete text archiving; in the automatic archiving stage, the handler scans the materials and determines the label to which the material belongs through the automatic verification module. The background program stores the scanned materials in the designated path according to the label; The automatic archiving phase starts the self-learning mechanism. First, the feature vectors of the text materials under each tag are extracted, then the relationship between the tags and the feature vectors of the text materials is established, and finally the relationship table is updated to the automatic verification module, and the cycle is repeated to realize the dynamic management of tags. ; Among them, the text material comes from manual calibration text and automatically calibrated text. The automatic calibration text adds a penalty mechanism to control the noise text information entering the training machine; after training, the model is obtained and updated to the automatic verification module, and the cycle is repeated to achieve uninterrupted Model optimization. 2. A government affairs text archiving method based on semi-supervised learning according to claim 1, characterized in that the automatic verification module processes the obtained real-time material information, and extracts strings for text materials as One of the basis for classifying text; then convert this set of strings describing text characteristics into feature vectors. 3. A method for archiving government affairs texts based on semi-supervised learning according to claim 2, characterized in that the obtained real-time material information is processed, and for text material extraction strings, a plurality of strings are extracted through a design strategy. Characters that describe the characteristics of the text. 4. A method for archiving government affairs texts based on semi-supervised learning according to claim 2, characterized in that a set of tag sets is created in advance, which corresponds to the directory to be archived; and a feature vector corresponding to each tag is pre-constructed at the same time. , a set of feature vectors that form a set of labels; The feature vectors generated by the newly entered materials are analyzed one by one in the feature vector set of the tags; a set of relevant tags are obtained from this, and the most relevant tags are obtained by controlling the correlation degree. If there are no tags that meet the conditions , choose to create this label, or define it as something else; Query the archived information resource library through the acquired tags to obtain archived information; Finally, based on the archived information, the backend program performs the archiving operation. 5. A government text archiving method based on semi-supervised learning according to claim 1 or 2 or 3 or 4, characterized in that the implementation business of the method includes data collection, information entry, archiving management and data storage management, Archive texts including licenses, contracts, letters of attorney, policies and regulations, and supporting materials; among them, Data collection methods include using high-definition cameras, scanners, electronic materials, and portable devices to capture government documents; Information entry includes manual review entry and automatic review entry mode; Archive management includes new archive directory and existing archive directory management based on whether there is an archive directory in advance; Data management includes data storage, ER index, data query, and data deletion. 6. A government affairs text archiving method based on semi-supervised learning according to claim 5, characterized in that the implementation of the method includes a task scheduling module, a business processing module, a data management module and an AI service module, wherein, The task scheduling module serves as the main control, coordinating the operation between various modules, including turning on or off automatic archiving mode, turning on or off self-learning mode; The business processing module is responsible for matters during business processing, including entering/selecting labels, scanning materials, and storage operations; The data management module is responsible for adding, deleting, modifying and checking data and coordinating data resources; The AI service module is responsible for intelligent computing services, including providing text recognition and strategic judgment. 7. A government affairs text archiving method based on semi-supervised learning according to claim 6, characterized in that the specific implementation of the method includes a manual recruitment stage, a semi-supervised learning stage and an unsupervised learning stage, In the manual entry stage, material images are entered according to conventional methods, and a large number of valid government affairs text images with labels are gradually accumulated; In the semi-supervised learning stage, on the one hand, manual input is continued, and on the other hand, the self-learning model is turned on, making full use of the accumulated sample images for classification learning, and gradually optimizing the recognition accuracy, that is, manual input and self-learning are carried out at the same time; in After the accumulation amount is reached, the self-learning function is turned on to assist manual entry; In the unsupervised learning stage, there is no need for manual input and archiving. The clerk only needs to submit materials to realize automatic archiving of text and image materials. 8. A government text archiving system based on semi-supervised learning, characterized in that the system implements the semi-supervised learning-based government text archiving method described in any one of claims 1 to 7; the system includes an interactive client, an application Server clusters, AI server clusters, various data and database systems, and components used to improve functions; The interactive client includes a business lobby, a mobile client, a Web client and an administrator client, providing user information entry and review functions, and administrator user operation and maintenance functions; The application server cluster is used to implement the basic functions of the system, and the AI server cluster is used to provide AI computing services for the system; through the configuration center, customized archiving tasks and control of system operating parameters are realized; Deploy database services to provide data storage, addition, deletion, modification and query operations. At the same time, deploy message queue services and cache services to enhance the stability of the system. 9. A government text archiving system based on semi-supervised learning according to claim 8, characterized in that various clients connect to the gateway cluster through the Nginx+firewall mode; Personal customers' mobile devices and Web clients access the system from the public cloud through the firewall to complete business processing; the business hall, internal private equipment and operation and maintenance clients access the system from the private cloud. 10. A government text archiving system based on semi-supervised learning according to claim 9, characterized in that the back-end server includes an application server, an AI server and a database server, and runs application program interface services respectively according to different task types. , AI services and database operation tasks.