CN115688026A - Model training method, text classification method and related equipment for audit field - Google Patents

Abstract

According to the model training method, the text classification method and the related equipment for the audit field, the obtained original audit data are preprocessed to obtain the original training data, the neural network model is pre-trained by adopting the original training data to obtain the pre-trained audit model of the audit field, and a foundation is provided for text classification of the audit field. Marking original audit data to form training sample data, training a pre-training audit model by adopting the training sample data, finely adjusting parameters of the pre-training audit model to obtain an audit text classification model, and accurately classifying the audit data to be classified through the audit text classification model.

Description

Model training method, text classification method and related equipment for audit field

technical field

The present application relates to the technical field of auditing, in particular to a model training method, a text classification method and related equipment used in the auditing field.

Background technique

The starting point of the pre-trained language model is a general language model with semantic analysis capabilities, which can be understood as a nerve center for language processing. Since the release of the first pre-trained language model BERT, it has been applied in many popular tasks. Whether it is industry or scientific research, the use of pre-training models is becoming more and more flexible. Parts suitable for tasks can be disassembled from pre-training models and assembled into actual task models in this field, so that they can be more effective in this field. work well. However, there are not many applications of pre-training models in the audit field at present, and how to train a classification model suitable for the audit field is an urgent problem to be solved.

Contents of the invention

In view of this, the purpose of this application is to propose a model training method, text classification method and related equipment used in the field of auditing.

Based on the above purpose, the first aspect of the present application provides a model training method for the field of auditing, including:

Obtain raw audit data;

Preprocessing the original audit data to obtain original training data;

Pre-training the neural network model based on the original training data to obtain a pre-training audit model;

Marking the original audit data to obtain training sample data;

The pre-trained audit model is trained based on the training sample data to obtain an audit text classification model.

Optionally, the preprocessing of the original audit data includes: performing data deduplication, format conversion and data filling on the original audit data.

Optionally, the labeling of the original audit data to obtain training sample data includes;

Perform data cleaning operations on the original audit data;

The original audit data after the data cleaning operation is marked based on the preset audit classification labels to obtain the training sample data.

Optionally, the training of the pre-trained audit model based on the training sample data includes:

Superimposing a softmax layer on the pre-trained audit model to obtain an initial classification model;

The initial classification model is trained by using the training sample data to obtain the audit model for training.

Optionally, the preset audit classification tags include at least financial audit, internal control audit, infrastructure project audit, contract audit, responsibility audit, special audit, annual internal control test and self-evaluation audit and other audits.

The second aspect of the present application provides a method for text classification using the audit text classification model described in the first aspect, including:

Obtain audit data to be classified;

The audit data to be classified is input into the audit text classification model, and the audit type corresponding to the audit data to be classified is input via the audit text classification model.

The third aspect of the present application provides a model training device for the field of auditing, including:

The first acquisition module is configured to acquire original audit data;

A preprocessing module configured to preprocess the original audit data to obtain original training data;

The first training module is configured to pre-train the neural network model based on the original training data to obtain a pre-trained audit model;

An annotation module configured to annotate the original audit data to obtain training sample data;

The second training module is configured to train the pre-trained audit model based on the training sample data to obtain an audit text classification model.

The fourth aspect of the present application provides a device for text classification using the audit text classification model described in the third aspect, including:

The second acquisition module is configured to acquire audit data to be classified;

The classification module is configured to input the audit data to be classified into the audit text classification model, and input the audit type corresponding to the audit data to be classified via the audit text classification model.

The present application also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable by the processor, and the processor implements the above-mentioned method when executing the computer program .

The present application also provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the above method.

It can be seen from the above that the model training method, text classification method and related equipment used in the audit field provided by the application preprocess the acquired original audit data to obtain the original training data, and use the original training data to train the neural network model. Perform pre-training to obtain a pre-trained audit model in the audit field, which provides a basis for text classification in the audit field. Label the original audit data to form training sample data, use the training sample data to train the pre-training audit model, fine-tune the parameters of the pre-training audit model to obtain the audit text classification model, and use the audit text classification model to treat classified audit data for precise text classification. This application obtains the text classification model in the audit field through two training processes, and trains the parameters of the neural network model through pre-training and fine-tuning to improve the applicability of the neural network model in the audit field, so that it can output more accurate audit types .

Description of drawings

In order to more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are only for this application Embodiments, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a schematic flow diagram of a model training method for the audit field according to an embodiment of the present application;

FIG. 2 is a schematic flow diagram of a method for text classification using an audit text classification model according to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a model training device used in the audit field according to an embodiment of the present application;

4 is a schematic structural diagram of a device for text classification using an audit text classification model according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application shall have the usual meanings understood by those skilled in the art to which the present application belongs. "First", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

Embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

This application provides a model training method for the audit field, referring to Figure 1, including the following steps:

Step 102, obtaining original audit data.

Specifically, the original audit data can be acquired through web crawling or from an existing database, and this embodiment does not limit the specific acquisition method. The original audit data includes the audit unit, the audited unit, audit questions and so on.

Step 104: Perform preprocessing on the original audit data to obtain original training data.

The obtained original audit data has the problem of duplication, missing or inconsistent format. It is necessary to preprocess the original audit data to obtain the original training data to provide basic data for the training of the subsequent neural network model.

Step 106, pre-training the neural network model based on the original training data to obtain a pre-trained audit model.

The neural network model in this step is the BERT (Bidirectional Encoder Representation from Transformers) model, and the BERT model is a pre-trained language representation model. The model has the following main advantages: 1) MLM (masked language model) is used to pre-train bidirectional Transformers to generate deep bidirectional language representations; 2) After pre-training, only an additional output layer needs to be added for fine-tuning, and the Optimal performance can be achieved in a variety of downstream tasks. However, the BERT model without pre-training is not suitable for the audit field. It is necessary to pre-train the BERT model through the original training data in the audit field, so that the SJ-BERT model implicitly learns the grammatical and semantic knowledge of the audit field, so that subsequent Transfer the learned knowledge to downstream tasks to form a pre-trained audit model SJ-BERT. The SJ-BERT in this step is a pre-trained large model, and the pre-trained large model refers to a pre-trained model that is not related to specific tasks obtained from large-scale data through self-supervised learning. The amount of model parameters of the large model and the number of layers of the neural network are larger than those of ordinary deep learning models. After the pre-trained audit model is obtained, the final audit text classification model can be further obtained through fine-tuning.

Step 108: Mark the original audit data to obtain training sample data. The original audit data is unlabeled data. By labeling the original audit data, each original audit data corresponds to a fixed type of label. After all the labels are completed, the training sample data is obtained.

Step 110: Train the pre-trained audit model based on the training sample data to obtain an audit text classification model.

SJ-BERT is trained through labeled training sample data, that is, the parameters of the SJ-BERT model are fine-tuned. After fine-tuning, the SJ-BERT model can output more accurate text types, and the fine-tuned model is the audit text model.

Based on the above steps 102 to 110, preprocess the acquired original audit data to obtain the original training data, use the original training data to pre-train the neural network model, obtain the pre-trained audit model in the audit field, and provide a basis for text classification in the audit field . Label the original audit data to form training sample data, use the training sample data to train the pre-training audit model, fine-tune the parameters of the pre-training audit model to obtain the audit text classification model, and use the audit text classification model to treat classified audit data for precise text classification. This application obtains the text classification model in the audit field through two training processes, and trains the parameters of the neural network model through pre-training and fine-tuning to improve the applicability of the neural network model in the audit field, so that it can output more accurate audit types .

In some embodiments, the preprocessing of the original audit data includes: performing data deduplication, format conversion and data filling on the original audit data. The obtained original audit data has the problem of duplication, missing or inconsistent format, and it needs to be preprocessed. The specific preprocessing includes data deduplication, format conversion and data filling. Data deduplication is used to remove duplicate data in the original audit data, filter out unnecessary data, and reduce the amount of subsequent data processing. Format conversion is to unify the data format in the original audit data. Since the source of the original audit data is different, the data format is not exactly the same. In order to facilitate subsequent data labeling and model training, it is necessary to unify the format of the original audit data. Due to the lack of data in the original audit data, the missing data is filled by matching rules or manually to further improve the original audit data.

In some embodiments, the labeling of the original audit data to obtain training sample data includes:

Perform data cleaning operations on the original audit data;

The original audit data after the data cleaning operation is marked based on the preset audit classification labels to obtain the training sample data.

The data cleaning operation in this embodiment is the same as the preprocessing operation in the above embodiment, and will not be repeated here. The technicians in the audit field formulate audit classification labels based on the conventional audit types in this field, and then mark the original audit data according to the audit classification labels. The technicians manually mark the original audit data through the AI platform, and the label format is label + separator + Audit data. Among them, the audit classification labels include at least financial audit, internal control audit, infrastructure project audit, contract audit, responsibility audit, special audit, annual internal control test and self-evaluation audit, and other audits. The interpretation of the above audit classification labels is shown in Table 1 below.

Table 1 Audit classification labels and interpretation

In some embodiments, the training of the pre-trained audit model based on the training sample data includes:

Superimposing a softmax layer on the pre-trained audit model to obtain an initial classification model;

The initial classification model is trained through the training sample data to obtain the audit text classification model.

Specifically, the output of the pre-trained audit model SJ-BERT includes 4 types, namely: last_hidden_state: indicates the output of the sequence of the last hidden layer; pooler_output: the output of the flag [CLS]; hidden_states: indicates a tuple, its The first element is embedding, and the remaining elements are the output of each layer; attentions: Represents a tuple whose elements are the attention weights of each layer, which are used to calculate the weighted average of the self-attention heads. Superimpose the softmax layer on the pre-trained audit model to obtain the initial classification model, that is, the [CLS] in the SJ-BERT output pooler_output is used as the input of the softmax layer, and the classification result of the model is output through the softmax layer. The training sample data is used to train the initial classification model, so that the SJ-BERT model superimposed with the softmax layer learns the classification process. During training, the training sample data is divided into training data and test data. Exemplarily, the division ratio may be 9:1. The specific model parameter table is shown in Table 2. After the training, the audit text classification model is obtained. The audit text classification model can be used to classify the text types of the audit data to be classified, and assist the technicians in the audit field to classify the classified audit data, which greatly improves the classification efficiency and provides auditors with follow-up The job provides the underlying data.

Table 2 Model parameter list

parameter class value word vector dimension 512 hidden layer size 768 training batch 10 batch_size twenty four Hidden Layer Dropout 0.1 Chinese dictionary size 21128 learning rate 0.00003

The present application also provides a method for text classification using the audit text classification model described in the above embodiment, referring to Figure 2, including:

Step 202. Obtain the audit data to be classified. The audit data to be classified can be the currently obtained audit data, or the audit data that auditors currently need to classify. The audit data can specifically be an audit report or a piece of audit text.

Step 204: Input the audit data to be classified into the audit text classification model, and input the audit type corresponding to the audit data to be classified via the audit text classification model. The audit data to be classified is input into the audit text classification model, embedded through the model, and transmitted in each neural layer of the model in turn, and the text type corresponding to the audit data to be classified is output through the output layer of the audit text classification model. For example, the audit data to be classified is output to the audit text classification model, and the text label type output through the model is financial audit.

It should be noted that the method in the embodiment of the present application may be executed by a single device, such as a computer or a server. The method of this embodiment can also be applied in a distributed scenario, and is completed by cooperation of multiple devices. In the case of such a distributed scenario, one of the multiple devices may only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete all described method.

It should be noted that some embodiments of the present application are described above. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from those in the above-described embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

The present application also provides a model training device used in the field of auditing.

Referring to Fig. 3, the model training device for the audit field includes:

The first obtaining module 302 is configured to obtain original audit data;

The preprocessing module 304 is configured to preprocess the original audit data to obtain original training data;

The first training module 306 is configured to pre-train the neural network model based on the original training data to obtain a pre-trained audit model;

The labeling module 308 is configured to label the original audit data to obtain training sample data;

The second training module 310 is configured to train the pre-trained audit model based on the training sample data to obtain an audit text classification model.

Based on the above-mentioned model training device used in the audit field, the obtained original audit data is preprocessed to obtain the original training data, and the original training data is used to pre-train the neural network model to obtain the pre-trained audit model in the audit field, which is the audit field text Classification provides the basis. Label the original audit data to form training sample data, use the training sample data to train the pre-training audit model, fine-tune the parameters of the pre-training audit model to obtain the audit text classification model, and use the audit text classification model to treat classified audit data for precise text classification. This application obtains the text classification model in the audit field through two training processes, and trains the parameters of the neural network model through pre-training and fine-tuning to improve the applicability of the neural network model in the audit field, so that it can output more accurate audit types .

In some embodiments, the preprocessing module 304 is further configured to perform data deduplication, format conversion and data filling on the original audit data.

In some embodiments, the labeling module 308 is further configured to perform a data cleaning operation on the original audit data;

The original audit data after the data cleaning operation is marked based on the preset audit classification labels to obtain the training sample data.

In some embodiments, the second training module 310 is further configured to superimpose a softmax layer on the pre-trained audit model to obtain an initial classification model;

The initial classification model is trained through the training sample data to obtain the audit text classification model.

In some embodiments, the preset audit classification tags include at least financial audit, internal control audit, infrastructure project audit, contract audit, responsibility audit, special audit, annual internal control test and self-evaluation audit and other audits.

The present application also provides a device for classifying text by applying the audit text classifying model.

Referring to Fig. 4, the device for applying the audit text classification model to perform text classification includes:

The second obtaining module 402 is configured to obtain audit data to be classified;

The classification module 404 is configured to input the audit data to be classified into the audit text classification model, and input the audit type corresponding to the audit data to be classified through the audit text classification model.

For the convenience of description, when describing the above devices, functions are divided into various modules and described separately. Of course, when implementing the present application, the functions of each module can be realized in one or more pieces of software and/or hardware.

The device of the above-mentioned embodiment is used to implement the corresponding model training method for the audit field or the method of applying the audit text classification model for text classification in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, and it is not mentioned here Let me repeat.

Based on the same inventive concept, and corresponding to the method in any of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor When the program is executed, the model training method for the audit field or the method for text classification using the audit text classification model described in any one of the above embodiments is realized.

FIG. 5 shows a schematic diagram of a more specific hardware structure of an electronic device provided by this embodiment. The device may include: a processor 1010 , a memory 1020 , an input/output interface 1030 , a communication interface 1040 and a bus 1050 . The processor 1010 , the memory 1020 , the input/output interface 1030 and the communication interface 1040 are connected to each other within the device through the bus 1050 .

The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of this specification.

The memory 1020 may be implemented in the form of ROM (Read Only Memory, read only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, and the like. The memory 1020 can store operating systems and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and invoked by the processor 1010 for execution.

The input/output interface 1030 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The communication interface 1040 is used to connect a communication module (not shown in the figure), so as to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 1050 includes a path that carries information between the various components of the device (eg, processor 1010, memory 1020, input/output interface 1030, and communication interface 1040).

It should be noted that although the above device only shows the processor 1010, the memory 1020, the input/output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

The electronic device of the above-mentioned embodiment is used to implement the corresponding model training method for the audit field or the method of applying the audit text classification model for text classification in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, here No longer.

Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium corresponding to the method in any of the above-mentioned embodiments, the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions use The purpose is to make the computer execute the model training method for the audit field or the method for text classification using the audit text classification model as described in any one of the above embodiments.

The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The computer instructions stored in the storage medium of the above-mentioned embodiments are used to make the computer execute the model training method for the audit field or the method for text classification using the audit text classification model as described in any of the above embodiments, and has a corresponding method The beneficial effects of the embodiment will not be repeated here.

Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is exemplary only, and is not intended to imply that the scope of the application (including claims) is limited to these examples; under the thinking of the application, the above embodiments or Combinations of technical features in different embodiments are also possible, steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the application as described above, which are not provided in detail for the sake of brevity.

In addition, for simplicity of illustration and discussion, and so as not to obscure the embodiments of the present application, well-known power/connections associated with integrated circuit (IC) chips and other components may or may not be shown in the provided figures. ground connection. Furthermore, devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and this also takes into account the fact that details regarding the implementation of these block diagram devices are highly dependent on the implementation of the embodiments of the present application to be implemented. platform (ie, the details should be well within the purview of those skilled in the art). Where specific details (eg, circuits) have been set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that other embodiments may be implemented without or with variations from these specific details. Implement the embodiment of the present application below. Accordingly, these descriptions should be regarded as illustrative rather than restrictive.

Although the application has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of those embodiments will be apparent to those of ordinary skill in the art from the foregoing description. For example, other memory architectures such as dynamic RAM (DRAM) may use the discussed embodiments.

The embodiments of the present application are intended to embrace all such alternatives, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent replacements, improvements, etc. within the spirit and principles of the embodiments of the present application shall be included within the protection scope of the present application.

Claims (10)

1. A method for model training in the field of auditing, characterized in that it comprises: Obtain raw audit data; Preprocessing the original audit data to obtain original training data; Pre-training the neural network model based on the original training data to obtain a pre-training audit model; Marking the original audit data to obtain training sample data; The pre-trained audit model is trained based on the training sample data to obtain an audit text classification model. 2. The method according to claim 1, wherein the preprocessing of the original audit data comprises: Perform data deduplication, format conversion and data filling on the original audit data. 3. The method according to claim 1, wherein said marking said original audit data to obtain training sample data comprises: Perform data cleaning operations on the original audit data; The original audit data after the data cleaning operation is marked based on the preset audit classification labels to obtain the training sample data. 4. The method according to claim 1, wherein said pre-training audit model is trained based on said training sample data to obtain an audit text classification model, comprising: Superimposing a softmax layer on the pre-trained audit model to obtain an initial classification model; The initial classification model is trained through the training sample data to obtain the audit text classification model. 5. The method according to claim 3, wherein the preset audit classification labels include at least financial audit, internal control audit, infrastructure project audit, contract audit, responsibility audit, special audit, annual internal control test and self-assessment. Evaluation audits and other audits. 6. A method for text classification using the audit text classification model described in any one of claims 1-5, characterized in that, comprising: Obtain audit data to be classified; The audit data to be classified is input into the audit text classification model, and the audit type corresponding to the audit data to be classified is input via the audit text classification model. 7. A model training device for the field of auditing, characterized in that it comprises: The first acquisition module is configured to acquire original audit data; A preprocessing module configured to preprocess the original audit data to obtain original training data; The first training module is configured to pre-train the neural network model based on the original training data to obtain a pre-trained audit model; An annotation module configured to annotate the original audit data to obtain training sample data; The second training module is configured to train the pre-trained audit model based on the training sample data to obtain an audit text classification model. 8. a kind of device that the audit text classification model described in application claim 7 carries out text classification, is characterized in that, comprises: The second acquisition module is configured to acquire audit data to be classified; The classification module is configured to input the audit data to be classified into the audit text classification model, and input the audit type corresponding to the audit data to be classified via the audit text classification model. 9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor implements any one of claims 1 to 6 when executing the program. method described in the item. 10. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium stores computer instructions, characterized in that the computer instructions are used to make a computer execute the computer described in any one of claims 1 to 6. Methods.

Images