CN110288081A - A recurrent network model and learning method based on FW mechanism and LSTM - Google Patents

Abstract

The present invention relates to a kind of Recursive Networks model and learning method based on FW mechanism and LSTM, belongs to recurrent neural network and natural language processing technique field.Learning method including Recursive Networks model and support based on FW mechanism and LSTM；The former includes data import modul, data generation module, load and iteration module, parameter setting module, definition module, Recursive Networks training, assessment and test module；Learning method includes: 1 importing data；2, which will import data, is split to obtain training data, assessment data and test data；3, according to data are imported, obtain pre-set configuration parameter；4 complete the initialization of weight parameter；Training, assessment and test data are sent into LSTM unit by 5 calculates output vector；6 calculate loss function, optimize to network parameter, export complexity.The network model and learning method further improve the accuracy and convergence rate of LSTM model treatment.

Description

A recurrent network model and learning method based on FW mechanism and LSTM

technical field

The invention relates to a recursive network model and a learning method based on a FW mechanism and an LSTM, and belongs to the technical field of recursive neural networks and natural language processing.

Background technique

Natural language processing models usually adopt a recurrent neural network (Recurrent Neural Network, RNN) structure. RNN consists of two time-scale variables, hidden layer states and weights. The hidden layer state is updated at each time step; the weights are updated after all information about the sequence has been fed into the network. Therefore, the weight representing the connection relationship between layers in the network often corresponds to the "long-term memory" of the network. However, the change and progress of the relationship between each layer of the actual network is often related to the length of the input sequence. It may be 3 or 5 time steps, or it may be 30 or 50 time steps that need to be updated.

The language model based on the LSTM unit is one of the widely used improved networks of RNN. The model predicts the next word that will appear in the text according to the input text through the training of the text data. The initial state of network parameters is initialized with a zero vector and updated after each word is read. The model uses the backpropagation method to optimize the network parameters when processing the input data. Divide the input data, that is, paragraphs composed of several sentences, into fixed-length input blocks, and each input block has fixed-length words. After each input block is processed, backpropagation is performed to update the network parameters.

Jimmy Ba et al. proposed the Fast Weights (FW) mechanism, which introduces a new variable between the hidden layer state and the weight of the update period to store the fast updated hidden layer state. For sequence-to-sequence models learning has been proven to be very effective. For the above considerations, new variables are introduced while retaining the existing hidden layer state and standard weights. The update cycle of this variable is longer than the hidden layer update cycle, but shorter than the standard weight update cycle, also known as fast Weights.

In terms of neural network training, generally complex and time-consuming processing is required to obtain better learning performance, that is, higher time and computing costs are required. Therefore, researchers often choose batch processing in order to reduce this time and computational cost.

Among them, batch regularization is a typical technique, but its role in recurrent neural networks is not obvious. Therefore, G. Hinton and others proposed layer normalization (layer normalization, LN), which is specifically implemented to calculate the mean and standard deviation of the states of all hidden units on a hidden layer in a training sample in a recurrent neural network. LN is used to solve the overflow problem during the update value of the hidden layer as the training increases in the fast weight mechanism.

The evaluation index parameters to measure the performance of the language model are complexity perplexity and loss. Among them, perplexity represents the average number of options for the language model to predict the next word based on the words in front of the sentence after learning the text data. For example, if a sequence is randomly composed of five letters A, B, C, D, and E, then when predicting the next letter, there are 5 options with equal probability, and the complexity value is 5. Therefore, if the complexity of a language model is K, it means that when the language model predicts upcoming words, on average, K words have the same probability as reasonable prediction choices. Among them, K is an integer, which is the total number of target words. Taking the PTB model as an example, the calculation formula for evaluating the complexity perplexity value of the PTB model performance index is (1):

Among them, Ptarget _i represents the i-th target word, and ln is a logarithmic function;

Another evaluation index parameter loss to measure the performance of the language model is defined as the average negative logarithm of the target word occurrence probability, the expression is as (2):

The relationship between the perplexity value and loss is (3):

perplexity=e ^loss (3)

When the language model learns the logical relationship between words and words in a sentence, the stronger the learning ability of the model is, when predicting the next word based on the previous word, the fewer the number of alternative words, the corresponding complexity perplexity lower. So the complexity perplexity can well reflect the learning performance of the network. The lower the complexity perplexity, the stronger the ability of the network to predict the next word in the sentence, and the better the effect.

Contents of the invention

The purpose of the present invention is to further improve the technical status of the existing LSTM-based recursive neural network when processing natural language with strong temporal correlation, and the complexity performance needs to be further improved. A recursive network model and learning method based on FW mechanism and LSTM is proposed. .

The recursive network model and learning method based on the FW mechanism and LSTM include the recurrent network model based on the FW mechanism and LSTM and the learning method it relies on;

Wherein, the recursive network model based on the FW mechanism and LSTM includes a data import module, a data generation module, a loading and iteration module, a parameter setting module, a model definition module, a recursive network training module, a recursive network evaluation module and a recursive network testing module ;

Among them, the data generation module includes a data splitting unit; the loading and iteration module includes a data loading unit and an iteration unit;

The data splitting unit includes a training data generating unit, an evaluation data generating unit and a test data generating unit;

The recursive network training module includes a dropout unit, an update unit and a result storage unit; the recursive network evaluation module and the recursive network test module only include an update unit and a result storage unit;

Wherein, the update unit includes a long short-term memory unit and a fast weight unit;

The connection relationship of each module in the recursive network model based on FW mechanism and LSTM is as follows:

The data import module is connected to the data generation module, the data generation module is connected to the loading and iteration module, the parameter setting module is connected to the loading and iteration module and the model definition module, the recursive network training module is connected to the loading and iteration module, the recursive network evaluation module and the model The definition module is connected, the recursive network evaluation module is connected with the loading and iteration module, the recursive network training module, the recursive network testing module and the model definition module; the recursive network testing module is connected with the loading and iteration module, the recursive network evaluation module and the model definition module;

The connection relationship of each unit in the data generation module is as follows: the training data, evaluation data and test data in the data splitting unit are respectively connected to the training label generation unit, evaluation label generation unit and test label generation unit;

Loading and the connection relation of each unit in the iterative module are as follows: the data loading unit is connected with the iterative unit;

The signal generation and output relationship of each module in the recursive network model based on the FW mechanism and LSTM is as follows:

The output of the data import module is connected to the data generation module; the data generation module is connected to the loading and iteration module after processing; the parameter setting module provides input parameters and FW model parameters for the loading and iteration module and the model definition module; the loading and iteration modules are respectively The recursive network training module, the recursive network evaluation module and the recursive network testing module provide training data and training labels, evaluation data and evaluation labels, and test data and test labels; the model definition module inputs the FW model parameters into the recurrent network training module and the recursive network evaluation module respectively. module and the recursive network testing module; the recursive network training module sends the trained network parameters into the recursive network evaluation module; the recursive network evaluation module sends the evaluated network parameters into the recursive network testing module;

The connection relationship of each unit in the recurrent network training module, evaluation module and test module is as follows:

The dropout unit receives data and is connected with the long-short-term memory unit, the long-short-term memory unit is connected with the data input and the fast weight unit, and the result storage unit is connected with the fast weight unit and the result.

The recursive network model based on the FW mechanism and LSTM and the learning method it relies on include the following steps:

Step 1. The data to be trained and tested is imported through the data import module, specifically:

Obtain text data by reading the text path;

Step 2, the data generation module splits the data imported by the data import module through the data splitting unit to obtain training data, evaluation data and test data respectively;

Wherein, the splitting is specifically: splitting the text data imported in step 1 according to every j characters into a sentence;

Wherein, the value range of j is from 5 to 50;

Step 3, the training data generation unit randomly selects the x% ratio of the data split by the data splitting unit to generate a training set; the evaluation data generation unit randomly selects the y% ratio of the data split by the data splitting unit to generate an evaluation set; test The data generating unit randomly selects the z% ratio to generate a test set from the data split by the data splitting unit;

Wherein, x%+y%+z%=1;

Step 4, the training label generating unit shifts each data in the training set generated by the training data generating unit by one bit to obtain the training label; the evaluation label generating unit shifts each data in the evaluation set generated by the evaluation data generating unit by one bit to obtain the evaluation label ; The test label generation unit shifts each data in the test set by one bit to obtain the test label;

Step 5, the parameter setting module obtains configuration parameters according to the model scale of the text imported by the data import module, and then inputs the obtained configuration parameters into the parameter setting module;

Among them, the configuration parameters include initial scale, learning rate, maximum gradient regular value, number of layers, number of steps, hidden layer size, maximum epoch number, maximum epoch value, dropout rate, decay rate, batch size, and vocab size;

Step 6. The data loading unit in the loading and iteration module loads the data in the training set, evaluation set and test set according to the configuration parameters obtained in the parameter setting module, and sets the initialization data sequence number i to 1;

Step 7, the model definition module uses the pseudo-random function to generate random values within the configuration range as the weight matrix parameters according to the configuration parameters in the parameter setting module, and completes the initialization of the weight parameters;

Step 8. The iteration module in the loading and iteration module judges whether the data in the current data set has been sent, and operates according to the judgment result, specifically:

If the data in the current data set has not been sent, send the i-th group of data, judge whether to train, evaluate or test, and skip to step 9, and skip to step 8; otherwise, stop the iteration;

Step 9. Determine whether the current data is training data. If so, extract the input data according to the dropout rate. After the extracted data, skip to step 10; otherwise, skip to step 10;

Step 10. Send the data input in step 9 to the long-short-term memory unit and the fast weight unit in the update unit to calculate the output vector, and at the same time use the gradient descent method to optimize the network, specifically:

Step 10.1 The update unit calculates the initial hidden layer state based on the input layer weight Wx and the standard weight Wh, and calculates the initial hidden state at the current time t by formula (4):

h ⁰ _t ＝f(LN(W _x *x _t +W _h *h _t-1 )) (4)

Among them, the input layer weight is recorded as Wx, and the standard weight is recorded as Wh; h ₀ is the initial hidden layer state, LN is the layer regularization function; f is the activation function; x _t is the input layer data at the current time t; h _{t- 1} is the data corresponding to the state of the hidden layer at the moment before the current moment, that is, time t-1, referred to as the state of the hidden layer;

Preferably, the activation function f is one of SeLU function, Leaky Relu function and Swish function;

The standard weight Wh is the weight propagated from the hidden layer to the next time step in the RNN network; the input layer weight Wx is the weight propagated from the input layer to the hidden layer;

Step 10.2, the fast weight unit calculates the fast weight, specifically calculated by formula (5):

W _A (t)＝λW _A (t-1)+ηh _t-1 h ^T _t-1 (5)

Among them, W _A (t) is the fast weight at the tth moment, which is the weight that only acts on the hidden layer in each time step; the total number of updates in a time step is recorded as s+1 times; λ is the decay rate , η is the learning rate, h _t-1 is the hidden layer state corresponding to the t-1 moment; h ^T _t-1 is h _t-1 , that is, the transposition of the hidden layer state corresponding to the t-1 moment;

Among them, s in the total number of times of time step update s+1 is the number of steps;

Among them, the decay rate ranges from 0.9 to 0.995, and the learning rate ranges from 0.3 to 0.8;

Step 10.3, the fast weight unit calculates the state of the hidden layer and updates the state of the hidden layer for s times;

Step 10.4, the slow weight unit calculates the normalized output;

Among them, the normalized output of the network is realized by either Softmax or sigmoid function;

Step 10.5, the result storage unit calculation is based on the normalized output calculated in step 10.4 to calculate loss and complexity perplexity;

Step 10.6. The slow weight unit judges whether the last Epoch has been reached. If not, the update unit updates the state of the hidden layer and the training parameters or test parameters, adds 1 to the current i, and skips to step 8.

Beneficial effect

A recursive network model and learning method based on the FW mechanism and LSTM of the present invention, compared with the prior art, has the following beneficial effects:

1. The recursive network model introduces fast weight and LSTM mechanism, and through parameter optimization of attenuation coefficient and learning rate, the learning accuracy of the network model with stored short-term memory information is greatly improved;

2. Compared with the existing LSTM model and the RNN model that introduces fast weights, the method of the present invention adopts LSTM in combination with SeLU activation function and layer regularization to greatly improve the convergence speed of training, evaluation and testing.

Description of drawings

Fig. 1 is the composition and the connection schematic diagram of each module of the recursive network model based on FW mechanism and LSTM of the present invention;

Fig. 2 is the composition and connection schematic diagram of the data generation module in the recursive network model based on FW mechanism and LSTM of the present invention;

Fig. 3 is a composition schematic diagram of loading and iteration modules in the recursive network model based on FW mechanism and LSTM of the present invention and data generation module, parameter setting module, model definition module, recursive network training module, recursive network evaluation module and recursive network test The connection relationship of the modules;

4 is a schematic diagram of the relationship and composition of the recursive network training module, the recursive network evaluation module and the recursive network testing module in the recursive network model based on the FW mechanism and LSTM of the present invention;

Fig. 5 is a schematic diagram of the composition of the long-short-term memory unit and the fast weight unit in the recursive network model based on the FW mechanism and LSTM of the present invention;

Fig. 6 is a comparison of the learning effects of different batch sizes of short sentence text data sets with a large correlation degree based on the method of the present invention based on the FW mechanism and the recursive network model of LSTM;

Fig. 7 is a log (perplexity) comparison of different models of short sentence text data sets with high relevance by the method based on the FW mechanism and the recursive network model of LSTM in the present invention.

Detailed ways

The recurrent network model and learning method based on the FW mechanism and LSTM of the present invention will be further explained and described in detail below in conjunction with the drawings and embodiments.

Example 1

This embodiment describes the composition and workflow of the recurrent network model based on the FW mechanism and LSTM of the present invention.

In the specific implementation, the corpus uses the representative short sentence library in the NLTK text corpus popularly used in natural language processing - the EU national conference corpus europarl_raw for experiments.

The text data of the europarl_raw corpus comes from conference conversations. Most of the sentences are short and medium sentences, about ten words in length. The sentence structure is relatively simple, and most of them have a subject-predicate-object structure. Specifically in this embodiment, the data set is processed by using each module in FIG. 1 .

Figure 1 schematically shows the composition of the recursive network model based on the FW mechanism and LSTM and the connection of each module. It can be seen from Figure 1 that the data imported by the data import module is sent to the data generation module; the data generation module generates training data, evaluates The data and test data and their labels are input into the loading and iteration module; the loading and iteration module and the model definition module receive the parameters of the parameter setting module, and are respectively connected to the recurrent network training module, evaluation module and test module for training and evaluation and testing.

. First, the text data is imported through the data import module by reading the text path; after import, it is output to the data generation module, and the data generation module further splits the original data into training data, evaluation data and test data, and then passes through the training label generation unit, evaluation The label generation unit and the test label generation unit generate labels for each data set, and their structure is shown in the connection schematic diagram of the data generation module in FIG. 2 .

Among them, the configuration parameters preset in the parameter setting module include four types as described in Table 1 below:

Table 1 Specific parameter settings for each configuration

According to the model size of the text imported by the data import module, the parameter setting module obtains the appropriate configuration parameters shown in Table 1, and inputs them into the parameter setting module, and then sends them to the model definition module and the loading and iteration module. According to the configuration parameters in the parameter setting module, the model definition module uses the pseudo-random function to generate random values within the configuration range as the parameters of the weight matrix to complete the initialization of the weight parameters.

The data loading unit in the loading and iteration module loads the data in the training set, evaluation set, and test set according to the configuration parameters obtained in the parameter setting module; the iteration module judges whether the data in the current data set has been sent, and operates according to the judgment result. If the current data set is training data, it is output to the recursive network training module; if it is evaluation data, it is output to the recurrent network evaluation module; if it is test data, it is output to the recurrent network testing module. From Figure 3, we can see the working diagram of the loading and iteration module and the connection relationship with the data generation module, parameter setting module, model definition module, recursive network training module, recursive network evaluation module and recursive network testing module.

Figure 4 illustrates the relationship and composition of the recurrent network training module, recurrent network evaluation module and recurrent network testing module in the recurrent network model based on the FW mechanism and LSTM. The difference between the recurrent network evaluation module and the recurrent network test module and the recurrent network training module is that the dropout unit is not included but only the update unit and the result storage unit are included. The recursive network training module sends the trained network parameters to the recursive network evaluation module; the recursive network evaluation module sends the evaluated network parameters to the recursive network testing module.

As can be seen from Figure 4, the update unit includes a long-short-term memory unit and a fast weight unit; as can be seen in Figure 4, the difference between the recursive network evaluation module and the recursive network test module and the recursive network training module is that the dropout unit is not included; only the update unit is included and result storage unit.

Figure 5 illustrates the composition of the long short-term memory unit and the fast weight unit in this model. In Figure 5, X _t corresponds to the input layer data at time t; C(t-1) and C'(t) correspond to the input and output of LSTM memory unit C at time t; Update and generate C(t); as the input of the LSTM memory unit C at the next moment; h _t-1 and h _t are the outputs of the LSTM cell at time t-1 and time t respectively. σ in Figure 5 is the activation function sigmoid; tanh is the tanh activation function.

In FIG. 5 , C'(t)=h ₀ (t) and C(t)h _s (t) respectively correspond to the input of the memory unit at time t before the initial fast weight update and the input of the memory unit at time t after the update.

Example 2

This example illustrates the comparison of the learning effects of processing the sentence sentence data sets with high relevance based on the method relied on by the recursive network model of the present invention.

We turn our attention to the processing of text data with strong correlation between sentences and short sentences. Due to the short sentences, we pay more attention to the connection between input words and words in a short period of time. We conduct experiments using the European Union National Assembly corpus europarl_raw, a representative short sentence corpus from the NLTK text corpus popularly used in natural language processing.

When using the europarl_raw corpus, set num_steps to 10, which means that the network processes every ten words as a complete sentence.

Firstly, it is necessary to determine the appropriate number of updates s.

When the fast weight is updated at the current moment, the hidden state will be updated s times. Compared with the sample data of the toy game scene, the correlation between the words before and after the text data is more complicated. We need to speed up the update frequency, that is, add Larger values of s give greater play to the function of fast weight processing short-term memory. We adjust the number of updates of the hidden state within a time step, fix the number of hidden units to 50, batch_size to 20, change S=5, 6, 7, 8, and record the model training effect, as shown in Table 2 below:

Table 2 Comparison of the complexity of the model when it is trained to the 5th, 10th, and 13th epochs under different update times

Update times s Complexity -5 Complexity -10 Complexity - 13 5 189.380 108.083 105.231 6 145.939 73.875 71.331 7 138.889 68.323 65.946 8 139.400 70.049 67.642

As shown in Table 2, when the number of updates s=7, the complexity of the fast weight model is 138.889 when training to the fifth epoch, drops to 68.323 at the 10th epoch, and converges to 65.946 at the 13th epoch.

Thereafter we will determine the appropriate batch size.

An appropriate batch size is crucial to the learning performance of a network. If the batch size is too large, it will cause the model to find the local minimum instead of the global minimum when the gradient descent method is used to find the optimal solution. If the batch size is too small, it will As a result, the convergence speed is slow and the model learning effect is poor. Therefore, in order to improve the performance of the new model that introduces fast weights, we fix the number of hidden units to 50, set the number of updates s to the optimal value 7 verified in the previous article, the number of words that make up the sentence num_steps=10, and change the batch size to 10 ,20,30,50, record the model training effect, as shown in Table 3 below:

Table 3 Comparison of the complexity of the model at the 10th epoch under different batch sizes

As can be seen from Table 3, when batch_size=20, the complexity of the model after convergence is the lowest. When training to the 10th epoch, the complexity is 45.139. When training to the 13th epoch, the complexity is as low as 43.344. The batch size is equal to 10 and 30, the complexity of model training to the 13th epoch is about 51. In order to more intuitively show the difference in complexity under different batch sizes, we take the logarithm log(perplexity) with the base 10 as the complexity, and compare the log(perplexity) difference of the fast weight model under different batch sizes, as shown in Figure 6 shown.

In Figure 6, the abscissa is the number of training epochs, and the ordinate is the logarithm log(perplexity) of the complexity based on 10. It can be seen that when every 20 epochs are used as a batch of data for training, the complexity of the model is the lowest, and the learning effect is the best, and the language model is compared.

The number of fixed hidden units is 50, the number of words that make up a sentence is num_steps=10, and the SeLU function is used as the activation function. Compare the training effects of the LSTM model, the RNN model, the model combining fast weight and LSTM network, and the model combining fast weight and RNN. The model training complexity is shown in Table 4:

Table 4: Comparison of training complexity of different models based on europarl_raw database

model name Complexity -5 Complexity -10 Complexity -15 Complexity - 20 LSTMs 267.602 178.175 174.935 174.824 LSTM+FW 90.945 45.139 43.280 43.208 RNN 1037.719 421.531 412.841 412.510 RNN+FW 533.806 378.564 369.842 369.474

As can be seen from Table 4, the complexity of the LSTM model that introduces fast weights is 90.945 when it is trained to the 5th epoch, and further reduces to 45.139 at the 10th epoch. When it is trained to the 15th epoch, the model complexity is 43.280, The model reaches convergence. Similarly, when training to the 15th epoch, the complexity of the LSTM model converges to 174.824, which is 131 higher than the LSTM model with fast weights, followed by the RNN network with fast weights, the complexity converges to 369.474, and the worst effect is RNN model, the complexity converges to 412.510.

In order to express the complexity difference of different models more intuitively, the complexity is taken as the logarithm to the base 10, and the log (perplexity) difference of different models is compared, as shown in Figure 7.

It can be seen from Figure 7 that the LSTM model with fast weights introduced has the lowest complexity after convergence, and the model learning effect is the best, and it is very different from the LSTM model without fast weights, which shows that the introduction of fast weights in the LSTM network, the model The training effect is significantly improved. The complexity of the RNN model after convergence is the highest, and the training effect of the RNN model after adding fast weights is slightly improved, but the effect is not obvious.

The above description is only a preferred embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. All equivalents or modifications accomplished without departing from the disclosed spirit of the present invention fall within the protection scope of the present invention.

Claims (9)

1. The recursive network model based on the FW mechanism and LSTM is characterized in that it includes a data import module, a data generation module, a loading and iteration module, a parameter setting module, a model definition module, a recursive network training module, a recursive network evaluation module and a recursive network model. Network test module; Among them, the data generation module includes a data splitting unit; the loading and iteration module includes a data loading unit and an iteration unit; The data splitting unit includes a training data generating unit, an evaluation data generating unit and a test data generating unit; The recursive network training module includes a dropout unit, an update unit and a result storage unit; the recursive network evaluation module and the recursive network test module only include an update unit and a result storage unit; Wherein, the update unit includes a long short-term memory unit and a fast weight unit; The connection relationship of each module in the recursive network model based on FW mechanism and LSTM is as follows: The data import module is connected to the data generation module, the data generation module is connected to the loading and iteration module, the parameter setting module is connected to the loading and iteration module and the model definition module, the recursive network training module is connected to the loading and iteration module, the recursive network evaluation module and the model The definition module is connected, the recursive network evaluation module is connected with the loading and iteration module, the recursive network training module, the recursive network testing module and the model definition module; the recursive network testing module is connected with the loading and iteration module, the recursive network evaluation module and the model definition module; The connection relationship of each unit in the data generation module is as follows: the training data, evaluation data and test data in the data splitting unit are respectively connected to the training label generation unit, evaluation label generation unit and test label generation unit; Loading and the connection relation of each unit in the iterative module are as follows: the data loading unit is connected with the iterative unit; The signal generation and output relationship of each module in the recursive network model based on the FW mechanism and LSTM is as follows: The output of the data import module is connected to the data generation module; the data generation module is connected to the loading and iteration module after processing; the parameter setting module provides input parameters and FW model parameters for the loading and iteration module and the model definition module; the loading and iteration modules are respectively The recursive network training module, the recursive network evaluation module and the recursive network testing module provide training data and training labels, evaluation data and evaluation labels, and test data and test labels; the model definition module inputs the FW model parameters into the recurrent network training module and the recursive network evaluation module respectively. module and the recursive network testing module; the recursive network training module sends the trained network parameters into the recursive network evaluation module; the recursive network evaluation module sends the evaluated network parameters into the recursive network testing module; The connection relationship of each unit in the recurrent network training module, evaluation module and test module is as follows: The dropout unit receives data and is connected with the long-short-term memory unit, the long-short-term memory unit is connected with the data input and the fast weight unit, and the result storage unit is connected with the fast weight unit and the result. 2. the learning method based on the recursive network model of FW mechanism and LSTM as claimed in claim 1, is characterized in that: comprise the steps: Step 1. The data to be trained and tested is imported through the data import module, specifically: Step 2, the data generation module splits the data imported by the data import module through the data splitting unit to obtain training data, evaluation data and test data respectively; Step 3, the training data generation unit randomly selects the x% ratio of the data split by the data splitting unit to generate a training set; the evaluation data generation unit randomly selects the y% ratio of the data split by the data splitting unit to generate an evaluation set; test The data generating unit randomly selects the z% ratio to generate a test set from the data split by the data splitting unit; Step 4, the training label generating unit shifts each data in the training set generated by the training data generating unit by one bit to obtain the training label; the evaluation label generating unit shifts each data in the evaluation set generated by the evaluation data generating unit by one bit to obtain the evaluation label ; The test label generation unit shifts each data in the test set by one bit to obtain the test label; Step 5, the parameter setting module obtains configuration parameters according to the model scale of the text imported by the data import module, and then inputs the obtained configuration parameters into the parameter setting module; Among them, the configuration parameters include initial scale, learning rate, maximum gradient regular value, number of layers, number of steps, hidden layer size, maximum epoch number, maximum epoch value, dropout rate, decay rate, batch size, and vocab size; Step 6. The data loading unit in the loading and iteration module loads the data in the training set, evaluation set and test set according to the configuration parameters obtained in the parameter setting module, and sets the initialization data sequence number i to 1; Step 7, the model definition module uses the pseudo-random function to generate random values within the configuration range as the weight matrix parameters according to the configuration parameters in the parameter setting module, and completes the initialization of the weight parameters; Step 8. The iteration module in the loading and iteration module judges whether the data in the current data set has been sent, and operates according to the judgment result, specifically: If the data in the current data set has not been sent, send the i-th group of data, judge whether to train, evaluate or test, and skip to step 9, and skip to step 8; otherwise, stop the iteration; Step 9. Determine whether the current data is training data. If so, extract the input data according to the dropout rate. After the extracted data, skip to step 10; otherwise, skip to step 10; Step 10. Send the data input in step 9 to the long-short-term memory unit and the fast weight unit in the update unit to calculate the output vector, and at the same time use the gradient descent method to optimize the network, specifically: Step 10.1 The update unit calculates the initial hidden layer state based on the input layer weight Wx and the standard weight Wh, and calculates the initial hidden state at the current time t by formula (4): h ⁰ _t ＝f(LN(W _x *x _t + W _h *h _t-1 )) (4) Among them, the input layer weight is recorded as Wx, and the standard weight is recorded as Wh; h ₀ is the initial hidden layer state, LN is the layer regularization function; f is the activation function; x _t is the input layer data at the current time t; h _{t- 1} is the data corresponding to the state of the hidden layer at the moment before the current moment, that is, time t-1, referred to as the state of the hidden layer; The standard weight Wh is the weight propagated from the hidden layer to the next time step in the RNN network; the input layer weight Wx is the weight propagated from the input layer to the hidden layer; Step 10.2, the fast weight unit calculates the fast weight, specifically calculated by formula (5): W _A (t)＝λW _A (t-1)+ηh _t-1 h ^T _t-1 (5) Among them, W _A (t) is the fast weight at the tth moment, which is the weight that only acts on the hidden layer in each time step; the total number of updates in a time step is recorded as s+1 times; λ is the decay rate , η is the learning rate, h _t-1 is the hidden layer state corresponding to the t-1 moment; h ^T _t-1 is h _t-1 , that is, the transposition of the hidden layer state corresponding to the t-1 moment; Among them, s in the total number of times of time step update s+1 is the number of steps; Step 10.3, the fast weight unit calculates the state of the hidden layer and updates the state of the hidden layer for s times; Step 10.4, the slow weight unit calculates the normalized output; Among them, the normalized output of the network is realized by either Softmax or sigmoid function; Step 10.5, the calculation of the result storage unit is based on the normalized output calculated in step 10.4 to calculate loss and complexity perplexity; Step 10.6. The slow weight unit judges whether the last Epoch has been reached. If not, the update unit updates the state of the hidden layer and the training parameters or test parameters, adds 1 to the current i, and skips to step 8. 3. The learning method based on the recursive network model of FW mechanism and LSTM as claimed in claim 2, characterized in that: Step 1 obtains the text data by reading the text path. 4. the learning method based on the recursive network model of FW mechanism and LSTM as claimed in claim 2 is characterized in that: in step 2, splitting is specifically: the text data that step 1 imports is according to every j character is a sentence Words are split. 5. The learning method based on FW mechanism and LSTM recursive network model as claimed in claim 4, characterized in that: the value range of j is 5 to 50. 6. The learning method based on FW mechanism and LSTM recursive network model as claimed in claim 2, characterized in that: in step 3, x%+y%+z%=1. 7. The learning method based on the recursive network model of FW mechanism and LSTM as claimed in claim 2, characterized in that: in step 10.1, the activation function f is one of SeLU function, Leaky Relu function and Swish function. 8. The learning method based on FW mechanism and LSTM recursive network model as claimed in claim 2, characterized in that: in step 10.2, the decay rate ranges from 0.9 to 0.995. 9. The learning method based on FW mechanism and LSTM recursive network model as claimed in claim 2, characterized in that: in step 10.2, the learning rate ranges from 0.3 to 0.8.