CN110619886A - End-to-end voice enhancement method for low-resource Tujia language - Google Patents

Abstract

The invention discloses an end-to-end speech enhancement method for low-resource Tujia language, belongs to the field of speech signal processing, relates to the speech enhancement technology of low-resource language, and aims at the diversity, randomness and non-stationary of environmental noise in Tujia language data performance, to achieve end-to-end rapid speech enhancement processing. Including: Based on the deep convolutional generation confrontation network, establish an end-to-end low-resource Tujia speech enhancement model, perform rapid enhancement processing, realize end-to-end Tujia speech rapid enhancement processing, and effectively remove Tujia speech with almost no distortion environmental noise.

Description

An end-to-end speech enhancement method for low-resource Tujia languages

technical field

The invention belongs to the field of speech signal processing, and relates to speech enhancement technology for low-resource languages, in particular to an end-to-end speech enhancement method for low-resource Tujia languages based on deep convolution generation confrontation networks.

Background technique

Speech enhancement technology is the preprocessing part of the voice digital signal. It mainly extracts the pure original voice signal from the noisy voice signal as much as possible. Fatigue, which is a subjective measurement; the second is to improve the intelligibility of speech, which is an objective measurement. Now the speech recognition technology has entered the practical stage, but many recognition systems have higher requirements on the environment. In practical applications, environmental noise pollution will degrade the speech processing system performance. Therefore, the speech enhancement technology can effectively solve the noise pollution and improve the accuracy of the speech recognition system. At present, the speech enhancement system has been widely used in the fields of speech communication and multimedia technology.

Traditional speech enhancement algorithms include spectral subtraction, which has a small amount of calculation, and can easily control speech signal distortion and residual noise, but is prone to residual music noise; adaptive filters such as Wiener filtering and Kalman filtering need to know some characteristics or statistical properties of noise. Time-domain based subspace decomposition can also be used for speech enhancement, but works better with low SNR or white noise. With the rapid development of deep learning, the method of using deep neural network for speech enhancement has also attracted widespread attention. Compared with traditional methods in non-stationary noise processing, it has obvious advantages, but deep network models are mostly supervised. For training, the model relies on a large amount of labeled data and long-term training.

As the language passed down from generation to generation by the Tujia people in my country, Tujia language contains rich national cultural connotations. However, due to the sharp decrease in the number of users, there is a gap in the inheritance of spoken language, and there is no form of textual records. It is already facing a crisis of extinction. In addition, the scope of use of Tujia language is also extremely limited, and the areas with better preservation are located in the mountains and deep valleys with inconvenient transportation and very closed. In this case, not only the amount of data that can be collected is very limited, but also it is difficult to find a professional recording studio. The process of investigating and collecting Tujia language is in a natural environment, and it is inevitable that audio files contain noise. Animal calls, motor vehicle sounds, current sounds from acquisition equipment, and interference from multiple people talking at the same time will drown useful voice information in the noise, which will also affect the subsequent tasks of Tujia language labeling and voice recognition.

To ensure high-quality corpus, removing noise from Tujia speech data is a challenging research. Using the existing speech denoising method, it is difficult to realize Tujia language annotation and speech recognition, and the accuracy rate of Tujia speech recognition is low.

Contents of the invention

In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides an end-to-end speech enhancement method for low-resource Tujia language based on deep convolutional generative adversarial network, aiming at the diversity, randomness and abnormality of environmental noise in Tujia language data. Stability, to achieve end-to-end rapid speech enhancement processing.

The invention can lay a research foundation for the language digital resource library, improve the accuracy of subsequent speech recognition, and help phoneticians complete the recording and preservation of endangered languages, more intuitively and vividly display the language appearance and its cultural connotation, and protect the language and culture. Inheritance has important practical significance.

The technical scheme provided by the invention is:

An end-to-end speech enhancement method for low-resource Tujia language, based on a deep convolutional generative adversarial network, establishes an end-to-end low-resource Tujia speech enhancement model, realizes end-to-end Tujia speech rapid enhancement processing, and effectively removes Tujia language Ambient noise for speech, comprising the following steps:

1) Construct the Tujia language corpus, classify and segment the Tujia language recording data, obtain the original noisy corpus of Tujia language and the original clean corpus of Tujia language, and intercept the pure noise segment from the original noisy corpus of Tujia language:

11) First, according to the quality of Tujia language recording data, the Tujia language corpus is divided into two parts: noise-free data (Tujia original clean corpus) and noisy data (Tujia original noisy corpus). In the noisy data, the unvoiced segments between sentences also contain environmental noise, so the noise segments can be intercepted according to speech processing tools (such as ELAN software) to obtain pure noise segments. Specifically, in the original noisy corpus of Tujia language, there are noises in places where there are voices and where there are no voices. The segment without voice is intercepted as a pure noise segment of Tujia language.

12) The noisy data and noise-free data of Tujia language are both long storytelling (voice long sentences), which need to be segmented using voice processing tools (such as the cross-platform multi-functional phonetics professional software Praat script). The independent short sentences obtained after classification are still divided into two categories: the original noisy corpus of Tujia language and the original clean corpus of Tujia language.

2) Extended corpus:

Due to the limited amount of Tujia language voice data, Chinese voice data sets (for example: Tsinghua University 30-hour (thchs30) Chinese voice data set) will be used as the extended data of Tujia language, called Chinese original clean corpus, in order to solve the problem of Tujia language voice data Insufficient problem. The pure noise fragments intercepted in step 11) are added to the original clean corpus of Tujia language and the original clean corpus of Chinese respectively to obtain new corpus respectively, and the new corpus obtained are respectively called Tujia synthetic noise corpus and Chinese synthetic band noisy corpus.

3) Establish an end-to-end speech enhancement model:

The present invention adopts Deep Convolutional Generative Adversarial Network (DCGAN) to establish an end-to-end Tujia language speech enhancement model, and performs speech enhancement on Tujia language corpus;

The end-to-end Tujia speech enhancement model includes: a generation network and a discriminant network; the generation network adopts an end-to-end fully convolutional network structure of encoding-decoding. Using the confrontation training setting, the enhanced speech and the real clean speech are input into the discriminant network for classification, and the authenticity of the input signal is judged as much as possible, and then passed to the generation network, so that the enhanced model can direct its output waveform towards the real one. The distribution is fine-tuned until the discriminative network is difficult to distinguish the authenticity of the input signal, so as to achieve the purpose of removing the noise signal. The present invention adds spectral normalization (SpectralNormalization, SN) to each convolutional layer of the network, and constrains the Lipschitz constant of the network by limiting the spectral norm of each layer. During model training, using an unbalanced learning rate can make the model training more stable, that is, setting the learning rate and different update rates for the generation network and the discriminant network respectively.

Specifically, perform the following operations:

31) Firstly, the time-domain waveform graph of the Chinese synthetic noisy corpus is used as the input of the generating network. The waveform is divided into frames by overlapping sliding windows. The specific implementation time window is 1 second, and the overlap between frames is 500 milliseconds; then input the 11 convolutional layers in the generation network coding stage to obtain the compression vector, and the compression vector enters Generate network decoding stage. The decoding stage and the encoding stage are in a mirror image relationship. There are 11 deconvolution layers and the convolution kernel parameters are consistent with the convolution layer parameters corresponding to the encoding stage. Each deconvolution layer simultaneously receives the result of the previous deconvolution layer and the encoding The results of the symmetrical convolutional layer in the stage, the two results are passed to the next deconvolutional layer through weighted addition, and finally the enhanced Chinese clean corpus is obtained;

32) The discriminant network receives the original Chinese clean corpus and the enhanced Chinese clean corpus obtained in step 31), classifies and obtains the discriminant result (output 0 or 1) through multi-layer convolution of the discriminant network, and passes the discriminant result to the generating network to generate The network calculates the loss value according to the loss function, performs backpropagation to update the weights of each layer, and starts a new round of enhanced training on the noisy corpus; the discriminant network continues to receive the enhanced results of the generated network, and calculates the loss value according to the loss function. Iterate repeatedly in this way until the discriminant network cannot distinguish the input source (the output is set to 0.5 at this time), and then the end-to-end speech enhancement model is obtained;

4) Fine-tuning the speech enhancement model obtained in step 3) to continue training to obtain an end-to-end Tujia speech enhancement model called Fine-tuning DCGAN (FDCGAN), the specific operation is: use the Tujia speech enhancement model in step 1) The original clean corpus of the language and the synthesized noisy corpus of Tujia language obtained in step 2) are used as training data to input the end-to-end speech enhancement model obtained in step 3), and the learning rate and batch processing parameters of the model are modified for training, and finally the trained End-to-end Tujia Speech Enhancement Model FDCGAN;

5) Input the Tujia language data to be voice-enhanced into the trained end-to-end Tujia language speech enhancement model FDCGAN obtained in step 4), that is, output the enhanced Tujia language speech.

During concrete implementation, the present invention adopts the original noisy corpus of Tujia language in step 1) as test data, step 4) the Tujia language speech enhancement model that obtains is tested, and adopts speech quality evaluation tool to the Tujia language speech enhancement that the present invention provides Model validation and evaluation.

Compared with prior art, the beneficial effect of the present invention is:

Aiming at the diversity, randomness and non-stationarity of environmental noise in the Tujia speech data, the present invention proposes a speech enhancement model based on the improved deep convolution confrontation generation network, which can perform rapid enhancement processing and realize the Tujia audio files undergo end-to-end enhanced processing. Since Tujia language has low resource and very limited data volume, the present invention adopts the Chinese speech data set as an extension to make the model more generalizable. Compared with the prior art, the present invention adds spectral normalization (Spectral Normalization, SN) to each convolutional layer, and constrains the Lipschitz constant of the network by limiting the spectral norm of each layer. During model training, using an unbalanced learning rate can make the model training more stable, that is, setting the learning rate and different update rates for the generation network and the discriminant network respectively. By comparing with the existing mainstream speech enhancement methods, the results show that it can effectively remove the environmental noise in Tujia speech with almost no distortion.

Description of drawings

Fig. 1 is a block flow diagram of a specific embodiment of the method of the present invention.

Fig. 2 is a schematic structural diagram of an end-to-end speech enhancement model used in an embodiment of the present invention.

FIG. 3 is a schematic diagram of loss function value changes during network training in an embodiment of the present invention.

Fig. 4 is a schematic diagram of loss function value changes during discriminant network training in an embodiment of the present invention.

In Figure 3-Figure 4, the abscissa is the number of iterations (passes), and the ordinate is the loss function value (loss).

Fig. 5 is a spectrogram of Tujia language before enhancement in the embodiment of the present invention.

Fig. 6 is an enhanced Tujia language spectrogram in the embodiment of the present invention.

In FIGS. 5-6 , the abscissa is time (Time), and the ordinate is frequency (Hz).

Detailed ways

The present invention will be further described through the embodiments below in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

The following embodiment adopts the thchs30 Chinese corpus that comprises 27 short spoken language corpora in total, and the total duration is 7 hours, 8 minutes and 59 seconds, and the thchs30 Chinese corpus with a total duration of more than 30 hours recorded by 25 people describes in detail the speech enhancement method provided by the present invention. Implementation process.

The flow chart of the specific implementation of the method is shown in Figure 1. The invention provides an end-to-end speech enhancement method based on deep convolution confrontation generation network for low-resource Tujia language. Since Tujia language has low resource, the experimental data is expanded to build a database; the deep convolution network and generation Combining adversarial training to enhance the speech signal, and fine-tuning and retraining for Tujia language data, the final model has stronger generalization and better enhancement effect. The model directly inputs the original speech signal and outputs the enhanced speech signal. The end-to-end method can preserve the phase details of the original speech signal in the time domain. In the deep convolutional generative adversarial network, each convolutional layer uses spectral normalization, and the cost of model training is reduced by modifying the loss function and network layer parameters. The unbalanced learning rate is used when training the generation network and the discriminative network, which makes the training of the two more stable. The specific implementation steps are as follows:

Data preprocessing and database construction:

1) Divide the Tujia language dataset into two parts, one part is noisy data and the other part is noise-free data. Utilize ELAN software (annotation tool for creating, editing, visualizing and searching signs of video and audio data, providing sound technology for signs and exploiting multimedia clips) and phonetic software Praat script (a cross-platform multi-functional phonetics professional software) divide the voice data into short sentences and call them Tujia original noisy corpus and Tujia original clean corpus, and manually intercept the noise segments in the noisy data. The types of noise include rooster crowing, The quantity of chicken crowing, motor vehicle acoustic electronic equipment interference noise and other noises is shown in Table 1:

Table 1 The types and numbers of noises in Tujia language

2) Superimpose the noise segment into the original clean corpus of Tujia language and the original clean corpus of Chinese through the audio conversion and processing program sox tool. The noise superposition method is to randomly select the starting position at the sampling point, and inject different noises into the recordings of each person in the original clean Chinese corpus according to the ratio of the number of each type of noise to the total number of noises, as shown in formula (1):

Among them, N _i represents the number of noise i, M _j represents the number of recordings of the jth person in the original clean corpus of Chinese, m _ij represents the number of noise i injected into the recording of the jth person in the thchs30 corpus; the specific implementation , i=1,...,5, j=1,...,25. The noise injection method of the original clean corpus of Tujia language is the same, and the new corpus obtained in this way is called Tujia synthetic noisy corpus and Chinese synthetic noisy corpus.

The speech enhancement model training process, the end-to-end speech enhancement model is shown in Figure 2:

1) The speech waveform (denoted by z) of the original Chinese noisy corpus is input into the generation network, and its encoding end is composed of 11 1-dimensional step convolution layers with a width of 31 and a step size of 2, and each layer has a filter The numbers are: 16, 32, 32, 64, 64, 128, 128, 256, 256, 512, 1024. The decoding end and the encoding end maintain a mirror image relationship, and also contain 11 deconvolution layers with the same parameters. The arrow in Figure 2 indicates a skip connection, that is, the information of the convolutional feature map is passed to the corresponding deconvolution layer, and at the same time, the knot of the previous deconvolution layer is received, and the two results are passed to the next deconvolution through weighted addition. layers to avoid loss of detail. The activation function of each convolutional layer uses the PReLU function. The output end of the generating network is the Chinese enhanced speech waveform denoted as G(z).

2) Input the generated Chinese enhanced speech G(z) and Chinese original clean speech into the discriminant network. The discriminant network consists of a 1-dimensional binary classification convolutional network with two channels for obtaining input sources, each of which has 16384 sampling points, and the last layer is a 1×1 convolution, and each layer uses an alpha value of A LeakyRelu nonlinear activation function of 0.3. The loss function L _D of the discriminant network is written as formula (2):

Among them, x represents the pure speech; P _data represents the distribution function that the pure speech x obeys; z represents the noisy speech; P _z represents the distribution function that the noisy speech z obeys. If the input is G(z), the discriminant network output D(G(z)) is 0; if the input is x, the discriminative network output D(x) is 1.

3) The discriminant network passes the discriminative result to the generating network, and the generating network calculates the loss function L _G according to formula (3):

The two networks perform backpropagation to update the weights of each layer according to the loss value until D(G(z))=D(x)=0.5, that is, the discriminative network cannot recognize whether the input signal is the original clean speech signal or the generated network enhanced If the speech is clean, the training is complete.

4) Set the generation network and the discriminant network to update at a rate of 1 to 1. When the two networks are training one, the other network remains frozen. The generative network learning rate is 0.0001, the discriminative network learning rate is 0.0003, and the batching parameter is 24. During the model training process, the present invention selects spectral normalization and unbalanced learning rate to make the training process more stable. Spectral normalization limits the spectral norm of each layer to constrain the Lipschitz constant of the discriminant network. The upper bound of the gradient of the Lipschitz continuity function is limited, so the function is smoother, and the parameter changes will be more smooth during the optimization process of the neural network. Stable and not prone to gradient explosions. Use the learning rate a(n) and b(n) to update the parameters of the generation network and the discriminant network, expressed as formula (4) and formula (5):

Among them, θ _n , h(θ _n ,ω _n ), They are the parameter vector, stochastic descent gradient, and random vector of the nth update of the generating network; ω _n , g(θ _n ,ω _n ), is the parameter vector, stochastic descent gradient, and random vector for the nth update of the discriminant network.

5) After the above-mentioned speech enhancement model is trained using Chinese corpus, input Tujia synthetic noisy corpus and Tujia original clean corpus, and set the learning rate of generation network to 0.00006 and the learning rate of discriminant network to 0.0001 when other parameters are consistent. The batch parameter is 16, and the model is trained again to make the model generalization better. Figure 3 and Figure 4 show the loss function changes during the training of the generative network and the discriminative network in the model.

6) Finally, the original noisy corpus of Tujia language is used to test the model. Fig. 5 is a noisy Tujia language spectrogram without enhancement, and Fig. 6 is an enhanced Tujia language spectrogram. According to the comparison method proposed by the present invention, Effectively remove environmental noise in Tujia language data.

The enhancement method proposed by the present invention for Tujia language voice data is compared with conventional traditional voice enhancement methods and the voice enhancement method based on deep recurrent neural network, and the evaluation index selects subjective voice quality evaluation (Perceptual evaluation of speech quality, PESQ) And the average opinion score - voice quality index (Mean Opinion Score Listening Quality Objective, MOSLQO). PESQ is a typical algorithm in voice quality evaluation. It adopts a linear scoring system and is widely used. The value is between -0.5 and 4.5, indicating the voice quality of the input test voice compared with the output voice. The higher the score , the better the voice quality. The evaluation results are shown in Table 2:

Table 2 Comparison of evaluation results of different enhancement methods

The results in Table 2 show that the end-to-end speech enhancement method based on the deep convolutional generative adversarial network proposed by the present invention can effectively remove the environmental noise in Tujia language, has a better enhancement effect, and lays a stable foundation for speech recognition.

It should be noted that the purpose of the disclosed embodiments is to help further understand the present invention, but those skilled in the art can understand that various replacements and modifications are possible without departing from the spirit and scope of the present invention and the appended claims of. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the protection scope of the present invention is subject to the scope defined in the claims.

Claims (10)

1. An end-to-end speech enhancement method for low-resource Tujia language, which is characterized in that, based on deep convolution generation confrontation network, an end-to-end low-resource Tujia speech enhancement model is established to realize end-to-end Tujia speech rapid enhancement Process, effectively remove the environmental noise of Tujia speech; comprise the following steps: 1) Construct the Tujia language corpus, classify and segment the Tujia language recording data, obtain the original noisy corpus of Tujia language and the original clean corpus of Tujia language, and obtain pure noise fragments from the original noisy corpus of Tujia language; 2) Extended corpus: use the original clean corpus of Chinese as the extended data of Tujia language, add the pure noise fragments to the original clean corpus of Tujia language and the original clean corpus of Chinese respectively, and call the new corpus of Tujia synthetic noise corpus respectively Synthesize noisy corpus with Chinese; 3) Establish and train an end-to-end speech enhancement model; including: Establish an end-to-end Tujia speech enhancement model using deep convolutional generative confrontation network DCGAN; The end-to-end Tujia language speech enhancement model includes: a generation network and a discrimination network; The generation network adopts an encoding-decoding end-to-end fully convolutional network structure; Add spectral normalization to each convolutional layer of the network, and constrain the Lipschitz constant of the network by limiting the spectral norm of each convolutional layer; Using the adversarial training setting, the enhanced speech and the real clean speech are input into the discriminant network for classification, and the authenticity of the input signal is judged, and then passed to the generation network, so that the end-to-end Tujia speech enhancement model will model the output waveform towards Real distribution fine-tuning, thereby achieving the purpose of removing noise signals; 4) continue fine-tuning training to the end-to-end speech enhancement model obtained in step 3), obtain the trained end-to-end Tujia speech enhancement model FDCGAN; specific operations are: Use the original clean corpus of Tujia language in step 1) and the synthesized noisy corpus of Tujia language obtained in step 2) as training data input step 3) The end-to-end speech enhancement model obtained, and modify the learning rate and batch processing parameters of the model to carry out Training, and finally get the trained end-to-end Tujia speech enhancement model FDCGAN; 5) Input the Tujia language data to be voice-enhanced into the trained end-to-end Tujia language speech enhancement model FDCGAN obtained in step 4), that is, output the enhanced Tujia language speech. 2. as claimed in claim 1 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, step 1) builds Tujia language corpus, specifically comprises the following operations: 11) First, according to the quality of Tujia language recording data, the Tujia language corpus is divided into two parts: noise-free data and noisy data, which are the original noisy corpus of Tujia language and the original clean corpus of Tujia language; Then use speech processing tools to intercept the noise fragments in the noisy data to obtain pure noise fragments; 12) Segment the long sentences of the speech data to obtain independent short sentences; the short sentences are still divided into two categories: the original noisy corpus of Tujia language and the original clean corpus of Tujia language. 3. as claimed in claim 1 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, step 2) expands corpus, specifically adopts Tsinghua University 30 hours Chinese speech data set thchs30 as the extended data of Tujia language. 4. as claimed in claim 1 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, step 3) trains the speech enhancement model of end-to-end, specifically performs the following operations: 31) using the time-domain waveform diagram of the Chinese synthetic noise corpus as the input of the generation network; The time-domain waveform diagram is input to generate multiple convolutional layers in the network coding stage by overlapping sliding windows to obtain a compressed vector; The compressed vector enters the generation network decoding stage; The generation network decoding stage is in a mirror image relationship with the encoding stage, where the deconvolution layer and convolution kernel parameters are consistent with the convolution layer parameters corresponding to the encoding stage; each deconvolution layer simultaneously receives the results of the previous deconvolution layer and The results of the relatively symmetrical convolutional layer in the encoding stage are passed to the next deconvolutional layer through weighted addition, thereby obtaining an enhanced Chinese clean corpus; 32) The discriminative network receives the original Chinese clean corpus and the enhanced Chinese clean corpus obtained in step 31), and classifies through multi-layer convolution of the discriminant network to obtain a discriminative result; Pass the discriminative result to the generation network; By calculating the network loss function, the loop training is passed to each other until the discriminant network cannot distinguish the source of the input, and an end-to-end speech enhancement model is obtained. 5. the end-to-end speech enhancement method for low-resource Tujia language as claimed in claim 4, is characterized in that, described multiple convolution layers are 11 convolution layers. 6. as claimed in claim 1 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, step 2), the pure noise segment is added respectively in the original clean corpus of Tujia language and the original clean corpus of Chinese, specifically by Audio conversion and processing tools for overlay; use the following methods: Randomly select the starting position at the sampling point, and inject different noises into the recordings of each person in the original clean Chinese corpus according to the ratio of the number of each type of noise to the total number of noises, expressed as formula (1): Among them, N _i represents the number of noise i, M _j represents the number of recordings of the jth person in the original clean Chinese corpus, and m _ij represents the number of noise i injected into the recordings of the jth person in the thchs30 corpus. 7. the end-to-end speech enhancement method for low-resource Tujia language as claimed in claim 1, is characterized in that, in step 3), the activation function of each convolutional layer of generation network adopts PReLU function. 8. as claimed in claim 1, for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, in step 3), the discriminant network is composed of a 1-dimensional binary classification convolutional network, and there are two sources of input The channel, each channel is 16384 sampling points, and the last layer is a 1×1 convolutional layer; Each layer of the discriminant network uses the LeakyRelu nonlinear activation function; The loss function L _D of the discriminant network is expressed as formula (2): Among them, x represents pure speech; P _data represents the distribution function that pure speech x obeys; z is noisy speech; let the distribution of pure speech x obey P _data , and the distribution of noisy speech z obey p _z ; if the input is G(z) then The discriminative network output D(G(z)) is 0; if the input is x, the discriminative network output D(x) is 1; The discriminant network passes the discriminative result to the generating network, and the generating network calculates the loss function L _G according to formula (3): The two networks perform backpropagation to update the weights of each layer according to the loss value until the network cannot identify whether the input signal is the original clean speech signal or the clean speech after network enhancement is generated, then the training is completed. 9. as claimed in claim 8 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, the specific setting generation network and the discriminant network are updated at a rate of 1 to 1, and one of the generation network and the discriminant network is training , the other network remains frozen. 10. as claimed in claim 9 for the end-to-end speech enhancement method of low-resource Tujia language, it is characterized in that, specifically adopt spectrum normalization and unbalanced learning rate to make training process stable; Described spectrum normalization limits each The spectral norm of the layer, the Lipschitz constant of the constrained discriminant network; Use the learning rate a(n) and b(n) to update the parameters of the generation network and the discriminant network, expressed as formula (4) and formula (6): Among them, θ _n , h(θ _n ,ω _n ), They are the parameter vector, stochastic descent gradient, and random vector of the nth update of the generating network; ω _n , g(θ _n ,ω _n ), They are the parameter vector, random descent gradient, and random vector of the nth update of the discriminant network.