CN118335092A - Voice compression method and system based on multi-scale residual error attention - Google Patents

Abstract

The invention belongs to the technical field of voice signal processing, and provides a voice compression method and a voice compression system based on multi-scale residual error attention, wherein the voice compression method comprises the steps of obtaining a voice signal; performing convolution operation on the voice signal to obtain a first feature, and performing operation on the first feature to obtain residual error and identity mapping of the first feature; adding the residual error and the identity mapping to obtain a first output characteristic, extracting the first output characteristic, obtaining an attention score through multiple operations, multiplying the attention score by the residual error and the identity mapping respectively, and obtaining a third output characteristic through multiple operations; performing multistage iterative quantization on the third output characteristic to obtain a first vector, finding a corresponding quantized vector in a codebook according to the index of the received first vector by a second network, and adding all the quantized vectors to obtain a reconstructed vector; and decoding the reconstructed vector to output the synthesized voice, and judging the authenticity of the generated voice through a discriminator. The invention can improve the quality of the synthesized voice.

Description

Speech compression method and system based on multi-scale residual attention

Technical Field

The present disclosure relates to the technical field of speech signal processing, and in particular to a speech compression method and system based on multi-scale residual attention.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

Ultra-low-rate speech compression coding technology based on deep learning is also called neural vocoder. Low-rate speech coding technology has a wide range of application needs in satellite communications, shortwave communications, underwater acoustic communications, and confidential communications. For example, in extremely harsh mountain communication environments, communication equipment must not only withstand extreme low temperatures and strong winds, but also maintain stable operation under limited energy supply. The coding rate of speech coding is often lower than 2400bps. When the speech coding rate is reduced, the quality of speech synthesis will be affected, so speech compression technology is particularly important.

Speech signals are complex signals that contain rich information, such as pitch, rhythm, timbre, etc., which are expressed in different frequency and time dimensions. Current speech coding methods usually only focus on features at a certain scale, while ignoring multi-scale information, resulting in limited speech synthesis quality and loss of feature information integrity. In addition, hierarchical information has not received sufficient attention, resulting in the model being unable to fully extract detailed information in speech. Therefore, it is necessary to explore more comprehensive and integrated feature representation methods to improve the system's ability to understand and express speech signals.

Summary of the invention

In response to the above-mentioned defects, the present invention proposes a speech compression method and system based on multi-scale residual attention, introduces multi-scale residual attention technology into a low-rate neural vocoder, and uses multi-scale residual attention to learn more refined features, so as to improve the quality of synthesized speech.

In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

The first aspect of the present disclosure provides a speech compression method based on multi-scale residual attention, comprising the following steps:

S1 obtains multi-frame speech signals at a low rate;

S2 inputs the speech signal into the first network to perform a convolution operation to obtain a first feature, performs multiple operations on the first feature, and obtains a residual of the first feature and an identity mapping of the first feature;

S3 adds the residual and the identity mapping to obtain a first output feature, extracts global and local features of the first output feature, obtains a fused feature according to the global and local features, operates the fused feature to obtain an attention score of the fused feature, multiplies the attention score by the residual and the identity mapping respectively, and adds the two results to obtain a second output feature;

S4 inputs the second output feature into a full-band feature extractor to obtain a third output feature;

S5 performs multi-level iterative quantization on the third output feature to obtain a first vector, transmits the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, adds all the quantization vectors, and obtains a reconstructed vector;

S6 decodes the reconstructed vector to output the synthesized speech, and determines the authenticity of the generated speech through a discriminator.

As a further implementation method, the first feature is operated multiple times to obtain the residual of the first feature. Specifically, two groups of convolution operations with convolution kernel sizes of three and five are performed on the first feature, and the convolution operations between the groups are multiplied by two. The convolved features are concatenated, and multi-scale fusion is performed through one-dimensional convolution to obtain the residual of the first feature.

As a further implementation manner, a skip connection operation is performed on the first feature to obtain an identity mapping of the first feature.

As a further implementation, the global feature of the first output feature is extracted, specifically:

An average pooling operation is performed on the first output feature, and a convolution operation with a convolution kernel of 1 is performed twice on the first output feature after average pooling, and batch normalization is performed on the convolved feature.

As a further implementation, extracting local features of the first output feature is specifically as follows:

The first output feature is convolved twice with a convolution kernel of 1, and the convolved features are batch normalized.

As a further implementation, before the second output feature is input into the full-band feature extractor, the following steps are also included:

The second output feature is input into the multi-scale residual attention block again for feature extraction, and the operation is repeated multiple times.

As a further implementation method, the authenticity of the generated speech is judged by a discriminator, specifically:

Multi-scale STFT discriminators and multi-cycle discriminators are introduced to judge the authenticity of generated speech, and adversarial training is used to encourage the generator to produce more realistic synthetic speech.

A second aspect of the present disclosure provides a speech compression system based on multi-scale residual attention, comprising:

The data acquisition module is configured to: acquire a multi-frame speech signal at a low rate;

The multi-scale residual module is configured to: input the speech signal into the first network to perform a convolution operation to obtain a first feature, perform multiple operations on the first feature to obtain a residual of the first feature and an identity mapping of the first feature;

A channel attention module is configured to: add the residual and the identity mapping to obtain a first output feature, extract global and local features of the first output feature, obtain a fused feature according to the global and local features, operate the fused feature to obtain an attention score of the fused feature, multiply the attention score by the residual and the identity mapping respectively, and add the two results to obtain a second output feature;

The full-band feature extraction module is configured to: input the second output feature into the full-band feature extractor to obtain a third output feature;

The quantization and dequantization module is configured to: perform multi-level iterative quantization on the third output feature to obtain a first vector, transmit the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, and adds all the quantization vectors to obtain a reconstructed vector;

The speech synthesis module is configured to: decode the reconstructed vector to output the synthesized speech, and judge the authenticity of the generated speech through a discriminator.

The third aspect of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the speech compression method based on multi-scale residual attention described in the first aspect of the present disclosure are implemented.

The fourth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multi-scale residual attention-based speech compression method described in the first aspect of the present disclosure are implemented.

Compared with the prior art, the present invention has the following beneficial effects:

The speech compression method based on multi-scale residual attention proposed in the present disclosure introduces multi-scale residual attention technology into the low-rate neural vocoder, and uses multi-scale residual attention to learn more important features. The multi-scale residual attention block consists of two parts: a multi-scale residual block and an attention fusion module. The multi-scale residual block captures speech detail features of different scales through cross-learning and fuses them. The attention feature fusion module further focuses on finer features by applying different attention to the residual block and the identity mapping to solve the problem of inconsistent hierarchical information. The multi-level vector quantization technology is used to quantize the encoded features, and the quantization index is transmitted to the decoding end. The decoding end retrieves the corresponding quantization vector in the codebook according to the received index to reconstruct the input vector and thus reconstruct the speech. By introducing multi-scale residual attention, the synthesis quality of speech is improved.

Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

FIG1 is a schematic diagram of the structure of the neural vocoder codec disclosed in the present invention;

FIG2 is a flowchart of multi-stage vector quantization of the present disclosure;

FIG3 is a flow chart of a speech compression method based on multi-scale residual attention disclosed in the present invention;

FIG4 is a schematic diagram of the structure of a multi-scale residual block disclosed in the present invention;

FIG5 is a schematic diagram of the structure of the channel attention block of the present invention.

Detailed ways

The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

Embodiment 1

As shown in FIG1 , the first embodiment of the present disclosure provides a speech compression method based on multi-scale residual attention, comprising the following steps:

S1 obtains multi-frame speech signals at a low rate;

S2 inputs the speech signal into the first network to perform a convolution operation to obtain a first feature, performs multiple operations on the first feature, and obtains a residual of the first feature and an identity mapping of the first feature;

S3 adds the residual and the identity mapping to obtain a first output feature, extracts global and local features of the first output feature, obtains a fused feature according to the global and local features, operates the fused feature to obtain an attention score of the fused feature, multiplies the attention score by the residual and the identity mapping respectively, and adds the two results to obtain a second output feature;

S4 inputs the second output feature into a full-band feature extractor to obtain a third output feature;

S5 performs multi-level iterative quantization on the third output feature to obtain a first vector, transmits the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, adds all the quantization vectors, and obtains a reconstructed vector;

S6 decodes the reconstructed vector to output the synthesized speech, and determines the authenticity of the generated speech through a discriminator.

First, the acquired speech signal is a speech sampled at 8KHz, and the speech sequence is composed of express, is the number of voice channels, is the total sample points of speech, where d is the duration of speech, is the sampling rate of the speech.

As shown in FIG3 , the first network (encoder) of the present disclosure is composed of a one-dimensional convolution, a multi-scale residual attention block, a full-band feature extractor, and a gated recurrent unit. First, the speech signal passes through the channel =32, and a one-dimensional convolution with a kernel size of K to obtain the first feature, and then the first feature is sequentially input into four multi-scale residual attention blocks, each of which consists of a multi-scale residual block and a channel attention block. As shown in FIG4, the multi-scale residual block contains two groups of convolutions with kernel sizes of 3 and 5, respectively. The convolutions between the groups are multiplied by each other, and the features after convolution are spliced. After multi-scale fusion through a one-dimensional convolution with a kernel size of 1, the residual R is obtained, and the identity map I of the input feature is obtained by skipping the connection. Then, after the channel attention module shown in FIG5, the residual R is added to the identity map to obtain the output feature M. The channel attention module extracts global and local features from M. For global features, it first undergoes average pooling with an output of 1, then passes through two layers of convolution layers with a convolution kernel of 1, and the features after convolution are batch normalized. For local features, after two layers of convolution layers with a convolution kernel of 1, the features after convolution are also batch normalized. The global features and local features are added together to obtain the fused feature T, and Sigmoid activation is performed on it to obtain the attention score of the fused feature. Then multiply the attention score by the residual to get , multiply the attention score difference by the identity mapping to get , and finally add them together to get the multi-scale residual attention block output , which is the second output feature. The multi-scale feature fusion module is followed by a one-dimensional convolution with downsampling, with a stride of S and a convolution kernel twice the stride S. The number of channels is doubled during the downsampling process.

The downsampled second output feature is input into the multi-scale residual attention block again for feature extraction, and the operation is repeated multiple times. In this embodiment, four multi-scale residual attention blocks are preferably used to perform feature extraction in sequence.

The features extracted by multiple multi-scale residual attention blocks are input into the full-band feature extractor, which consists of four stacked temporal convolutional network blocks with different dilation factors, with a kernel of 3, a dilation factor of D, and a channel of The temporal convolutional network block consists of an input one-dimensional convolution, a depth-separable convolution, and an output one-dimensional Conv. A parameterized PReLU activation function and a normalization layer are inserted between adjacent convolutions, and a residual connection is added. The full-band feature extractor is followed by a gated recurrent unit with a channel number of Finally, after the kernel size is K and the output channel is The one-dimensional convolution layer of , obtains the third output feature. =32, K=7, S=(1, 4, 5, 8), D=(1, 2, 5, 9), =512.

As shown in Figure 2, the third output feature is quantized using a multi-level quantizer. The codebook size is designed to be M, the codebook dimension is C, and the number of potential frame vectors is N. Each codebook can encode For codebook initialization, the K-Means algorithm is used to cluster M clusters to obtain the initialized codebook First, obtain the encoded latent frame vector , the multi-level quantizer quantizes the potential frame vector z. For example, the first-level quantizer obtains the best matching quantization vector The second-level quantizer obtains the most matching quantization vector , the third-level quantizer obtains the most matching quantization vector , the remaining quantizers are iterated in turn, and the codebook search algorithm uses the Euclidean distance. The index is transmitted to the second network. Where M=256, C=512, N=50.

The quantized vector is packaged into a binary byte stream for transmission, and the transmitted binary bytes are unpacked. The second network (decoding end) obtains the dequantized features in the multi-level quantizer according to the index. , and add them up in sequence to get the final potential frame vector .

The dequantized speech features The speech is reconstructed by inputting it into the second network (decoder) which has the same structure as the first network (encoder) but is symmetrically inverted. First, it is convolved in one dimension with a kernel size of 7 and a channel of . Following the gated recurrent unit and the full-band feature extractor, the channels remain unchanged. Then, after four multi-scale residual attention modules, each multi-scale residual attention module is followed by a one-dimensional transposed convolution with a stride of E and a kernel twice the stride of E, and the number of channels is doubled during the upsampling process. Finally, through a one-dimensional convolution with a kernel of 7, the number of channels is Finally, we get the reconstructed speech. =512, E=(8,5,4,1), =32.

To further improve the performance of the generator, a multi-scale STFT discriminator (MS-STFT) and a multi-period discriminator (MPD) are introduced to judge the authenticity of the generated speech, and the generator is encouraged to produce more realistic synthetic speech through adversarial training. The MS-STFT discriminator consists of the same structured network operating on a multi-scale complex-valued STFT, where the real and imaginary parts are connected. Each sub-network consists of a 2D convolution layer (using a kernel size of 3x8 with 32 channels), followed by a 2D convolution, increasing at a dilation rate of D in the time dimension and a stride of 2 in the frequency axis. The final 2D convolution with a kernel size of 3x3 and a stride of (1,1) provides the final prediction. We use 5 different scales with STFT window lengths of (2048, 1024, 51, 256, 128). MPD is a mixture of sub-discriminators, each of which only accepts equally spaced samples of the input speech. The space is given as the period p. The sub-discriminators are designed to capture different implicit structures from each other by looking at different parts of the input audio. We set the period p to [2, 3, 5, 7, 11] to avoid overlap as much as possible. The 1D original audio of length T is first reshaped into 2D data of height T/p and width p, and then 2D convolution is applied to the reshaped data. In each convolution layer of MPD, the kernel size on the width axis is restricted to 1 to process periodic samples independently.

Embodiment 2

Embodiment 2 of the present disclosure provides a speech compression system based on multi-scale residual attention, including:

The data acquisition module is configured to: acquire a multi-frame speech signal at a low rate;

The multi-scale residual module is configured to: input the speech signal into the first network to perform a convolution operation to obtain a first feature, perform multiple operations on the first feature to obtain a residual of the first feature and an identity mapping of the first feature;

A channel attention module is configured to: add the residual and the identity mapping to obtain a first output feature, extract global and local features of the first output feature, obtain a fused feature according to the global and local features, operate the fused feature to obtain an attention score of the fused feature, multiply the attention score by the residual and the identity mapping respectively, and add the two results to obtain a second output feature;

The full-band feature extraction module is configured to: input the second output feature into the full-band feature extractor to obtain a third output feature;

The quantization and dequantization module is configured to: perform multi-level iterative quantization on the third output feature to obtain a first vector, transmit the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, and adds all the quantization vectors to obtain a reconstructed vector;

The speech synthesis module is configured to: decode the reconstructed vector to output the synthesized speech, and judge the authenticity of the generated speech through a discriminator.

The more detailed steps are the same as those in the first embodiment and will not be repeated here.

Embodiment 3

Embodiment 3 of the present disclosure provides a medium on which a program is stored. When the program is executed by a processor, the steps in the speech compression method based on multi-scale residual attention as described in Embodiment 1 of the present disclosure are implemented.

The more detailed steps are the same as those in the first embodiment and will not be repeated here.

Embodiment 4

Embodiment 4 of the present disclosure provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the speech compression method based on multi-scale residual attention as described in Embodiment 1 of the present disclosure are implemented.

The more detailed steps are the same as those in the first embodiment and will not be repeated here.

The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims (10)

1. A speech compression method based on multi-scale residual attention, characterized in that it comprises the following steps: S1 obtains multi-frame speech signals at a low rate; S2 inputs the speech signal into the first network to perform a convolution operation to obtain a first feature, performs multiple operations on the first feature, and obtains a residual of the first feature and an identity mapping of the first feature; S3 adds the residual and the identity mapping to obtain a first output feature, extracts global and local features of the first output feature, obtains a fused feature according to the global and local features, operates the fused feature to obtain an attention score of the fused feature, multiplies the attention score by the residual and the identity mapping respectively, and adds the two results to obtain a second output feature; S4 inputs the second output feature into a full-band feature extractor to obtain a third output feature; S5 performs multi-level iterative quantization on the third output feature to obtain a first vector, transmits the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, adds all the quantization vectors, and obtains a reconstructed vector; S6 decodes the reconstructed vector to output the synthesized speech, and determines the authenticity of the generated speech through a discriminator. 2. The speech compression method based on multi-scale residual attention as described in claim 1 is characterized in that the first feature is operated multiple times to obtain the residual of the first feature, specifically: two groups of convolution operations with convolution kernel sizes of three and five are performed on the first feature, and the convolution operations between the groups are multiplied by two, the features after convolution are spliced, and multi-scale fusion is performed through one-dimensional convolution to obtain the residual of the first feature. 3. The speech compression method based on multi-scale residual attention as described in claim 2 is characterized in that a skip connection operation is performed on the first feature to obtain an identity mapping of the first feature. 4. The speech compression method based on multi-scale residual attention as claimed in claim 1, characterized in that the global feature of the first output feature is extracted, specifically: An average pooling operation is performed on the first output feature, and a convolution operation with a convolution kernel of 1 is performed twice on the first output feature after average pooling, and batch normalization is performed on the convolved feature. 5. The speech compression method based on multi-scale residual attention as claimed in claim 4, characterized in that the local features of the first output features are extracted, specifically: The first output feature is convolved twice with a convolution kernel of 1, and the convolved features are batch normalized. 6. The speech compression method based on multi-scale residual attention as described in claim 1, characterized in that the second output feature is input into the full-band feature extractor, and further comprises the following steps: The second output feature is input into the multi-scale residual attention block again for feature extraction, and the operation is repeated multiple times. 7. The speech compression method based on multi-scale residual attention as claimed in claim 1, characterized in that the authenticity of the generated speech is judged by a discriminator, specifically: Multi-scale STFT discriminators and multi-cycle discriminators are introduced to judge the authenticity of generated speech, and adversarial training is used to encourage the generator to produce more realistic synthetic speech. 8. A speech compression system based on multi-scale residual attention, characterized by comprising: The data acquisition module is configured to: acquire a multi-frame speech signal at a low rate; The multi-scale residual module is configured to: input the speech signal into the first network to perform a convolution operation to obtain a first feature, perform multiple operations on the first feature to obtain a residual of the first feature and an identity mapping of the first feature; A channel attention module is configured to: add the residual and the identity mapping to obtain a first output feature, extract global and local features of the first output feature, obtain a fused feature according to the global and local features, operate the fused feature to obtain an attention score of the fused feature, multiply the attention score by the residual and the identity mapping respectively, and add the two results to obtain a second output feature; The full-band feature extraction module is configured to: input the second output feature into the full-band feature extractor to obtain a third output feature; The quantization and dequantization module is configured to: perform multi-level iterative quantization on the third output feature to obtain a first vector, transmit the index of the first vector to the second network, and the second network finds the corresponding quantization vector in the codebook according to the received index, and adds all the quantization vectors to obtain a reconstructed vector; The speech synthesis module is configured to: decode the reconstructed vector to output the synthesized speech, and judge the authenticity of the generated speech through a discriminator. 9. A medium having a program stored thereon, characterized in that when the program is executed by a processor, the steps in the speech compression method based on multi-scale residual attention as described in any one of claims 1 to 7 are implemented. 10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multi-scale residual attention-based speech compression method as described in any one of claims 1 to 7 are implemented.