JP7567716B2 - Estimation method, estimation program, deep neural network device, and estimation device - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies [Disclosure 1] 1. Issue date: August 26, 2020 2. Publication: The Acoustical Society of Japan, General Incorporated Foundation, 2020 Autumn Research Presentation Meeting (Proceedings of Lectures) https://acoustics.jp/annualmeeting/past-meetings/ 3. Disclosure by: Yuma Koizumi, Noboru Harada, Kohei Yatabe, Yoshinori Masuyama, Yasuhiro Oikawa [Disclosure 2] 1. Date held: September 9, 2020 to September 11, 2020 (publication date: September 10, 2020) 2. Name of meeting: The Acoustical Society of Japan, General Incorporated Foundation, 2020 Autumn Research Presentation Meeting (held online) https://acoustics.jp/annualmeeting/past-meetings/ jp/annualmeeting/past-meetings/ 3. Disclosure: Yuma Koizumi, Noboru Harada, Kohei Yatabe, Yoshinori Masuyama, Yasuhiro Oikawa [Disclosure fact 3] 1. Website posting date: October 28, 2020 2. Website address IEEE signal processing (vol15, No1, Jan 2021) https://ieeexplorer.ieee.org/document/9242279 3. Disclosure: Yuma Koizumi, Noboru Harada, Kohei Yatabe, Yoshinori Masuyama, Yasuhiro Oikawa

The present invention relates to an acoustic signal restoration technique that restores a phase spectrum from only the amplitude spectrum.

Many acoustic signal processing applications, such as speech synthesis and speech enhancement, convert the observed signal (waveform) in the time domain into the time-frequency domain using a short-time Fourier transform (STFT) or similar to perform processing. The STFT spectrum is a complex number, and both the amplitude spectrogram and phase spectrogram are required to restore the time signal from the STFT spectrogram. However, the phase spectrum is difficult to handle. For this reason, in speech synthesis and speech enhancement, only the amplitude spectrum is estimated and controlled, and the phase spectrum is often substituted with the minimum phase or observed phase before being converted back into a time signal.

The amplitude spectrogram and phase spectrogram are not independent variables, so when one is controlled, the other must be a corresponding variable. Therefore, in speech synthesis and speech enhancement, inconsistencies between amplitude and phase can sometimes result in a decrease in the quality of the output sound.

The technology disclosed in Non-Patent Document 1 is known as a technology for estimating a phase spectrogram that is consistent with the amplitude from an amplitude spectrogram. The technology disclosed in Non-Patent Document 1 (called the Griffin-Lim algorithm) is a technology for estimating a consistent phase spectrogram from an amplitude spectrogram A by repeating the following steps:

Here, X is a complex spectrogram with amplitude A, G is STFT, G ^† is inverse STFT, ◎ is element-wise multiplication, : is element-wise division, and |·| is element-wise absolute value calculation. Equations (1) and (2) are equivalent to solving the following optimization problem.

Here, |||| _Fro represents the Frobenius norm. Note that B is a set of spectrograms with amplitude A. As described above, when the STFT and inverse STFT of equation (1) are performed on complex spectrogram X in order to substitute the minimum phase or the observed phase for the phase spectrum, the original complex spectrogram X cannot be restored. Therefore, the amplitude is fixed to the given amplitude spectrogram A by equation (2), and the phase is found by equation (3) so as to obtain a correct short-time Fourier transform representation.

The method disclosed in Non-Patent Document 1 is applicable to any acoustic signal, but requires a huge number of iterations. The reason for the huge number of iterations is that the optimization framework does not make any assumptions about the statistical properties of the desired acoustic signal to be restored.

On the other hand, Non-Patent Document 2 proposes a method of incorporating deep learning into the Griffin-Lim algorithm disclosed in Non-Patent Document 1. In other words, the statistical properties of the signal to be restored are incorporated using a deep neural network (DNN) trained using learning data. Figure 8 shows the configuration of the estimation device disclosed in Non-Patent Document 2. The estimation device 100 includes M estimation units 110-m (m = 0, 1, 2, ..., M-1, M is an integer equal to or greater than 1) and a phase assignment unit 120.

Figure 9 is a block diagram showing the configuration of the estimation unit 110-m. The estimation unit 110-m is composed of a phase assignment unit 111 corresponding to equation (2), a conversion unit 112 corresponding to equation (1), and a phase modification unit 113 that brings the phase of the complex spectrogram X closer to the phase of the desired acoustic signal based on the statistical properties of the training acoustic signal corresponding to the desired acoustic signal. The phase modification unit 113 is composed of a DNN 114 and a subtraction unit 115.

In the configuration shown in Figures 8 and 9, by performing processing using a DNN after one iteration of the Griffin-Lim algorithm, consistent phase estimation that takes into account the statistical properties of the signal to be restored is realized. The configuration in Figures 8 and 9 is equivalent to stacking the internal DNN for several M iterations. In other words, by controlling the number of iterations M of the estimation unit 110-m, the scale of the DNN during processing can be changed. Reducing the number of iterations M is equivalent to using a shallow DNN, and although processing performance decreases, high-speed calculations become possible. On the other hand, increasing the number of iterations M is equivalent to using a deep DNN, and although the processing speed becomes slower, high-quality output sound can be obtained.

The DNN used here can be one that has been trained in some way from the training data of the signal to be restored, and can be any process that brings the phase of the output sound of the Griffin-Lim algorithm closer to the signal to be restored. As an example of a training method for a DNN, the following example of residual learning is shown.

Y ^[m] = P _B (X ^[m] ) ... (4)
Z ^[m] = P _C (Y ^[m] ) ... (5)
X ^[m+1] = E (X ^[m] ) ... (6)
=Z ^[m] -F _θ (X ^[m] , Y ^[m] , Z ^[m] ) ... (7)

Y ^[m] is the output signal of the phase adding unit 111, Z ^[m] is the output signal of the conversion unit 112, and X ^[m+1] is the output signal of the phase changing unit 113. Here, _Fθ is a DNN consisting of a real convolution layer and a gated linear layer. In other words, the configuration of FIG. 9 is configured such that the DNN, which has been trained based on the statistical properties of the signal to be restored, removes (subtracts) the distortion and estimation error generated by the Griffin-Lim algorithm. Here, the DNN does not directly estimate the signal to be restored, but estimates components that are not the signal to be restored. The DNN is trained to minimize, for example, the following objective function.

where X ^* is the true complex spectrogram. X , Y , and Z are given by:

N is complex Gaussian noise. However, the Griffin-Lim algorithm is a process for restoring only the phase spectrum. Therefore, the amplitude of Y is set to match the amplitude of the true complex spectrogram X ^* .

The learning stage of the DNN disclosed in Non-Patent Document 2 will be described. Fig. 10 is a block diagram showing the configuration of a conventional learning device. The learning device 200 shown in Fig. 10 receives as input learning data of a signal to be restored (clean audio signal X ^(L)* , expressed as a complex spectrogram), the amplitude spectrogram A(L ^{) of the clean audio signal X(L )} *, noise N, and parameters required for various optimizations.

The learning device 200 is composed of a noise addition unit 209, a phase assignment unit 211, a conversion unit 212, a DNN 213, a subtraction unit 214, and a parameter update unit 215.

Figure 11 is a block diagram showing the configuration of the DNN 213. The DNN 213 is composed of multiple real-valued convolutional layers 2130-2134 and multiple gated linear layers 2135-2138. In Figure 11, s represents the convolution stride, c represents the number of channels, and k represents the kernel size.

Fig. 12 is a flowchart for explaining the operation of the learning device 200. For example, an initialization unit (not shown) initializes a parameter θ of the DNN 213 with a random number (step S100 in Fig. 12).
The noise adding unit 209 receives the clean audio signal X ^(L)* and noise N, adds the noise N to the clean audio signal X ^(L)* , and outputs a complex spectrogram X tilde (Step S101 in FIG. 12).

The phase assigning unit 211 receives the complex spectrogram X{tilde} and the amplitude spectrogram A ^(L) , assigns the phase of the complex spectrogram X{tilde} to the amplitude spectrogram A ^(L) , as shown in the following equation, and outputs the phase-assigned signal Y{tilde} (step S102 in FIG. 12).

As described above, ◎ represents multiplication for each element, : represents division for each element, and |·| represents an absolute value calculation for each element. Since equation (13) multiplies each element of complex spectrogram X{tilde} by each element of amplitude spectrogram A ^(L) and divides the multiplication result by the amplitude spectrogram |X{tilde} of complex spectrogram X{tilde}, it can be said that this is a process of converting the amplitude of complex spectrogram X{tilde} into the magnitude of amplitude spectrogram A ^(L) .

The conversion unit 212 receives the signal Y tilde as input, converts the signal Y tilde into a time waveform by the inverse short-time Fourier transform G† according to the following formula, and converts the converted time waveform into a frequency domain signal Z tilde by the short-time Fourier transform G corresponding to the inverse short-time Fourier transform G† and outputs it (Figure 12, step S103).

The DNN 213 receives the complex spectrogram X tilde, the signal Y tilde, and the signal Z tilde as input, estimates the distortion or estimation error generated by the Griffin-Lim algorithm, and outputs an estimated value F _θ (X tilde, Y tilde, Z tilde) (step S104 in FIG. 12).

On the other hand, the subtraction unit 214 obtains and outputs the difference Z tilde -X ^{( L)* between the signal Z tilde and the clean audio signal X (L} )* (step S105 in FIG. 12).

The parameter update unit 215 receives the difference Z tilde -X ^(L)* and the estimated value F _θ (X tilde, Y tilde, Z tilde) as input, and uses these values to update the parameter θ of the DNN 213 so as to minimize the following objective function (step S106 in FIG. 12).

As a learning method, a stochastic steepest descent method or the like may be used. The learning rate may be set to, for example, about 10 ^-5 .
The parameter update unit 215 determines whether or not a predetermined condition is satisfied (step S107 in FIG. 12), and if the predetermined condition is satisfied, the DNN 213 at that point in time is regarded as a trained DNN.

If the predetermined condition is not satisfied, the process of steps S101 to S106 is carried out again using a new clean acoustic signal X ^(L)* , new noise N, and updated parameters θ. For example, when the process of steps S101 to S106 is repeated 100,000 times, it is determined that the predetermined condition is satisfied and the learning of the DNN 213 ends.

13 is a flowchart explaining the operation of the estimation device 100. The estimation device 100 receives as input a complex spectrogram X ^[0] in which the phase and amplitude are inconsistent, and an amplitude spectrogram A of a desired acoustic signal, and obtains and outputs a complex spectrogram Y ^[M] having a phase spectrogram that is consistent with the amplitude spectrogram A. The amplitude of the complex spectrogram X ^[0] is the amplitude spectrogram A.

Each of M estimation units 110-m (m=0, 1, 2, ..., M-1, M is an integer equal to or greater than 1) receives as input a complex spectrogram X ^[m] having a conflicting phase and amplitude and an amplitude spectrogram A of a desired acoustic signal, and determines and outputs a complex spectrogram X ^[m+1] having an estimated phase spectrogram.

As described above, the estimation unit 110-m is composed of a phase assignment unit 111, a conversion unit 112, and a phase change unit 113. The phase change unit 113 is composed of a DNN 114 and a subtraction unit 115. A DNN learned by the learning device 200 in FIG. 10 is set in the DNN 114.

The phase assigning unit 111 receives as input a complex spectrogram X ^[m] whose phase and amplitude are inconsistent, and an amplitude spectrogram A of a desired acoustic signal, assigns the phase of the complex spectrogram X ^[m] to the amplitude spectrogram A as shown in the following equation, and outputs a signal Y ^[m] = _P (X ^[m] ) after the phase assignment (step S201 in FIG. 13).

Since equation (16) multiplies each element of complex spectrogram X ^[m] by each element of amplitude spectrogram A and divides the multiplication result by the amplitude spectrogram |X ^[m] | of complex spectrogram X[ ^m] , it can be said that this is a process of converting the amplitude of complex spectrogram X ^[m] into the magnitude of amplitude spectrogram A.

The conversion unit 112 receives the signal Y ^[m] as input, converts the signal Y ^[m] into a time waveform using the inverse short-time Fourier transform G† according to the following equation, and converts the converted time waveform into a frequency domain signal Z ^[m] = P _C (Y ^[m] ) using the short-time Fourier transform G corresponding to the inverse short-time Fourier transform G†, and outputs it (step S202 in Figure 13).

The process of step S202 corresponds to a process of converting a complex spectrogram Y ^[m] in which the phase and amplitude are inconsistent into a time waveform, and then converting the converted time waveform into a complex spectrogram Z ^[m] in which the phase and amplitude are consistent.

The phase modification unit 113 uses the complex spectrogram X ^[m] , the signal Y ^[m], and the signal Z ^[m] to bring the phase of the complex spectrogram X ^[m] closer to the phase of the desired acoustic signal based on the statistical properties of the training acoustic signal corresponding to the desired acoustic signal.

The DNN 114 receives the complex spectrogram X ^[m] , the signal Y ^[m], and the signal Z ^[m] , estimates the distortion or estimation error (Z ^[m] -X ^[m] ) generated by the Griffin-Lim algorithm, and outputs the estimated value _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) (step S203 in FIG. 13).

The subtraction unit 115 obtains and outputs the difference X ^[m+1] = Z ^[m] - _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) between the signal Z ^[m] and the estimated value _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) (step S204 in FIG. 13). This subtraction process corresponds to the process of removing the distortion or estimation error generated by the Griffin-Lim algorithm, and corresponds to the process of bringing the phase spectrogram of the complex spectrogram X ^[m] closer to the desired acoustic signal.

The process of steps S201 to S204 is repeated M times, the number of times being the number of estimation units 110-m, and when the process is completed M times (YES in step S206 in FIG. 13), a complex spectrogram X ^[M] is output from the final stage estimation unit 110-(M−1). The number of repetitions M may be set to, for example, about 5.

The phase assigning unit 120 receives the complex spectrogram X ^[M] and the amplitude spectrogram A as input, assigns the phase of the complex spectrogram X ^[M] to the amplitude spectrogram A as shown in the following equation, and outputs the signal Y ^[M] = _P (X ^[M] ) after the phase assignment (step S207 in FIG. 13).

In step S207, the amplitude of the complex spectrogram X ^[M] is converted into the magnitude of the amplitude spectrogram A again.
With the above configuration, the technique disclosed in Non-Patent Document 2 makes it possible to reconstruct a consistent phase spectrum from only the amplitude spectrum by utilizing the statistical properties of the signal to be reconstructed.

However, in the DNN structure used in Non-Patent Document 2, the real and imaginary parts of the spectrogram are concatenated as real numbers and input to the real convolution layer. This type of DNN cannot take into account the algebraic structure of the spectrogram as complex numbers. The relationship between the phase and amplitude of the spectrogram is obtained from the structure of the spectrogram as complex numbers. For this reason, with a DNN structure that is only real numbers, there is a possibility that the complex spectrogram of the noise component cannot be adequately estimated.

D. Griffin and J. Lim, “Signal estimation from modified shorttime Fourier transform”, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236-243, Apr. 1984. Y.Masuyama, K.Yatabe, Y.Koizumi, Y.Oikawa, N.Harada, “DEEP GRIFFIN-LIM ITERATION”, International Conference on Acoustics, Speech, and Signal Processing, Oct.2019

The present invention has been made to solve the above-mentioned problems, and aims to provide an estimation method, an estimation program, a deep neural network device , and an estimation device that are capable of performing calculations for acoustic signal restoration taking into account the algebraic structure of the spectrogram as a complex number.

The estimation method of the present invention includes a plurality of gated complex convolution steps, and a first complex convolution step for performing a convolution operation on the output of the final gated complex convolution step, the first gated complex convolution step including a second complex convolution step for performing a convolution operation on a first complex spectrogram, a first amplitude gate operation step for inputting the first complex spectrogram and an amplitude spectrogram of a desired acoustic signal and selecting only an area of the first complex spectrogram that requires correction, and a first multiplication step for outputting a result of multiplying the output of the second complex convolution step by the output of the first amplitude gate operation step; The gated complex convolution step other than the first includes a third complex convolution step that performs a convolution operation on the output of the immediately preceding gated complex convolution step, a second amplitude gate operation step that uses the output of the immediately preceding gated complex convolution step and the amplitude spectrogram as inputs to select only an area in which correction of the output of the immediately preceding gated complex convolution step is required, and a second multiplication step that outputs a result of multiplying the output of the third complex convolution step by the output of the second amplitude gate operation step, and is characterized in that a complex spectrogram of the noise component of the desired acoustic signal is estimated by a deep neural network.

The estimation method of the present invention includes a gated complex convolution step performed multiple times, and a first complex convolution step of performing a convolution operation on an output of the gated complex convolution step performed last time, and the first gated complex convolution step includes a second complex convolution step of performing a convolution operation on a first complex spectrogram, a first amplitude gate operation step of selecting only an area of the first complex spectrogram that needs to be corrected, and a first multiplication step of outputting a result of multiplying an output of the second complex convolution step by an output of the first amplitude gate operation step. a second amplitude gate calculation step for selecting only an area that needs to be corrected from the output of the immediately preceding gated complex convolution step; and a second multiplication step for outputting a result of multiplying the output of the third complex convolution step by the output of the second amplitude gate calculation step; the first amplitude gate calculation step includes a step of performing an amplitude gate calculation using Sigmoid(|C|*W _R ) when the first complex spectrogram is C and a real weighting parameter is W _R ; and the second amplitude gate calculation step includes a step of performing an amplitude gate calculation using Sigmoid(|C|*W _R ) when the complex spectrogram that is the output of the immediately preceding gated complex convolution step is C; and the complex spectrogram of a noise component of a desired acoustic signal is estimated by a deep neural network.

The estimation method of the present invention further includes a phase assignment step of receiving as input a second complex spectrogram having a conflicting phase and amplitude and an amplitude spectrogram of a desired acoustic signal, assigning the phase of the second complex spectrogram to the amplitude spectrogram to obtain a first signal after the assignment, a transformation step of converting the first signal into a time waveform by an inverse short-time Fourier transform and converting the converted time waveform into a second signal in the frequency domain by a short-time Fourier transform corresponding to the inverse short-time Fourier transform, and a phase modification step of bringing the phase of the second complex spectrogram closer to the phase of the desired acoustic signal based on statistical properties of a learning acoustic signal corresponding to the desired acoustic signal, wherein the phase modification step includes the gated complex convolution step multiple times and the first complex convolution step, and includes a step of using the second complex spectrogram, the first signal, and the second signal as the first complex spectrogram to be input to the deep neural network, and outputting a difference between an output of the transformation step and an output of the deep neural network.
In addition, in one configuration example of the estimation method of the present invention, the deep neural network is trained in advance using a complex spectrogram obtained from a training acoustic signal and its amplitude spectrogram, and is characterized in that it outputs an estimate of the residual between the second signal and the second complex spectrogram.
The present invention also provides an estimation program for causing a computer to execute each of the above steps.

The deep neural network device of the present invention includes a plurality of gated complex convolutional layers connected in cascade, and a first complex convolutional layer configured to perform a convolutional operation on an output of the gated complex convolutional layer at a final stage, and the gated complex convolutional layer at the initial stage includes a second complex convolutional layer configured to perform a convolutional operation on a first complex spectrogram, a first amplitude gated layer configured to receive the first complex spectrogram and an amplitude spectrogram of a desired acoustic signal as input and select only an area of the first complex spectrogram that requires correction, and outputs a result of multiplying an output of the second complex convolutional layer by an output of the first amplitude gated layer. and a first multiplication unit configured to perform a convolution operation on an output of the gated complex convolution layer of a previous stage, and the gated complex convolution layers other than the first stage each include a third complex convolution layer configured to perform a convolution operation on an output of the gated complex convolution layer of a previous stage, a second amplitude gating layer configured to receive as input the output of the gated complex convolution layer of the previous stage and the amplitude spectrogram and select only a region in which correction of the output of the gated complex convolution layer of the previous stage is required, and a second multiplication unit configured to output a result of multiplying the output of the third complex convolution layer by the output of the second amplitude gating layer, and to estimate a complex spectrogram of a noise component of the desired acoustic signal.

The deep neural network device of the present invention includes a plurality of gated complex convolutional layers connected in cascade, and a first complex convolutional layer configured to perform a convolutional operation on an output of the gated complex convolutional layer at a final stage, and the gated complex convolutional layer at the initial stage includes a second complex convolutional layer configured to perform a convolutional operation on a first complex spectrogram, a first amplitude gated layer configured to select only an area of the first complex spectrogram that requires correction, and outputs a result of multiplying an output of the second complex convolutional layer by an output of the first amplitude gated layer. and a first multiplication unit configured to output a result of multiplying the output of the gated complex convolution layer of the previous stage by an output of the gated complex convolution layer of the previous stage, the second amplitude gate layer configured to select only a region that needs to be corrected from the output of the gated complex convolution layer of the previous stage, and a second multiplication unit configured to output a result of multiplying the output of the third complex convolution layer by the output of the second amplitude gate layer, and the first amplitude gate layer is configured to output a result of multiplying the output of the third complex convolution layer by the output of the second amplitude gate layer, the first amplitude gate layer being configured to multiply the first complex spectrogram by C and a real weighting parameter by W. The first amplitude gating layer performs an amplitude gating operation using Sigmoid (|C|*WR ₎ when C is a complex spectrogram that is _the output of the gated complex convolution layer of the previous stage, and is characterized in that it _estimates a complex spectrogram of the noise component of a desired acoustic signal.

The estimation device of the present invention further includes a phase assigning unit configured to receive as input a second complex spectrogram having a conflicting phase and amplitude and an amplitude spectrogram of a desired acoustic signal, assign the phase of the second complex spectrogram to the amplitude spectrogram, and obtain a first signal after the assignment; a conversion unit configured to convert the first signal into a time waveform by an inverse short-time Fourier transform, and convert the converted time waveform into a second signal in the frequency domain by a short-time Fourier transform corresponding to the inverse short-time Fourier transform; and a phase changing unit configured to bring the phase of the second complex spectrogram closer to the phase of the desired acoustic signal based on statistical properties of a training acoustic signal corresponding to the desired acoustic signal, wherein the phase changing unit includes the deep neural network device , and uses the second complex spectrogram, the first signal, and the second signal as a first complex spectrogram to be input to the deep neural network device , and outputs a difference between an output of the conversion unit and an output of the deep neural network device .

According to the present invention, by executing a gated complex convolution step in which the result of the complex convolution step is multiplied by the result of the amplitude gate calculation step, it becomes possible to perform calculations for restoring an acoustic signal taking into account the algebraic structure of the spectrogram as a complex number. As a result, the present invention makes it possible to accurately estimate the complex spectrogram of the noise component, and to obtain an output sound of higher quality compared to conventional techniques.

FIG. 1 is a block diagram showing the configuration of a DNN according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating the operation of a DNN according to an embodiment of the present invention. FIG. 3 is a block diagram showing a configuration of an estimation device according to an embodiment of the present invention. FIG. 4 is a block diagram showing a configuration of an estimation unit of an estimation device according to an embodiment of the present invention. FIG. 5 is a block diagram showing the configuration of a learning device according to an embodiment of the present invention. FIG. 6 is a block diagram showing another configuration of a DNN according to an embodiment of the present invention. FIG. 7 is a block diagram showing an example of the configuration of a computer that realizes an estimation device and a learning device according to an embodiment of the present invention. FIG. 8 is a block diagram showing the configuration of a conventional estimation device. FIG. 9 is a block diagram showing a configuration of an estimation unit of a conventional estimation device. FIG. 10 is a block diagram showing the configuration of a conventional learning device. FIG. 11 is a block diagram showing the configuration of a conventional DNN. FIG. 12 is a flowchart illustrating the operation of a conventional learning device. FIG. 13 is a flowchart illustrating the operation of a conventional estimation device.

Below, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, we propose an amplitude-informed gated complex convolutional neural network (AI-GCNN) as the DNN structure in Non-Patent Document 2.

The complex convolution for the complex spectrogram C is defined as follows:
Conv _C (C)=(W _Re *C _Re -W _Im *C _Im )+i(W _Re *C _Im +W _Im *C _Re )
...(19)

i is the imaginary unit, C _Re is the real part of the complex spectrogram C, C _Im is the imaginary part of the complex spectrogram C, W _Re is a weighting parameter for the real part, and W _Im is a weighting parameter for the imaginary part.

Figure 1 is a block diagram showing the configuration of AI-GCNN, a DNN according to this embodiment. AI-GCNN is composed of multiple gated complex convolutional layers 1000-1002 connected in cascade, and a complex convolutional layer 1003 that performs a convolution operation on the output of the final gated complex convolutional layer 1002.

Each of the gated complex convolution layers 1000 to 1002 is composed of a complex convolution layer 1004 that performs a convolution operation on a complex spectrogram, an amplitude gate layer 1005 that receives as input the complex spectrogram and the amplitude spectrogram of the desired acoustic signal and selects only the areas of the complex spectrogram that require correction, and a multiplication unit 1006 that multiplies the output of the complex convolution layer 1004 and the output of the amplitude gate layer 1005 element by element and outputs the result.

In Figure 1, c represents the number of channels and k represents the kernel size. The complex convolution layer 1004 and the amplitude gate layer 1005 have 64 channels and a kernel size of 5 x 3. The complex convolution layer 1003 has 1 channel and a kernel size of 1 x 1. The convolution stride of each layer is 1.

In AI-GCNN, an amplitude gate layer defined as follows is used as a nonlinear layer.
AmpGate _WR (C)=Sigmoid(|C|*W _R )...(20)

C is the input complex spectrogram, and W _R is a real-valued weighting parameter. The amplitude gate layer acts like a time-frequency mask on the complex spectrogram, and is expected to select only the areas of the spectrogram that require correction. As an operation of applying this amplitude gate layer to the complex convolution layer Conv _WC , an amplitude-based gated complex convolution (AGC) layer is defined as follows:
AGC _WC,WR (C)=Conv _WC (C)◎AmpGate _WR (C)...(21)

As above, ◎ denotes element-wise multiplication. Furthermore, to more directly incorporate amplitude information, which is known to be useful in estimating the time-frequency mask, the AGC layer is extended to an amplitude-informed gated complex convolution (AI-GC) layer, defined as follows:

[A, C] in equation (23) indicates a channel-wise combination of the amplitude spectrogram A and the complex spectrogram C. According to equations (22) and (23), it is possible to compare the amplitudes of the target amplitude spectrogram A and the complex spectrogram C and extract the residual.
In this way, the gated complex convolutional layers 1000 to 1002 of this embodiment receive the amplitude spectrogram A as input in addition to the complex spectrogram C.

Figure 2 is a flowchart explaining the operation of AI-GCNN, which is a DNN according to this embodiment. Here, the variable that counts the number of times the gated complex convolution step is executed is n.

In the first gated complex convolution step (n=0), the complex convolution layer 1004 of the gated complex convolution layer 1000 performs a convolution operation on the complex spectrogram C (step S11 in FIG. 2), and the amplitude gate layer 1005 of the gated complex convolution layer 1000 performs an amplitude gate operation to select only the area of the complex spectrogram C that needs to be corrected (step S12 in FIG. 2). The multiplication unit 1006 of the gated complex convolution layer 1000 outputs the result of multiplying the outputs of the complex convolution layer 1004 and the amplitude gate layer 1005 of the gated complex convolution layer 1000 element by element (step S13 in FIG. 2).

In the second gated complex convolution step (n=1), the complex convolution layer 1004 of the gated complex convolution layer 1001 performs a convolution operation on the complex spectrogram output from the previous gated complex convolution layer 1000 (step S11), and the amplitude gate layer 1005 of the gated complex convolution layer 1001 performs an amplitude gate operation to select only the region of the complex spectrogram output from the previous gated complex convolution layer 1000 that needs correction (step S12). The multiplication unit 1006 of the gated complex convolution layer 1001 outputs the result of multiplying the outputs of the complex convolution layer 1004 and the amplitude gate layer 1005 of the gated complex convolution layer 1001 element by element (step S13).

In the third gated complex convolution step (n=2), the complex convolution layer 1004 of the gated complex convolution layer 1002 performs a convolution operation on the complex spectrogram output from the preceding gated complex convolution layer 1001 (step S11), and the amplitude gate layer 1005 of the gated complex convolution layer 1002 performs an amplitude gate operation to select only the region of the complex spectrogram output from the preceding gated complex convolution layer 1001 that needs correction (step S12). The multiplication unit 1006 of the gated complex convolution layer 1002 outputs the result of multiplying the outputs of the complex convolution layer 1004 and the amplitude gate layer 1005 of the gated complex convolution layer 1002 by element (step S13).

After the three gated complex convolution steps are completed, the complex convolution layer 1003 performs a convolution operation on the complex spectrogram output from the preceding gated complex convolution layer 1002 (step S16 in FIG. 2).
Thus, in this embodiment, it becomes possible to extract unnecessary residuals contained in the output (Z ^[m] ) of the Griffin-Lim algorithm using a DNN.

In this embodiment, the number of gated complex convolutional layers 1000 to 1002 is three (the number of gated complex convolutional steps is three), but this is not limited to three, and the number of layers may be three or more.
Furthermore, the parameters used in the complex convolution operation and the amplitude gate operation are given by learning, and different parameters may be used for each nth layer (each layer).

3 is a block diagram showing the configuration of an estimation device according to the present embodiment. The estimation device 100a includes M estimation units 110a-m (m=0, 1, 2, ... ^{, M-1,} M is an integer of 1 or more) that receive a complex spectrogram X ^[m] with a contradiction between phase and amplitude and an amplitude spectrogram A of a desired acoustic signal, and output a complex spectrogram X[m+1] having an estimated phase spectrogram, and a phase assigning unit 120 that assigns the phase of the complex spectrogram X ^[M] output from the final estimation unit 110a-(M-1) to the amplitude spectrogram A.
The spectrogram X ^[0] input to the processing block where m=0, ie, the first-stage estimation section 110a-0, may be the amplitude spectrogram A.

4 is a block diagram showing the configuration of the estimation unit 110a-m. The estimation unit 110a-m of this embodiment includes a phase assigning unit 111 that receives a complex spectrogram X ^[m] and an amplitude spectrogram A and assigns the phase of the complex spectrogram X ^[m] to the amplitude spectrogram A, a conversion unit 112 that converts the output signal of the phase assigning unit 111 into a time waveform by an inverse short-time Fourier transform and converts the converted time waveform into a frequency domain signal by a short-time Fourier transform corresponding to the inverse short-time Fourier transform, and a phase changing unit 113a that brings the phase of the complex spectrogram X ^[m] closer to the phase of the desired acoustic signal based on the statistical properties of a learning acoustic signal corresponding to the desired acoustic signal. The phase changing unit 113a includes a DNN 114a and a subtraction unit 115.

Figure 5 is a block diagram showing the configuration of a learning device according to this embodiment. The learning device 200a is composed of a noise addition unit 209, a phase assignment unit 211, a conversion unit 212, a DNN 213a, a subtraction unit 214, and a parameter update unit 215. The DNNs 114a and 213a have the configuration shown in Figure 1.

Next, the learning stage of the DNNs 114a and 213a in this embodiment will be described. The flow of operation of the learning device 200a is similar to that of the learning device 200 disclosed in Non-Patent Document 2, so the operation of the learning device 200a will be described using FIG. 12.

An initialization unit (not shown) of the learning device 200a initializes a parameter θ of the DNN 213a with a random number (step S100 in FIG. 12).
The noise adding unit 209 receives the clean audio signal X ^(L)* and noise N as input, adds the noise N to the clean audio signal X ^(L)* as shown in equation (12), and outputs a complex spectrogram X tilde (Step S101 in FIG. 12).

The phase assigning unit 211 receives the complex spectrogram X tilde and the amplitude spectrogram A ^(L) , assigns the phase of the complex spectrogram X tilde to the amplitude spectrogram A ^(L) as shown in equation (13), and outputs the assigned signal Y tilde (step S102 in FIG. 12). Since equation (13) multiplies each element of the complex spectrogram X tilde by each element of the amplitude spectrogram A ^(L) and divides the multiplication result by the amplitude spectrogram |X tilde| of the complex spectrogram X tilde, it can be said that this is a process of converting the amplitude of the complex spectrogram X tilde into the magnitude of the amplitude spectrogram A ^(L) .

The conversion unit 212 receives the signal Y tilde as input, converts the signal Y tilde into a time waveform by an inverse short-time Fourier transform G† as shown in equation (14), and converts the converted time waveform into a frequency domain signal Z tilde by a short-time Fourier transform G corresponding to the inverse short-time Fourier transform G†, and outputs the signal Z tilde (step S103 in FIG. 12).

DNN 213a receives as input complex spectrogram X tilde, signal Y tilde, signal Z tilde, and amplitude spectrogram A ^(L) , estimates the distortion or estimation error generated by the Griffin-Lim algorithm, and outputs an estimated value F _θ (X tilde, Y tilde, Z tilde) (step S104 in FIG. 12).

On the other hand, the subtraction unit 214 obtains and outputs the difference Z tilde -X ^{( L)* between the signal Z tilde and the clean audio signal X (L} )* (step S105 in FIG. 12).

The parameter update unit 215 receives the difference Z tilde -X ^(L)* and the estimated value F _θ (X tilde, Y tilde, Z tilde) as input, and uses these values to update the parameter θ of the DNN 213a so as to minimize the objective function shown in equation (15) (step S106 in Figure 12).

As a learning method, a stochastic steepest descent method or the like may be used. The learning rate may be set to, for example, about 10 ^-5 .
The parameter update unit 215 determines whether or not a predetermined condition is satisfied (step S107 in FIG. 12), and if the predetermined condition is satisfied, the DNN 213a at that point in time is regarded as a trained DNN.

If the predetermined condition is not satisfied, the process of steps S101 to S106 is carried out again using a new clean acoustic signal X ^(L)* , new noise N, and updated parameters θ. For example, when the process of steps S101 to S106 is repeated 100,000 times, it is determined that the predetermined condition is satisfied and the learning of the DNN 213a ends.

Next, the operation of the estimation device 100a will be described. The flow of operation of the estimation device 100a is similar to that of the estimation device 100 disclosed in Non-Patent Document 2, so the operation of the estimation device 100a will be described using FIG. 13.

The estimation device 100a receives as input a complex spectrogram X ^[0] in which the phase and amplitude are inconsistent, and an amplitude spectrogram A of a desired acoustic signal, and calculates and outputs a complex spectrogram Y ^[M] having a phase spectrogram that is consistent with the amplitude spectrogram A.

Each of M estimation units 110a-m (m=0, 1, 2, ..., M-1, M is an integer equal to or greater than 1) receives as input a complex spectrogram X ^[m] having a conflicting phase and amplitude and an amplitude spectrogram A of a desired acoustic signal, and determines and outputs a complex spectrogram X ^[m+1] having an estimated phase spectrogram.

As shown in FIG. 4, the estimation units 110a-m of this embodiment are composed of a phase assignment unit 111, a conversion unit 112, and a phase change unit 113a. The phase change unit 113a is composed of a DNN 114a and a subtraction unit 115. The DNN learned by the learning device 200a is set in the DNN 114a.

The phase assigning unit 111 receives as input a complex spectrogram X ^[m] whose phase and amplitude are inconsistent and an amplitude spectrogram A of a desired acoustic signal, assigns the phase of the complex spectrogram X ^[m] to the amplitude spectrogram A as shown in equation (16), and outputs a signal Y ^[m] = _P (X ^[m] ) after the phase assignment (step S201 in FIG. 13).

The conversion unit 112 receives the signal Y ^[m] as input, converts the signal Y ^[m] into a time waveform using the inverse short-time Fourier transform G† as shown in equation (17), and converts the converted time waveform into a frequency domain signal Z ^[m] = P _C (Y ^[m] ) using the short-time Fourier transform G corresponding to the inverse short-time Fourier transform G†, and outputs it (step S202 in Figure 13).

The process of step S202 corresponds to a process of converting a complex spectrogram Y ^[m] in which the phase and amplitude are inconsistent into a time waveform, and then converting the converted time waveform into a complex spectrogram Z ^[m] in which the phase and amplitude are consistent.

The phase changing unit 113a uses the complex spectrogram X ^[m] , the signal Y ^[m] , the signal Z ^[m], and the amplitude spectrogram A to bring the phase of the complex spectrogram X ^[m] closer to the phase of the desired acoustic signal, based on the statistical properties of the training acoustic signal corresponding to the desired acoustic signal.

The DNN 114a receives the complex spectrogram X ^[m] , the signal Y ^[m] , the signal Z ^[m], and the amplitude spectrogram A as input, estimates the distortion or estimation error (Z ^[m] -X ^[m] ) generated by the Griffin-Lim algorithm, and outputs the estimated value _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) (step S203 in FIG. 13).

The subtraction unit 115 obtains and outputs the difference X ^[m+1] = Z ^[m] - _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) between the signal Z ^[m] and the estimated value _Fθ (X ^[m] , Y ^[m] , Z ^[m] ) (step S204 in FIG. 13). This subtraction process corresponds to the process of removing the distortion or estimation error generated by the Griffin-Lim algorithm, and corresponds to the process of bringing the phase spectrogram of the complex spectrogram X ^[m] closer to the desired acoustic signal.

The process of steps S201 to S204 is repeated M times for the number of estimation units 110a-m, and when the process is completed M times (YES in step S206 in FIG. 13), a complex spectrogram X ^[M] is output from the final stage estimation unit 110a-(M-1). The number of repetitions M may be set to about 5, for example.

The phase assigning unit 120 receives the complex spectrogram X ^[M] and the amplitude spectrogram A, assigns the phase of the complex spectrogram X ^[M] to the amplitude spectrogram A as shown in equation (18), and outputs the signal Y ^[M] = _P (X ^[M] ) after the phase assignment (step S207 in FIG. 13). The amplitude of the complex spectrogram X ^[M] is converted again into the magnitude of the amplitude spectrogram A by the processing in step S207.

In this embodiment, the DNN structure shown in FIG. 1 makes it possible to perform calculations for restoring an acoustic signal taking into account the algebraic structure of complex numbers. As a result, in this embodiment, the complex spectrogram of the noise component can be estimated with high accuracy, and it is possible to obtain output sound of higher quality compared to the technology disclosed in Non-Patent Document 2.

The DNN shown in Figure 1 uses the amplitude spectrogram A, so high accuracy can be achieved with a small number of iterations (the number of iterations M mentioned above). However, if true amplitude information is not available, the DNN shown in Figure 1 cannot be used.

On the other hand, the DNN using the AGC layer shown in equation (21) is less accurate than the DNN shown in FIG. 1 at a small number of iterations (iteration number M), but can be used even when true amplitude information is not available. Therefore, it is desirable to use either the DNN shown in FIG. 1 or the DNN using the AGC layer as DNNs 114a and 213a depending on the purpose.

When using a DNN with an AGC layer as DNN 114a, 213a, the amplitude gate layer shown in equation (20) can be used instead of the amplitude gate layer 1005 of each gated complex convolution layer 1000-1002 in the configuration shown in Figure 1. The configuration of the DNN in this case is shown in Figure 6.

The AGCNN, which is a DNN shown in Figure 6, is composed of multiple gated complex convolutional layers 1000a to 1002a connected in cascade, and a complex convolutional layer 1003 that performs a convolution operation on the output of the final gated complex convolutional layer 1002a.

Each of the gated complex convolutional layers 1000a to 1002a is composed of a complex convolutional layer 1004, an amplitude gate layer 1005a corresponding to equation (20), and a multiplication unit 1006 that outputs the result of multiplying the output of the complex convolutional layer 1004 and the output of the amplitude gate layer 1005a for each element. The only input to the amplitude gate layer 1005a is a complex spectrogram C. The complex spectrogram C in the learning device 200a is X tilde, Y tilde, Z tilde, and the complex spectrogram C in the estimation device 100a is X ^[m] , Y ^[m] , Z ^[m] .

The estimation device 100a and learning device 200a described in this embodiment can be realized by a computer equipped with a CPU (Central Processing Unit), a storage device, and an interface, and a program that controls these hardware resources. An example of the configuration of this computer is shown in FIG. 7. The computer is equipped with a CPU 300, a storage device 301, and an interface device (I/F) 302.

The I/F 302 is connected to, for example, a network. An estimation program for implementing the estimation method of the present invention is stored in the storage device 301. The program may also be provided via the network. The CPU 300 executes the processing described in this embodiment according to the program stored in the storage device 301.

In addition, the estimation device 100a and the learning device 200a are configured by executing a specific program on a computer, but at least a portion of the processing content of the estimation device and the learning device may be realized by hardware.

The present invention can be applied to acoustic signal restoration technology that restores the phase spectrum from only the amplitude spectrum.

100a...Estimation device, 110a-0 to 110a-(M-1)...Estimation unit, 111, 120, 211...Phase assignment unit, 112, 212...Conversion unit, 113a...Phase change unit, 114a, 213a...DNN, 115, 214...Subtraction unit, 200a...Learning device, 209...Noise addition unit, 215...Parameter update unit, 1000, 1000a, 1001, 1001a, 1002, 1002a...Gated complex convolution layer, 1003, 1004...Complex convolution layer, 1005, 1005a...Amplitude gate layer, 1006...Multiplication unit.

Claims (8)

multiple gated complex convolution steps;
a first complex convolution step for performing a convolution operation on an output of the final gated complex convolution step;
The first gated complex convolution step is
a second complex convolution step of convolving the first complex spectrogram;
a first amplitude gate calculation step of inputting the first complex spectrogram and an amplitude spectrogram of a desired acoustic signal to select only a region of the first complex spectrogram that needs to be corrected;
a first multiplication step of multiplying an output of the second complex convolution step by an output of the first amplitude gate operation step and outputting the result;
The gated complex convolution step other than the first one may be
a third complex convolution step for convolving the output of the immediately preceding gated complex convolution step;
a second amplitude gate operation step for selecting only a region in which the output of the immediately preceding gated complex convolution step needs to be corrected by using the output of the immediately preceding gated complex convolution step and the amplitude spectrogram as input;
a second multiplication step of multiplying an output of the third complex convolution step by an output of the second amplitude gate operation step and outputting the result;
An estimation method comprising estimating a complex spectrogram of a noise component of the desired acoustic signal using a deep neural network. multiple gated complex convolution steps;
a first complex convolution step for performing a convolution operation on an output of the final gated complex convolution step;
The first gated complex convolution step is
a second complex convolution step of convolving the first complex spectrogram;
a first amplitude gate calculation step for selecting only a region of the first complex spectrogram that needs to be corrected;
a first multiplication step of multiplying an output of the second complex convolution step by an output of the first amplitude gate operation step and outputting the result;
The gated complex convolution step other than the first one may be
a third complex convolution step for convolving the output of the immediately preceding gated complex convolution step;
a second amplitude gating step for selecting only those regions of the output of the immediately preceding gated complex convolution step that require correction;
a second multiplication step of multiplying an output of the third complex convolution step by an output of the second amplitude gate operation step and outputting the result;
the first amplitude gate calculation step includes a step of performing an amplitude gate calculation by Sigmoid (|C|*W _R ), where C is the first complex spectrogram and W _R is a real-number weighting parameter;
the second amplitude gate calculation step includes a step of performing an amplitude gate calculation by Sigmoid(|C|*W _R ), where C is a complex spectrogram that is an output of the immediately preceding gated complex convolution step;
An estimation method comprising estimating a complex spectrogram of a noise component of a desired acoustic signal using a deep neural network. 3. The estimation method according to claim 1,
a phase assignment step of inputting a second complex spectrogram having a conflicting phase and amplitude and an amplitude spectrogram of a desired acoustic signal, assigning the phase of the second complex spectrogram to the amplitude spectrogram, and obtaining a first signal after the phase assignment;
a transforming step of transforming the first signal into a time waveform by an inverse short-time Fourier transform, and transforming the transformed time waveform into a second signal in the frequency domain by a short-time Fourier transform corresponding to the inverse short-time Fourier transform;
and a phase changing step of changing a phase of the second complex spectrogram to approach a phase of the desired acoustic signal based on a statistical property of a training acoustic signal corresponding to the desired acoustic signal,
the phase changing step includes the gated complex convolution step performed multiple times and the first complex convolution step, and includes a step of using the second complex spectrogram, the first signal, and the second signal as the first complex spectrogram to be input to the deep neural network, and outputting a difference between an output of the conversion step and an output of the deep neural network. 4. The estimation method according to claim 3,
The deep neural network includes:
The system is trained in advance using a complex spectrogram obtained from a training acoustic signal and its amplitude spectrogram,
and outputting an estimate of a residual between the second signal and the second complex spectrogram. An estimation program that causes a computer to execute each step described in any one of claims 1 to 4. a plurality of cascaded gated complex convolutional layers;
a first complex convolution layer configured to perform a convolution operation on an output of the gated complex convolution layer at a final stage;
The gated complex convolutional layer in the first stage is
a second complex convolutional layer configured to perform a convolution operation on the first complex spectrogram; and
a first amplitude gate layer configured to receive the first complex spectrogram and an amplitude spectrogram of a desired acoustic signal as input and select only a region of the first complex spectrogram that needs to be corrected;
a first multiplication unit configured to output a result of multiplying an output of the second complex convolution layer by an output of the first amplitude gate layer;
The gated complex convolutional layer other than the first stage is
a third complex convolution layer configured to perform a convolution operation on an output of the gated complex convolution layer of the previous stage;
a second amplitude gating layer configured to receive an output of the gated complex convolution layer of a previous stage and the amplitude spectrogram as input and select only a region in which correction of the output of the gated complex convolution layer of the previous stage is required;
a second multiplication unit configured to output a result of multiplying an output of the third complex convolution layer by an output of the second amplitude gate layer;
A deep neural network device that estimates a complex spectrogram of a noise component of the desired acoustic signal. a plurality of cascaded gated complex convolutional layers;
a first complex convolution layer configured to perform a convolution operation on an output of the gated complex convolution layer at a final stage;
The gated complex convolutional layer in the first stage is
a second complex convolutional layer configured to perform a convolution operation on the first complex spectrogram; and
a first amplitude gating layer configured to select only areas of the first complex spectrogram that require correction;
a first multiplication unit configured to output a result of multiplying an output of the second complex convolution layer by an output of the first amplitude gate layer;
The gated complex convolutional layer other than the first stage is
a third complex convolution layer configured to perform a convolution operation on an output of the gated complex convolution layer of the previous stage;
A second amplitude gated layer configured to select only a region that needs to be corrected from the output of the gated complex convolutional layer of the previous stage;
a second multiplication unit configured to output a result of multiplying an output of the third complex convolution layer by an output of the second amplitude gate layer;
the first amplitude gate layer performs an amplitude gate operation by Sigmoid (|C|*W _R ), where C is the first complex spectrogram and W _R is a real-number weighting parameter;
The second amplitude gate layer performs an amplitude gate operation by Sigmoid (|C|*W _R ), where C is a complex spectrogram that is an output of the gated complex convolution layer in the previous stage,
A deep neural network device that estimates a complex spectrogram of a noise component of a desired audio signal. a phase assigning unit configured to receive a second complex spectrogram having a conflicting phase and amplitude and an amplitude spectrogram of a desired acoustic signal, assign the phase of the second complex spectrogram to the amplitude spectrogram, and obtain a first signal after the phase assignment;
a transform unit configured to transform the first signal into a time waveform by an inverse short-time Fourier transform, and to transform the transformed time waveform into a second signal in a frequency domain by a short-time Fourier transform corresponding to the inverse short-time Fourier transform;
a phase changing unit configured to change a phase of the second complex spectrogram to be closer to a phase of a desired acoustic signal based on a statistical property of a training acoustic signal corresponding to the desired acoustic signal,
The phase change unit includes the deep neural network device according to claim 6 or 7, and uses the second complex spectrogram, the first signal, and the second signal as a first complex spectrogram to be input to the deep neural network device , and outputs a difference between an output of the conversion unit and an output of the deep neural network device .

Images