CN115706756A - Abnormal echo delay identification method, device, terminal and storage medium - Google Patents

Abstract

The application discloses an abnormal echo delay identification method, an abnormal echo delay identification device, a terminal and a storage medium, and relates to the technical field of communication. The method comprises the following steps: performing audio feature extraction on an input audio frame acquired by a microphone to obtain a first audio feature; in response to reaching the target delay, determining second audio features from the candidate audio features based on the first audio features, wherein the candidate audio features are audio features corresponding to output audio frames for loudspeaker playing, and the second audio features are matched with the first audio features; determining echo delay of the output audio frame corresponding to the second audio feature; and determining that abnormal echo delay exists in response to the echo delay being smaller than the target delay. The method for detecting the negative delay in the echo cancellation process is provided, so that the situation that the echo cancellation module continues to perform echo cancellation work under wrong echo delay or cannot perform echo cancellation due to the fact that the echo delay cannot be calculated is avoided, and the accuracy of echo cancellation can be improved.

Description

Abnormal echo delay identification method, device, terminal and storage medium

technical field

The embodiment of the present application relates to the technical field of communication, and in particular to a method, device, terminal and storage medium for identifying abnormal echo delay.

Background technique

During the voice call through the audio terminal equipment, the sound played by the speaker, especially the sound played by the speaker in the hands-free mode is relatively loud, which is easy to be collected by the microphone; the sound played by the speaker is collected by the microphone and then fed back to the remote end, the person speaking at the far end will hear his own voice, forming an echo and seriously affecting the quality of the voice call.

In the related art, the audio terminal equipment is equipped with a software or hardware echo cancellation module to cancel the echo collected by the microphone. The echo cancellation process generally adopts the echo cancellation method. By comparing the reference point signal and the receiving point signal, the echo delay from the reference point signal to the receiving point signal is estimated, and the reference point signal is delayed according to the echo delay. Finally, the transfer function is calculated jointly with the signal at the receiving point, and the transfer function is used to predict the echo replica, so that the echo replica can be used to perform echo cancellation on the signal at the receiving point.

It can be seen from the above echo cancellation process that accurate echo delay estimation is a prerequisite for improving the echo cancellation effect.

Contents of the invention

Embodiments of the present application provide a method, device, terminal and storage medium for identifying abnormal echo delay. Described technical scheme is as follows:

According to one aspect of the present application, a method for identifying abnormal echo delay is provided, the method comprising:

Perform audio feature extraction on the input audio frame collected by the microphone to obtain the first audio feature;

In response to reaching the target delay, based on the first audio feature, determining a second audio feature from candidate audio features, the candidate audio feature is an audio feature corresponding to an output audio frame, and the output audio frame is used for speaker playback, the second audio characteristic matches the first audio characteristic;

Determining the echo delay of the output audio frame corresponding to the second audio feature;

In response to the echo delay being less than the target delay, it is determined that there is an abnormal echo delay.

According to another aspect of the present application, an abnormal echo delay identification device is provided, the device comprising:

The feature extraction module is used to extract the audio feature from the input audio frame collected by the microphone to obtain the first audio feature;

The first determination module is configured to determine a second audio feature from candidate audio features based on the first audio feature in response to reaching the target delay, the candidate audio feature is an audio feature corresponding to an output audio frame, and the output The audio frame is used for speaker playback, and the second audio feature matches the first audio feature;

The second determination module is used to determine the echo delay of the output audio frame corresponding to the second audio feature;

A third determining module, configured to determine that there is an abnormal echo delay in response to the echo delay being less than the target delay.

According to another aspect of the present application, a terminal is provided, the terminal includes a processor and a memory, at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to realize the above aspects The method for identifying the abnormal echo delay.

According to another aspect of the present application, a computer-readable storage medium is provided. At least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to realize the abnormal echo delay as described above. time identification method.

According to another aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the terminal reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the terminal executes the abnormal echo delay identification method provided in the above optional implementation manner.

The beneficial effects brought by the technical solutions provided by the embodiments of the present application at least include:

In the embodiment of the present application, in view of the fact that negative delay occurs in the process of echo cancellation, after the input audio frame is received at the receiving point, it needs to wait for a period of time to find the matching output audio frame, and proposes a A method of detecting negative delay in the process of echo cancellation, by obtaining the first audio feature corresponding to the input audio frame and delaying the target delay, and then performing feature matching based on the first audio feature, so that in the negative delay The echo delay can still be estimated in the case of , and the delay feature matching makes the calculated echo delay the sum of the target delay and the transfer delay, then the transfer delay can be determined based on the relationship between the echo delay and the target delay Positive or negative of the time, so that it can be determined in time whether there is an abnormal echo delay (negative delay), avoiding the echo cancellation module to continue the echo cancellation work under the wrong echo delay, or avoiding the echo delay caused by the inability to calculate the echo delay The situation where echo cancellation cannot be performed, so that the accuracy of echo cancellation can be improved.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 shows a schematic structural diagram of an echo cancellation system in the related art;

Fig. 2 shows a schematic structural diagram of an echo cancellation system shown in an exemplary embodiment of the present application;

Fig. 3 shows the flowchart of the abnormal echo delay identification method provided by an exemplary embodiment of the present application;

Fig. 4 shows a schematic diagram of the process of determining the echo delay shown in an exemplary embodiment of the present application;

FIG. 5 shows a flow chart of an abnormal echo delay identification method provided by another exemplary embodiment of the present application;

Fig. 6 shows a working diagram of an exemplary delay estimation process and abnormal delay detection process of the present application;

FIG. 7 shows a schematic diagram of an audio feature extraction process shown in an exemplary embodiment of the present application;

Fig. 8 shows a schematic diagram of the principle of implementing a delay function in a feature storage area shown in an exemplary embodiment of the present application;

Fig. 9 shows a schematic diagram of a delay feature matching process shown in an exemplary embodiment of the present application;

Fig. 10 shows a schematic diagram of a feature matching process shown in an exemplary embodiment of the present application;

Fig. 11 shows a schematic diagram of a delay estimation process and an abnormal delay detection process shown in another exemplary embodiment of the present application;

Fig. 12 shows a schematic diagram of the audio feature extraction process in the delay estimation process shown in an exemplary embodiment of the present application;

FIG. 13 shows a flow chart of an abnormal echo delay identification method provided by another exemplary embodiment of the present application;

Fig. 14 is a structural block diagram of an abnormal echo delay identification device provided by an exemplary embodiment of the present application;

Fig. 15 shows a structural block diagram of a terminal provided by an exemplary embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

During a voice call through the audio terminal device, the sound played by the speaker (speaker), especially the sound played by the speaker in the external mode, is louder, and the played sound is easy to be re-collected by the microphone of the audio terminal device, so that The sound played by the speaker will be collected by the microphone and then sent to the remote end. The speaker at the far end will hear his own voice, which will form an echo and seriously affect the call quality. Therefore, a general audio terminal device (communication device) will include a software or hardware echo cancellation system, and the echo cancellation system will perform echo cancellation on the sound collected by the microphone. Fig. 1 shows a schematic structural diagram of an echo cancellation system in the related art. During a voice call, the audio signal received by the terminal from the remote end needs to pass through a reference point before being sent to the speaker for playback. The audio signal collected at the reference point is generally called the reference point signal, and the reference point signal passes through the software and hardware playback logic. It is sent to the speaker for playback, propagates into the microphone through air and other media, and then reaches the receiving point through software and hardware acquisition logic. The audio signal obtained from the receiving point is called the receiving point signal. It can be seen that when the reference point signal arrives at the receiving point, it needs to go through the software and hardware playback delay (the time passed from the reference point to the speaker), and the acoustic path delay (the time passed from the speaker to the microphone through air and other transmission media). ) and software and hardware acquisition delay (the time from the microphone to the receiving point through the software and hardware acquisition logic).

As shown in FIG. 1 , in order to implement echo cancellation, a delay estimation module 101 , a delay alignment module 102 and an echo cancellation module 103 are provided in an audio terminal device (communication device). Wherein, the delay estimation module 101 is used for comparing the reference point signal and the receiving point signal, thereby estimating the echo delay (Tde) between the reference point signal and the receiving point signal, and passing the Tde to the delay alignment module 102 The delay alignment module 102 is used to obtain the Tde estimated according to the delay estimation module 101, delaying the reference point signal by Tde to obtain a delayed reference point signal, and sending the delayed reference point signal to the echo cancellation module 103 , the echo cancellation module 103 estimates the transfer function of the echo path based on the delayed reference point signal and the receiving point signal, and then when the reference point signal receives a new reference point signal, it can be based on the new reference point signal and transfer function The function predicts the echo copy corresponding to the reference point signal, and subtracts the echo copy from the receiving point signal to eliminate the echo signal in the receiving point signal.

As can be seen from the echo cancellation system in FIG. 1, whether the delay estimation module 101 can accurately estimate the echo delay Tde will affect the accuracy of the echo cancellation, and the operating system may be affected by the operating system during the delay estimation process. Due to the impact of speaker playback thread scheduling and application stability, the echo delay from the reference point to the receiving point is not constant. The newly collected receiving point signal is obtained in advance, but the matching audio feature cannot be found from the feature storage area that stores the audio feature corresponding to the reference point signal, so the echo delay Tde cannot be estimated or accurately calculated, and negative delay occurs situation, which seriously affects the effect of echo cancellation, or cannot achieve echo cancellation.

It can be seen that in the process of echo cancellation, how to effectively and timely detect whether there is an abnormal echo delay in the echo cancellation system can avoid the situation that the echo cancellation system adopts a wrong delay for echo cancellation, or cannot perform echo cancellation, and then can take effective measures in time. It is the key to improve the accuracy of echo cancellation to eliminate the abnormal delay by means. Based on the above problems, as shown in FIG. 2 , it shows a schematic structural diagram of an echo cancellation system shown in an exemplary embodiment of the present application. The echo cancellation system mainly includes an abnormality monitoring module 201, a delay estimation module 202, a delay alignment module 203, and an echo cancellation module 204. Compared with FIG. 1, the present application adds an abnormality monitoring module 201 to the echo cancellation system. Module 201 can monitor the possible abnormal delay (negative delay) based on the reference point signal and the receiving point signal. When a negative delay is detected, a reset instruction can be sent to the delay estimation module, so that the delay estimation Both the module 202 and the delay alignment module 203 can reset the algorithm and clear the cache, so that the subsequent echo cancellation process can normally estimate the echo delay.

Please refer to FIG. 3 , which shows a flowchart of a method for identifying abnormal echo delay provided by an exemplary embodiment of the present application. In this embodiment of the application, the method is applied to a terminal as an example for illustration, and the method includes:

Step 301 , extracting audio features from input audio frames collected by a microphone to obtain first audio features.

Among them, the input audio frame is collected by the microphone. During the microphone collection process, the input audio signal (input audio frame) collected by the microphone will first be stored in the buffer corresponding to the hardware and software collection logic, and then call the application thread from the Read input audio frames from this buffer.

As shown in Figure 2, the reference point signal needs to go through software and hardware playback logic, air and other media propagation, and software and hardware acquisition logic to reach the receiving point. The corresponding audio signal is not exactly the same at the reference point and the receiving point, but it should be the most similar Therefore, in a possible implementation, in order to determine the audio frame most similar to the input audio frame from the reference point signal, an audio feature matching method can be used, that is, by performing audio feature extraction on the input audio frame , to obtain the first audio feature, and then perform feature matching based on the first audio feature and the audio feature corresponding to the historical output audio frame (reference point signal), so as to determine the audio feature most similar to the first audio feature, for subsequent Perform echo delay estimation.

Since the first audio feature is used for audio frame matching, in order to improve the accuracy of audio matching, the first audio feature generally selects the unique features of the audio signal. Schematically, the first audio feature can be Mel frequency Cepstral coefficients (Mel-frequency Cepstral Coefficients, MFCC), Fourier coefficients, linear prediction coefficients (LinearPrediction Coefficients, LPC), audio features based on spectral energy, etc.

Step 302, in response to reaching the target delay, based on the first audio feature, determine the second audio feature from the candidate audio features, the candidate audio feature is the audio feature corresponding to the output audio frame, the output audio frame is used for speaker playback, the second audio The feature matches the first audio feature.

During the echo cancellation process, when the process of calling the application thread to read the input audio frame from the buffer (the buffer corresponding to the hardware and software acquisition logic) is stuck, the buffer follows the first-in-first-out principle, and the data storage capacity in the buffer is limited. If the data reading thread freezes, but the data writing thread continues, when the data in the buffer is full, the newly collected input audio frames will overwrite the historical input audio frames stored in the buffer, and if the subsequent freeze recovers , will directly read the new input audio frame, so that the new input audio frame arrives at the receiving point in advance; at the same time, the freezing process will also cause the historical output audio frame corresponding to the candidate audio feature to stop writing to the historical feature storage area, so that it cannot be recorded in the historical feature storage area. An output audio frame matching the current input audio frame is found in the feature storage area, resulting in the inability to perform delay estimation, or a delay estimation error, that is, negative delay occurs (the audio feature of the current input audio frame corresponding to the output audio frame needs to be received A period of time after the current input audio frame will be written into the historical feature storage area); therefore, based on the characteristics of the negative delay, in order to accurately detect the negative delay, that is, in order to The echo delay can still be calculated. In a possible implementation, after extracting the first audio feature corresponding to the input audio frame, the feature matching will not be performed immediately based on the first audio feature, but the delay target After a delay, feature matching is performed based on the first audio feature, so that a matching candidate audio feature can be found.

Among them, the target delay can be set by the developer, or set by the business personnel based on the actual situation. Since the target delay is to ensure that the second audio feature that matches the first audio feature can still be found in the event of a negative delay, the corresponding target delay must be greater than or equal to the maximum negative delay that can occur in the operating system , illustratively, if the maximum negative delay is 100ms, the target delay may be 120ms. The embodiment of the present application does not limit the specific value of the target delay.

Optionally, the candidate audio feature is the audio feature corresponding to the output audio frame, and the output audio frame is obtained before being sent to the speaker for playback, that is, when the output audio signal passes the reference point, it is stored in the buffer, and in When the receiving point receives a frame of input audio frame, it reads a frame of historical output audio frame from the buffer, performs audio feature extraction, and stores the extracted candidate audio feature in the historical feature storage area, so that the follow-up can be based on the first The audio feature searches for a matching second audio feature from the historical feature storage area.

Step 303, determine the echo delay of the output audio frame corresponding to the second audio feature.

In a possible implementation, when the second audio feature matching the first audio feature is determined from the candidate audio features, it means that the input audio frame is the second audio feature and the corresponding output audio frame reaches the receiving point through the echo path , and during the echo delay period, the storage position of the second audio feature in the historical feature memory moves with time, correspondingly, based on the storage position of the second audio feature in the historical feature memory, it can be determined The second audio feature corresponds to an echo delay of the output audio frame.

Schematically, as shown in FIG. 4 , it shows a schematic diagram of a process of determining an echo delay in an exemplary embodiment of the present application. The output audio frame Q at the reference point, after the transmission delay Td, arrives at the receiving point to obtain the input audio frame Q'. Due to the distortion and noise of the transmission channel, Q and Q' are different but similar signals. By extracting the audio features of the input audio frame Q' at the receiving point, the first audio feature G is obtained; assuming that the output audio frame Q is the 80th audio frame, the second audio feature obtained by extracting the audio features of the output audio frame Q is F(80), and store F(80) in the tail position (tail) of the historical feature memory. After the transmission delay Td, the reference point newly acquires 31 frames of output audio frames and calculates 31 candidate audio features in turn. Put it into the historical feature storage area; therefore, when the receiving point obtains the first audio feature G corresponding to the input audio frame Q', the historical feature storage has newly stored a total of 31 candidates from F(81) to F(111) Audio features; due to the presence of target delay, after the target delay, feature matching will be performed based on the first audio feature G and the candidate audio features stored in the historical feature storage area, and after the target delay, the historical feature memory Newly stored from F(111) to F(141) candidate audio features; then the echo delay at this time is the length of 61 output audio frames from F(80) to F(141), then the corresponding can be based on The sampling interval and frame number of each output audio frame to calculate the echo delay.

Step 304, in response to the fact that the echo delay is less than the target delay, it is determined that there is an abnormal echo delay.

Since the application performs audio feature matching after the target delay is delayed after receiving the input audio frame at the receiving point, the estimated The echo delay should be the sum of the transmission delay and the target delay. The transmission delay is the delay from the reference point signal to the receiving point signal. Under normal circumstances, the transmission delay is a positive integer, that is, the echo delay must be is greater than the target delay; on the contrary, if the echo delay is less than the target delay, it means that the transmission delay is negative, that is, there is a negative delay. Therefore, in a possible implementation, when it is determined that the echo When the delay is less than the target delay, it means that the system has a negative delay caused by an abnormal thread call, that is, there is an abnormal echo delay. The corresponding delay estimation module cannot accurately estimate the transmission delay, and the delay estimation algorithm needs to be reset. And clear its cache to eliminate the case of negative latency.

To sum up, in the embodiment of the present application, in view of the negative delay in the echo cancellation process, after receiving the input audio frame at the receiving point, it takes a while to find the matching output audio frame. feature, a method for detecting negative delay in the echo cancellation process is proposed, by obtaining the first audio feature corresponding to the input audio frame and delaying the target delay, and then performing feature matching based on the first audio feature, so that In the case of negative delay, the echo delay can still be estimated; and the delay feature matching makes the calculated echo delay the sum of the target delay and the transfer delay, which can be based on the relationship between the echo delay and the target delay , to determine the positive or negative of the transmission delay, so that it can be determined in time whether there is an abnormal echo delay (negative delay), so as to prevent the echo cancellation module from continuing to perform echo cancellation work under the wrong echo delay, or avoid In the event that the echo is delayed and the echo cannot be canceled, the accuracy of the echo cancellation can be improved.

Since when the receiving point acquires the first audio feature corresponding to the input audio frame, it will not perform feature matching immediately, so as to avoid the inability to calculate the echo delay in the case of negative delay, and then after reaching the target delay, then Perform feature matching on it, and during the target delay process, the receiving point will still receive new input audio frames and perform audio feature extraction on it, so that the first audio feature in history can still be used after the target delay. For feature matching, in a possible implementation manner, a feature memory for storing the first audio feature corresponding to the input audio frame is added, and the delay function for feature matching is realized through the feature memory.

In an exemplary example, as shown in FIG. 5 , it shows a flowchart of a method for identifying an abnormal echo delay provided by another exemplary embodiment of the present application. In this embodiment of the application, the method is applied to a terminal as an example for illustration, and the method includes:

Step 501, perform time-frequency conversion and frequency band division on the input audio frame collected by the microphone, and determine M sub-bands.

It should be noted that in the embodiment of the present application, an abnormality monitoring module (abnormal delay detection module) is added to the original echo cancellation system, and both the abnormality monitoring module and the original delay estimation module need to perform audio feature extraction. In one possible scenario, the audio features extracted in the abnormal monitoring module can be the same as those extracted in the original delay estimation module; optionally, the audio features extracted in the abnormal monitoring module can also be the same as the original delay estimation module different.

When the audio features extracted by the abnormality monitoring module and the original delay estimation module are different, different feature storage areas need to be allocated for the audio features. Correspondingly, at least two feature storage areas need to be added for storing audio features. , the first feature storage area is used to store audio features corresponding to input audio frames, and the second feature storage area is used to store audio features corresponding to output audio frames.

Schematically, as shown in FIG. 6 , it shows a working diagram of an exemplary delay estimation process and abnormal delay detection process in the present application. It can be seen from FIG. 6 that the feature storage area of the audio feature extracted in the delay estimation process 601 is different from the feature storage area of the audio feature extracted in the abnormal delay detection process 602, wherein the delay estimation process corresponds to the feature storage area 1, and the abnormal delay detection process 602 corresponds to feature storage area 3 and feature storage area 4.

In the delay estimation process 601, the audio feature extraction is carried out to the reference point signal (output audio frame) by the feature extraction module 1, and the candidate audio features extracted are stored in the feature storage area 1; when the receiving point signal is obtained , the audio feature extraction is carried out to the receiving point signal (input audio frame) through the feature extraction module 2, and the first audio feature extracted is sent to the feature matching module 1 for feature matching, specifically based on the first audio feature from the feature storage Search for the matching second audio feature in zone 1, and then determine the transmission delay Tde based on the delay determination strategy 1.

In the abnormal delay detection process 602, the audio feature extraction is carried out to the reference point signal (output audio frame) by the feature extraction module 3, and the extracted candidate audio features are stored in the feature storage area 3, and passed through the feature extraction module 4 Carry out audio feature extraction to the receiving point signal (input audio frame), and store the extracted first audio feature in the feature storage area 4, and then read from the feature storage area 4 by the feature matching module 3 after the target delay The first audio feature, based on the first audio feature, finds the matching second audio feature from the feature storage area 3, and then calculates the echo delay Tde3 through the delay determination strategy 3; judge the echo delay Tde3 at the abnormal delay detection place Whether it is an abnormal echo delay, if so, reset each module in the delay estimation process.

The audio features extracted in this embodiment are frequency-domain features. In a possible implementation, firstly, time-frequency conversion and frequency band division are performed on the input audio frame to obtain the required M sub-bands, and then based on the M sub-bands, The band extracts the corresponding first audio features.

Schematically, the value of M can be set by the developer. Since the first audio feature needs to be stored in the feature storage area, in order to improve calculation efficiency, M can be a quantity that is convenient for subsequent storage and calculation. Schematically, the value of M is 36, and 32 binary audio features can be obtained from 36 subbands, and 32 bits can be stored in a 32-bit integer variable, which is convenient for improving storage and calculation efficiency.

Optionally, short-time Fourier transform can be used to convert the input audio frame to the frequency domain to obtain the spectrum signal of the input audio frame, and then divide the frequency spectrum signal into frequency bands. The frequency band division method can be linear division or based on psychoacoustic theory. The Mel frequency band is divided to obtain M subbands.

Optionally, in a voice call scenario, the sampling rate corresponding to the reference point signal and the receiving point signal is 16kHz or 32kHz, etc. Taking 32kHz as an example, according to the Nyquist sampling theorem, the effective bandwidth of the voice collected at the 32kHz sampling rate is 16KHz; and the voice bandwidth required to extract audio features does not need to be very high, because in the actual voice communication system, the frequency range that can reliably characterize voice is about 300Hz to 3kHz, so the actual bandwidth required is only about 3kHz, so , in order to reduce the amount of calculation, during the audio feature extraction process, the audio frame with a high sampling rate of the reference point or the receiving point is down-sampled to about 6khz (the effective audio bandwidth corresponding to the 6kHz sampling rate is 3kHz), and then the subsequent audio feature extraction process is performed.

Schematically, as shown in FIG. 7 , it shows a schematic diagram of an audio feature extraction process shown in an exemplary embodiment of the present application. The audio frame after the downsampling of the input audio frame is converted to the frequency domain through a time-frequency domain conversion algorithm (such as short-time Fourier transform) to obtain the spectral signal of the input audio frame; the spectral signal is divided into 36 frequency bands. subbands (subband 1 to subband 36).

In step 502, N first frequency-domain feature scores are obtained by comparing sub-band energies of M sub-bands in frequency domain, where N is a positive integer, and M-N is a positive integer.

In a possible implementation manner, after time-frequency conversion and frequency band division are performed on the input audio frame to obtain M subbands, the subband energy corresponding to each subband is calculated respectively, and then the frequency domain energy comparison is performed on the subband energy, Thus, N frequency-domain feature scores are obtained.

Among them, the specific value of N can be set by the developer. In order to improve storage and calculation efficiency, the value of N can be 32, because 32 bits can be stored in a 31-bit integer variable; optionally, the value of N Values can also be 16, 64, etc.

Optionally, the process of calculating the feature score in the frequency domain may include the following steps, that is, step 502 may include step 502A and step 502B.

Step 502A, in response to the energy of the jth subband being the maximum value among the subband energies corresponding to the j-ith subband to the j+ith subband, determine the first score as the first frequency-domain feature score corresponding to the jth subband value, the jth subband energy is the subband energy corresponding to the jth subband, where i is a positive integer, and j-i is a positive integer, and j+i is less than or equal to M.

In a possible implementation, when calculating the frequency-domain feature score, the energy of the current subband is compared with the energy of the upper and lower subbands adjacent to it, and if the energy of the current subband has the maximum value, a binary The value is 1, otherwise the binary value 0 is output; that is, for the jth subband, if the jth subband energy corresponding to the jth subband is the energy of the j-ith subband to the j+ith subband corresponding to the subband energy maximum value, the first score is determined as the first frequency-domain feature score corresponding to the jth subband. Schematically, the first score may be 1.

Among them, the value of i can be set by the developer. Schematically, the value of i can be 1, which corresponds to comparing the energy of the current subband with the energy of the upper and lower subbands adjacent to it; if the value of i is The value is 2, which corresponds to comparing the energy of the current sub-band with the energy of two sub-bands adjacent to it.

Step 502B, in response to the energy of the jth subband being less than the maximum value of the subband energies corresponding to the j-ith subband to the j+ith subband, determine the second score as the first frequency-domain feature score corresponding to the jth subband value.

Conversely, for the jth subband energy, if the jth subband energy is less than the maximum value of the j-ith subband to the j+ith subband corresponding to the subband energy, the second score is determined as the jth subband corresponding The first frequency-domain feature score. Schematically, the second score may be 0.

As shown in Figure 7, when M subbands are obtained, the energy of each subband is calculated separately to obtain 36 subband energies (subband energy 1 to subband energy 36); if the value of i is 2, the comparator will The energy of the current subband is compared with the two subband energies adjacent to it; for example, comparator 1 compares the subband energy 3 with the two subband energies adjacent to it, that is, if the subband energy 3 is in If subband energy 1 to subband energy 5 has the maximum value, then comparator 1 sets the frequency-domain feature score corresponding to subband energy 3 to 1, that is, comparator 1 outputs G(n, 1), otherwise comparator 1 1 outputs G(n, 0); since subband energy 4 has the maximum value among subband energy 2 to subband energy 6, comparator 2 outputs G(n, 1); where n is the nth audio frame.

Step 503, determining a set of N first frequency-domain feature scores as a first audio feature.

In a possible implementation manner, the subband energies of the M subbands are energy compared to obtain N first frequency domain feature scores, and then the set of N first frequency domain feature scores is the input audio frame corresponding to the first audio feature.

Exemplarily, if N is 32, the first audio feature is a set of 32-bit binary values.

It should be noted that, the feature extraction process of the output audio frame may refer to the feature extraction process of the input audio frame, which will not be described in detail here in this embodiment.

Step 504, the first audio feature is stored in the tail storage position of the first feature storage area, the first storage capacity of the first feature storage area is determined by the target delay, and the storage position of the first audio feature changes from the tail storage position to Head storage location moved.

In a possible implementation manner, a first feature storage area is provided for storing the first audio feature corresponding to the input audio frame, and the first feature storage area is a first-in-first-out type storage area, that is, newly added audio The feature will be stored at the end of the first feature storage area, and every time a new frame of the first audio feature corresponding to the input audio frame is added, the first feature storage area will correspondingly delete a frame of the first audio corresponding to the input audio frame from the head feature.

In order to enable the first feature storage area to achieve the function of delaying the target delay for feature matching, in a possible implementation manner, the first storage capacity of the first feature storage area needs to be determined by the target delay, that is, The feature matching is performed when the first audio feature moves from the tail storage position of the first feature storage area to the head storage position over time, so that the feature matching can be performed on the first audio feature just after a target delay.

Optionally, the first storage capacity is determined by the target delay and the sampling time interval between adjacent input audio frames. Schematically, the target delay is 120ms, and each input audio frame reaches the next audio frame after 4ms, then Indicates that the first feature storage area needs to delay the first audio feature by 30 frames before performing feature matching. Correspondingly, the first feature storage area needs to store the first audio feature corresponding to 31 frames of audio input frames.

In a possible implementation, after the first audio feature corresponding to the input audio frame is extracted, feature matching is not performed directly based on the first audio feature, but the first audio feature is stored at the end of the first feature storage area Storage location, during the target delay process, the first storage area continuously adds the first audio feature, and the storage location of the first audio feature will move from the tail storage location to the head storage location as the audio feature increases. That is, the storage location of the first audio feature will move from the tail storage location to the head storage location over time, and when the first audio feature moves to the head storage location, it can be determined that the target delay is reached.

Schematically, as shown in FIG. 8 , it shows a schematic diagram of the principle of implementing the delay function in the characteristic storage area shown in an exemplary embodiment of the present application. If the first feature storage area is written before the first audio feature G (130) corresponding to the 130th frame input audio frame, 31 frames of input audio frames such as G (99) to G (129) are stored in the first feature storage area The corresponding first audio feature; when the first audio feature G (130) corresponding to 130 frames is obtained, G (130) is written into the tail storage area (tail) of the first feature storage area, then the first feature storage The first audio feature stored in the area is: G(100)～G(130); in the target delay process, G(130) moves from the tail storage area to the head storage area position with time, when G(130) When moving to the head storage position, it is determined that the target delay is reached. At this time, the first audio features stored in the first feature storage area are: G(130)-G(160).

Step 505, in response to the first audio feature moving to the head storage location of the first feature storage area, based on the first audio feature, determine the second audio feature from the candidate audio features.

In a possible implementation manner, when the first audio feature moves to the head storage position of the first feature storage area, based on the relationship between the first storage capacity of the first feature storage area and the target delay time, it is determined that the target delay time is reached. , the second audio feature may be determined from the candidate audio features based on the first audio feature.

Wherein, the process of determining the second audio feature from candidate audio features based on the first audio feature (feature matching process) may include step 1 and step 2.

1. Perform feature matching on the first audio feature and candidate audio features to obtain at least one candidate matching score. The candidate matching score is used to indicate the degree of matching between the first audio feature and the candidate audio features. The candidate matching score and the matching into a negative correlation.

In a possible implementation manner, after the target delay, based on the first audio feature and each candidate audio feature stored in the current historical feature memory (the historical feature memory stores the candidate audio feature corresponding to the output audio frame) Perform feature matching to obtain at least one candidate matching score, and then determine a second audio feature matching the first audio feature from the candidate audio features according to the candidate matching score.

Wherein, the number of candidate matching scores is determined by the storage capacity of the historical feature storage area. Schematically, the candidate audio features corresponding to 75 frames of output audio frames are stored in the historical feature storage area, and the first audio features are respectively matched with the candidate audio features corresponding to the 75 frames of output audio frames, so that 75 audio features can be obtained Candidate match score.

Schematically, as shown in FIG. 10 , it shows a schematic diagram of a feature matching process shown in an exemplary embodiment of the present application. When after the target delay time, based on the first audio feature G (80), when searching for a matching second audio feature from the candidate audio features, 75 such as F (67) to F (141) are stored in the historical feature memory. Candidate audio features correspond to feature matching between G(80) and 75 candidate audio features to obtain 75 candidate matching scores: S(1)～S(75).

It can be seen from the above embodiments that each audio feature contains N frequency-domain feature scores. Correspondingly, in the process of feature matching, it is also necessary to perform matching operations on each frequency-domain feature score. In an example In a specific example, Step 1 may also include Step 1 and Step 2 (that is, the process of determining the candidate matching score may also include Step 1 and Step 2).

1. Perform a matching operation on the k-th first frequency-domain feature score and the k-th second frequency-domain feature score to obtain the k-th sub-matching score, k is a positive integer less than or equal to N, and the sub-matching score is the first The matching degree of the audio feature and the score of the kth frequency domain feature in the second audio feature.

In order to perform feature matching, it is necessary to ensure that the candidate audio feature and the first audio feature are extracted using the same feature extraction method, and the number of frequency domain feature scores contained in the candidate audio feature needs to be the same as that of the first audio feature The number of frequency-domain feature scores contained in is the same. Schematically, the first audio feature contains N first frequency-domain feature scores, and the corresponding candidate audio feature also contains N second frequency-domain feature scores. , N is a positive integer. Schematically, when N is 32, the first audio feature contains 32 binary values, and the candidate audio feature also contains 32 binary values.

In a possible implementation manner, during the matching operation, the matching operation is performed on the N first frequency-domain feature scores and the N second frequency-domain feature scores respectively, that is, for the kth first The frequency domain feature score is matched with the kth second audio feature to obtain the kth sub-matching score, and then the sum of the N sub-matching scores is determined as the matching score of the first audio feature and the candidate audio feature.

Schematically, if N is 5, the first audio features are: G ₁ (n, 0), G ₂ (n, 1), G ₃ (n, 1), G ₄ (n, 0), G ₅ ( n, 1), the candidate audio features are: F ₁ (n, 1), F ₂ (n, 0), F ₃ (n, 0), F ₄ (n, 1), F ₅ (n, 0); Then, when performing the matching operation on the first audio feature and the candidate audio feature, G ₁ (n, 0) and F ₁ (n, 1) are matched to obtain the first sub-matching score, and G ₂ (n, 1 ) and F ₂ (n, 0) to perform matching operations to obtain the second sub-matching score, similarly the third sub-matching score, the fourth sub-matching score and the fifth sub-matching score can be obtained, and then the first The sum of the sub-matching scores to the fifth sub-matching scores is determined as a candidate matching score.

Optionally, the matching operation may use an exclusive OR operation or other matching algorithms, which is not limited in this embodiment of the present application.

2. Determine the sum of the N sub-matching scores as the candidate matching score.

In a possible implementation manner, the sum of N sub-matching scores is determined as a candidate matching score. Schematically, if the matching operation uses an XOR operation, the more similar the first audio feature is to the candidate audio feature, the lower the corresponding candidate matching score, that is, the candidate matching score is negatively correlated with the matching degree.

2. Determine a second audio feature from the candidate audio features based on at least one candidate matching score.

The purpose of feature matching is to find the second audio feature that matches the first audio feature from the candidate audio features, and based on the relationship between the matching degree and the candidate matching score, the higher the matching degree, the lower the candidate matching score , correspondingly, the candidate audio feature corresponding to the minimum value among the candidate matching scores may be determined as the second audio feature.

Optionally, in other possible implementation manners, after the candidate matching score is calculated, it may be smoothed based on historical matching results to obtain a feature matching score, and then the second audio feature is determined based on the feature matching score. In an exemplary example, Step 2 may further include Step 3-Step 5.

3. Perform smoothing processing on at least one candidate matching score to obtain at least one feature matching score.

In a possible implementation manner, each candidate matching score is smoothed to obtain at least one feature matching score.

In an illustrative example, the calculation process of the feature matching score can be expressed as:

Sm(n)=S(n)*b+Sm(n)'*(1-b)

Among them, Sm(n) represents the feature matching score, S(n) represents the candidate matching score, b represents the smoothing coefficient, b is a decimal between 0 and 1, the smaller b is, the higher the smoothing degree is, and Sm( n)' represents the previous smoothing result.

Schematically, as shown in FIG. 10 , 75 candidate matching scores S(1)˜S(75) are smoothed to obtain 75 feature matching scores: Sm(1)˜Sm(75).

4. Determine the minimum value among the feature matching scores as the target matching score.

Since the candidate matching score is negatively correlated with the matching degree, the feature matching score is also negatively correlated with the matching degree. The smaller the feature matching score corresponding to the first audio feature, therefore, in a possible implementation, the minimum value of the feature matching score is determined as the target matching score, and then the candidate audio corresponding to the target matching score The characteristic determines a second audio characteristic.

5. Determine the candidate audio feature corresponding to the target matching score as the second audio feature.

In a possible implementation manner, since the candidate audio feature corresponding to the target matching score is the audio feature that best matches the first audio feature, it may be directly determined as the second audio feature.

Optionally, in order to further improve the accuracy of feature matching, in a possible implementation, a matching score threshold is set, and when the target matching score is smaller than the matching score threshold, the corresponding candidate audio The feature is determined as a second audio feature.

Step 506, acquiring a target storage location of the second audio feature in the second feature storage area.

In a possible implementation manner, after the second audio feature is determined, since the second audio feature is also stored in the first-in-first-out second feature storage area, and the storage position of the second audio feature is stored by the tail The location is shifted towards the head storage location, so the target echo delay can be determined based on the location of the second audio feature in the second feature storage area.

Step 507, based on the target storage location, determine the target echo delay of the output audio frame corresponding to the second audio feature.

As shown in FIG. 9 , it shows a schematic diagram of a delayed feature matching process according to an exemplary embodiment of the present application. The output audio frame Q at the reference point, after the transmission delay Td, arrives at the receiving point to obtain the input audio frame Q'. The feature G (100) is obtained by extracting the audio features of the input audio frame Q' at the receiving point, and stored in the tail storage position of the first feature storage area (where G (130) is located in the figure); the output audio frame Q is assumed to be The 100th frame audio frame, the second audio feature obtained by extracting the audio feature of the output audio frame Q is F (100), and F (100) is stored in the tail storage position (tail) of the second feature storage area, after passing After the transmission delay Td, the reference point newly acquires 31 output audio frames and calculates 31 candidate audio features and puts them into the second feature storage area in turn; therefore, when the receiving point obtains the first audio frame corresponding to the input audio frame Q' During the feature G (100), 31 candidate audio features from F (100) to F (131) have been newly stored in the second feature storage area; When the position of G (130) moves to the head of the first feature storage area, feature matching is performed based on the first audio feature G (100) and the candidate audio features stored in the second feature storage area; and after the target delay , newly stored from F (131) to F (161) candidate audio features in the second feature storage area; then the echo delay at this time is 61 output audio frames from F (100) to F (161) The corresponding echo delay can be calculated based on the sampling interval of each output audio frame and the number of audio frames.

In a possible implementation manner, based on the delay feature matching process shown in FIG. 9 , the echo delay of the output audio frame corresponding to the second audio feature can be determined based on the target storage location of the second audio feature in the second feature storage area. hour.

Schematically, as shown in Figure 10, the first audio feature G (80) matches F (80) in the second feature storage area, and F (80) corresponds to Sm (14) in the feature matching score, and the echo delay The time is the length of 61 audio frames from Sm(14) to Sm(75); if each audio frame reaches the next audio frame after 4ms, the echo delay is 61×4ms.

Step 508: In response to the echo delay being greater than the target delay, perform echo cancellation processing on the input audio frame based on the echo delay obtained by the echo delay estimation.

In a possible implementation manner, if the echo delay is greater than the target delay, it means that no negative delay occurs, and echo cancellation processing may be performed on the input audio frame based on the echo delay obtained in the original delay estimation process.

Step 509, in response to the fact that the echo delay is less than the target delay, it is determined that there is an abnormal echo delay.

For the implementation manner of step 509, reference may be made to the foregoing embodiments, and details are not described in this embodiment here.

Step 510, re-estimating the echo delay in response to the abnormal echo delay.

In a possible implementation, if there is an abnormal echo delay, it is considered that the echo delay generated in the original delay estimation process is unreliable, and a reset command is correspondingly generated to reset the delay estimation module and delay alignment module, clear their buffers to eliminate the negative delay case, so that the delay estimation module can re-echo delay estimation.

In this embodiment, by setting the first feature storage area for the first audio feature corresponding to the input audio frame, in order to realize the feature delay matching of the first audio feature by storing the first audio feature in the first feature storage area function; in addition, by using an audio feature extraction method different from the original delay estimation process in the abnormal delay monitoring process, the accuracy of feature matching in the abnormal delay monitoring process can be improved, thereby improving the abnormal delay monitoring process. Determine the accuracy of the echo delay, and then use the more accurate echo delay to guide whether to continue the original delay estimation process.

In another possible application scenario, in order to save the amount of calculation and data storage space, the exception monitoring module and the original delay estimation module are set to use the same feature extraction method. Correspondingly, only one feature storage area needs to be added. It only needs to store the first audio feature corresponding to the input audio frame.

Corresponding to Figure 6, if the same feature extraction method is used, correspondingly, the abnormal delay detection process does not need to use a separate feature extraction module, which can reduce the amount of calculations generated by additional feature extraction; at the same time, the candidate audio features corresponding to the reference point signal Therefore, there is no need to additionally use a new feature storage area for storage, and it is only necessary to increase the feature storage area for storing the first audio feature, which can reduce the storage space occupied by repeated audio feature storage.

On the basis of Figure 6, after deleting the feature extraction module 3, the feature extraction module 4 and the feature storage area 3, as shown in Figure 11, it shows the delay estimation process and abnormalities shown in another exemplary embodiment of the present application Schematic diagram of the delay detection process. In FIG. 11 , the candidate audio features stored in the feature storage area 1 can be used for feature matching module 1 to perform feature matching, so as to calculate the echo delay Tde generated in the delay estimation process 1101; The matching module 2 performs feature matching for calculating the echo delay Tde2 generated in the abnormal delay detection process 1102; and the first audio feature stored in the feature storage area 2 can be used immediately when the first audio feature is obtained. The feature matching is performed in the feature matching module 1 for estimating the echo delay Tde; after the target delay is reached, the feature matching module 2 can be used for feature matching for estimating the echo delay Tde2.

Schematically, in the delay estimation process 1101, the audio feature extraction is performed on the reference point signal (output audio frame) by the feature extraction module 1, and the extracted candidate audio features are stored in the feature storage area 1; After receiving the point signal, the audio feature extraction is carried out to the receiving point signal (input audio frame) by the feature extraction module 2, and the first audio feature extracted is sent to the feature matching module 1 for feature matching, specifically based on the first audio The feature finds the matching second audio feature from the feature storage area 1, and then determines the transmission delay Tde based on the delay determination strategy 1.

In the abnormal delay detection process 1102, after the first audio feature corresponding to the input audio frame is obtained, the first audio feature is stored in the feature storage area 2, and then after the target delay, the feature matching module 2 is used to store the first audio feature Read the first audio feature in area 2, search for the second audio feature matching it from feature storage area 1 based on the first audio feature, and then calculate the echo delay Tde2 through the delay determination strategy 2; in abnormal delay detection It is judged whether the echo delay Tde2 is an abnormal echo delay, and if so, each module in the delay estimation process is reset.

It can be seen from FIG. 11 that the delay estimation process and the abnormal delay detection process both use the same feature extraction method. FIG. 12 shows a schematic diagram of the audio feature extraction process in the delay estimation process shown in an exemplary embodiment of the present application. The audio frame of the reference point or the receiving point is converted to the frequency domain by downsampling and time-frequency domain conversion algorithm (such as short-time Fourier transform), and the spectral signal of the audio frame is obtained; the spectral signal is divided into frequency bands, such as linear division or according to psychological Acoustic theory divides the Mel frequency band and divides the spectrum into M subbands (M is 32), because 32 bits can be stored in a 32-bit plastic variable, which is convenient for improving calculation efficiency. M can also be 16, 64, etc. Amount of storage and computation. Calculate the subband energy E(m) for each subband, and calculate the smoothed subband energy 1 of each subband through the smooth subband energy module, smooth subband energy Ep(m)=E(m)*a +Ep(m)'*(1-a), where a is a decimal between 0 and 1, and the smaller a is, the higher the smoothness is. The comparator compares the sub-band energy with the smooth sub-band energy, and outputs a binary value 1 if the sub-band energy is greater than the smooth sub-band energy, otherwise outputs a binary value 0. Assuming that the current frame is the nth frame, the binarized output of the first subband is stored in the 0th bit of the nth frame of F, that is, F(n, 0), and the binarized output of the second subband is stored in the same way In the first bit of the nth frame of F, that is, F(n, 1), by analogy, M binary values are obtained, which are the extracted audio features.

In another possible application scenario, in order to further reduce the amount of calculation in the feature matching process, the second storage capacity of the second feature storage area may be smaller than the first storage capacity corresponding to the first feature storage area. , if an output audio frame matching the input audio frame can be found in the second feature storage area, it indicates that a negative delay occurs.

On the basis of FIG. 3 , as shown in FIG. 13 , step 302 to step 304 may be replaced by step 1301 and step 1302 .

Step 1301, in response to reaching the target delay, search for candidate audio features matching the first audio feature from the second feature storage area.

Wherein, the first storage capacity of the first feature storage area is determined by the target delay and the sampling time interval between adjacent input audio frames, then the number of frames of the corresponding input audio frames stored in the second feature storage area is less than the first The frame number of the output audio frame corresponding to the audio feature is stored in the feature storage area. Schematically, if the first feature storage area can be used to store audio features of 31 audio frames, then the second feature storage area can be set to be able to store audio features of 30 audio frames.

In a possible implementation manner, after the first audio feature is extracted and the target delay is reached, the second feature storage area is searched based on the first audio feature, that is, the first audio feature and the second Each candidate audio feature stored in the feature storage area is matched, and a matching score corresponding to the first audio feature and each candidate audio feature is obtained.

It should be noted that when the second storage capacity of the second feature storage area is smaller than the first feature storage area, the second feature storage area cannot be shared in the abnormal delay detection process and the delay estimation process, that is, even if the abnormal The same audio feature extraction method is used in the delay detection process and the delay estimation process, and the candidate audio features to be used in the delay estimation module also need to be stored in other feature storage areas.

Step 1302, in response to finding a candidate audio feature that matches the first audio feature, determine that there is an abnormal echo delay.

Since the abnormal delay detection process only needs to detect the situation that the echo delay is less than the target delay, when the second storage capacity of the second feature storage area is smaller than the first storage capacity, under normal circumstances, feature matching is performed based on the first audio feature When , there is no matching candidate audio feature in the second storage area. On the contrary, if a matching candidate audio feature can be found in the second feature storage area, a value of an echo delay that is less than the target delay can be obtained , so it can be explained that there is a negative delay; therefore, in a possible implementation, when a candidate audio feature matching the first audio feature is found in the second storage area, it is determined that there is an abnormal echo delay .

In this embodiment, by setting the storage capacity of the second feature storage area, it is not necessary to calculate the echo delay, and it is only necessary to determine whether there is a candidate audio feature that matches the first audio feature in the second feature storage area. If there is, then Determining the existence of abnormal echo delay can simplify the detection logic of abnormal delay and further improve the detection efficiency of abnormal echo delay; in addition, reducing the storage capacity of the second feature storage area can also reduce the number of feature matching. It is recommended not to reduce the abnormal echo delay. Echo delay detection efficiency.

The following are device embodiments of the present application. For details not described in detail in the device embodiments, reference may be made to the above method embodiments.

Fig. 14 is a structural block diagram of an abnormal echo delay identification device provided by an exemplary embodiment of the present application. The unit includes:

The feature extraction module 1401 is used to extract the audio feature from the input audio frame collected by the microphone to obtain the first audio feature;

The first determining module 1402 is configured to determine a second audio feature from candidate audio features based on the first audio feature in response to reaching the target delay, the candidate audio feature being an audio feature corresponding to an output audio frame, the outputting an audio frame for playback by a speaker, the second audio feature matching the first audio feature;

The second determination module 1403 is configured to determine the echo delay of the output audio frame corresponding to the second audio feature;

The third determination module 1404 is configured to determine that there is an abnormal echo delay in response to the echo delay being less than the target delay.

Optionally, the first determining module 1402 includes:

a storage unit, configured to store the first audio feature in the tail storage position of the first feature storage area, the first storage capacity of the first feature storage area is determined by the target delay, and the first audio feature The storage location of moves from the tail storage location to the head storage location over time;

A first determining unit, configured to determine the second audio feature from the candidate audio features based on the first audio feature in response to the first audio feature moving to the head storage location of the first feature storage area. audio characteristics.

Optionally, the candidate audio feature is stored in the second feature storage area, and the storage location of the candidate audio feature moves from the tail storage location to the head storage location over time;

The second determining module 1403 includes:

an acquiring unit, configured to acquire a target storage location of the second audio feature in the second feature storage area;

The second determining unit is configured to determine, based on the target storage location, a target echo delay of the output audio frame corresponding to the second audio feature.

Optionally, the second storage capacity of the second feature storage area is smaller than the first storage capacity;

The device also includes:

a search module, configured to search for a candidate audio feature matching the first audio feature from the second feature storage area in response to reaching the target delay;

The fourth determining module is configured to determine that there is an abnormal echo delay in response to finding a candidate audio feature that matches the first audio feature.

Optionally, the first storage capacity is determined by the target delay and the sampling time interval between adjacent input audio frames, and the number of frames corresponding to the input audio frames stored in the second feature storage area is less than the specified The audio feature is stored in the first feature storage area corresponding to the frame number of the output audio frame.

Optionally, the feature extraction module 1401 includes:

A third determining unit, configured to perform time-frequency conversion and frequency band division on the input audio frame collected by the microphone, to determine M subbands;

The fourth determining unit is configured to obtain N first frequency-domain feature scores by comparing the sub-band energies of the M sub-bands in the frequency domain, where N is a positive integer, and M-N is a positive integer;

The fifth determining unit is configured to determine a set of N first frequency-domain feature scores as the first audio feature.

Optionally, the fourth determination unit is also used for:

Responding to the fact that the jth subband energy is the maximum value among the subband energies corresponding to the j-ith subband to the j+ith subband, determining a first score as the first frequency domain feature corresponding to the jth subband Score, the jth subband energy is the subband energy corresponding to the jth subband, where i is a positive integer, and j-i is a positive integer, and j+i is less than or equal to M;

In response to the energy of the jth subband being less than the maximum value of the subband energies corresponding to the j-ith subband to the j+ith subband, determining a second score as the first frequency corresponding to the jth subband domain feature score.

Optionally, the first determining module 1402 includes:

A feature matching unit, configured to perform feature matching on the first audio feature and the candidate audio feature to obtain at least one candidate matching score, the candidate matching score being used to indicate the first audio feature and the candidate audio feature The degree of matching between audio features, the candidate matching score is negatively correlated with the degree of similarity;

A sixth determining unit, configured to determine the second audio feature from the candidate audio features based on at least one candidate matching score.

Optionally, the first audio features include N first frequency-domain feature scores, and the candidate audio features include N second frequency-domain feature scores, where N is a positive integer;

The feature matching unit is also used for:

Perform a matching operation on the kth first frequency domain feature score and the kth second frequency domain feature score to obtain the kth sub-matching score, k is a positive integer less than or equal to N, and the sub-matching score is the The matching degree of the first audio feature and the kth frequency domain feature score in the second audio feature;

The sum of the N sub-matching scores is determined as the candidate matching score.

Optionally, the fifth determining unit is also used for:

smoothing at least one of the candidate matching scores to obtain at least one feature matching score;

determining the minimum value among the feature matching scores as the target matching score;

The candidate audio feature corresponding to the target matching score is determined as the second audio feature.

Optionally, the device also includes:

The reset module is used for re-estimating the echo delay in response to the abnormal echo delay.

Optionally, the device also includes:

The echo cancellation module is configured to perform echo cancellation processing on the input audio frame based on the echo delay obtained by echo delay estimation in response to the echo delay being greater than the target delay.

To sum up, in the embodiment of the present application, in view of the negative delay in the echo cancellation process, after receiving the input audio frame at the receiving point, it takes a while to find the matching output audio frame. feature, a method for detecting negative delay in the echo cancellation process is proposed, by obtaining the first audio feature corresponding to the input audio frame and delaying the target delay, and then performing feature matching based on the first audio feature, so that In the case of negative delay, the echo delay can still be estimated; and the delay feature matching makes the calculated echo delay the sum of the target delay and the transfer delay, which can be based on the relationship between the echo delay and the target delay , to determine the positive or negative of the transmission delay, so that it can be determined in time whether there is an abnormal echo delay (negative delay), so as to prevent the echo cancellation module from continuing to perform echo cancellation work under the wrong echo delay, or avoid In the event that the echo is delayed and the echo cannot be canceled, the accuracy of the echo cancellation can be improved.

Fig. 15 shows a structural block diagram of a terminal 1500 provided by an exemplary embodiment of the present application. The terminal 1500 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, moving picture experts compress standard audio layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, moving picture experts compress standard audio layer 4 ) player, laptop or desktop computer. The terminal 1500 may also be called user equipment, portable terminal, laptop terminal, desktop terminal and other names.

Generally, the terminal 1500 includes: a processor 1501 and a memory 1502 .

The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1501 can adopt at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 1501 may also include a main processor and a coprocessor, the main processor is a processor for processing data in the wake-up state, and is also called a CPU (Central Processing Unit, central processing unit); the coprocessor is used to Low-power processor for processing data in standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1501 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is configured to process computing operations related to machine learning.

Memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory, and non-volatile memory, such as one or more magnetic disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1502 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 1501 to implement the information processing provided by the method embodiments in this application method.

In some embodiments, the terminal 1500 may optionally further include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, the memory 1502, and the peripheral device interface 1503 may be connected through buses or signal lines. Each peripheral device can be connected to the peripheral device interface 1503 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1504 , a display screen 1505 , a camera component 1506 , an audio circuit 1507 , a positioning component 1508 and a power supply 1509 .

The peripheral device interface 1503 may be used to connect at least one peripheral device related to I/O (Input/Output, input/output) to the processor 1501 and the memory 1502 . In some embodiments, the processor 1501, memory 1502 and peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 1501, memory 1502 and peripheral device interface 1503 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 1504 is used to receive and transmit RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 1504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: World Wide Web, Metropolitan Area Network, Intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1504 may also include circuits related to NFC (Near Field Communication, short-range wireless communication), which is not limited in this application.

The display screen 1505 is used for displaying a UI (User Interface, user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1505 is a touch display screen, the display screen 1505 also has the ability to collect touch signals on or above the surface of the display screen 1505 . The touch signal can be input to the processor 1501 as a control signal for processing. At this time, the display screen 1505 can also be used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 1505, which is provided on the front panel of the terminal 1500; in other embodiments, there may be at least two display screens 1505, which are respectively provided on different surfaces of the terminal 1500 or in a folding design; In some other embodiments, the display screen 1505 may be a flexible display screen, which is arranged on the curved surface or the folding surface of the terminal 1500 . Even, the display screen 1505 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1505 may be made of LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light-emitting diode) and other materials.

The camera assembly 1506 is used to capture images or videos. Optionally, the camera component 1506 includes a front camera and a rear camera. Usually, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of the main camera, depth-of-field camera, wide-angle camera, and telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function. Combined with the wide-angle camera to realize panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 1507 may include a microphone and speakers. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals and input them to the processor 1501 for processing, or input them to the radio frequency circuit 1504 to realize voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, which are respectively set at different parts of the terminal 1500 . The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1501 or the radio frequency circuit 1504 into sound waves. The loudspeaker can be a conventional membrane loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it is possible not only to convert electrical signals into sound waves audible to humans, but also to convert electrical signals into sound waves inaudible to humans for purposes such as distance measurement. In some embodiments, audio circuitry 1507 may also include a headphone jack.

The positioning component 1508 is used to locate the current geographic location of the terminal 1500 to implement navigation or LBS (Location Based Service, location-based service). The positioning component 1508 may be a positioning component based on the GPS (Global Positioning System, Global Positioning System) of the United States, the Beidou system of China or the Galileo system of Russia.

The power supply 1509 is used to supply power to various components in the terminal 1500 . Power source 1509 may be alternating current, direct current, disposable batteries, or rechargeable batteries. When the power source 1509 includes a rechargeable battery, the rechargeable battery may be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the terminal 1500 further includes one or more sensors 1510 . The one or more sensors 150 include, but are not limited to: an acceleration sensor 1511 , a gyro sensor 1512 , a pressure sensor 1513 , a fingerprint sensor 1514 , an optical sensor 1515 and a proximity sensor 1516 .

The acceleration sensor 1511 can detect the acceleration on the three coordinate axes of the coordinate system established by the terminal 1500 . For example, the acceleration sensor 1511 can be used to detect the components of the acceleration of gravity on the three coordinate axes. The processor 1501 can control the touch display screen 1505 to process information in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1511 . The acceleration sensor 1511 can also be used for collecting game or user's motion data.

The gyro sensor 1512 can detect the body direction and rotation angle of the terminal 1500 , and the gyro sensor 1512 can cooperate with the acceleration sensor 1511 to collect 3D actions of the user on the terminal 1500 . According to the data collected by the gyroscope sensor 1512, the processor 1501 can realize the following functions: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control and inertial navigation.

The pressure sensor 1513 may be disposed on the side frame of the terminal 1500 and/or the lower layer of the touch display screen 1505 . When the pressure sensor 1513 is set on the side frame of the terminal 1500 , it can detect the user's grip signal on the terminal 1500 , and the processor 1501 performs left and right hand recognition or shortcut operation according to the grip signal collected by the pressure sensor 1513 . When the pressure sensor 1513 is arranged on the lower layer of the touch screen 1505, the processor 1501 controls the operable controls on the UI interface according to the user's pressure operation on the touch screen 1505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 1514 is used to collect the user's fingerprint, and the processor 1501 recognizes the identity of the user according to the fingerprint collected by the fingerprint sensor 1514, or, the fingerprint sensor 1514 recognizes the user's identity according to the collected fingerprint. When the identity of the user is recognized as a trusted identity, the processor 1501 authorizes the user to perform related sensitive operations, such sensitive operations include unlocking the screen, viewing encrypted information, downloading software, making payment, and changing settings. The fingerprint sensor 1514 may be provided on the front, back or side of the terminal 1500 . When the terminal 1500 is provided with a physical button or a manufacturer's Logo, the fingerprint sensor 1514 may be integrated with the physical button or the manufacturer's Logo.

The optical sensor 1515 is used to collect ambient light intensity. In one embodiment, the processor 1501 can control the display brightness of the touch screen 1505 according to the ambient light intensity collected by the optical sensor 1515 . Specifically, when the ambient light intensity is high, the display brightness of the touch screen 1505 is increased; when the ambient light intensity is low, the display brightness of the touch screen 1505 is decreased. In another embodiment, the processor 1501 may also dynamically adjust shooting parameters of the camera assembly 1506 according to the ambient light intensity collected by the optical sensor 1515 .

The proximity sensor 1516 , also called a distance sensor, is usually arranged on the front panel of the terminal 1500 . The proximity sensor 1516 is used to collect the distance between the user and the front of the terminal 1500 . In one embodiment, when the proximity sensor 1516 detects that the distance between the user and the front of the terminal 1500 gradually decreases, the processor 1501 controls the touch display screen 1505 to switch from the bright screen state to the off-screen state; when the proximity sensor 1516 detects When the distance between the user and the front of the terminal 1500 gradually increases, the processor 1501 controls the touch display screen 1505 to switch from the off-screen state to the on-screen state.

Those skilled in the art can understand that the structure shown in FIG. 15 does not constitute a limitation to the terminal 1500, and may include more or less components than shown in the figure, or combine certain components, or adopt a different component arrangement.

The present application also provides a computer-readable storage medium, wherein at least one instruction, at least one program, code set or instruction set is stored in the readable storage medium, the at least one instruction, the at least one program, the The code set or instruction set is loaded and executed by the processor to implement the abnormal echo delay identification method provided by any of the above exemplary embodiments.

An embodiment of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the terminal reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the terminal executes the abnormal echo delay identification method provided in the above optional implementation manner.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above are only optional embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection of the application. within range.

Claims (15)

1. An abnormal echo delay identification method, characterized in that the method comprises:

performing audio feature extraction on an input audio frame acquired by a microphone to obtain a first audio feature;

in response to reaching a target delay, determining a second audio feature from candidate audio features based on the first audio feature, wherein the candidate audio features are audio features corresponding to an output audio frame, the output audio frame is used for loudspeaker playing, and the second audio feature is matched with the first audio feature;

determining the echo delay of the output audio frame corresponding to the second audio characteristic;

and determining that abnormal echo delay exists in response to the echo delay being less than the target delay.

2. The method of claim 1, wherein determining a second audio feature from the candidate audio features based on the first audio feature in response to reaching the target latency comprises:

storing the first audio feature in a tail storage location of a first feature store, a first storage capacity of the first feature store being determined by the target latency, the storage location of the first audio feature moving from the tail storage location to a head storage location over time;

determining the second audio feature from the candidate audio features based on the first audio feature in response to the first audio feature moving to a head storage location of the first feature store.

3. The method of claim 2, wherein the candidate audio feature is stored in a second feature storage region, the storage location of the candidate audio feature moving from a tail storage location to a head storage location over time;

the determining that the second audio feature corresponds to the echo delay of the output audio frame includes:

acquiring a target storage position of the second audio feature in the second feature storage area;

and determining the target echo delay of the output audio frame corresponding to the second audio characteristic based on the target storage position.

4. The method of claim 3, wherein the second storage capacity of the second feature storage area is less than the first storage capacity;

after the audio feature extraction is performed on the input audio frame acquired by the microphone to obtain the first audio feature, the method further includes:

in response to reaching the target delay, searching the second feature store for candidate audio features that match the first audio feature;

determining that an abnormal echo delay exists in response to finding a candidate audio feature that matches the first audio feature.

5. The method of claim 4,

the first storage capacity is determined by the target delay and the sampling time interval between adjacent input audio frames, and the number of the input audio frames corresponding to the audio features stored in the second feature storage area is smaller than the number of the output audio frames corresponding to the audio features stored in the first feature storage area.

6. The method according to any one of claims 1 to 5, wherein the performing audio feature extraction on the input audio frame collected by the microphone to obtain the first audio feature comprises:

performing time-frequency conversion and frequency band division on the input audio frame acquired by the microphone to determine M sub-bands;

obtaining N first frequency domain feature scores by comparing the frequency domain energy of the sub-band energy of the M sub-bands, wherein N is a positive integer, and M-N is a positive integer;

determining a set of N first frequency-domain feature scores as the first audio feature.

7. The method of claim 6, wherein obtaining N first frequency-domain feature scores by frequency-domain energy comparison of subband energies for M subbands comprises:

determining a first score as the first frequency domain feature score corresponding to the jth sub-band in response to that the jth sub-band energy is the maximum value of sub-band energies corresponding to the jth sub-band from the jth sub-band to the jth + i sub-band, wherein i is a positive integer, j-i is a positive integer, and j + i is less than or equal to M;

and determining a second score as the first frequency domain feature score corresponding to the jth sub-band in response to the jth sub-band energy being less than the maximum of sub-band energies corresponding to the jth sub-band to the jth + i sub-band.

8. The method of any of claims 1 to 5, wherein determining the second audio feature from the candidate audio features based on the first audio feature comprises:

performing feature matching on the first audio feature and the candidate audio feature to obtain at least one candidate matching score, wherein the candidate matching score is used for indicating the matching degree between the first audio feature and the candidate audio feature, and the candidate matching score and the matching degree form a negative correlation relationship;

determining the second audio feature from the candidate audio features based on at least one of the candidate match scores.

9. The method of claim 8, wherein the first audio feature comprises N first frequency-domain feature scores, wherein the candidate audio features comprise N second frequency-domain feature scores, and wherein N is a positive integer;

the performing feature matching on the first audio feature and the candidate audio feature to obtain at least one candidate matching score includes:

performing matching operation on the kth first frequency domain feature score and the kth second frequency domain feature score to obtain a kth sub-matching score, wherein k is a positive integer less than or equal to N, and the sub-matching score is the matching degree of the kth frequency domain feature score in the first audio feature and the second audio feature;

determining a sum of the N sub-match scores as the candidate match score.

10. The method of claim 8, wherein determining the second audio feature from the candidate audio features based on at least one of the candidate match scores comprises:

performing smoothing processing on at least one candidate matching score to obtain at least one feature matching score;

determining the minimum value in the feature matching scores as a target matching score;

and determining the candidate audio features corresponding to the target matching scores as the second audio features.

11. The method of any of claims 1 to 5, wherein after determining that an abnormal echo delay exists in response to the echo delay being less than the target delay, the method further comprises:

and responding to the abnormal echo delay, and re-estimating the echo delay.

12. The method of any of claims 1 to 5, wherein after determining that the second audio feature corresponds to an echo delay of an output audio frame, the method further comprises:

and in response to the echo delay being larger than the target delay, performing echo cancellation processing on the input audio frame based on the echo delay obtained by echo delay estimation.

13. An abnormal echo delay identifying apparatus, comprising:

the characteristic extraction module is used for extracting audio characteristics of the input audio frames collected by the microphone to obtain first audio characteristics;

a first determining module, configured to determine, in response to reaching a target delay, a second audio feature from candidate audio features based on the first audio feature, where the candidate audio feature is an audio feature corresponding to an output audio frame, the output audio frame is used for speaker playing, and the second audio feature is matched with the first audio feature;

a second determining module, configured to determine an echo delay of the output audio frame corresponding to the second audio feature;

and a third determining module, configured to determine that an abnormal echo delay exists in response to the echo delay being smaller than the target delay.

14. A terminal, characterized in that the terminal comprises a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the abnormal echo delay identifying method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein at least one program is stored in the computer-readable storage medium, and the at least one program is loaded and executed by a processor to implement the method for identifying abnormal echo delay according to any one of claims 1 to 12.

Images