CN115038014B - Audio signal processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the invention discloses an audio signal processing method, an audio signal processing device, electronic equipment and a storage medium, wherein the audio signal processing method comprises the following steps: acquiring a first audio signal acquired by a directional microphone and acquiring a second audio signal acquired by an omni-directional microphone; determining a sound source direction according to the acquisition parameters of the second audio signal; determining a speaking mode based on the sound source direction and the intensity of the first audio signal; and carrying out fusion processing on the first audio signal and the second audio signal based on the speaking mode to obtain a target audio signal. The embodiment of the invention can receive the audio signals in all directions without transmitting the microphone, thereby improving the user experience. And obtaining the target audio signal by carrying out fusion processing on the first audio signal and the second audio signal based on the speaking mode. The received audio signals can be processed more accurately, the signal to noise ratio of the target audio signals is effectively improved, and the user experience is further improved.

Description

Audio signal processing method, device, electronic device and storage medium

Technical Field

The embodiments of the present invention relate to signal processing technology, and in particular to an audio signal processing method, device, electronic device and storage medium.

Background Art

Microphones for collecting audio signals are generally divided into directional microphones and omnidirectional microphones. Among them, directional microphones are used to collect the audio signals of the speaker's voice. Omnidirectional microphones are used to collect the audio signals of multiple people's voices.

In daily meetings, directional microphones are often used to pick up the speaker's voice and audio signals. When other participants are speaking, the user needs to pass the microphone on, which is very inconvenient. Omnidirectional microphones can pick up voice and audio signals from multiple people, but they will also pick up more ambient noise, resulting in poor signal quality.

Summary of the invention

The embodiments of the present invention provide an audio signal processing method, an apparatus, an electronic device and a storage medium. The embodiments of the present invention can be applied to perform different noise reduction processes on received audio signals according to different speech modes.

In a first aspect, an embodiment of the present invention provides an audio signal processing method, the method comprising:

Acquire a first audio signal collected by the directional microphone, and acquire a second audio signal collected by the omnidirectional microphone;

determining a sound source direction according to an acquisition parameter of the second audio signal;

determining a speech mode according to the direction of the sound source and the strength of the first audio signal;

The first audio signal and the second audio signal are fused based on the speech mode to obtain a target audio signal.

In a second aspect, an embodiment of the present invention provides an audio signal processing device, the device comprising:

A signal acquisition module, used to acquire a first audio signal collected by the directional microphone and to acquire a second audio signal collected by the omnidirectional microphone;

A direction determination module, used to determine the direction of the sound source according to the acquisition parameters of the second audio signal;

a mode determination module, configured to determine a speech mode according to the direction of the sound source and the intensity of the first audio signal;

The target signal acquisition module is used to perform fusion processing on the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal.

In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an audio signal processing method as described in any one of the embodiments of the present invention is implemented.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio signal processing method as described in any one of the embodiments of the present invention.

In an embodiment of the present invention, an audio device including a directional microphone and an omnidirectional microphone is applied to obtain a first audio signal collected by the directional microphone and a second audio signal collected by the omnidirectional microphone; the direction of the sound source is determined according to the collection parameters of the second audio signal; the speech mode is determined according to the direction of the sound source and the intensity of the first audio signal; the first audio signal and the second audio signal are fused based on the speech mode to obtain a target audio signal. That is, in an embodiment of the present invention, the audio device has both a directional microphone and an omnidirectional microphone, and can obtain audio signals received by the directional microphone and the omnidirectional microphone, combining the advantages of these two types of microphones. While not needing to transmit the microphone, it can also receive audio signals from all directions, thereby improving the user experience; the audio signals collected by these two types of microphones are fused based on the speech mode to obtain the target audio signal, and the quality of the obtained target audio signal is improved through signal fusion, further improving the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic flow chart of an audio signal processing method provided by an embodiment of the present invention;

FIG2 is a flow chart of obtaining a target audio signal according to an embodiment of the present invention;

FIG3 is another flow chart of the audio signal processing method provided by an embodiment of the present invention;

FIG4 is a schematic diagram of a signal extraction process according to an embodiment of the present invention;

FIG5 is a structural diagram of an audio signal processing device provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

FIG1 is a flow chart of an audio signal processing method provided in an embodiment of the present invention. The method in an embodiment of the present invention is applicable to an audio device having two types of microphones. The method can be executed by an audio signal processing device provided in an embodiment of the present invention, and the device can be implemented in software and/or hardware. In a specific embodiment, the device can be integrated in an electronic device, and the electronic device can be, for example, an audio device. The following embodiments will be described by taking the device integrated in an audio device as an example. The audio device includes a directional microphone and an omnidirectional microphone, and the number of directional microphones can be one or more.

Referring to FIG1 , the method may specifically include the following steps:

Step 101: Acquire a first audio signal collected by a directional microphone, and acquire a second audio signal collected by an omnidirectional microphone.

Among them, the directional microphone is based on multi-diaphragm technology, and receives sound signals by changing the cavity setting of the microphone head and mixing and modulating the multi-diaphragm. The directional microphone has high sensitivity to the sound signal in its specified direction and low sensitivity to the sound signal in other directions. The omnidirectional microphone can receive sound signals in any direction, and has the same sensitivity to the sound signals in all directions, which is suitable for use in places such as conference rooms and offices. For example, in a meeting with multiple participants, a directional microphone and an omnidirectional microphone are set. The directional microphone can collect the sound signal of the person who is speaking with the directional microphone, and the omnidirectional microphone can collect the sound signal of the person who is speaking with the directional microphone and the sound signal emitted by other participants. Of course, the directional microphone will receive the ambient audio signal in its direction while receiving the human voice audio signal. The omnidirectional microphone will also receive the ambient audio signal in the meeting while receiving all human voice audio signals. Specifically, the human voice audio signal and the ambient audio signal received by the directional microphone are the first audio signal, and the human voice audio signal and the ambient audio signal received by the omnidirectional microphone are the second audio signal.

Step 102: Determine the sound source direction according to the acquisition parameters of the second audio signal.

Among them, the second audio signal includes a human voice audio signal and an environmental audio signal collected by an omnidirectional microphone, and the collection parameters include the time when the audio signal is received, the speed of sound of the audio signal, the simulated angular frequency, amplitude and phase of the audio signal, etc. Specifically, the sound source direction of the audio signal can be determined by a controllable beamforming method based on maximum output power, a high-resolution spectrogram estimation method and a sound source localization method based on sound time difference. Among them, the controllable beamforming method based on maximum output power is to form a beam by weighted summing of the collected audio signal, guide the beam by searching for the possible position of the sound source, and then modify the weight to maximize the output signal power of the microphone array. The high-resolution spectrum estimation method estimates the direction of the sound source by decomposing the covariance matrix. The high-resolution spectrum estimation method is not limited by the sampling frequency, and can achieve any degree of positioning to a certain extent. The sound source localization method based on sound time difference can locate the direction of the sound source by estimating the time difference of the signal reaching each microphone, and can also determine the sound source position by using the geometric relationship between the sound signal and the microphone. Furthermore, the direction of the sound source is determined by using the above-mentioned sound source localization method according to the acquisition parameters such as the time when the audio signal is received, the sound speed of the audio signal, the analog angular frequency, amplitude and phase of the audio signal.

Step 103: Determine a speech mode according to the direction of the sound source and the strength of the first audio signal.

Among them, the speech mode includes a single-person speech mode and a multi-person speech mode. The single-person speech mode is suitable for scenes where only one person or a specified direction of audio signals need to be collected. For example, in a lecture venue, only the audio signal emitted by the speaker needs to be collected, and in scenes such as theater performances and singing performances, only the audio signal emitted by the actors on the stage needs to be collected. The multi-person speech mode is suitable for scenes where audio signals from multiple people or from multiple directions need to be collected. For example, in a multi-person meeting, the audio signals emitted by each person attending the meeting need to be collected. The speech mode can be determined according to the direction of the sound source and the intensity of the first audio signal. Specifically, when the intensity of the first audio signal collected by the directional microphone reaches a certain level, and the direction of the audio signal is within the direction specified by the directional microphone. It means that in the scene at this time, someone is making a sound in the direction where the directional microphone is set, and the speech mode can be further determined to be a single-person mode. If the intensity of the first audio signal collected by the directional microphone is relatively weak, or the intensity of the first audio signal reaches a certain level but its signal direction does not belong to the direction pointed by the directional microphone, it means that the person speaking is not in the direction where the directional microphone is set, and the speech mode can be further determined to be a multi-person speech mode.

Step 104: perform fusion processing on the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal.

Among them, the fusion processing includes fusing the amplified human voice audio signal and the noise-suppressed environmental audio signal in the first audio signal and the second audio signal to finally obtain the target audio signal. Figure 2 is a flowchart of obtaining the target audio signal provided by an embodiment of the present invention. As shown in Figure 2, after obtaining the first audio signal and the second audio signal, the human voice audio signal and the environmental audio signal in the first audio signal and the second audio signal are extracted, the extracted human voice audio signal is amplified to obtain a human voice enhancement signal, and the extracted environmental audio signal is noise-suppressed to obtain a noise-suppressed audio signal. Then the human voice enhancement signal and the noise-suppressed audio signal are fused to finally obtain the target audio signal.

The technical solution of this embodiment can obtain a first audio signal collected by a directional microphone, and obtain a second audio signal collected by an omnidirectional microphone. The direction of the sound source is determined according to the collection parameters of the second audio signal. The speech mode is determined according to the direction of the sound source and the strength of the first audio signal. The first audio signal and the second audio signal are fused based on the speech mode to obtain a target audio signal. Using the technical solution of this embodiment, audio signals from various directions can be received, and the speech mode can be determined based on the direction of the sound source and the strength of the first audio signal, which can improve the accuracy of determining the speech mode based on the received audio signal. The target audio signal is obtained by fusing the first audio signal and the second audio signal based on the speech mode. The received audio signal can be processed more accurately, the signal-to-noise ratio of the target audio signal can be improved, and the user experience is further improved.

FIG3 is another flow chart of an audio signal processing method provided by an embodiment of the present invention. This embodiment is a refinement based on the above embodiment. The specific method can be shown in FIG3, and the method can include the following steps:

Step 201: Acquire a first audio signal collected by a directional microphone, and acquire a second audio signal collected by an omnidirectional microphone.

Step 202: Determine the sound source direction according to the acquisition parameters of the second audio signal.

The acquisition parameters include the time when the second audio signal is received, the speed of sound of the audio signal, the analog angular frequency, amplitude and phase of the audio signal, etc. The omnidirectional microphone can receive audio signals from any direction, so the second audio signal received by the omnidirectional microphone can determine the sound source direction of the first audio signal.

In this solution, there are at least two omnidirectional microphones, and the sound source direction is determined according to the collection parameters of the second audio signal, including: calculating the sum of the generalized cross-correlation GCC-PHAT functions of the phase transformation weighted by at least two omnidirectional microphones for the collected signals based on the collection parameters of the second audio signal collected by each of the at least two omnidirectional microphones when assuming the sound source direction; and finding the direction in the sound source space that maximizes the controllable response frequency SRP value based on the sum of the GCC-PHAT functions to obtain the sound source direction.

Specifically, _xm (n) represents the received signal of the mth omnidirectional microphone, _rl and _rm represent the rectangular coordinate vector of the lth element and the mth element respectively, c is the sound speed in air (about 342m/s), q is the rectangular coordinate vector of the imaginary sound source, _τlm (q) represents the arrival time difference from the imaginary sound source to the lth and mth microphones, then _τlm (q) is:

Among them, ||qr _m || represents the norm of qr _m , and ||qr _l || represents the norm of qr _l . represents the GCC-PHAT function of the received signals of the lth microphone and the mth microphone, The expression is:

Wherein, _Xm (k) is the fast Fourier transform of _xm (n), _Xl (k) is the fast Fourier transform of _xl (n), “*” indicates taking the conjugate, k is the number of fast Fourier transform points, w is the analog angular frequency, and ejwτ represents the integral coefficient.

Then the SRP-PHAT function based on GCC-PHAT is:

Where q is the rectangular coordinate vector of the imaginary sound source, τ _lm (q) represents the arrival time difference between the imaginary sound source and the lth and mth microphones, is the GCC-PHAT function of the received signals of the lth microphone and the mth microphone. Then the estimation of the sound source position can be expressed as:

Among them, arg represents the complex argument, and Q is the preset sound source space.

Step 203: Determine the speech mode according to the sound source direction and the strength of the first audio signal.

After the sound source direction of the first audio signal is determined, the speech mode is determined according to the sound source direction and the strength of the first audio signal.

In the embodiment of the present solution, optionally, determining the speech mode according to the sound source direction and the strength of the first audio signal includes the following steps A1 to A3:

Step A1: determining whether the intensity of the first audio signal is greater than a preset intensity, and determining whether the direction of the first audio signal belongs to the direction of the sound source.

Among them, the preset strength can be set according to the environment and specific needs of the actual application scenario. The omnidirectional microphone can receive audio signals from any direction. Therefore, the sound source direction of the first audio signal received by the directional microphone can be determined based on the audio signal received by the omnidirectional microphone. Furthermore, after receiving the first audio signal, it is determined whether the strength of the first audio signal is greater than the preset strength, and whether the direction of the first audio signal belongs to the sound source direction based on the second audio signal received by the omnidirectional microphone.

Step A2: If the intensity of the first audio signal is greater than a preset intensity and the direction of the first audio signal belongs to the direction of the sound source, then determining that the speech mode is a single-person speech mode.

Specifically, when the intensity of the first audio signal is greater than the preset intensity and the direction of the first audio signal belongs to the direction of the sound source, it means that the directional microphone in this scenario can receive the audio signal with a strong signal from its specified direction, and the speech mode is determined to be the single-person speech mode. For example, during a speech performance, the directional microphone set on the stage can receive the speaker's voice audio signal.

Step A3: If the intensity of the first audio signal is not greater than the preset intensity, or if the intensity of the first audio signal is greater than the preset intensity but the direction of the first audio signal does not belong to the sound source direction, determine that the speech mode is a multi-person speech mode.

Specifically, if the intensity of the first audio signal is not greater than the preset intensity, it means that the directional microphone cannot receive an audio signal of a certain intensity, and the speech mode is determined to be a multi-person speech mode. If the intensity of the first audio signal is greater than the preset intensity but the direction of the first audio signal does not belong to the sound source direction, it means that the person sending the audio signal is not within the range specified by the directional microphone, and the speech mode is determined to be a multi-person speech mode. For example, the lecture hall in the conference room is equipped with a directional microphone, and the seat is equipped with an omnidirectional microphone. When a discussion meeting is held, users generally discuss in their seats. At this time, even if the directional microphone can receive the audio signal, the direction of the audio signal comes from the seat rather than the lecture hall, and the speech mode is determined to be a multi-person speech mode.

Determining the speech mode according to the above steps can improve the accuracy of determining the speech mode, and further lay the foundation for subsequently determining different audio processing methods according to different speech modes.

Step 204: extract a human voice audio signal from the first audio signal and the second audio signal based on the speech mode.

After receiving the first audio signal and the second audio signal, a human voice audio signal is extracted from the first audio signal and the second audio signal.

In the embodiment of the present solution, optionally, when the speech mode is a multi-person speech mode, the audio signals of all human voices are extracted from the first audio signal and the second audio signal to obtain a human voice audio signal. When the speech mode is a single-person speech mode, the audio signal of human voice is extracted from the first audio signal to obtain a human voice audio signal.

Among them, when the speech mode is determined to be a multi-person speech mode, it is necessary to extract the human voice audio signals received by the directional microphone and the omnidirectional microphone. For example, in a press conference, the podium is provided with a directional microphone, and the press box is provided with an omnidirectional microphone. At this time, both the human voice audio signal of the speaker and the human voice audio signal of the reporter received by the directional microphone need to be extracted. Furthermore, when the speech mode is a multi-person speech mode, the audio signals of all human voices are extracted from the first audio signal and the second audio signal to obtain the human voice audio signal.

Among them, when the speech mode is determined to be a single-person speech mode, it is necessary to extract the human voice audio signal received by the directional microphone. For example, during a speech performance, the directional microphone set on the podium can receive the speaker's human voice audio signal. However, the human voice audio signal received by the omnidirectional microphone set in the auditorium does not need to be extracted as a human voice audio signal. Furthermore, when the speech mode is a single-person speech mode, the human voice audio signal is extracted from the first audio signal to obtain the human voice audio signal.

In the above method description, different human voice audio signals can be simply and accurately extracted from the first audio signal and the second audio signal according to different speech modes, thereby improving the signal-to-noise ratio of the target audio signal obtained subsequently.

Step 205: extract an ambient audio signal from the first audio signal and the second audio signal based on the speech mode.

After the speech mode is determined and the human voice audio signal is extracted according to the speech mode, it is necessary to extract the environmental audio signal from the first audio signal and the second audio signal.

In the embodiment of the present solution, optionally, extracting the ambient audio signal from the first audio signal and the second audio signal includes the following steps B1-B2:

Step B1: Determine the similarity between the first audio signal and the second audio signal and a preset environment signal.

Among them, by pre-training various ambient sound signals based on deep learning, a preset ambient sound signal can be established according to the type of ambient sound. After obtaining the preset ambient sound signal, the first audio signal and the second audio signal are similarly matched with the ambient sound signal to obtain a similarity matching result.

Step B2: extracting a signal whose similarity with the preset environment signal exceeds a preset similarity value from the first audio signal and the second audio signal.

Among them, the preset similarity value can be set according to the actual environment and specific needs, and the preset similarity value can also be set by training the preset environmental sound signal multiple times. Specifically, after obtaining the similarity matching result between the first audio signal and the second audio signal and the preset environmental signal, it is determined whether the similarity extracted from the first audio signal and the second audio signal with the preset environmental signal exceeds the preset similarity value. If it exceeds the preset similarity value, it means that the audio signal is an environmental audio signal. Furthermore, a signal whose similarity with the preset environmental signal exceeds the preset similarity value is extracted from the first audio signal and the second audio signal.

In the above method, various ambient sound signals are pre-trained using a deep learning-based method, and a signal whose similarity with a preset ambient signal exceeds a preset similarity value is extracted from the first audio signal and the second audio signal, which can improve the accuracy of identifying whether an audio signal is an ambient audio signal.

Among them, for different speech modes, the extracted environmental audio signals are different.

In the embodiment of the present solution, optionally, when the speech mode is a single-person speech mode, an ambient audio signal is extracted from the first audio signal and the second audio signal, and an audio signal of a human voice is extracted from the second audio signal to obtain an ambient audio signal. When the speech mode is a multi-person speech mode, an ambient audio signal is extracted from the first audio signal and the second audio signal to obtain an ambient audio signal.

Specifically, when the speech mode is a single-person speech mode, it means that the human voice audio signal in the second audio signal is not needed as a human voice audio signal. For example, during a speech performance, the directional microphone installed on the podium can receive the speaker's human voice audio signal. The human voice audio signal received by the omnidirectional microphone installed in the auditorium can be regarded as environmental noise. Therefore, when the speech mode is a single-person speech mode, the environmental audio signal is extracted from the first audio signal and the second audio signal, and the human voice audio signal is extracted from the second audio signal to obtain the environmental audio signal.

Specifically, when the speech mode is a multi-speech mode, the human voice audio signal in the second audio signal needs to be extracted as a human voice audio signal. Therefore, when the speech mode is a multi-speech mode, the ambient audio signal is extracted from the first audio signal and the second audio signal to obtain the ambient audio signal.

In the above method description, different environmental audio signals can be simply and accurately extracted from the first audio signal and the second audio signal according to different speech modes. In the single-person speech mode, the human voice signal in the second audio signal is called the environmental audio signal, which can improve the signal-to-noise ratio of the target audio signal obtained subsequently.

Step 206: amplify the human voice audio signal to obtain a human voice enhanced signal.

After obtaining the human voice audio signal, the human voice audio signal is amplified according to a preset amplification ratio to obtain a human voice enhanced signal. Specifically, the human voice audio signals extracted in the single-speaking mode and the multi-speaking mode can be amplified according to the same amplification ratio or different amplification ratios. The amplification ratio can be set according to the actual environment and specific needs.

Step 207: Determine the sound wave amplitude value of the ambient audio signal.

The sound wave amplitude value can be understood as the sound amplitude of the audio signal, and the sound wave amplitude value can reflect the range and intensity of the sound vibration. Specifically, after determining the ambient audio signal of the first audio signal and the second audio signal, the sound wave amplitude value of the ambient audio signal is determined.

Step 208: Filter out signals whose sound wave amplitude values exceed a preset sound wave amplitude value from the ambient audio signal to obtain a noise-suppressed audio signal; or use a cancellation signal to cancel the signal value of a designated signal in the ambient audio signal to no more than the preset sound wave amplitude value to obtain a noise-suppressed audio signal, where the designated signal is a signal whose sound wave amplitude value exceeds the preset signal value in the ambient audio signal.

Among them, the preset sound wave amplitude value can be set according to the actual environment, specific needs and professional experience. If there is a signal in the ambient audio signal whose sound wave amplitude value exceeds the preset sound wave amplitude value, it means that the signal value can be regarded as a noise signal, and the signal in the ambient audio signal whose sound wave amplitude value exceeds the preset sound wave amplitude value is filtered out. Alternatively, a specified signal is determined in the ambient sound signal, and a signal with a sound wave amplitude value similar to the specified signal but opposite in polarity is simulated, and the signal value is used as a cancellation signal. The cancellation signal is used to offset the specified signal value in the ambient audio signal to no more than the preset sound wave amplitude value, so as to obtain a noise-suppressed audio signal. Figure 4 is a flow chart of signal extraction provided by the implementation of the present invention. As shown in Figure 4, when the speech mode is determined to be a multi-person speech mode, the audio signals of all human voices are extracted from the first audio signal and the second audio signal as human voice audio signals, and the human voice signals are amplified to obtain human voice enhancement signals. An ambient audio signal is extracted from the first audio signal and the second audio signal, and it is determined whether the similarity between the ambient audio signal and the preset ambient sound signal exceeds the pre-emitted similarity value. If so, the ambient audio signal is filtered out or offset, and finally a noise-suppressed audio signal is obtained. When the speech mode is determined to be a single-person speech mode, an audio signal of a human voice is extracted from the first audio signal, an ambient audio signal is extracted from the first audio signal and the second audio signal, and an audio signal of a human voice is extracted from the second audio signal as the ambient audio signal, and it is determined whether the similarity between the ambient audio signal and the preset ambient sound signal exceeds the pre-emitted similarity value. If so, the ambient audio signal is filtered out or offset, and finally a noise-suppressed audio signal is obtained.

Step 209: fusing the human voice enhancement signal and the noise suppression audio signal to obtain a target audio signal.

After obtaining the human voice enhancement signal and the noise suppression audio signal, the human voice enhancement signal and the noise suppression audio signal are fused. By fusion processing the human voice enhancement signal and the noise suppression audio signal, a clearer human voice audio signal can be obtained while ensuring that the target audio signal has good transparency.

In an embodiment of the present invention, a first audio signal collected by a directional microphone is obtained, and a second audio signal collected by an omnidirectional microphone is obtained. The sum of the generalized cross-correlation GCC-PHAT functions of the phase transformation weighted by at least two omnidirectional microphones for the collected signals is calculated based on the collection parameters of the second audio signal collected by each of the at least two omnidirectional microphones in the assumed sound source direction. Based on the sum of the GCC-PHAT functions, the direction that maximizes the controllable response frequency SRP value is searched in the sound source space to obtain the sound source direction. The speech mode is determined according to the sound source direction and the intensity of the first audio signal. A human voice audio signal is extracted from the first audio signal and the second audio signal based on the speech mode. An ambient audio signal is extracted from the first audio signal and the second audio signal based on the speech mode. The human voice audio signal is amplified to obtain a human voice enhancement signal. The sound wave amplitude value of the ambient audio signal is determined. Signals whose sound wave amplitude values exceed the preset sound wave amplitude value are filtered out from the ambient audio signal to obtain a noise-suppressed audio signal; or the signal value of a designated signal in the ambient audio signal is offset to not exceed the preset sound wave amplitude value by using a cancellation signal to obtain a noise-suppressed audio signal, wherein the designated signal is a signal whose sound wave amplitude value exceeds the preset signal value in the ambient audio signal. The human voice enhancement signal and the noise-suppressed audio signal are fused to obtain a target audio signal. In the technical solution of this embodiment, the speech mode can be accurately determined according to the first audio signal and the second audio signal, and different processing methods can be used for the first audio signal and the second audio signal according to different speech modes, which can effectively improve the signal-to-noise ratio of the target audio signal. By fusing the human voice enhancement signal and the noise-suppressed audio signal, a clearer human voice audio signal can be obtained, while ensuring that the target audio signal has good transparency, thereby improving the user experience.

FIG5 is a structural diagram of an audio signal processing device provided by an embodiment of the present invention, and the device is suitable for executing the audio signal processing method provided by an embodiment of the present invention, and the device includes a directional microphone and an omnidirectional microphone. As shown in FIG5 , the device may specifically include:

A signal acquisition module 501 is used to acquire a first audio signal collected by the directional microphone and to acquire a second audio signal collected by the omnidirectional microphone;

A direction determination module 502, configured to determine a sound source direction according to acquisition parameters of the second audio signal;

A mode determination module 503, configured to determine a speech mode according to the sound source direction and the strength of the first audio signal;

The target signal acquisition module 504 is configured to perform fusion processing on the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal.

Optionally, the mode determination module 503 is specifically configured to:

determining whether the intensity of the first audio signal is greater than a preset intensity, and determining whether the direction of the first audio signal belongs to the sound source direction;

If the intensity of the first audio signal is greater than the preset intensity and the direction of the first audio signal belongs to the sound source direction, determining that the speech mode is a single-person speech mode;

If the intensity of the first audio signal is not greater than the preset intensity, or if the intensity of the first audio signal is greater than the preset intensity but the direction of the first audio signal does not belong to the sound source direction, the speech mode is determined to be a multi-speech mode.

Optionally, the target signal acquisition module 504 specifically includes:

a signal extraction unit, configured to extract a human voice audio signal from the first audio signal and the second audio signal based on the speech mode, and to extract an environmental audio signal from the first audio signal and the second audio signal based on the speech mode;

A signal amplifying unit, used to amplify the human voice audio signal to obtain a human voice enhancement signal;

A signal noise suppression unit, used for performing noise suppression processing on the ambient audio signal to obtain a noise-suppressed audio signal;

The target signal acquisition unit is used to fuse the human voice enhancement signal and the noise suppression audio signal to obtain the target audio signal.

Optionally, the signal noise suppression unit is specifically used for:

Determining the sound wave amplitude value of the ambient audio signal;

The signal whose sound wave amplitude value exceeds the preset sound wave amplitude value is filtered out from the ambient audio signal to obtain the noise-suppressed audio signal; or the signal value of a designated signal in the ambient audio signal is offset to not exceed the preset sound wave amplitude value by using a cancellation signal to obtain the noise-suppressed audio signal, wherein the designated signal is a signal in the ambient audio signal whose sound wave amplitude value exceeds the preset signal value.

Optionally, the signal extraction unit is specifically used for:

When the speech mode is a multi-person speech mode, extracting audio signals of all human voices from the first audio signal and the second audio signal to obtain the human voice audio signal;

When the speech mode is a single-person speech mode, an audio signal of a human voice is extracted from the first audio signal to obtain the human voice audio signal.

Optionally, the signal extraction unit is further used for:

When the speech mode is a multi-person speech mode, extracting an ambient audio signal from the first audio signal and the second audio signal to obtain the ambient audio signal;

When the speech mode is a single-person speech mode, an ambient audio signal is extracted from the first audio signal and the second audio signal, and an audio signal of a human voice is extracted from the second audio signal to obtain the ambient audio signal.

Optionally, the signal extraction unit is further used for:

Determining similarities between the first audio signal and the second audio signal and a preset environment signal;

A signal whose similarity to the preset environment signal exceeds a preset similarity value is extracted from the first audio signal and the second audio signal.

Optionally, the direction determination module 502 is specifically configured to:

Calculate the sum of the weighted generalized cross-correlation GCC-PHAT functions of the phase transformation of the collected signals by the at least two omnidirectional microphones based on the collection parameters of the second audio signal collected by each of the at least two omnidirectional microphones under the assumption of the sound source direction;

Based on the sum of the GCC-PHAT functions, the direction that maximizes the controllable response frequency SRP value is found in the sound source space to obtain the sound source direction.

The audio signal processing device provided in the embodiment of the present invention can execute the audio signal processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, reference can be made to the description in any method embodiment of the present invention.

FIG6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Referring to FIG6 , a schematic diagram of the structure of a computer system 12 suitable for implementing an electronic device of an embodiment of the present invention is shown. The electronic device shown in FIG6 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present invention. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and/or cache memory 32. The electronic device 12 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG. 6 , commonly referred to as a “hard drive”). Although not shown in FIG. 6 , a disk drive for reading and writing to a removable non-volatile disk (e.g., a “floppy disk”) and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data media interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

A program/utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and/or methods of the embodiments described herein.

The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 12, and/or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input/output (I/O) interface 22. In addition, the electronic device 12 in this embodiment, the display 24 does not exist as an independent individual, but is embedded in the mirror surface, and when the display surface of the display 24 is not displayed, the display surface of the display 24 and the mirror surface are visually integrated. In addition, the electronic device 12 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and/or public network, such as the Internet) through a network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

The processing unit 16 executes various functional applications and audio signal processing by running the program stored in the system memory 28, for example, implementing an audio signal processing method provided in an embodiment of the present invention: obtaining a first audio signal collected by a directional microphone, and obtaining a second audio signal collected by an omnidirectional microphone; determining the direction of the sound source according to the collection parameters of the second audio signal; determining a speech mode according to the direction of the sound source and the intensity of the first audio signal; and fusing the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal.

The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, an audio signal processing method as provided in all the embodiments of the present invention is implemented: obtaining a first audio signal collected by a directional microphone, and obtaining a second audio signal collected by an omnidirectional microphone; determining the direction of the sound source according to the collection parameters of the second audio signal; determining the speech mode according to the direction of the sound source and the intensity of the first audio signal; fusing the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal. Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, computer readable storage media may be any tangible media that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (9)

1. An audio signal processing method, characterized in that it is applied to an audio device including a directional microphone and an omnidirectional microphone, the method comprising: Acquire a first audio signal collected by the directional microphone, and acquire a second audio signal collected by the omnidirectional microphone; determining a sound source direction according to an acquisition parameter of the second audio signal; Determining a speech mode according to the direction of the sound source and the strength of the first audio signal; determining the speech mode according to the direction of the sound source and the strength of the first audio signal includes: determining whether the strength of the first audio signal is greater than a preset strength, and determining whether the direction of the first audio signal belongs to the direction of the sound source; if the strength of the first audio signal is greater than the preset strength, and the direction of the first audio signal belongs to the direction of the sound source, determining that the speech mode is a single-person speech mode; if the strength of the first audio signal is not greater than the preset strength, or if the strength of the first audio signal is greater than the preset strength but the direction of the first audio signal does not belong to the direction of the sound source, determining that the speech mode is a multi-person speech mode; The first audio signal and the second audio signal are fused based on the speech mode to obtain a target audio signal. 2. The audio signal processing method according to claim 1, wherein the fusing the first audio signal and the second audio signal based on the speech mode to obtain the target audio signal comprises: extracting a human voice audio signal from the first audio signal and the second audio signal based on the speech pattern, and extracting an environmental audio signal from the first audio signal and the second audio signal based on the speech pattern; Amplifying the human voice audio signal to obtain a human voice enhancement signal; Performing noise suppression processing on the ambient audio signal to obtain a noise-suppressed audio signal; The human voice enhancement signal and the noise suppression audio signal are fused to obtain the target audio signal. 3. The audio signal processing method according to claim 2, wherein the step of performing noise suppression on the ambient audio signal to obtain the noise suppressed audio signal comprises: Determining the sound wave amplitude value of the ambient audio signal; Filtering out signals whose sound wave amplitude values exceed a preset sound wave amplitude value from the ambient audio signal to obtain the noise-suppressed audio signal; or The cancellation signal is used to cancel the signal value of the designated signal in the ambient audio signal to not exceed the preset sound wave amplitude value, thereby obtaining the noise suppression audio signal. The designated signal is a signal in the ambient audio signal whose sound wave amplitude value exceeds the preset signal value. 4. The audio signal processing method according to claim 2, wherein extracting a human voice audio signal from the first audio signal and the second audio signal based on the speech mode comprises: When the speech mode is a multi-speech mode, extracting audio signals of all human voices from the first audio signal and the second audio signal to obtain the human voice audio signal; When the speech mode is a single-person speech mode, an audio signal of a human voice is extracted from the first audio signal to obtain the human voice audio signal. 5. The audio signal processing method according to claim 2, wherein extracting the ambient audio signal from the first audio signal and the second audio signal based on the speech mode comprises: When the speech mode is a multi-person speech mode, extracting an ambient audio signal from the first audio signal and the second audio signal to obtain the ambient audio signal; When the speech mode is a single-person speech mode, an ambient audio signal is extracted from the first audio signal and the second audio signal, and an audio signal of a human voice is extracted from the second audio signal to obtain the ambient audio signal. 6. The audio signal processing method according to claim 5, wherein extracting the ambient audio signal from the first audio signal and the second audio signal comprises: Determining similarities between the first audio signal and the second audio signal and a preset environment signal; A signal whose similarity to the preset environment signal exceeds a preset similarity value is extracted from the first audio signal and the second audio signal. 7. An audio signal processing device, characterized in that it is applied to an audio device including a directional microphone and an omnidirectional microphone, and the audio signal processing device comprises: A signal acquisition module, used to acquire a first audio signal collected by the directional microphone and to acquire a second audio signal collected by the omnidirectional microphone; A direction determination module, used to determine the direction of the sound source according to the acquisition parameters of the second audio signal; A mode determination module, configured to determine a speech mode according to the direction of the sound source and the strength of the first audio signal; the step of determining the speech mode according to the direction of the sound source and the strength of the first audio signal comprises: determining whether the intensity of the first audio signal is greater than a preset intensity, and determining whether the direction of the first audio signal belongs to the direction of the sound source; if the intensity of the first audio signal is greater than the preset intensity, and the direction of the first audio signal belongs to the direction of the sound source, determining that the speech mode is a single-person speech mode; if the intensity of the first audio signal is not greater than the preset intensity, or if the intensity of the first audio signal is greater than the preset intensity but the direction of the first audio signal does not belong to the direction of the sound source, determining that the speech mode is a multi-person speech mode; The target signal acquisition module is used to perform fusion processing on the first audio signal and the second audio signal based on the speech mode to obtain a target audio signal. 8. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the audio signal processing method according to any one of claims 1 to 6 when executing the program. 9. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the audio signal processing method according to any one of claims 1 to 6 is implemented.