CN118841022A - Audio processing method, processing system, medium and program product - Google Patents

Abstract

The application relates to the technical field of audio processing, in particular to an audio processing method, a processing system, a medium and a program product, wherein the method comprises the steps of acquiring a propagation environment parameter, and determining a target array microphone configuration parameter corresponding to the propagation environment based on the propagation environment parameter, wherein the target array microphone configuration parameter comprises the number, the type and the distance; acquiring propagation environment audio corresponding to a propagation environment based on the configuration parameters of the target array microphone; identifying audio features contained in the propagation environment audio, and determining master audio and slave audio from the propagation environment audio based on the audio features, wherein the audio features comprise frequency, time domain and variation amplitude; determining a secondary audio signal of the secondary audio based on the primary audio, the audio features and a preset mathematical model; noise reduction processing is carried out on transmission environment audio based on the slave audio signals, noise reduction audio is obtained, and audio transmission is carried out according to the noise reduction audio. The application is convenient for improving the accuracy of the audio processing result.

Description

Audio processing method, processing system, medium and program product

Technical Field

The present application relates to the field of audio processing technology, and in particular to an audio processing method, processing system, medium and program product.

Background Art

During the process of sound propagation, sound may encounter obstacles such as walls, ceilings, and floors, which may produce reverberation echoes. These reverberation echoes may interfere with the original audio, making the original audio blurred and unclear. In addition, there may be a lot of ambient noise in the on-site environment where sound is propagated, such as air conditioning sounds, conversation sounds, walking sounds, etc. The ambient noise may also cover up the original audio, causing the original audio to be unable to be clearly propagated.

In the related art, array microphone technology is generally used to suppress reverberation echoes and environmental noise encountered during sound propagation in order to improve the quality of audio propagation. However, when there are multiple sound sources and noise sources in the propagation environment, using only array microphone technology to process the propagated audio may affect the accuracy of the audio processing results due to the inability to effectively distinguish and process multiple different sound sources and noise sources, thereby affecting the quality of audio propagation.

Summary of the invention

In order to improve the accuracy of audio processing results and thus improve the quality of audio transmission, the present application provides an audio processing method, a processing system, a medium and a program product.

In a first aspect, the present application provides an audio processing method, which adopts the following technical solution:

An audio processing method, comprising:

Acquire propagation environment parameters, and determine target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters, wherein the propagation environment parameters include site size, site layout, and site material, and the target array microphone configuration parameters include quantity, type, and spacing;

Based on the target array microphone configuration parameters, collecting propagation environment audio corresponding to the propagation environment;

Identify audio features contained in the propagation environment audio, and determine the main audio and the slave audio from the propagation environment audio based on the audio features, wherein the audio features include frequency, time domain, and variation amplitude;

Determine a slave audio signal corresponding to the slave audio based on the master audio, the audio feature and a preset mathematical model;

The propagation environment audio is subjected to noise reduction processing based on the secondary audio signal to obtain noise-reduced audio, and audio propagation is performed based on the noise-reduced audio.

By adopting the above technical scheme, the audio collection range can be improved by adopting microphone configuration parameters corresponding to the propagation environment for audio collection, rather than adopting fixed configuration parameters for audio collection, so as to provide high-quality original audio data for subsequent audio processing. In addition, the main audio and the slave audio in the propagation environment audio are distinguished by audio features, that is, the priority in the propagation environment audio is distinguished, so as to provide priority propagation resources for the main audio. In addition, after the slave audio is processed by a preset mathematical model to obtain a slave audio signal, the propagation environment audio is denoised based on the slave audio signal, so as to improve the audio noise reduction processing efficiency and accuracy, thereby improving the audio propagation quality.

In a possible implementation, determining the target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters includes:

Determining a corresponding target site model based on the propagation environment parameters and a preset standard site model;

Determine a plurality of initial array microphone configurations according to the propagation environment parameters, and perform audio propagation simulation based on each initial array microphone configuration and the target site model to obtain initial simulation parameters corresponding to each initial array microphone configuration, wherein each initial simulation parameter includes a simulated echo parameter and a simulated reverberation parameter;

The preset propagation standard parameters are matched with each initial simulation parameter to obtain a simulation score corresponding to each initial simulation parameter, and the initial simulation parameter with the highest simulation score is determined as the target array microphone configuration parameter corresponding to the propagation environment.

By adopting the above technical scheme, the corresponding target array microphone configuration is determined according to the propagation environment parameters, so as to improve the compatibility between the array microphone configuration and the propagation environment, thereby facilitating the improvement of the audio coverage in the propagation environment. In addition, the target array microphone configuration corresponding to the propagation environment is jointly determined through model simulation and preset propagation standard parameters, so as to improve the speed and accuracy of determining the target array microphone configuration.

In a possible implementation, determining the slave audio signal corresponding to the slave audio based on the master audio, the audio feature, and a preset mathematical model includes:

Identify a main audio feature corresponding to the main audio from the audio feature, and determine relevant slave audio and irrelevant slave audio from the slave audio based on the main audio feature and the audio feature;

A noise adjustment quantization parameter is obtained, and a slave audio signal corresponding to the slave audio is determined according to the relevant slave audio, the unrelated slave audio, the noise adjustment parameter, and a preset mathematical model.

By adopting the above technical solution, since different noises correspond to different noise reduction processing methods, the slave audio is further subdivided according to the main audio characteristics, so as to improve the efficiency and accuracy of the noise reduction process, and then determine the final slave audio signal based on the noise adjustment quantization parameter and the preset mathematical calculation model, so as to reduce or eliminate the noise components in a targeted manner and improve the accuracy of determining the slave audio signal.

In a possible implementation, when it is detected that the main audio has at least two main audio features, the method further includes:

Acquire sound wave recording information, and determine the sound source position corresponding to each main audio feature based on the sound wave recording information, wherein the sound wave recording information includes the arrival time of the main audio at each microphone in the array microphone;

Acquire the sound source environment audio generated within the corresponding range of each sound source position, and determine the sound source secondary audio from the corresponding sound source environment audio based on each main audio feature;

Identify the noise information of each sound source from the audio, and determine whether the noise information corresponding to the audio of all sound sources is the same by comparing the noise information, the noise information includes the noise type and the noise loudness;

If so, the same noise reduction method is used to perform noise reduction processing on the sound source environment audio corresponding to the at least two main audio features.

By adopting the above technical solution, the sound source position corresponding to each main audio is accurately located by the arrival time of the main audio at each microphone. In addition, by comparing whether the noise information at the sound source positions corresponding to different main audios is consistent, it is convenient to judge whether the main audios corresponding to different sound source positions can use the same noise reduction processing operation, instead of performing complicated noise reduction processing determination operations for each main audio, so as to improve the noise reduction processing rate.

In a possible implementation, the identifying the noise information of each sound source from the audio, and determining whether the noise information corresponding to all the sound source from the audio is the same by comparing the noise information, includes:

Identify the frequency distribution and peak value of each sound source from the audio in the spectrum diagram, and determine a first noise display diagram corresponding to each sound source from the audio based on the frequency distribution and peak value corresponding to each sound source from the audio;

Identify the audio loudness of each sound source slave audio, and determine a second noise display map corresponding to each sound source slave audio based on a loudness mapping relationship, wherein the loudness mapping relationship is a correspondence between the audio loudness and the second noise display map;

Superimpose the first noise display image and the second noise display image of each sound source from the audio to obtain a noise superposition display image of each sound source from the audio;

Perform image matching on the noise superposition display map corresponding to the audio of each sound source, and determine whether the noise information corresponding to the audio of all sound sources is the same based on the image matching results. When the matching value between the noise superposition display maps corresponding to the audio of all sound sources is not lower than the preset matching value, it indicates that the noise information corresponding to the audio of all sound sources is the same.

By adopting the above technical solution, by visually displaying the noise information corresponding to each sound source from the audio in the form of an image, it is convenient for relevant staff to intuitively view the characteristics and similarities between each sound source from the audio. In addition, since the noise superposition display diagram corresponding to the sound source from the audio is generally determined by extracting key features, when judging whether the noise information corresponding to multiple sound sources from the audio is similar based on image matching, it is convenient to ignore secondary or redundant data, thereby facilitating improving the accuracy of the judgment result.

In a possible implementation, when there is a person of interest in the communication environment, the method further includes:

Acquiring the concerned position and hearing loss parameters of the concerned person;

Determine a sound source interval and a sound source direction based on the focus position and the sound source position corresponding to the main audio, and determine a loudness adjustment parameter corresponding to the focus position based on the sound source interval and the sound source direction and the loudness mapping relationship;

The noise reduction audio is optimized based on the loudness adjustment parameter and the hearing loss parameter to obtain an optimized noise reduction audio, and the optimized noise reduction audio is fed back to the hearing aid device worn by the concerned person.

By adopting the above technical solution, when the broadcast environment includes persons of interest, the noise reduction audio is optimized in a timely manner according to the location and hearing loss of the persons of interest, so as to determine the audio output suitable for the hearing condition of the persons of interest. Through personalized adjustment, the audio experience of the persons of interest is improved, and the robustness of the audio processing process is also improved.

In a second aspect, the present application provides a processing system, which adopts the following technical solution:

A processing system, the processing system comprising:

at least one processor;

Memory;

At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the above audio processing method.

In a third aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

A computer-readable storage medium includes: a computer program that can be loaded by a processor and execute the above audio processing method.

In a fourth aspect, the present application provides a computer program product, which adopts the following technical solution:

A computer program product comprises a computer program, wherein the computer program implements the above-mentioned audio processing method when executed by a processor.

In summary, the present application includes at least one of the following beneficial technical effects:

By adopting microphone configuration parameters corresponding to the propagation environment for audio collection, instead of adopting fixed configuration parameters for audio collection, it is convenient to improve the audio collection range, so as to provide high-quality original audio data for subsequent audio processing. In addition, the main audio and the slave audio in the propagation environment audio are distinguished by audio characteristics, that is, the priority in the propagation environment audio is distinguished, so as to provide priority propagation resources for the main audio. In addition, after the slave audio is processed by a preset mathematical model to obtain the slave audio signal, the propagation environment audio is denoised based on the slave audio signal, so as to improve the audio noise reduction processing efficiency and accuracy, thereby improving the audio propagation quality.

By knowing the arrival time of the main audio at each microphone, it is convenient to accurately locate the sound source position corresponding to each main audio. In addition, by comparing whether the noise information at the sound source positions corresponding to different main audios is consistent, it is convenient to judge whether the main audios corresponding to different sound source positions can use the same noise reduction processing operation, instead of performing complicated noise reduction processing determination operations for each main audio, thereby facilitating the improvement of the noise reduction processing rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of an audio processing method in an embodiment of the present application;

FIG2 is a flow chart of another audio processing method in an embodiment of the present application;

FIG. 3 is a schematic diagram of the structure of a processing system in an embodiment of the present application.

DETAILED DESCRIPTION

The present application is further described in detail below in conjunction with Figures 1-3.

After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

It should be noted that in the optional embodiments of the present application, the object information and other related data involved, when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

Specifically, the embodiment of the present application provides an audio processing method, which is executed by a processing system, and the processing system can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and the embodiment of the present application does not limit this.

Referring to FIG. 1 , FIG. 1 is a schematic flow chart of an audio processing method in an embodiment of the present application, the method comprising steps S110 to S150, wherein:

Step S110: Acquire propagation environment parameters, and determine target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters, the propagation environment parameters include venue size, venue layout, venue material, and the target array microphone configuration parameters include quantity, type, and spacing.

Specifically, the propagation environment is a venue where audio processing is required. For example, the propagation environment can be an interactive classroom, an auditorium, a lecture hall, etc. The specific propagation environment is not specifically limited in the embodiments of this application. The propagation environment parameters include but are not limited to the venue size, venue layout, and venue materials. The propagation environment parameters may affect the arrangement of the array microphone. Therefore, the propagation environment parameters need to be analyzed during the audio processing. The more propagation environment parameters there are, the better the effect of audio processing based on the propagation environment parameters. Different propagation environments correspond to different propagation environment parameters, and the propagation environment parameters can be uploaded by relevant staff.

An array microphone consists of multiple microphones. The array microphone can collect audio at different locations. Due to different requirements for audio propagation in different propagation environments, for example, in shopping malls, airports and other environments, the requirements for noise reduction are high, and a 6-microphone configuration is often used. In homes, offices and other environments, the requirements for noise reduction are relatively low, and a dual-microphone or 4-microphone configuration is often used. The array microphone configuration parameters include not only the number, but also the microphone type, the spacing between microphones, and the direction in which the microphones are set. Among them, the microphone type can include unidirectional microphones, bidirectional microphones, and cardioid microphones. The characteristic of a unidirectional microphone is that it can only receive sound from a specified direction; the characteristic of a bidirectional microphone is that it can receive sound from the front and rear, which is suitable for meetings, interviews, etc.; the characteristic of a cardioid microphone is that it can receive sound from the front and both sides, which is suitable for recording studios, concerts, etc. The target array microphone configuration may include different types of microphones, that is, in the target array microphone configuration, suitable microphone types are usually selected for combination according to actual needs. In the target array microphone configuration, the larger the spacing between different microphones, the greater the gain that the target array microphone can achieve.

When determining the corresponding target array microphone configuration parameters according to the propagation environment parameters, the determination can be made according to the parameter mapping relationship uploaded to the processing system in advance by the relevant staff, wherein the parameter mapping relationship includes the target array microphone configuration parameters corresponding to different propagation environment parameters, and the parameter mapping relationship can be determined by the relevant staff according to the experimental data. Further, in order to facilitate improving the speed and accuracy of determining the target array microphone configuration, determining the target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters can specifically include:

Based on the propagation environment parameters and the preset standard site model, the corresponding target site model is determined; multiple initial array microphone configurations are determined according to the propagation environment parameters, and audio propagation simulation is performed based on each initial array microphone configuration and the target site model to obtain initial simulation parameters corresponding to each initial array microphone configuration, each initial simulation parameter includes a simulated echo parameter and a simulated reverberation parameter; the preset propagation standard parameters are matched with each initial simulation parameter to obtain a simulation score corresponding to each initial simulation parameter, and the initial simulation parameter with the highest simulation score is determined as the target array microphone configuration parameter corresponding to the propagation environment.

Among them, the preset standard site model is a basic model corresponding to the propagation environment, and the corresponding target site model is determined based on the propagation environment parameters and the preset standard site model, that is, the basic model is optimized according to the propagation environment parameters to optimize the basic model to the target site model corresponding to the propagation environment parameters, and the model parameters of the target site model are consistent with the propagation environment parameters. There is a corresponding relationship between the propagation environment parameters and the initial array microphone configuration, and the initial array microphone configuration is an array microphone configuration that can be used in the propagation environment. The corresponding relationship between the propagation environment parameters and the initial array microphone configuration can be determined by relevant staff based on experimental data or historical operating experience and then uploaded to the processing system. The specific content of the corresponding relationship is not specifically limited in the embodiment of the present application. The initial array microphone configuration can be used in the propagation environment. Different initial array microphone configurations produce different audio propagation effects in the propagation environment. Therefore, after determining the corresponding multiple initial array microphone configurations according to the propagation environment parameters, it is also necessary to perform audio propagation simulation according to each initial array microphone configuration, and then determine the target array microphone configuration that is compatible with the propagation environment from the initial array microphone configuration according to the simulation results.

Since echo parameters and reverberation parameters will affect the audio propagation quality during the audio processing process, when determining the target array microphone configuration from the initial array microphone configuration, it is necessary to analyze the initial simulation parameters generated by the audio propagation simulation of each initial array microphone configuration, wherein the initial simulation parameters include simulated echo parameters and simulated reverberation parameters. The simulated echo parameters are the sounds reflected back after the simulated sound waves encounter obstacles during the audio propagation simulation process. The simulated echo parameters usually include echo attenuation rate and threshold value, etc. The echo attenuation rate can determine the ratio of the echo audio to the original audio, and the threshold value can determine the maximum signal amplitude for the start of dynamic reverberation. Reverberation can characterize the reflection phenomenon of simulated sound in the target venue model. Reverberation parameters may include reverberation duration, reverberation volume, and reverberation decay rate. A reverberation duration that is too long may cause the audio to sound too persistent, that is, it may blur the clarity of the original audio; a reverberation volume that is too high may cause the audio to be too noisy during propagation, while a reverberation volume that is too low may cause the audio to lack layers during propagation, that is, it may cause distortion of the original audio; the reverberation decay rate is the ratio of high-frequency attenuation to low-frequency attenuation. Adjusting the ratio parameter of high-frequency attenuation to low-frequency attenuation facilitates balancing the timbre and brightness of the original audio.

The initial simulation parameters include but are not limited to simulation echo parameters and simulation reverberation parameters. The preset propagation standard parameters also include at least preset standard simulation echo parameters and preset standard simulation reverberation parameters. By matching the preset propagation standard parameters with each initial simulation parameter, a simulation score corresponding to each initial simulation parameter is obtained. The higher the simulation score, the closer the corresponding initial simulation parameter is to the preset standard simulation echo parameter, that is, the more the corresponding initial simulation parameter meets the preset standard. The initial simulation parameter with the highest simulation score is determined as the target initial simulation parameter, and the initial array microphone configuration parameter corresponding to the target initial simulation parameter is determined as the target array microphone configuration parameter.

Step S120: Based on the target array microphone configuration parameters, the propagation environment audio corresponding to the propagation environment is collected.

Specifically, after determining the target array microphone configuration parameters, microphone control instructions can be generated according to the target array microphone configuration parameters to prompt relevant staff to adjust and set the microphones in the propagation environment based on the microphone control instructions, so that the propagation environment can collect audio based on the target array microphone configuration parameters.

Step S130: Identify audio features contained in the propagation environment audio, and determine the main audio and the slave audio from the propagation environment audio based on the audio features, where the audio features include frequency, time domain, and variation amplitude.

Specifically, the propagation environment audio is the audio generated in the propagation environment, including the main audio and the slave audio, wherein the slave audio can also be called noise audio, and the main audio can be the audio generated by the teacher or the lecturer. Since the main audio generally has a high degree of recognition, the main audio and the slave audio in the propagation environment audio can be distinguished by identifying the audio features in the propagation environment audio, wherein the audio features can be frequency, time domain, and change amplitude, etc. Regarding frequency features: the main audio usually has a specific frequency in different time periods, such as a unique tone or sound quality, while the frequency distribution of the slave audio may be wider and irregular, such as traffic noise, crowd noise, etc.; Regarding time domain features: the main audio generally has continuity and recognizable speech syllables, while the slave audio may be continuous or intermittent, and usually does not have recognizable speech syllables; Regarding change amplitude: the change amplitude corresponding to the main audio is generally larger, including from soft voice to loud voice, while the change amplitude of the slave audio is generally smaller, and the volume is relatively stable. Therefore, by identifying the audio features contained in the propagation environment audio, the propagation environment audio can be divided, wherein the slave audio is the audio that is not the main audio in the propagation environment audio, but the noise sources corresponding to the slave audio may be different, for example, air conditioning noise, vehicle traffic noise, conversation noise, etc.

Step S140: Determine a slave audio signal corresponding to the slave audio based on the master audio, the audio feature and the preset mathematical model.

Specifically, the preset mathematical model may be a mathematical model for suppressing the slave audio. The slave audio may be determined from the propagation environment audio based on the audio features. By converting the slave audio into a slave audio signal, it is convenient to provide a noise reduction basis for subsequent noise reduction processing. When converting the slave audio into a slave audio signal, in order to improve the conversion accuracy and the noise reduction processing effect, the slave audio may be further divided. Based on the main audio, the audio features and the preset mathematical model, the slave audio signal corresponding to the slave audio is determined, which may specifically include:

Identify main audio features corresponding to the main audio from the audio features, and determine relevant slave audio and irrelevant slave audio from the slave audio based on the main audio features and the audio features; obtain noise adjustment quantization parameters, and determine the slave audio signal corresponding to the slave audio based on the relevant slave audio, the irrelevant slave audio, the noise adjustment parameters and a preset mathematical calculation model.

Specifically, the slave audio includes related slave audio and unrelated slave audio, wherein the related slave audio is the slave audio related to the main audio. For example, when the main audio is the audio emitted by the speaker using a microphone, the related slave audio can be the noise of the microphone itself, or the breathing sound of the speaker, and the unrelated slave audio can be the background noise unrelated to the main audio, such as the fan sound, traffic sound and other random noise. In order to improve the accuracy of determining the slave audio signal.

The preset mathematical model can be n(t)=αPi(t)+Ni(t), where α is used to characterize the noise adjustment quantization parameter; Pi(t) follows Poisson distribution and is used to characterize the signal corresponding to the relevant slave audio; Ni(t) follows Gaussian distribution and is used to characterize the signal corresponding to the unrelated slave audio; and n(t) is used to characterize the slave audio signal. The noise adjustment quantization parameter can be input by relevant staff. The larger the noise adjustment quantization parameter, the greater the weight of the relevant slave audio in the slave audio. Conversely, the smaller the noise adjustment quantization parameter, the smaller the weight of the relevant slave audio in the slave audio. Finally, the relevant slave audio and the unrelated slave audio are weighted and summed according to the obtained noise adjustment quantization parameter through the preset mathematical model to obtain the final slave audio signal.

When converting the related slave audio to an audio signal that follows the Poisson distribution, the parameters of the Poisson distribution can be determined first, which are used to characterize the reference value of the audio amplitude per unit time. It is also necessary to determine the expected number of occurrences in which the audio amplitude reaches the reference value within a fixed time period. Based on the reference value and the expected number of occurrences, the related slave audio signal that follows the Poisson distribution can be obtained, wherein the reference value includes but is not limited to the audio amplitude. When converting the unrelated slave audio to an audio signal that follows the Gaussian distribution, the expected value and standard deviation set by the relevant staff can be obtained, wherein the expected value is the center point of the Gaussian distribution, which is used to characterize the average amplitude of the audio signal, and the standard deviation is the degree of discreteness of the Gaussian distribution, which is used to characterize the volume fluctuation of the audio signal. After determining the parameters of the Gaussian distribution, a random number generator can be used to determine a Gaussian distribution random sample that meets the expected value and standard deviation. By converting the generated random sample into a time series, the random sample is the amplitude value at each time point in the unrelated slave audio, and the signal corresponding to the unrelated slave audio is determined based on the random sample.

The specific method of converting the relevant slave audio into an audio signal that follows the Poisson distribution is not specifically limited in the embodiments of the present application, and the specific method of converting the unrelated slave audio into an audio signal that follows the Gaussian distribution is not specifically limited in the embodiments of the present application, as long as the signal conversion can be performed. Since the relevant slave audio is usually related to the main audio signal, if the relevant slave audio is caused by a series of independent triggering events, and these independent triggering events are evenly distributed in time, then it is more appropriate to summon the relevant slave audio as an audio signal that follows the Poisson distribution. Since the Gaussian distribution can well describe the probability distribution of random variables, it is more reasonable to convert the unrelated slave audio into an audio signal that follows the Gaussian distribution. Since different noises correspond to different noise reduction processing methods, the slave audio is further subdivided by the main audio features to facilitate the improvement of efficiency and accuracy in the noise reduction process, and then the final slave audio signal is determined based on the noise adjustment quantization parameter and the preset mathematical calculation model, so as to facilitate the targeted reduction or elimination of noise components and improve the accuracy of determining the slave audio signal.

Step S150: performing noise reduction processing on the propagation environment audio based on the audio signal to obtain noise-reduced audio, and performing audio propagation based on the noise-reduced audio.

Specifically, when performing noise reduction processing on the propagation environment audio based on the slave audio signal, the signal features contained in the slave audio signal can be first identified. The signal features can include signal period, signal phase, signal spectrum, etc., and targeted noise reduction processing is performed on the propagation environment audio based on the signal features to suppress the slave audio in the propagation environment audio, thereby facilitating amplification or enhancement of the main audio. The noise reduction algorithm used in the process of performing targeted noise reduction processing based on signal features can be a deep learning algorithm, wherein the signal features and the propagation environment audio can be imported into a trained noise reduction model to obtain noise-reduced audio. The noise reduction model is obtained after training with a large number of noise reduction samples, and the noise reduction samples include noise audio and noise audio features.

For the embodiments of the present application, the audio collection range can be improved by using microphone configuration parameters corresponding to the propagation environment for audio collection, rather than using fixed configuration parameters for audio collection, so as to provide high-quality original audio data for subsequent audio processing. In addition, the main audio and the slave audio in the propagation environment audio are distinguished by audio features, that is, the priority in the propagation environment audio is distinguished, so as to provide priority propagation resources for the main audio. In addition, after the slave audio is processed by a preset mathematical model to obtain a slave audio signal, the propagation environment audio is denoised based on the slave audio signal, so as to improve the audio noise reduction processing efficiency and accuracy, thereby improving the audio propagation quality.

Further, when it is detected that the main audio has at least two main audio features, the method provided in the embodiment of the present application further includes steps S1 to S4, as shown in FIG2 , wherein:

Step S1: Acquire sound wave recording information, and determine the sound source position corresponding to each main audio feature based on the sound wave recording information, wherein the sound wave recording information includes the arrival time of the main audio at each microphone in the array microphone.

Specifically, when the communication environment is an interactive classroom, an interview room or other places, there will be at least two main audio features in the communication environment audio. When there is a discussion or debate in the interactive classroom, more than two main audio features may appear. When at least two main audio features appear in the communication environment audio, the sound source positions corresponding to the two main audio features may be different. By analyzing the sound wave recording information, it is easy to determine the arrival time of each main audio feature at the microphone. Based on the arrival time corresponding to each main audio feature, it is easy to determine the sound source position corresponding to each main audio feature, that is, the sound source position corresponding to each main audio.

For any main audio, the specific steps for determining the corresponding sound source position based on the sound wave recording information can be: calculating the time difference between the same main audio arriving at different microphones, and then using the target array microphone configuration parameters, sound wave propagation speed, and the time difference between the same main audio arriving at different microphones to estimate the sound source position corresponding to the main audio. The estimation method can adopt a sound source localization algorithm based on arrival time difference. The specific estimation method is not specifically limited in the embodiments of the present application, as long as the sound source position corresponding to the main audio can be determined. The above steps can be used to obtain the sound source position corresponding to each main audio.

Step S2: Acquire the sound source environment audio generated within the corresponding range of each sound source position, and determine the sound source slave audio from the corresponding sound source environment audio based on each main audio feature.

Specifically, the range corresponding to each sound source position is an area with the sound source position as the center and a preset distance as the radius. After determining the area corresponding to each sound source position, the sound source environment audio within the corresponding range is obtained according to the microphone set in the area. The specific preset distance is not specifically limited in the embodiment of the present application and can be set by relevant technical personnel. The specific method of determining the sound source secondary audio from the corresponding sound source environment audio based on the main audio feature can refer to the method disclosed in step S130, which will not be repeated here.

Step S3: Identify the noise information of each sound source from the audio, and determine whether the noise information corresponding to all the sound source from the audio is the same by comparing the noise information. The noise information includes the noise type and the noise loudness.

Specifically, when the sound source positions are different, the noise information contained in the corresponding sound source slave audio may be the same or different. For example, when the propagation environment is an interactive classroom, the sound source position corresponding to the main audio a is the podium, and the sound source position corresponding to the main audio b is the back of the classroom, and an air conditioner is installed at the back of the classroom. Therefore, the sound source slave audio corresponding to the main audio b may have a louder air conditioning noise than the sound source slave audio corresponding to the main audio a.

Noise information includes but is not limited to noise type and noise loudness, wherein noise type includes low-frequency noise, medium-frequency noise and high-frequency noise, wherein low-frequency noise is noise with a main frequency lower than 300Hz, medium-frequency noise is noise with a main frequency between 300-800Hz, and high-frequency noise is noise with a main frequency higher than 800Hz. Whether the noise type and noise loudness are consistent can be used to determine whether the audio from different sound sources is consistent. In addition to noise type and noise loudness, time domain features of the audio from different sound sources can also be compared and analyzed. Furthermore, in order to facilitate improving the accuracy of the judgment result, the noise information of each sound source from the audio is identified, and by comparing the noise information, it is determined whether the noise information corresponding to all sound sources from the audio is the same. Specifically, it may include:

Identify the frequency distribution and peak value of each sound source from the audio in the spectrum diagram, and determine the first noise display diagram corresponding to each sound source from the audio based on the frequency distribution and peak value corresponding to each sound source from the audio.

Specifically, first, spectrum analysis can be performed on each sound source from the audio to obtain the spectrum diagram corresponding to each sound source from the audio, and the frequency distribution and peak value of the corresponding sound source from the audio can be determined from each spectrum diagram. The first noise display diagram is a three-dimensional display diagram, which is composed of a basic distribution diagram and a peak superposition, wherein the basic distribution diagram contains multiple scattered points, and the dispersion degree diagrams of the scattered points corresponding to different frequency distributions are different. When determining the dispersion degree diagram corresponding to the frequency distribution, it can be determined through a dispersion mapping relationship, wherein the dispersion mapping relationship contains dispersion degree diagrams of scattered points corresponding to different frequency distributions. After determining the dispersion degree diagram, the peak value parameters corresponding to the peak value can be superimposed at the preset position in the dispersion degree diagram to obtain the first noise display diagram. In the superposition process, the peak value can be first converted into a peak value parameter, so as to facilitate the comparison of the first noise display diagrams corresponding to different sound source audios, wherein the corresponding conversion ratio and preset position when converting the peak value into the peak value parameter can be uploaded to the processing system by relevant staff, and the specific content is not limited in the embodiments of the present application.

Identify the audio loudness of each sound source from the audio, and determine the second noise display map corresponding to each sound source from the audio based on the loudness mapping relationship, where the loudness mapping relationship is the corresponding relationship between the audio loudness and the second noise display map,

Specifically, different audio loudnesses correspond to different second noise display graphs. The louder the audio loudness, the longer the graphic edge length of the corresponding second noise display graph. The second noise display graph is a two-dimensional display graph, and different audio loudnesses are described by the image edge length, that is, different audio loudnesses are characterized by the area of the second noise display graph, which is convenient for relevant staff to intuitively view the differences between different sound sources from audio. For example, the second noise display graph corresponding to the sound source from audio a is a circle with a graphic edge length of 4 cm, and the second noise display graph corresponding to the sound source from audio b is a circle with a graphic edge length of 7 cm. For ease of comparison, the basic graphics corresponding to different second noise display graphs remain unchanged, and there is only a difference between the graphic edge lengths. The loudness mapping relationship includes second noise display graphs corresponding to different audio loudnesses. The specific content is not specifically limited in the embodiments of the present application. As long as each sound source can be determined according to the loudness mapping relationship,

The first noise display image and the second noise display image of each sound source from the audio are superimposed to obtain a noise superposition display image of each sound source from the audio.

Specifically, when superimposing the first noise display map and the second noise display map of each sound source from the audio, it can be done by pixel level superposition, position superposition, transparency superposition, etc. The specific superposition method is not specifically limited in the embodiment of the present application, as long as image superposition can be achieved. Among them, when the pixel level superposition method is adopted, the first noise display map and the second noise display map can be mathematically operated at the pixel level to obtain a noise superposition display map.

Perform image matching on the noise superposition display map corresponding to the audio of each sound source, and determine whether the noise information corresponding to the audio of all sound sources is the same based on the image matching results. When the matching value between the noise superposition display maps corresponding to the audio of all sound sources is not lower than the preset matching value, it indicates that the noise information corresponding to the audio of all sound sources is the same.

Specifically, when performing image matching on two noise superposition display images, the edge information of the two noise superposition display images can be determined first, and then the matching value between the two noise superposition display images can be determined by comparing the edge information of the two noise superposition display images. When the matching value between the two noise superposition display images is not lower than the preset matching value, it indicates that the noise information of the audio from the corresponding sound source of the two noise superposition display images is consistent. Among them, the edge information of each noise superposition display image can be determined according to the Canny edge detection algorithm, the gradient operator algorithm, etc., and the specific method is not specifically limited in the embodiment of the present application. The preset matching value can be 98% or 95%, and the specific value can be set by relevant staff according to actual needs. Using the above method, the matching value between any two noise superposition display images is obtained. When the matching values corresponding to any two noise superposition display images are not lower than the preset matching value, it indicates that all noise superposition display images are consistent, that is, the noise information of the audio from the corresponding sound source of all noise superposition display images is consistent.

Since the noise superposition display diagram corresponding to the sound source from the audio is generally determined by extracting key features, when judging whether the noise information corresponding to multiple sound sources from the audio is similar based on image matching, it is easy to ignore minor or redundant data, thereby facilitating improving the accuracy of the judgment result.

Step S4: If yes, the same noise reduction method is used to perform noise reduction processing on the sound source environment audio corresponding to at least two main audio features.

Specifically, if it is determined that the noise superposition display diagrams of the sound source environment audio corresponding to at least two main audio features are consistent, then the at least two sound source environment audios can be processed by noise reduction in the same manner, wherein the manner of performing noise reduction processing on the sound source environment audio corresponding to any main audio feature can refer to step S110-step S150 in the above embodiment, which will not be described in detail here. By comparing whether the noise information at the sound source positions corresponding to different main audios is consistent, it is convenient to judge whether the main audios corresponding to different sound source positions can be processed by the same noise reduction processing operation, rather than performing complicated noise reduction processing determination operations for each main audio, so as to improve the noise reduction processing rate.

Furthermore, when there is a person of interest in the communication environment, the method provided in the embodiment of the present application further includes:

The attention position and hearing loss parameters of the concerned person are obtained; the sound source interval and sound source direction are determined based on the attention position and the sound source position corresponding to the main audio, and the loudness adjustment parameters corresponding to the attention position are determined based on the sound source interval and sound source direction and the loudness mapping relationship; the noise reduction audio is optimized based on the loudness adjustment parameters and the hearing loss parameters to obtain the optimized noise reduction audio, and the optimized noise reduction audio is fed back to the hearing aid device worn by the concerned person.

Specifically, the person of concern may be a hearing-impaired person, and the position of concern may be the position of the person of concern in the communication environment. When the communication environment is monitored to contain the person of concern, the position of concern of the person of concern may be determined by identifying the person from the communication environment image, wherein the communication environment image may be collected by an image acquisition device set in the communication environment and uploaded to the processing system. The hearing loss parameter is used to characterize the degree of hearing loss of the person of concern, and may be obtained after obtaining authorization from the person of concern.

The sound source interval and sound source direction between the person of interest and the sound source position corresponding to the main audio may affect the loudness of the main audio received by the person of interest. Therefore, it is necessary to determine the loudness adjustment parameter corresponding to the position of interest based on the loudness mapping relationship. The longer the sound source interval and the larger the angle corresponding to the sound source direction, the higher the corresponding loudness adjustment parameter value. The specific content of the loudness mapping relationship is not specifically limited in the embodiment of the present application, and can be determined by relevant staff based on historical experimental data and uploaded to the processing system, as long as the loudness adjustment parameter value corresponding to the position of interest can be determined based on the loudness mapping relationship.

The loss adjustment loudness is determined according to the hearing loss parameters of the hearing-impaired person. For high-frequency hearing loss, the loudness of the high-frequency component may need to be increased; for low-frequency hearing loss, the loudness of the low-frequency component may need to be increased. Different hearing-impaired people have different corresponding loss adjustment loudness. After the loss adjustment loudness is determined, the noise reduction audio is optimized according to the loss adjustment loudness and the loudness adjustment parameters. The optimized noise reduction audio is convenient for adapting to the hearing characteristics and preferences of the corresponding hearing-impaired person. Among them, hearing aids may include hearing aids, cochlear implants, etc. Personalized adjustments can improve the audio experience of the concerned person and also improve the robustness of the audio processing process.

A processing system is provided in an embodiment of the present application, as shown in FIG3 , and the processing system 300 shown in FIG3 includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the processing system 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the processing system 300 does not constitute a limitation on the embodiment of the present application.

The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG. 3 only uses one line to represent, but does not mean that there is only one bus or one type of bus.

The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

The processing system includes, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. The processing system shown in FIG3 is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method in any of the above embodiments is implemented.

It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims (9)

1. An audio processing method, comprising: Acquire propagation environment parameters, and determine target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters, wherein the propagation environment parameters include site size, site layout, and site material, and the target array microphone configuration parameters include quantity, type, and spacing; Based on the target array microphone configuration parameters, collecting propagation environment audio corresponding to the propagation environment; Identify audio features contained in the propagation environment audio, and determine the main audio and the slave audio from the propagation environment audio based on the audio features, wherein the audio features include frequency, time domain, and variation amplitude; Determine a slave audio signal corresponding to the slave audio based on the master audio, the audio feature and a preset mathematical model; The propagation environment audio is subjected to noise reduction processing based on the secondary audio signal to obtain noise-reduced audio, and audio propagation is performed based on the noise-reduced audio. 2. The audio processing method according to claim 1, wherein determining the target array microphone configuration parameters corresponding to the propagation environment based on the propagation environment parameters comprises: Determining a corresponding target site model based on the propagation environment parameters and a preset standard site model; Determine a plurality of initial array microphone configurations according to the propagation environment parameters, and perform audio propagation simulations based on each initial array microphone configuration and the target site model to obtain initial simulation parameters corresponding to each initial array microphone configuration, wherein each initial simulation parameter includes a simulated echo parameter and a simulated reverberation parameter; The preset propagation standard parameters are matched with each initial simulation parameter to obtain a simulation score corresponding to each initial simulation parameter, and the initial simulation parameter with the highest simulation score is determined as the target array microphone configuration parameter corresponding to the propagation environment. 3. The audio processing method according to claim 1, wherein determining the slave audio signal corresponding to the slave audio based on the master audio, the audio feature and a preset mathematical model comprises: Identify a main audio feature corresponding to the main audio from the audio feature, and determine a relevant slave audio and a non-relevant slave audio from the slave audio based on the main audio feature and the audio feature; A noise adjustment quantization parameter is obtained, and a slave audio signal corresponding to the slave audio is determined according to the relevant slave audio, the unrelated slave audio, the noise adjustment parameter, and a preset mathematical model. 4. The audio processing method according to claim 1, characterized in that when it is detected that the main audio has at least two main audio features, it further comprises: Acquire sound wave recording information, and determine the sound source position corresponding to each main audio feature based on the sound wave recording information, wherein the sound wave recording information includes the arrival time of the main audio at each microphone in the array microphone; Acquire the sound source environment audio generated within the corresponding range of each sound source position, and determine the sound source secondary audio from the corresponding sound source environment audio based on each main audio feature; Identify the noise information of each sound source from the audio, and determine whether the noise information corresponding to the audio of all sound sources is the same by comparing the noise information, the noise information includes the noise type and the noise loudness; If so, the same noise reduction method is used to perform noise reduction processing on the sound source environment audio corresponding to the at least two main audio features. 5. The audio processing method according to claim 4, characterized in that the step of identifying the noise information of each sound source from the audio, and determining whether the noise information corresponding to all the sound source from the audio is the same by comparing the noise information, comprises: Identify the frequency distribution and peak value of each sound source from the audio in the spectrum diagram, and determine a first noise display diagram corresponding to each sound source from the audio based on the frequency distribution and peak value corresponding to each sound source from the audio; Identify the audio loudness of each sound source slave audio, and determine a second noise display map corresponding to each sound source slave audio based on a loudness mapping relationship, wherein the loudness mapping relationship is a correspondence between the audio loudness and the second noise display map; Superimpose the first noise display image and the second noise display image of each sound source from the audio to obtain a noise superposition display image of each sound source from the audio; Perform image matching on the noise superposition display map corresponding to the audio of each sound source, and determine whether the noise information corresponding to the audio of all sound sources is the same based on the image matching results. When the matching value between the noise superposition display maps corresponding to the audio of all sound sources is not lower than the preset matching value, it indicates that the noise information corresponding to the audio of all sound sources is the same. 6. The audio processing method according to claim 1, characterized in that when there is a person of interest in the communication environment, it also includes: Acquiring the concerned position and hearing loss parameters of the concerned person; Determine a sound source interval and a sound source direction based on the focus position and the sound source position corresponding to the main audio, and determine a loudness adjustment parameter corresponding to the focus position based on the sound source interval, the sound source direction and the loudness mapping relationship; The noise reduction audio is optimized based on the loudness adjustment parameter and the hearing loss parameter to obtain an optimized noise reduction audio, and the optimized noise reduction audio is fed back to the hearing aid device worn by the person of interest. 7. A processing system, characterized in that the processing system comprises: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute an audio processing method according to any one of claims 1-6. 8. A computer-readable storage medium, characterized in that it comprises: a computer program that can be loaded by a processor and execute an audio processing method according to any one of claims 1 to 6. 9. A computer program product, characterized in that it comprises a computer program, and when the computer program is executed by a processor, the steps of an audio processing method according to any one of claims 1 to 6 are implemented.