CN118692498A - Multi-channel audio and video signal processing method and device - Google Patents

Abstract

The invention provides a multichannel-based audio and video signal processing method and device, wherein the method comprises the following steps: calculating the audio signal energy and voice activity detection information of all input channels, and determining a plurality of target channels according to the calculation result of the audio signal energy and the calculation result of voice activity detection; and under the condition that the target channels meet the number requirement, the target channels carry out audio enhancement, carry out audio attenuation on the other channels, mix audio of all input channels and output the audio. According to the method and the device, aiming at the problem that the sound is noisy in a microphone-on scene of a plurality of people during audio-video conversation, audio-video teaching or audio-video conference, through reasonable audio channel selection and audio enhancement, a user can acquire needed information more easily, the hearing feeling of the user is improved, the online communication is smoother, and the time cost of the user is saved.

Description

Multi-channel audio and video signal processing method and device

Technical Field

The present invention relates to the technical field of audio and video processing, and in particular to a multi-channel-based audio and video signal processing method and device.

Background Art

In multi-person audio and video calls, audio and video teaching, or audio and video conferences, a common scenario is multi-person online interaction. However, when there are dozens of audio channels with microphones turned on at the same time, the sound heard is very messy, and the various types of equipment and different speaking environments further aggravate the noise. In actual application scenarios, at the same time, you often only need to pay attention to the speech content of a few people, and the information of most other users can be ignored, or users of multiple channels take turns to speak on stage.

When multiple audios exist simultaneously, how to select and enhance the multi-channel audio signals to highlight the target audio being spoken, weaken the interference of other audios, and ensure the reception quality of the target audio has become an urgent problem to be solved by technical personnel in this field.

Summary of the invention

The present invention provides a multi-channel-based audio and video signal processing method and device, which can significantly improve the user's auditory experience through the selection of target audio channels and audio enhancement, making the user's online communication more concise and smooth, and saving the user's communication costs.

The present invention provides a multi-channel audio and video signal processing method, the method comprising:

Calculate the energy and voice activity detection information of all input channel audio signals;

Determining a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;

When the target channel meets the quantity requirement, the target channel is audio enhanced, and the remaining channels are audio attenuated;

Mix the audio from all input channels and output them.

In some embodiments, calculating the audio signal energy of all input channels specifically includes:

Calculate the maximum energy of each audio sampling point in a single frame of the current channel;

Calculate the average value of the maximum energy of all frames within the preset duration of the current channel;

Perform quantization and shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;

Each input channel is used as the current channel in turn to obtain the audio signal energy of all input channels.

In some embodiments, calculating voice activity detection information of all input channels specifically includes:

Resample the audio signal of the current channel to a preset sampling frequency, divide it into multiple frequency bands according to the Mel-frequency cepstrum, and calculate the energy information in each frequency band;

The Gaussian mixture model is used to calculate the Gaussian probability density function of speech and noise in each sub-band respectively;

Calculate the log-likelihood ratio of speech and noise, and sum the log-likelihood ratios of all sub-bands;

Based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands, the speech activity detection of the current channel is calibrated;

Each input channel is used as the current channel in turn to obtain voice activity detection information of all input channels.

In some embodiments, determining multiple target channels according to the calculation results of the audio signal energy and the voice activity specifically includes:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.

In some embodiments, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, specifically including:

When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;

When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;

The number of candidate channels is the difference between the target channel and the preset number.

In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;

When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.

The present invention also provides a multi-channel audio and video signal processing device, the device comprising:

A parameter calculation unit, used to calculate the audio signal energy and voice activity detection information of all input channels;

A channel selection unit, configured to determine a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;

An audio processing unit, configured to perform audio enhancement on the target channel and audio attenuation on the remaining channels when the target channel meets the quantity requirement;

The result output unit is used to mix and output the audio of all input channels.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described above when executing the program.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described above when executed by a processor.

The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method described above is implemented.

The multi-channel audio and video signal processing method and device provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This method and device aims at the problem of noisy sound in the scenario where multiple people turn on the microphone at the same time during multi-person audio and video calls, audio and video teaching, or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves users' time costs.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a multi-channel audio and video signal processing method provided by the present invention;

FIG2 is a second flow chart of the multi-channel audio and video signal processing method provided by the present invention;

FIG3 is a third flow chart of the multi-channel audio and video signal processing method provided by the present invention;

FIG4 is a fourth flow chart of the multi-channel audio and video signal processing method provided by the present invention;

FIG5 is a fifth flow chart of the multi-channel audio and video signal processing method provided by the present invention;

FIG6 is a sixth flowchart of the multi-channel audio and video signal processing method provided by the present invention;

FIG7 is a diagram showing the effects of audio enhancement and compression processing;

FIG8 is a diagram showing the effect of compression after mixing;

9 is a block diagram of a multi-channel audio and video signal processing device provided by the present invention;

FIG. 10 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

For ease of understanding, FIG1 shows an online application scenario of the multi-channel audio and video signal processing method provided by the present invention. In this specific application scenario, the video window area of the client interface displays the multiple audio and video channels (i.e., all channels) received by the current user, and the on-stage user area displays the audio and video channels (i.e., the target channels) currently preparing to speak on stage. In this embodiment, the number of windows in the on-stage user area ranges from The purpose of the present invention is to select 6 channels from all video window channels for audio enhancement, and to perform audio attenuation on the remaining video window channels, so as to better obtain the information that needs attention.

The overall technical solution of the multi-channel audio and video signal processing method provided by the present invention is shown in Figure 2. First, the channels that need audio decoding are selected for the multi-channel input audio signals according to the energy information and the voice activity detection information, and then the channels that need audio enhancement are selected, and then the audio enhancement and audio attenuation are performed on the selected multi-channel audio channels, and finally the audio of all channels is mixed and output.

In a specific implementation, as shown in FIG3 , the multi-channel audio and video signal processing method provided by the present invention includes the following steps:

S310: Calculate audio signal energy and voice activity detection information of all input channels;

S320: determining a plurality of target channels according to the calculation result of the audio signal energy and the calculation result of the voice activity detection;

S330: When the target channel meets the quantity requirement, the target channel is audio enhanced, and the remaining channels are audio attenuated;

S340: Mix and output the audio of all input channels.

In step S310, the audio signal energy of all input channels is calculated, as shown in FIG4 , which specifically includes the following steps:

S410: Calculate the maximum energy of each audio sampling point in a single frame of the current channel;

S420: Calculate the average value of the maximum energy of all frames within the preset time length of the current channel;

S430: performing quantization shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;

S440: Taking each input channel as the current channel in turn to obtain audio signal energy of all input channels.

Specifically, in a specific usage scenario, when calculating the energy information of the audio signals of all channels, the purpose of calculating the audio energy is to report the audio energy information to the background, and then forward it to the receiving client by the background, so as to guide the routing strategy of the receiving end. The reporting of audio energy cannot be too frequent, otherwise the amount of data is too large, which will cause great pressure on the background. In this embodiment, the reporting period interval is selected to be reported once every 2 seconds, the period interval of the volume statistics is also 2 seconds, and the frame length of the audio frame is 20 milliseconds, so there are a total of 100 frames of audio data in 2 seconds.

In this embodiment, the method involves a method for calculating volume, which specifically includes the following steps:

First, calculate the maximum energy of the audio sampling points in a frame:

In the formula, is the maximum value of the current audio frame sampling point, is the audio sampling point, the value range is , The length of the audio frame.

Then, we average the maximum values of the audio data frames within 2 seconds:

In the formula, is the average value of the maximum value of the audio data frame within 2 seconds, It is the maximum value of the sampling point of the current audio frame.

Finally, calculate the volume:

In the formula, The maximum volume within 2 seconds after quantization and shaping, with a value range of , It is the average value of the maximum value of the audio data frame within 2 seconds.

In step S310, voice activity detection information of all input channels is calculated, as shown in FIG5 , which specifically includes the following steps:

S510: resampling the audio signal of the current channel to a preset sampling frequency, dividing it into multiple frequency bands according to the Mel-frequency cepstrum, and calculating the energy information in each frequency band;

S520: using a Gaussian mixture model to calculate the Gaussian probability density function of the speech and noise of each sub-band;

S530: Calculate the log likelihood ratio of speech and noise, and sum the log likelihood ratios of all sub-bands;

S540: calibrating the speech activity of the current channel based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands;

S550: Taking each input channel as the current channel in turn to obtain voice activity detection information of all input channels.

Still taking the above specific usage scenario as an example, voice activity detection is performed on the audio signals of all channels. The purpose of voice activity detection (VAD) is to distinguish between voice and non-voice. During audio transmission, VAD can guide the transmission of audio data. When the signal is judged to be voice, it is sent. When the signal is judged to be non-voice, it is stopped. This can save a lot of bandwidth and traffic.

This embodiment uses a statistical model method to perform voice activity detection, which specifically includes the following steps:

First, the audio signal is resampled to a sampling frequency of 8 kHz (Hertz), and the cutoff frequency corresponding to the signal with a sampling frequency of 8 kHz is 4 kHz. Then, the signal with a sampling frequency of 8 kHz is divided into multiple frequency bands according to the Mel cepstrum frequency, and the energy information in each frequency band is calculated. In this embodiment, the logarithmic energy is taken, and the calculation method is as follows:

In the formula, is the subband marker, is the length mark of the subband, represents the length of the subband, represents the number of subbands, represents the logarithmic energy of the subband, Represents the sampling points within the subband.

Then, the Gaussian mixture model is used to calculate the Gaussian probability density functions of speech and noise in the N subbands respectively. The Gaussian probability density function of speech is calculated as follows:

In the formula, represents the Gaussian probability density function of speech, Indicates the number of subbands, represents the order of Gaussian mixture model, represents the logarithmic energy of the subband, , , They respectively represent the mean, variance, and mixing weight of the Gaussian mixture model of speech.

The Gaussian probability density function of noise is calculated as follows:

In the formula, represents the Gaussian probability density function of the noise, Indicates the number of subbands, represents the order of Gaussian mixture model, represents the logarithmic energy of the subband, , , They represent the mean, variance, and mixing weight of the Gaussian mixture model of noise respectively.

Then the log-likelihood ratio of speech and noise is calculated and summed over all subbands:

In the formula, represents the sum of log-likelihood ratios of all subbands, Indicates the number of subbands, represents the Gaussian probability density function of speech, Gaussian probability density function representing the noise.

According to the result of VAD determined by the threshold, speech is marked as 1 and non-speech is marked as 0, as shown below:

In the formula, represents the result of VAD, represents the sum of log-likelihood ratios of all subbands, is the VAD decision threshold.

Finally, update the mean, variance, and mixing weight of the Gaussian mixture model for the next calculation.

In step S320, multiple target channels are determined according to the calculation results of the audio signal energy and the calculation results of the voice activity detection, specifically including:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.

Specifically, in the selection of decoding channels, the audio codec of this embodiment adopts the opus codec. Audio decoding is performed on the client side, which consumes a lot of performance in the whole system. The system is a multi-channel system. If dozens of audio encoding signals are to be decoded at the same time, the equipment is not enough to support such a large performance pressure. And if there are too many decoding channels, it will be very time-consuming and affect the processing of subsequent modules, so selective decoding is required.

In order to take into account the performance of various low-end devices, it is found through experiments and calculations that decoding 20 audio encoding signals at the same time can basically cover various low-end devices on the market.

The principle of decoding channel selection is:

Select a channel with speech according to the VAD information in step S310;

All channels are sorted according to the energy information in step S310, and a decoding channel is selected from the highest energy to the lowest energy.

In step S330, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, which specifically includes:

When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;

When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;

The number of candidate channels is the difference between the target channel and the preset number.

In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;

When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.

When calculating the signal-to-noise ratio of the audio signals of all channels, the signal-to-noise ratio information of the audio signals is the information outputted in the noise reduction module at the encoding end, which reflects the ratio of the signal strength to the background noise strength. This embodiment uses this information to guide the routing strategy of the receiving end, and the calculation method is as follows:

In the formula, represents the ratio of signal to noise, represents the power of the signal, Represents the power of the noise.

The calculation method is as follows:

In the formula, Represents the sampling point value of the signal, Indicates the frame length of the audio frame.

The calculation method is as follows:

In the formula, represents the power of the noise, represents the sampling point value of the noise, Indicates the frame length of the audio frame.

In the selection of enhanced channels, first of all, the definition of enhanced channels and non-enhanced channels is that according to the needs of the product, the sound of some channels needs to be amplified, while the sound of other channels needs to be reduced, so as to better focus on the audio information of the required channels. There are as many as dozens of channels with microphones turned on at the same time, which requires certain principles to select which channels' audio needs to be enhanced and which channels' audio needs to be attenuated.

The process of audio enhancement channel selection is shown in Figure 6:

First, check whether the number of users on the stage is more than 6 from the N audio channels. If the number of users on the stage is more than 6, the enhanced channels need to be selected from the users on the stage; if the number of users on the stage is less than 6, then the users on the stage are selected for enhancement first, and then the users who have not come on stage are selected, so that the total number of channels selected to be enhanced is 6;

The principle of selection is to first select the channel with VAD of 1 (i.e., with speech), and then select from large to small according to the energy information sorting; if the energy difference is not large, then select according to the signal-to-noise ratio, and give priority to the channel with a high signal-to-noise ratio;

Then there is the issue of switching to enhanced channel selection. If other users want to speak, they can also switch to the enhanced channel. The principle of switching is that after entering the enhanced queue, the channel will not be reselected within 2 seconds; the channel that has not spoken for more than 4 seconds enters the selection queue and waits for re-selection of the enhanced channel.

Finally, the channel enhancement strategy of the channel enhancement and mixing modules is configured to the client through the server background. The parameters of the enhancement configuration are: 1dB, 3dB, 5dB. The parameters of the attenuation configuration are: -3dB, -5dB, -10dB. The gain of all audio channels that need to be attenuated is attenuated, and the gain of all audio channels that need to be enhanced is increased. When increasing the gain, it should be noted that the increase in gain may cause the volume value to exceed the maximum limit of 0dB, so this method performs compression processing on the part above -10dB after the gain to prevent popping, and the part below -10dB is fixedly increased. As shown in Figure 4, the horizontal axis is the energy value of the audio signal before compression processing, and the vertical axis is the energy value of the audio signal after compression processing. The volume value after multi-channel mixing may also exceed the maximum limit of 0dB, so this method performs compression processing on the part above -10dB after mixing, while the gain of the part below -10dB remains unchanged, as shown in Figure 5, the horizontal axis is the energy value of the audio signal before compression processing, and the vertical axis is the energy value of the audio signal after compression processing.

In the above specific implementation, the multi-channel audio and video signal processing method provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This method aims at the problem of noisy sound when multiple people turn on their microphones at the same time during multi-person audio and video calls, audio and video teaching, or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves users' time costs.

In addition to the above method, the present invention also provides a multi-channel audio and video signal processing device, as shown in FIG9 , the device comprises:

A parameter calculation unit 910, configured to calculate audio signal energy and voice activity detection information of all input channels;

A channel selection unit 920, configured to determine a plurality of target channels according to a calculation result of audio signal energy and a calculation result of voice activity detection;

An audio processing unit 930 is configured to perform audio enhancement on the target channel and audio attenuation on the remaining channels when the target channel meets the quantity requirement;

The result output unit 940 is used to mix and output the audio of all input channels.

In some embodiments, calculating the audio signal energy of all input channels specifically includes:

Calculate the maximum energy of each audio sampling point in a single frame of the current channel;

Calculate the average value of the maximum energy of all frames within the preset duration of the current channel;

Perform quantization and shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;

Each input channel is used as the current channel in turn to obtain the audio signal energy of all input channels.

In some embodiments, calculating voice activity detection information of all input channels specifically includes:

Resample the audio signal of the current channel to a preset sampling frequency, divide it into multiple frequency bands according to the Mel-frequency cepstrum, and calculate the energy information in each frequency band;

The Gaussian mixture model is used to calculate the Gaussian probability density function of speech and noise in each sub-band respectively;

Calculate the log-likelihood ratio of speech and noise, and sum the log-likelihood ratios of all sub-bands;

Based on the relationship between a preset threshold and the sum of log-likelihood ratios of all sub-bands, the speech activity of the current channel is calibrated;

Each input channel is used as the current channel in turn to obtain voice activity detection information of all input channels.

In some embodiments, determining multiple target channels according to the calculation results of the audio signal energy and the calculation results of the voice activity detection specifically includes:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and a preset number of channels with larger audio signal energy are selected as the target channels.

In some embodiments, when the target channel meets the quantity requirement, the target channel is audio enhanced and the remaining channels are audio attenuated, specifically including:

When the number of the target channels is greater than or equal to a preset number, performing audio enhancement on all the target channels;

When the number of the target channels is less than the preset number, selecting candidate channels from the remaining channels as supplementary target channels, and performing audio enhancement on all the target channels;

The number of candidate channels is the difference between the target channel and the preset number.

In some embodiments, selecting a candidate channel from the remaining channels as a supplementary target channel specifically includes:

According to the calculation result of the voice activity detection, a channel marked as having voice is selected as a candidate channel;

According to the calculation result of the audio signal energy, all candidate channels are sorted according to the audio signal energy, and at least one channel with a larger audio signal energy is selected as the supplementary target channel;

When the audio signal energy values of the candidate channels are the same, the channel with a higher signal-to-noise ratio is selected as the supplementary target channel.

In the above specific implementation, the multi-channel audio and video signal processing device provided by the present invention calculates the audio signal energy and voice activity detection information of all input channels, and determines multiple target channels according to the calculation results of the audio signal energy and the voice activity detection; when the target channels meet the quantity requirements, the target channels perform audio enhancement, the remaining channels perform audio attenuation, and the audio of all input channels is mixed and output. This device aims at the problem of noisy sound in the scenario where multiple people turn on the microphone at the same time during multi-person audio and video calls, audio and video teaching or audio and video conferences. Through reasonable audio channel selection and audio enhancement, it makes it easier for users to obtain the information they need, improves the user's auditory experience, makes online communication smoother, and saves users' time costs.

FIG10 illustrates a schematic diagram of a physical structure of an electronic device. As shown in FIG10 , the electronic device may include: a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communications interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the above method.

In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the above method.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A multichannel-based audio/video signal processing method, the method comprising:

Calculating energy and voice activity detection information of all input channel audio signals;

determining a plurality of target channels according to the calculation result of the audio signal energy and the calculation result of voice activity detection;

Under the condition that the target channels meet the number requirement, the target channels carry out audio enhancement, and the rest channels carry out audio attenuation;

Audio of all input channels is mixed and output.

2. The multi-channel audio/video signal processing method according to claim 1, wherein calculating the audio signal energy of all input channels comprises:

Calculating the maximum energy value of each audio sampling point in a single frame of the current channel;

calculating the average value of the energy maximum values of all frames in the preset duration of the current channel;

Carrying out quantization shaping calculation on the average value of the maximum energy value to obtain the audio signal energy of the current channel;

and taking each input channel as the current channel in turn to obtain the audio signal energy of all the input channels.

3. The multi-channel audio/video signal processing method according to claim 1, wherein calculating the energy and voice activity detection information of all input channel audio signals comprises:

Resampling an audio signal of a current channel to a preset sampling frequency, dividing the audio signal into a plurality of frequency bands according to the Mel cepstrum frequency, and calculating energy information in each frequency band;

Respectively calculating Gaussian probability density functions of voice and noise of each sub-band by using a Gaussian mixture model;

Calculating the log-likelihood ratio of voice and noise, and summing the log-likelihood ratios of all the sub-bands;

calibrating voice activity of the current channel based on the magnitude relation between a preset threshold value and the sum of log likelihood ratios of all sub-bands;

And taking each input channel as the current channel in turn to obtain voice activity detection information of all the input channels.

4. The multi-channel-based audio/video signal processing method according to claim 1, wherein determining a plurality of target channels based on the calculation result of the audio signal energy and the calculation result of the voice activity detection, specifically comprises:

selecting a channel marked as having voice as an alternative channel according to the calculation result of voice activity detection;

and sorting all the alternative channels according to the energy of the audio signals according to the calculation result of the energy of the audio signals, and selecting a preset number of channels with larger energy of the audio signals as the target channels.

5. The method for processing audio and video signals based on multiple channels according to claim 4, wherein if the target channels meet a number requirement, the target channels perform audio enhancement, and perform audio attenuation on the remaining channels, specifically comprising:

Under the condition that the number of the target channels is greater than or equal to the preset number, carrying out audio enhancement on all the target channels;

Selecting candidate channels from the rest channels as complementary target channels under the condition that the number of the target channels is smaller than the preset number, and carrying out audio enhancement on all the target channels;

wherein the number of the candidate channels is the difference between the target channel and the preset number.

6. The audio-video signal processing method based on multiple channels according to claim 5, wherein the selecting candidate channels from the remaining channels as complementary target channels specifically comprises:

selecting a channel marked as voice as a candidate alternative channel according to the calculation result of voice activity detection;

according to the calculation result of the audio signal energy, sequencing all candidate alternative channels according to the audio signal energy, and selecting at least one channel with larger audio signal energy as the complementary target channel;

When the audio signal energy values of the candidate channels are the same, a channel having a high signal-to-noise ratio is selected as a complementary target channel.

7. An audio-visual signal processing apparatus based on multiple channels, said apparatus comprising:

A parameter calculation unit for calculating audio signal energy and voice activity detection information of all input channels;

a channel selection unit for determining a plurality of target channels according to the calculation result of the audio signal energy and the calculation result of the voice activity detection;

the audio processing unit is used for carrying out audio enhancement on the target channels and carrying out audio attenuation on the other channels under the condition that the target channels meet the quantity requirement;

And the result output unit is used for mixing and outputting the audio of all the input channels.

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 6 when the program is executed by the processor.

9. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1 to 6.