CN113299313B - Audio processing method, device and electronic equipment - Google Patents

Abstract

This application discloses an audio processing method, device and electronic equipment, which belongs to the field of signal processing. It can solve the problem of poor playback effect of broadband/full-band non-speech signals. The above method includes: performing resolution improvement processing on the first audio signal to obtain a second audio signal; performing low-pass filtering processing on the above-mentioned second audio signal to obtain a processed second audio signal; and performing a processing on the above-processed second audio signal. The signal is subjected to signal processing to obtain Y first sub-band signals with the same bandwidth; M high-frequency sub-band signals are generated based on the low-frequency sub-band signals among the above-mentioned Y first sub-band signals; based on the high-frequency sub-band signals of the above-mentioned first audio signal frequency characteristic information, perform spectrum adjustment on the above-mentioned M high-frequency sub-band signals, and obtain M target high-frequency sub-band signals; synthesize the above-mentioned M target high-frequency sub-band signals to obtain the target audio signal; among them, Y, M is a positive integer. The embodiments of this application are applied to audio processing scenarios.

Description

Audio processing method, device and electronic equipment

Technical Field

The present application belongs to the field of signal processing, and specifically relates to an audio processing method, device and electronic equipment.

Background Art

With the advancement of electronic technology and the continuous improvement of the performance of electronic equipment, high-definition televisions, headphones, speakers and mobile phones can already support the playback of high-definition audio, and people's demand for high-fidelity and high-expressiveness high-definition audio has become more urgent.

Generally, audio signals usually include speech signals and non-speech signals (e.g., music signals). In related technologies, electronic devices can generate models based on speech signals to expand narrowband speech signals into broadband speech signals to reduce the loss of sound information of speech signals and improve the fidelity of speech signals.

However, since the spectral characteristics of non-speech signals are different from those of speech signals, and the speech signal generation model in the electronic device is generated based on the spectral characteristics of the speech signal, it can only process audio signals with the same spectral characteristics as the speech signal. Therefore, the speech signal generation model in the electronic device is not applicable to non-speech signals (such as music signals and sound signals generated by nature). As a result, the electronic device cannot process the non-speech signal, which leads to poor playback effect of the non-speech signal.

Summary of the invention

The purpose of the embodiments of the present application is to provide an audio processing method that can solve the problem of poor playback effect of broadband/full-band non-voice signals.

In order to solve the above technical problems, this application is implemented as follows:

In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising: performing resolution enhancement processing on a first audio signal to obtain a second audio signal; performing low-pass filtering processing on the second audio signal to obtain a processed second audio signal; performing signal processing on the processed second audio signal to obtain Y first sub-band signals with the same bandwidth; generating M high-frequency sub-band signals according to low-frequency sub-band signals in the Y first sub-band signals; performing spectrum adjustment on the M high-frequency sub-band signals based on high-frequency feature information of the first audio signal to obtain M target high-frequency sub-band signals; synthesizing the M target high-frequency sub-band signals to obtain a target audio signal; wherein Y and M are positive integers.

In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising: a processing module, a generation module and a synthesis module, wherein:

The above-mentioned processing module is used to perform resolution enhancement processing on the first audio signal to obtain a second audio signal; the above-mentioned processing module is also used to perform low-pass filtering processing on the above-mentioned second audio signal to obtain a processed second audio signal; the above-mentioned processing module is also used to perform signal processing on the above-mentioned processed second audio signal to obtain Y first sub-band signals with the same bandwidth; the above-mentioned generation module is used to generate M high-frequency sub-band signals according to the low-frequency sub-band signals in the Y first sub-band signals obtained by the processing module; the above-mentioned processing module is also used to perform spectrum adjustment on the M high-frequency sub-band signals generated by the above-mentioned generation module based on the high-frequency feature information of the above-mentioned first audio signal to obtain M target high-frequency sub-band signals; the above-mentioned synthesis module is used to synthesize the M target high-frequency sub-band signals obtained by the above-mentioned processing module to obtain a target audio signal; wherein Y and M are positive integers.

In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a non-volatile storage medium and is executed by at least one processor to implement the method as described in the first aspect.

In an embodiment of the present application, the electronic device can perform resolution enhancement processing on a low-resolution first audio signal (such as a broadband/full-band non-speech signal) to obtain a high-resolution second audio signal, and perform low-pass filtering on the second audio signal to filter out high-frequency signals in the second audio signal. Then, the processed second audio signal is subjected to signal processing to obtain Y first sub-band signals with the same bandwidth, and M high-frequency sub-band signals are generated based on the low-frequency sub-band signals in the above Y first sub-band signals. Finally, based on the high-frequency spectrum information of the low-resolution first audio signal, the above M high-frequency sub-band signals are spectrally adjusted to obtain M target high-frequency sub-band signals, and the above M target high-frequency sub-band signals are synthesized to obtain a first audio signal whose harmonic characteristics of the high-frequency part are well reconstructed. In this way, a high-definition audio signal with high fidelity and high expressiveness can be obtained, thereby improving the playback effect of the non-speech signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an audio processing method provided by an embodiment of the present application;

FIG2 is a schematic diagram of a waveform diagram of an audio signal provided in an embodiment of the present application;

FIG3 is a second schematic diagram of a waveform diagram of an audio signal provided in an embodiment of the present application;

FIG4 is a schematic diagram of spectrum duplication/flipping provided in an embodiment of the present application;

FIG5 is a schematic diagram of a neural network topology structure provided in an embodiment of the present application;

FIG6 is an amplitude-frequency response curve of a low-pass prototype filter and a PQMF analysis filter bank provided in an embodiment of the present application;

FIG7 is a schematic diagram of a PQMF subband analysis/synthesis filter bank provided in an embodiment of the present application;

FIG8 is a block diagram of a high-definition audio generation system provided in an embodiment of the present application;

FIG9 is a schematic diagram of a structure of an audio processing device provided in an embodiment of the present application;

FIG10 is a second structural diagram of an audio processing device provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged when appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated before and after are in an "or" relationship.

The audio processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

The embodiment of the present application provides an audio processing method, which can be applied to an audio processing device. The figure shows a flow chart of the audio processing method provided by the embodiment of the present application. As shown in FIG1 , the audio processing method provided by the embodiment of the present application may include the following steps 101 to 106:

Step 101: Perform resolution enhancement processing on a first audio signal to obtain a second audio signal.

In an embodiment of the present application, the first audio signal includes at least one of the following: wideband audio (16kHz sampling), ultra-wideband audio (32kHz sampling) and full-band audio (44.1kHz sampling, 48kHz sampling).

In an embodiment of the present application, the resolution of the first audio signal is smaller than the resolution of the second audio signal.

It should be noted that the resolution of an audio signal is determined by the sampling rate and the bit depth. For two audio signals with the same bit depth, the resolution of the audio signal with a higher sampling rate is higher than that of the audio signal with a lower sampling rate. Therefore, the resolution of the audio signal can be improved by increasing the sampling rate of the audio signal. That is, the sampling rate of the first audio signal is less than the sampling rate of the second audio signal. For example, the sampling rate of the second audio signal can be 96 kHz.

In the embodiment of the present application, since the first audio signal is usually broadband/ultra-wideband/full-band audio, its playback effect is poor, therefore, the first audio signal needs to be adjusted to high-definition audio, however, generating a high-definition audio source has high requirements on the software and hardware environment. Therefore, the present application can increase the sampling rate of the above-mentioned first audio signal to reach the sampling rate of high-definition audio without changing the sampling rate and encoding format of the digital audio source and without increasing the network transmission bandwidth, so that the broadband/ultra-wideband/full-band audio can be adjusted to high-definition audio (96kHz sampling).

Generally, upsampling and downsampling both involve resampling of digital signals. Specifically, the resampled sampling rate is compared with the sampling rate of the original digital signal (for example, sampled from an analog signal). If the resampled sampling rate is greater than the sampling rate of the original digital signal, it is upsampling; otherwise, it is downsampling.

It is understandable that the above-mentioned resolution enhancement process can be considered as: performing up-sampling on the first audio signal. That is, the above-mentioned step 101 can include the following step 101a:

Step 101a: up-sample the first audio signal by L times to obtain a second audio signal of a predetermined sampling rate, where L is greater than 0.

Exemplarily, assuming that the first audio signal is a full-band audio signal with a sampling rate of 48 kHz, when it is upsampled (i.e., resampled) by 2 times, the sampling rate (48 kHz) of the full-band audio signal is converted to the sampling rate of high-definition audio (96 kHz).

Example 1, taking the generation of 96kHZ sampled high-definition audio through 48kHz sampled full-band audio (i.e., the first audio signal) as an example, the specific implementation process of the above step 101a is explained. The time domain waveform of the audio signal of the full-band audio is shown in (a) of Figure 2, and the spectrum of the audio signal of the full-band audio is shown in (b) of Figure 2. For example, assuming that the sampling rate of the full-band audio is 48kHZ and the effective bandwidth is 24kHz, the audio processing device will perform 2 times upsampling on the full-band audio input to obtain a 96kHz sampled signal (i.e., the second audio signal).

It should be noted that, since the up-sampling process of the audio signal will increase the bandwidth of the audio signal, the bandwidth of the second audio signal is greater than that of the first audio signal.

Step 102: Perform low-pass filtering on the second audio signal to obtain a processed second audio signal.

In the embodiment of the present application, the signal processing device can filter the high-frequency component (i.e., high-frequency signal) in the second audio signal through a low-pass filter, and only retain the low-frequency component (i.e., low-frequency signal) of the second audio signal. It should be noted that low-pass filtering can be simply considered as: setting a frequency point (i.e., cutoff frequency), when the signal frequency is higher than this frequency, it cannot pass, and when the frequency domain is higher than this cutoff frequency, all values are assigned to 0.

Example 2, in combination with the above-mentioned Example 1, illustrates the above-mentioned signal processing of the second audio signal. After obtaining the above-mentioned 96kHz sampled signal, the audio processing device can filter the audio signal through a low-pass filter with a cutoff frequency of 24kHz to remove the image frequency component of the high-frequency part after upsampling. The waveform diagram and spectrum diagram of the audio signal after upsampling and low-pass filtering are shown in (a) of FIG3 and (b) of FIG3 respectively.

It should be noted that the bandwidth of the first audio signal is the same as the bandwidth of the processed second audio signal. For example, referring to Fig. 3, the sampling rate of the processed audio signal is 96kHz and the effective bandwidth of the processed audio signal remains at 24kHz.

It should be noted that the bandwidth of an audio signal is defined as: the frequency range of the audio signal. According to the Nyquist theorem, the sampling frequency (i.e., sampling rate) of a signal is twice the bandwidth of the signal, i.e., the bandwidth of a signal is 1/2 of the sampling frequency of the signal. Assuming that the sampling rate of the first audio signal is 48kHz, its bandwidth is 48kHz/2, i.e., 24kHz.

Step 103: Perform signal processing on the second audio signal after the above processing to obtain Y first sub-band signals with the same bandwidth.

Wherein, Y is a positive integer.

In an embodiment of the present application, the above-mentioned Y first sub-band signals include a high-frequency sub-band signal and a low-frequency sub-band signal.

Optionally, in the embodiment of the present application, the above-mentioned signal processing of the processed second audio signal may be: performing filtering processing and down-sampling processing on the processed second audio signal.

Exemplarily, the above signal processing includes PQMF sub-band filtering and down-sampling. Further, the audio processing device can first divide the input processed second audio signal into Y sub-band signals of equal bandwidth through the PQMF sub-band filter bank, and then obtain Y first sub-band signals by down-sampling each sub-band signal.

It should be noted that PQMF sub-band analysis performs time-frequency transformation on the original signal, and its purpose is to obtain multiple sub-band signals that reflect the correlation between high and low frequencies, have good harmonic characteristics and are easy to analyze. At the analysis end, the input time domain signal is divided into multiple sub-band signals of equal bandwidth through the PQMF analysis filter group, and then each sub-band signal is down-sampled. At the synthesis end, each sub-band signal is first up-sampled, and then the up-sampled sub-band signal is converted into a time domain signal through the PQMF synthesis filter group.

It should be noted that the division of the high-frequency sub-band signal and the low-frequency sub-band signal in the Y first sub-band signals is determined according to the frequency range of the high-frequency component and the low-frequency component of the processed second audio signal. That is, the first sub-band signal whose signal frequency is within the frequency range of the low-frequency component is a low-frequency sub-band signal; the first sub-band signal whose signal frequency is within the frequency range of the high-frequency component is a high-frequency sub-band signal.

Step 104: Generate M high frequency sub-band signals according to the low frequency sub-band signals in the Y first sub-band signals.

In an embodiment of the present application, the audio processing device can generate one or more high-frequency sub-band signals based on a low-frequency sub-band signal, that is, each low-frequency sub-band signal in the Y first sub-band signals corresponds to one or more high-frequency sub-band signals, and Y is less than or equal to M.

In the embodiment of the present application, a high frequency generator can be used to generate a high frequency sub-band signal spectrum according to the spectrum of the low frequency sub-band signal in the Y first sub-band signals to generate a high frequency sub-band signal.

Exemplarily, the method for the audio processing device to generate M high frequency sub-band signals may include any one of the four methods shown in Table 1.

Table 1 High frequency sub-band spectrum generation method

Referring to Table 1 above, it can be seen that the spectrum processing type corresponding to the above methods 1 and 2 is spectrum replication, and the spectrum processing type corresponding to the above methods 3 and 4 is spectrum flipping. The difference between spectrum replication and spectrum flipping can be shown in FIG4 .

Step 105: Based on the high-frequency feature information of the first audio signal, perform spectrum adjustment on the M high-frequency sub-band signals to obtain M target high-frequency sub-band signals.

In an embodiment of the present application, the above-mentioned high-frequency feature information may be the signal gain of the above-mentioned M high-frequency sub-band signals.

In the embodiment of the present application, the audio processing device can adjust the amplitudes of the M high frequency sub-band signals through an envelope adjuster to obtain M reconstructed high frequency sub-band signals (i.e., the M target high frequency sub-band signals).

Step 106: synthesize the M target high frequency sub-band signals to obtain a target audio signal.

Wherein, M is a positive integer.

In an embodiment of the present application, the audio processing device can synthesize the above-mentioned M target high-frequency sub-band signals through a PQMF synthesis filter group to obtain a target audio signal.

In the audio processing method provided in the embodiment of the present application, the electronic device can perform resolution enhancement processing on a low-resolution first audio signal (such as a broadband/full-band non-speech signal) to obtain a high-resolution second audio signal, and perform low-pass filtering on the second audio signal to filter out the high-frequency signal in the second audio signal. Then, the processed second audio signal is subjected to signal processing to obtain Y first sub-band signals with the same bandwidth, and M high-frequency sub-band signals are generated based on the low-frequency sub-band signals in the above Y first sub-band signals. Finally, based on the high-frequency spectrum information of the low-resolution first audio signal, the above M high-frequency sub-band signals are spectrally adjusted to obtain M target high-frequency sub-band signals, and the above M target high-frequency sub-band signals are synthesized to obtain a first audio signal whose harmonic characteristics of the high-frequency part are well reconstructed. In this way, a high-definition audio signal with high fidelity and high expressiveness can be obtained, thereby improving the playback effect of the non-speech signal.

Optionally, in the embodiment of the present application, since there is a correlation between the high-frequency sub-band signal and the low-frequency sub-band signal of the audio signal, a corresponding high-frequency sub-band signal can be generated according to the low-frequency sub-band signal in the processed second audio signal.

Exemplarily, the above step 104 may include the following step 104a:

Step 104a: Perform spectrum replication on all low-frequency sub-band signals in the Y sub-band signals to generate M high-frequency sub-band signals.

Exemplarily, the audio processing device may use the spectrum replication method in Table 1 to generate the spectrum of M high-frequency sub-band signals. Exemplarily, the audio processing device may replicate the upper half of the spectrum of the low-frequency sub-band signal multiple times to generate the spectrum of M high-frequency sub-band signals to generate M high-frequency sub-band signals.

In this way, the audio processing device can obtain the high-frequency component in the processed second audio signal based on the low-frequency component in the processed second audio signal, thereby preliminarily obtaining the frequency spectrum of the processed second audio signal.

Optionally, in an embodiment of the present application, the audio processing device may extract the low-frequency features of the original audio signal based on the strong correlation between the low-frequency features and the high-frequency spectrum envelope of the audio signal, so as to predict the high-frequency features of the audio signal according to the low-frequency features.

Exemplarily, before the above step 105, the audio processing method provided in the embodiment of the present application further includes the following steps A1 and A2:

Step A1: extract features from the first audio signal to obtain low-frequency feature information of the first audio signal.

Step A2: Input the above low-frequency feature information into a preset neural network model to predict the high-frequency feature information of the first audio signal.

Exemplarily, the low-frequency feature information includes at least one of the following: a normalized autocorrelation coefficient (x _acf ) of the first audio signal, a gradient index x _gi , and a sub-band spectrum flatness (x _sfm ).

It should be noted that the above low-frequency feature information can be regarded as a feature parameter of the first audio signal. The selection of the feature parameter needs to consider the following three principles:

(1) The low-frequency characteristic parameters have a strong correlation with the high-frequency spectrum envelope;

(2) There is good independence between feature components;

(3) The characteristic components are easy to calculate.

Based on the above principles, the present embodiment selects the above three characteristic parameters to describe the audio characteristics from the perspective of time domain and frequency domain. In practical applications, other feasible characteristic parameters can also be selected, and the present embodiment does not make any limitation on this.

For further detailed description of the above three frequency characteristic information (ie, characteristic parameters), please see below.

Exemplarily, the above-mentioned preset neural network can be a DNN neural network. It should be noted that the DNN neural network is a multi-layer forward network with one-way propagation, which can efficiently abstract and model complex data. The topological structure of the DNN neural network is shown in Figure 5, which is divided into three categories: input layer, hidden layer and output layer. Usually, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. The neurons in each layer are fully connected, and there is no connection between the neurons in the same layer.

Exemplarily, the above-mentioned DNN neural network is used to establish a nonlinear mapping from the low-frequency features of the first audio signal to the high-frequency spectrum envelope of the first audio signal.

Exemplarily, the input of the above-mentioned DNN neural network is the high-frequency feature information of the above-mentioned first audio signal, including the normalized autocorrelation coefficient, the gradient index, and the sub-band spectrum flatness, and the output of the above-mentioned DNN neural network is the signal gain (represented by G) of the high-frequency sub-band signal of the first audio signal.

In this way, the audio processing device can predict the high-frequency feature information of the first audio signal based on the low-frequency feature information of the first audio signal, thereby adjusting the spectrum (i.e., spectrum envelope) of the processed second audio signal through the high-frequency feature information.

Optionally, in an embodiment of the present application, the audio processing device may frame the second audio signal after the above processing, and then perform audio signal processing based on each audio signal frame to reduce the impact of the overall non-steady state and time variation of the speech signal.

Exemplarily, the above step 103 may include the following steps 103a and 103b:

Step 103a: Divide the second audio signal after the above processing into frames to obtain X audio signal frames.

Step 103b: Filter and downsample each audio signal frame in turn to obtain N first subband signals corresponding to each audio signal frame.

The above-mentioned Y first sub-band signals include: N first sub-band signals corresponding to each audio signal frame.

Exemplarily, each audio signal frame includes a first predetermined number of sample points. For example, it can be preset that each signal frame includes 2048 sample points.

Exemplarily, X in the X audio signal frames is determined according to a sampling rate of the second audio signal and the number of sample points included in each of the audio signal frames.

Exemplarily, the audio processing device may number the obtained X audio signal frames, and each audio signal frame may correspond to a sequence number. For example, assuming that the processed second audio signal includes l audio signal frames, the l audio signal frames may be numbered from 1-1.

For example, take the case of generating a 96 kHz sampled high-definition audio through a 48 kHz sampled full-band audio (i.e., a first audio signal). In combination with the above examples 1 and 2, after up-sampling and low-pass filtering the first audio signal to obtain a processed second audio signal, the sampling rate of the processed second audio signal is 96 kHz, and it can be divided into 46 audio signal frames (i.e., X audio signal frames) with 2048 sample points per frame according to the sampling rate of 96 kHz.

Exemplarily, the audio processing device may perform the above-mentioned filtering and down-sampling processing on each audio signal frame in sequence according to the timing information of the X audio signal frames.

Exemplarily, each of the N first sub-band signals has an index, and one index corresponds to one first sub-band signal.

Exemplarily, the N first sub-band signals include P low-frequency sub-band signals and Q high-frequency sub-band signals, where P and Q are positive integers.

Exemplarily, the number of subband signals corresponding to each audio signal frame (i.e., N) is preset, and further, the number of subband signals is determined according to the parameters set for the PQMF subband filter bank. For example, the number of subbands of the PQMF subband filter bank is set to 64, and after each audio signal frame is processed by the PQMF subband filter bank, 64 subband signals corresponding to each audio signal frame can be obtained.

Exemplarily, for the above step 103b, the audio processing device may first perform PQMF filtering on each audio signal frame to obtain N sub-band signals corresponding to each audio signal frame, and then downsample the N sub-band signals to obtain N first sub-band signals. Further, the above downsampling process may be an N-fold downsampling process.

Exemplarily, each of the N first sub-band signals includes a second predetermined number of sample points. Further, the second predetermined number is determined according to a sampling multiple of downsampling.

Exemplarily, the second predetermined number of sample points in each of the first sub-band signals are arranged in time sequence within the frequency range where the first sub-band signal is located.

Example 3, taking the generation of 96kHZ sampled high-definition audio from 48kHz sampled full-band audio (i.e., the first audio signal) as an example. After the second audio signal after the above processing is framed, assuming that each signal frame includes 2048 sample points, 64 sub-band signals are obtained after filtering by the PQMF analysis filter bank, and then each sub-band signal is down-sampled 64 times to obtain 64 first sub-band signals, each of which includes 32 sample points. Among them, the 0th to 31st sub-band signals are low-frequency sub-band signals, and the 32nd to 63rd sub-band signals are high-frequency sub-band signals.

It should be noted that the N first sub-band signals corresponding to each audio signal frame belong to N different frequency ranges (i.e., frequency bands) of the second audio signal. For example, assuming that each audio signal frame corresponds to 64 first sub-band signals, the second audio signal is divided into 64 frequency ranges according to the signal frequency, and each first sub-band signal belongs to one frequency range of the 64 frequency ranges. In this way, the obtained N first sub-band signals have frequency characteristics that can reflect the signal and have good harmonic characteristics.

For ease of description, the PQMF analysis filter group output signal, that is, the above-mentioned N first subband signals are represented as x _l [k][n], where k represents the subband number, which ranges from 0≤k≤63, n represents the time sequence number of the time sequence sample point in each subband, which ranges from 0≤n≤31, and l represents the sequence number of the current audio signal frame.

It should be noted that, for each of the X audio signal frames, after being filtered by the PQMF analysis filter bank, the output subband signal (i.e., the first subband signal) forms an x[k][n] matrix, where k represents the transformed subband sequence number (the sequence number of the first subband signal), and n represents the sequence number of the transformed subband time series sample point (i.e., the time series sample point of the first subband signal). x[k][n] has dual resolutions of time and frequency, and has both frequency distribution characteristics in the frequency domain and waveform characteristics in the time domain.

For ease of understanding, the expressions of the PQMF analysis filter bank and the synthesis filter bank are explained below.

Exemplarily, the mathematical expressions of the PQMF analysis filter bank and the synthesis filter bank used in the embodiment of the present application are as follows:

Analysis Filters:

Synthesis Filter:

Wherein, N in equation (1) and equation (2) is the number of the first subband signal, p(n) is a low-pass prototype filter, whose normalized cutoff frequency is π/(2N), the filter length is M, M=LN, L is any positive integer, k=0,1,…,N-1, represents the subband number, and n identifies the number of the subband time series sample point after the transformation.

Exemplarily, the number of subbands of the above-mentioned PQMF subband filter bank can be set to N=64, the order of the low-pass prototype filter p(n) can be set to M=768, and the filter stopband attenuation is designed to be -90 dB.

(a) in FIG6 is the amplitude-frequency response curve of the low-pass prototype filter p(n), and (b) in FIG6 is the amplitude-frequency response curve of the PQMF analysis filter bank.

FIG7 is a schematic diagram of the principle of a PQMF subband analysis/synthesis filter bank. In FIG7 , H _k (z) is the Z transform of h _k (n), and F _k (z) is the Z transform of F _k (n).

It should be noted that the above-mentioned analysis filter group is used to divide the input time domain signal into N sub-band signals, and the above-mentioned synthesis filter group is used to synthesize the N sub-band signals into one time domain signal.

Further optionally, in combination with the above 103b, the above step 104a may include the following step 104a1:

Step 104a1: Generate at least one high frequency sub-band signal according to the low frequency sub-band signal in the N first sub-band signals of each audio signal frame.

Exemplarily, the number of high frequency sub-band signals finally generated in each audio signal frame is the same.

Example 4. In combination with the above Example 3, after the audio processing device obtains 64 first subband signals corresponding to each audio signal frame, the audio processing device can select 16 low-frequency subband signals with subband indexes of 15-30 (i.e., corresponding to the low-frequency source subband sequence numbers in Table 2) when copying, and copy the spectral coefficients of the low-frequency subband signals twice to generate 32 high-frequency subband spectral coefficients (i.e., corresponding to the high-frequency target subband sequence numbers in Table 2). The corresponding relationship during frequency band copying is shown in Table 2.

Low frequency source subband signal High frequency target subband signal 15 32, 48 16 33, 49 17 34, 50 18 35, 51 19 36, 52 20 37, 53 twenty one 38, 54 twenty two 39, 55 twenty three 40, 56 twenty four 41, 57 25 42, 58 26 43, 59 27 44, 60 28 45, 61 29 46, 62 30 47, 63

Table 2 High frequency and low frequency band replication correspondence table

It should be noted that the “low-frequency source sub-band sequence number” in Table 2 is the sequence number of the low-frequency sub-band signal, and the “high-frequency target sub-band sequence number” is the sequence number of the high-frequency sub-band signal.

Further optionally, in the embodiment of the present application, the above step A1 includes the following step B1:

Step B1: extracting features of the P low-frequency sub-band signals in the N first sub-band signals in each audio signal frame to obtain low-frequency feature information of each audio signal frame.

Exemplarily, the audio processing device may calculate the normalized autocorrelation coefficient and the gradient index of the first audio signal according to the number of samples of the first audio signal and the order of the autocorrelation function.

The definition of the above low-frequency feature information is described in detail below:

(1) The above normalized autocorrelation coefficient is used to describe the correlation of the signal in the time domain. Let x(n) be the input audio signal, N be the number of samples per frame, and m be the order of the autocorrelation function (m = 1, 2, ..., M, where M is the maximum autocorrelation order). The normalized autocorrelation coefficient is calculated as follows:

(2) The above gradient index is used to distinguish the harmonic and noise characteristics of the audio signal. It is defined as the sum of the gradient amplitudes of the audio signal in each change direction, that is:

Among them, the variable ψ(n) is the indicator function of the direction of signal change:

Where sign(x) is the sign function, which is defined as:

Where E is the total energy of the input signal of the current frame:

(3) The above sub-band spectrum flatness is used to distinguish the tone and noise characteristics of the audio signal in the sub-band. The greater the sub-band spectrum flatness, the more tone components are shown in the sub-band spectrum. Conversely, the more noise components are shown in the sub-band spectrum. It is defined as the ratio of the geometric mean to the algebraic mean of all spectra (MDTC spectrum coefficients) in each low-frequency PQMF sub-band.

The following combines the definition of the above low-frequency feature information to further illustrate the above low-frequency feature information in a specific example.

Exemplarily, the audio processing device may obtain the frequency spectrum coefficient of each low-frequency sub-band signal in the P low-frequency sub-band signals to calculate the sub-band spectrum flatness of each low-frequency sub-band signal.

Exemplarily, the low-frequency feature information of the first audio signal may be a set of 64-dimensional feature vectors of each audio signal frame.

Example 5, taking the generation of 96 kHz sampled high-definition audio from 48 kHz sampled full-band audio (i.e., the first audio signal) as an example. Assuming that each of the above signal frames corresponds to 64 sub-band signals (first sub-band signals), the audio processing device can obtain the spectral coefficients of 0-31 sub-band signals therein, and calculate the sub-band spectrum flatness of each of the 0-31 sub-band signals.

It should be noted that, when performing feature extraction, the maximum autocorrelation order of the above normalized autocorrelation coefficient can be set to M=31. The setting of the feature dimension in the embodiment of the present application is shown in Table 3.

Table 3 Feature names and dimensions

Further optionally, in the embodiment of the present application, in combination with the above step B1, the above step A2 includes the following step B2:

Step B2: Input the low-frequency feature information of each audio signal frame into a preset neural network model to predict the high-frequency feature information of each audio signal frame.

Exemplarily, the high-frequency feature information of each audio signal frame may be signal gains of the H high-frequency sub-band signals.

For example, assuming that the kth high-frequency sub-band signal among the M high-frequency sub-band signals is generated by the jth low-frequency sub-band signal among the low-frequency sub-band signals, the sub-band gain G[k] of the kth high-frequency sub-band is defined as:

In formula (9), En _k is the total energy of the k-th high-frequency sub-band spectrum coefficients, and En _j is the total energy of the j-th low-frequency PQMF sub-band MDCT spectrum coefficients.

It should be noted that audio signals are "serialized" data with time sequence, and the previous and next signals are related. In order to make full use of its contextual relevance, the DNN neural network (i.e., DNN model) uses frame splicing to consider the impact of contextual related information on the current frame. Specifically, assuming that the feature parameter vector extracted from the current frame signal is When splicing frames, it selects m frames forward and backward to form a superframe feature vector As the input of the DNN model, It is expressed as follows:

For example, in order to make full use of the contextual relevance of the audio signal (i.e., the relevance between multiple consecutive audio signal frames), the audio processing device can adopt a frame splicing strategy after obtaining the low-frequency feature information of each audio signal frame, and input multiple audio signal frames into the DNN neural network. For example, when splicing frames, it selects 3 frames forward and backward respectively, and a total of 7 frames of feature vectors including the current frame feature form a superframe feature vector As the input of the DNN model, its dimension is 64*7=448, that is:

Example 6, taking the generation of 96kHZ sampled high-definition audio from 48kHz sampled full-band audio (i.e., the first audio signal) as an example. Assuming that each audio signal frame corresponds to 64 sub-band signals (first sub-band signals), where sub-bands 32-63 are high-frequency sub-band signals, after processing each audio signal frame through the DNN neural network, the signal gain of the output high-frequency sub-band signal is is a 32-dimensional feature vector, and its mathematical expression is as follows:

Exemplarily, the hyperparameter settings of the above DNN neural network are shown in Table 4.

Table 4 Hyperparameters of DNN neural model

Further optionally, in the embodiment of the present application, the above step 105 includes the following step 105a:

Step 105a: According to the high-frequency feature information of each audio signal frame, spectrum adjustment is performed on the H high-frequency sub-band signals in each audio signal frame to obtain H target high-frequency sub-band signals.

Among them, the above-mentioned M target high-frequency sub-band signals include the above-mentioned H target high-frequency sub-band signals of each audio signal frame.

Exemplarily, the kth high frequency sub-band signal among the H high frequency sub-band signals obtained by the high frequency generator is Its total energy is Let the kth high-frequency subband gain obtained by the envelope predictor be G[k], and the kth reconstructed high-frequency subband signal (i.e., the target high-frequency subband signal) obtained by the envelope modulator be X[k][m], then:

Where N is the frame length of one frame of MDCT coefficients of the PQMF subband, k _l and k _h are the start index and end index of the high-frequency PQMF subband respectively.

Example 7, taking the generation of 96 kHz sampled high-definition audio from 48 kHz sampled full-band audio (i.e., the first audio signal) as an example. Assume that each audio signal frame corresponds to 64 sub-band signals (the first sub-band signal), where sub-bands 32-63 are high-frequency sub-band signals, and the kth high-frequency sub-band signal obtained by the high-frequency generator is Its total energy is Let the kth high-frequency sub-band gain obtained by the envelope predictor be G[k], and the kth reconstructed high-frequency sub-band signal obtained by the envelope modulator be X[k][m], then:

Further optionally, in the embodiment of the present application, after the processed second audio signal is framed, the audio signal at the boundary of two adjacent frames may have a large amplitude difference, thereby causing the audio signal to be discontinuous, thereby generating noise. In order to eliminate such noise, denoising processing may be performed on the X audio signal frames.

Exemplarily, after the above step 103a, the signal processing method provided in the embodiment of the present application further includes the following step C1:

Step C1: performing signal processing on N first sub-band signals in two adjacent audio signal frames in the X audio signal frames to obtain processed N first sub-band signals.

Exemplarily, the processed first subband signal includes a low-frequency subband signal in each audio signal frame.

Exemplarily, the signal processing may include MDCT transformation. Further, in the case of MDCT transformation, two first sub-band signals with the same frequency band in the two adjacent audio signal frames may be obtained in sequence, and then the two first sub-band signals may be windowed and MDCT transformed to obtain a first sub-band signal with MDCT spectrum coefficients (i.e., frequency spectrum).

For the convenience of subsequent description, the two first sub-band signals with the same frequency band in the two adjacent audio signal frames are recorded as two related sub-band signals.

Furthermore, each of the above subband signals includes N sample points. When performing MDCT transformation, the input sequence of the first audio signal frame (i.e., x(n)) and the N sample points of the input sequence of the first audio signal frame are combined to form 2N sample points, and then the signal of the 2N sample points is windowed, and then the windowed signal is transformed by MDCT to obtain MDCT spectrum coefficients of the N sample points.

The expression of MDCT is as follows:

Exemplarily, when windowing a signal, the window function selects a sine window, which is defined as:

Example 8, taking the generation of 96 kHz sampled high-definition audio through 48 kHz sampled full-band audio (i.e., the first audio signal) as an example. Assume that each audio signal frame corresponds to 64 sub-band signals (first sub-band signals), wherein each sub-band signal includes 32 sample points, and after windowing and MDCT transformation of the above two related sub-band signals, each sub-band signal obtains 32 sample points of MDCT spectrum coefficients, denoted as X _l [k][m], wherein k represents the sub-band number, and its range is 0≤k≤63, m represents the MDCT spectrum number, and its range is 0≤m≤31, and l represents the audio signal frame number.

Further optionally, in the embodiment of the present application, in combination with the above step 103a, the above step 106 includes the following steps 106a and 106b:

Step 106a: synthesize the H target high frequency sub-band signals in each audio signal frame to obtain a fourth audio signal corresponding to each audio signal frame.

Step 106b: synthesize the fourth audio signal corresponding to each audio signal frame to obtain a target audio signal.

Exemplarily, the audio processing device may synthesize H target high frequency sub-band signals in each audio signal frame through up-sampling and filtering to obtain a fourth audio signal corresponding to each audio signal frame.

Furthermore, when synthesizing the H target high-frequency sub-band signals in each of the above audio signal frames, the audio processing device first upsamples each sub-band signal by N times, and then converts the up-sampled sub-band signal into a time domain signal through a PQMF synthesis filter group.

The mathematical expression of the PQMF synthesis filter group used in the embodiment of the present application has been explained above and will not be repeated here.

Further optionally, in an embodiment of the present application, when performing MDCT transformation on the above-mentioned N first sub-band signals, the audio processing device can perform MDCT inverse transformation (i.e., IMDCT) on the H high-frequency sub-band signals after spectrum adjustment to restore the sub-band signal in each audio signal frame.

In combination with the above step 103a and step C1, after the spectrum of the H high frequency sub-band signals in each audio signal frame is adjusted in the above step 105a, the audio signal processing method provided in the embodiment of the present application further includes the following step D1:

Step D1: Perform IMDCT transformation on the H high-frequency sub-band signals after spectrum adjustment to obtain a sub-band reconstruction signal corresponding to each of the above high-frequency sub-band signals.

The H target high-frequency sub-band signals include the sub-band reconstructed signals.

Exemplarily, the audio processing device performs IMDCT transform and overlap-add operation on the MDCT spectrum coefficients of each subband when performing IDMT transform on the first subband signal after the above processing, to obtain N subband reconstructed signals x′ _l [k][n] of the current lth frame, where k represents the subband number, and its range is 0≤k≤63, n represents the time sequence number of the time sequence sample point in each subband, and its range is 0≤n≤31, and l represents the audio signal frame number.

The expression of IMDCT is as follows:

Where w(n) is the window function. The output signal after IMDCT transformation Perform Overlap-add operation to obtain the subband reconstructed signal x′ _l (n) of the current frame l, that is:

It should be noted that the overall flow chart of the audio processing method provided in the embodiment of the present application is shown in FIG8 .

It should be noted that the audio processing method provided in the embodiment of the present application can be executed by an audio processing device or a control module in the audio processing device for executing the audio processing method. In the embodiment of the present application, the audio processing device provided in the embodiment of the present application is described by taking the audio processing device executing the audio processing method as an example.

The embodiment of the present application provides an audio processing device, as shown in FIG9 , the device includes: a processing module 801, a generating module 802 and a synthesizing module 803, wherein:

The processing module 801 is used to perform resolution enhancement processing on the first audio signal to obtain a second audio signal; the processing module 801 is also used to perform low-pass filtering processing on the second audio signal to obtain a processed second audio signal; the processing module 801 is also used to perform filtering processing and down-sampling processing on the processed second audio signal to obtain Y first sub-band signals with the same bandwidth; the generating module 802 is used to generate M high-frequency sub-band signals according to the low-frequency sub-band signals in the Y first sub-band signals obtained by the processing module 801; the processing module 801 is also used to perform spectrum adjustment on the M high-frequency sub-band signals generated by the generating module 802 based on the high-frequency feature information of the first audio signal to obtain M target high-frequency sub-band signals; the synthesizing module 803 is used to synthesize the M target high-frequency sub-band signals obtained by the processing module 801 to obtain a target audio signal; wherein Y and M are positive integers.

In the audio processing device provided in the embodiment of the present application, the electronic device can perform resolution enhancement processing on a low-resolution first audio signal (such as a broadband/full-band non-speech signal) to obtain a high-resolution second audio signal, and perform low-pass filtering on the second audio signal to filter out the high-frequency signal in the second audio signal. Then, the processed second audio signal is subjected to signal processing to obtain Y first sub-band signals with the same bandwidth, and M high-frequency sub-band signals are generated based on the low-frequency sub-band signals in the above Y first sub-band signals. Finally, based on the high-frequency spectrum information of the low-resolution first audio signal, the above M high-frequency sub-band signals are spectrally adjusted to obtain M target high-frequency sub-band signals, and the above M target high-frequency sub-band signals are synthesized to obtain a first audio signal whose harmonic characteristics of the high-frequency part are well reconstructed. In this way, a high-definition audio signal with high fidelity and high expressiveness can be obtained, thereby improving the playback effect of the non-speech signal.

Optionally, in an embodiment of the present application, the above-mentioned generation module 802 is specifically used to perform spectrum replication on all low-frequency sub-band signals in the above-mentioned Y sub-band signals to generate M high-frequency sub-band signals, wherein one low-frequency sub-band signal corresponds to at least one high-frequency sub-band signal, and Y is less than or equal to M.

Optionally, in the embodiment of the present application, the audio processing device further includes: an extraction module 804 and a prediction module 805;

The extraction module 804 is used to extract features from the first audio signal to obtain low-frequency feature information of the first audio signal; the prediction module 805 is used to input the low-frequency feature information extracted by the extraction module into a preset neural network model to predict high-frequency feature information of the first audio signal.

Optionally, in an embodiment of the present application, the processing module 801 is specifically used to upsample the first audio signal by L times to obtain a second audio signal with a predetermined sampling rate, and the bandwidth of the first audio signal is the same as that of the second audio signal.

Optionally, in the embodiment of the present application, the processing module 801 is further used to frame the low-frequency component of the second audio signal to obtain X audio signal frames, each audio signal frame including a predetermined number of sample points; the processing module 801 is specifically used to filter and down-sample each audio signal frame in turn to obtain N first sub-band signals corresponding to each audio signal frame; wherein the Y first sub-band signals include: N first sub-band signals corresponding to each audio signal frame.

Optionally, in the embodiment of the present application, the processing module 801 is specifically configured to perform signal processing on N first subband signals of the first audio signal frame and N first subband signals in the second audio signal frame to obtain processed N first subband signals; wherein the first audio signal frame and the second audio signal frame are adjacent audio signal frames in the X audio signal frames.

The audio processing device in the embodiment of the present application can be a device, or a component, an integrated circuit, or a chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

The audio processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

The audio processing device provided in the embodiment of the present application can implement each process implemented by the method embodiments of Figures 1 to 8. To avoid repetition, they will not be described here.

Optionally, as shown in FIG10 , an embodiment of the present application further provides an electronic device 900, including a processor 901, a memory 902, and a program or instruction stored in the memory 902 and executable on the processor 901. When the program or instruction is executed by the processor 901, each process of the above-mentioned audio processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG. 11 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes but is not limited to: a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110 and other components.

Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 110 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The electronic device structure shown in FIG11 does not constitute a limitation on the electronic device, and the electronic device may include more or less components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.

The processor 110 is used to perform resolution enhancement processing on the first audio signal to obtain a second audio signal; the processor 110 is also used to perform low-pass filtering processing on the second audio signal to obtain a processed second audio signal, filter and downsample the processed second audio signal to obtain Y first sub-band signals with the same bandwidth, and generate M high-frequency sub-band signals according to the low-frequency sub-band signals in the Y first sub-band signals; the processor 110 is also used to perform spectrum adjustment on the M high-frequency sub-band signals generated based on high-frequency feature information of the first audio signal to obtain M target high-frequency sub-band signals, and synthesize the M target high-frequency sub-band signals to obtain a target audio signal; wherein Y and M are positive integers.

In the electronic device provided in the embodiment of the present application, the electronic device can perform resolution enhancement processing on a low-resolution first audio signal (such as a broadband/full-band non-speech signal) to obtain a high-resolution second audio signal, and perform low-pass filtering on the second audio signal to filter out the high-frequency signal in the second audio signal. Then, the processed second audio signal is subjected to signal processing to obtain Y first sub-band signals with the same bandwidth, and M high-frequency sub-band signals are generated based on the low-frequency sub-band signals in the above Y first sub-band signals. Finally, based on the high-frequency spectrum information of the low-resolution first audio signal, the above M high-frequency sub-band signals are spectrally adjusted to obtain M target high-frequency sub-band signals, and the above M target high-frequency sub-band signals are synthesized to obtain a first audio signal whose harmonic characteristics of the high-frequency part are well reconstructed. In this way, a high-definition audio signal with high fidelity and high expressiveness can be obtained, thereby improving the playback effect of the non-speech signal.

Optionally, in an embodiment of the present application, the processor 110 is specifically used to perform spectrum replication on all low-frequency sub-band signals in the Y sub-band signals to generate M high-frequency sub-band signals, wherein one low-frequency sub-band signal corresponds to at least one high-frequency sub-band signal, and Y is less than or equal to M.

Optionally, in an embodiment of the present application, the processor 110 is further used to perform feature extraction on the first audio signal to obtain low-frequency feature information of the first audio signal, input the low-frequency feature information into a preset neural network model, and predict high-frequency feature information of the first audio signal.

Optionally, in an embodiment of the present application, the processor 110 is specifically configured to upsample the first audio signal by L times to obtain a second audio signal of a predetermined sampling rate, and the first audio signal has the same bandwidth as the second audio signal.

Optionally, in the embodiment of the present application, the processor 110 is further configured to frame the low-frequency component of the second audio signal to obtain X audio signal frames, each audio signal frame including a predetermined number of sample points, and sequentially filter and downsample each audio signal frame to obtain N first sub-band signals corresponding to each audio signal frame; wherein the Y first sub-band signals include: N first sub-band signals corresponding to each audio signal frame.

Optionally, in the embodiment of the present application, the processor 110 is specifically configured to perform signal processing on N first subband signals of the first audio signal frame and N first subband signals in the second audio signal frame to obtain processed N first subband signals; wherein the first audio signal frame and the second audio signal frame are adjacent audio signal frames in the X audio signal frames.

It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include but are not limited to a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 109 can be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 110.

The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned audio processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

The embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

An embodiment of the present application provides a computer program product, which is stored in a non-volatile storage medium and is executed by at least one processor to implement the method as described in the first aspect.

It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above implementation methods can be implemented by means of software plus a necessary general hardware platform, or by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), and includes several instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present application and the claims, all of which are within the protection of the present application.

Claims (14)

1. A method of audio processing, the method comprising:

performing resolution enhancement processing on the first audio signal to obtain a second audio signal;

performing low-pass filtering processing on the second audio signal to obtain a processed second audio signal;

filtering the processed second audio signal to obtain Y subband signals with the same bandwidth, and performing downsampling on the Y subband signals with the same bandwidth to obtain Y first subband signals with the same bandwidth;

generating M high-frequency subband signals according to low-frequency subband signals in Y first subband signals in a spectrum copying or spectrum inversion mode, wherein each low-frequency subband signal in Y first subband signals corresponds to one or more high-frequency subband signals;

Performing frequency spectrum adjustment on the M high-frequency subband signals based on the high-frequency characteristic information of the first audio signal to obtain M target high-frequency subband signals, wherein the high-frequency characteristic information is the signal gain of the M high-frequency subband signals, and the signal gain is determined by the normalized autocorrelation coefficient, gradient index and subband spectrum flatness of the first audio signal;

synthesizing the M target high-frequency subband signals to obtain target audio signals;

wherein the first audio signal comprises at least one of: broadband audio, ultra-broadband audio, full-band audio, Y, M is a positive integer.

2. The method according to claim 1, wherein generating M high frequency subband signals from the low frequency subband signals in the Y first subband signals by spectral replication or spectral inversion comprises:

and performing spectrum copying on all low-frequency subband signals in the Y first subband signals to generate M high-frequency subband signals, wherein one low-frequency subband signal corresponds to at least one high-frequency subband signal, and Y is smaller than or equal to M.

3. The method of claim 1, wherein the performing spectral adjustment on the M high frequency subband signals based on the high frequency characteristic information of the first audio signal, before obtaining M target high frequency subband signals, further comprises:

Extracting the characteristics of the first audio signal to obtain low-frequency characteristic information of the first audio signal;

inputting the low-frequency characteristic information into a preset neural network model to predict the high-frequency characteristic information of the first audio signal.

4. The method of claim 1, wherein performing resolution enhancement processing on the first audio signal to obtain a second audio signal comprises:

and carrying out L times up-sampling on the first audio signal to obtain a second audio signal with a preset sampling rate, wherein the bandwidth of the first audio signal is the same as that of the second audio signal.

5. The method of claim 1, wherein filtering the processed second audio signal to obtain Y subband signals with the same bandwidth, and downsampling the Y subband signals with the same bandwidth to obtain Y first subband signals with the same bandwidth, comprises:

framing the low-frequency component of the second audio signal to obtain X audio signal frames, wherein each audio signal frame comprises a preset number of sample points;

sequentially carrying out filtering and downsampling on each audio signal frame to obtain N first sub-band signals corresponding to each audio signal frame;

Wherein the Y first subband signals include: n first sub-band signals corresponding to each audio signal frame.

6. The method of claim 5, wherein after framing the low frequency component of the second audio signal to obtain X audio signal frames, the method further comprises:

performing signal processing on N first sub-band signals of the first audio signal frame and N first sub-band signals in the second audio signal frame to obtain N processed first sub-band signals; wherein the first audio signal frame and the second audio signal frame are adjacent ones of the X audio signal frames.

7. An audio processing apparatus, the apparatus comprising: the device comprises a processing module, a generating module and a synthesizing module, wherein:

the processing module is used for carrying out resolution improvement processing on the first audio signal to obtain a second audio signal;

the processing module is further used for performing low-pass filtering processing on the second audio signal to obtain a processed second audio signal;

the processing module is further configured to perform filtering processing on the processed second audio signal to obtain Y subband signals with the same bandwidth, and perform downsampling processing on the Y subband signals with the same bandwidth to obtain Y first subband signals with the same bandwidth;

The generating module is configured to generate M high-frequency subband signals according to the low-frequency subband signals in the Y first subband signals obtained by the processing module, where each low-frequency subband signal in the Y first subband signals corresponds to one or more high-frequency subband signals in a spectrum duplication or spectrum inversion manner;

the processing module is further configured to perform spectrum adjustment on the M high-frequency subband signals generated by the generating module based on the high-frequency characteristic information of the first audio signal, to obtain M target high-frequency subband signals, where the high-frequency characteristic information is signal gains of the M high-frequency subband signals, and the signal gains are determined by a normalized autocorrelation coefficient, a gradient index and subband spectrum flatness of the first audio signal;

the synthesizing module is used for synthesizing the M target high-frequency subband signals obtained by the processing module to obtain target audio signals;

wherein the first audio signal comprises at least one of: broadband audio, ultra-broadband audio, full-band audio, Y, M is a positive integer.

8. The apparatus of claim 7, wherein the device comprises a plurality of sensors,

the generating module is specifically configured to perform spectrum duplication on all low-frequency subband signals in the Y first subband signals to generate M high-frequency subband signals, where one low-frequency subband signal corresponds to at least one high-frequency subband signal, and Y is smaller than or equal to M.

9. The apparatus of claim 7, wherein the audio processing apparatus further comprises: the device comprises an extraction module and a prediction module;

the extraction module is used for extracting the characteristics of the first audio signal to obtain low-frequency characteristic information of the first audio signal;

the prediction module is used for inputting the low-frequency characteristic information extracted by the extraction module into a preset neural network model to predict the high-frequency characteristic information of the first audio signal.

10. The apparatus of claim 7, wherein the device comprises a plurality of sensors,

the processing module is specifically configured to perform L times up-sampling on the first audio signal to obtain a second audio signal with a predetermined sampling rate, where the bandwidth of the first audio signal is the same as that of the second audio signal.

11. The apparatus of claim 7, wherein the device comprises a plurality of sensors,

the processing module is further configured to frame the low-frequency component of the second audio signal to obtain X audio signal frames, where each audio signal frame includes a predetermined number of sample points;

the processing module is specifically configured to sequentially perform filtering and downsampling processing on each audio signal frame to obtain N first subband signals corresponding to each audio signal frame;

Wherein the Y first subband signals include: n first sub-band signals corresponding to each audio signal frame.

12. The apparatus of claim 11, wherein the device comprises a plurality of sensors,

the processing module is specifically configured to perform signal processing on N first subband signals of the first audio signal frame and N first subband signals in the second audio signal frame, so as to obtain N processed first subband signals; wherein the first audio signal frame and the second audio signal frame are adjacent ones of the X audio signal frames.

13. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the audio processing method according to any of claims 1-6.

14. A readable storage medium, characterized in that the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the steps of the audio processing method according to any of claims 1-6.