CN114627882A - Audio processing method, electronic device and computer readable storage medium - Google Patents

Abstract

The embodiments of the present application disclose an audio processing method, an electronic device, and a computer-readable storage medium, wherein the method includes: in response to an audio enhancement enabling instruction for target audio, acquiring a first audio to be played from the target audio; the The target audio is the audio in the playing state; the first audio is input into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model; the mode of the second audio is processed according to the high frequency band mode of the first audio and The phase of the second audio is modified according to the phase of the low frequency band of the first audio, so as to process the second audio into a third audio; the third audio is super high-quality audio; and the playback target audio is replaced with the playback of the third audio. With the embodiments of the present application, the audio being played can be spectrum-extended in real time, so as to improve the audio playback effect.

Description

Audio processing method, electronic device, and computer-readable storage medium

technical field

The present application relates to the technical field of audio processing, and in particular, to an audio processing method, an electronic device, and a computer-readable storage medium.

Background technique

With the development of terminal equipment, various applications emerge one after another. When using a terminal device to play songs, for some old songs or songs recorded by devices with poor recording effect, because these songs are not rich enough in audio signal components, they cannot provide users with the ultimate hearing. feelings, thereby affecting the user experience.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an audio processing method, an electronic device, and a computer-readable storage medium, which can perform spectrum expansion on the audio being played in real time, so as to improve the audio playback effect.

In a first aspect, an embodiment of the present application discloses an audio processing method, which is applied to a terminal device, where the terminal device includes an audio enhancement model, and the method includes:

In response to the audio enhancement opening instruction for the target audio, the first audio to be played is obtained from the target audio; the target audio is the audio in the playing state;

Input the first audio into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model;

The mode of the second audio is processed according to the high-band mode of the first audio and the phase of the second audio is modified according to the low-band phase of the first audio, so as to process the second audio into a third audio; the third audio for ultra-high quality audio;

Replace playing the target audio with playing the third audio.

In a second aspect, an embodiment of the present application discloses an audio processing device, the device comprising:

The acquisition module is used to obtain the first audio to be played from the target audio in response to the audio enhancement opening instruction for the target audio; the target audio is the audio in the playing state;

a processing module for inputting the first audio into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model;

The processing module is further configured to process the mode of the second audio frequency according to the high frequency band mode of the first audio frequency and modify the phase of the second audio frequency according to the low frequency band phase of the first audio frequency, so as to process the second audio frequency as the first audio frequency. Three audios; the third audio is super high quality audio;

The playback module is used to replace the playback target audio with the playback of the third audio.

In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the audio processing provided in the foregoing first aspect method.

In a fourth aspect, an embodiment of the present invention provides a computer storage medium, characterized in that the computer storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, is used to execute the above-mentioned first aspect. audio processing method.

In a fifth aspect, an embodiment of the present invention provides a computer program product or a computer program, the computer program product includes a computer program, and the computer program is stored in a computer storage medium; the processor of the computer device reads the computer from the computer storage medium. When the instruction is executed, the processor executes the audio processing method provided by the foregoing first aspect.

In this embodiment of the present invention, by receiving and responding to an audio enhancement enable instruction input for the target audio in the playing state, the terminal device can acquire the first audio to be played from the target audio according to the audio enhancement enable instruction, so as to convert the first audio The audio is input to the pre-trained audio enhancement model, the second audio output by the audio enhancement model is obtained, and the mode of the second audio is processed according to the high frequency band mode of the first audio frequency, and the phase pair of the low frequency band according to the first audio frequency is processed. The phase of the second audio is corrected, so that the second audio is processed into a super high-quality third audio, and the playback target audio is replaced with the third audio, so that the terminal device can play the audio in real time. The target audio is subjected to audio enhancement processing in real time, thereby obtaining an ultra-high-quality third audio, so as to improve the user's listening experience.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1A is a schematic diagram of a time-domain spectrum spreading provided by an embodiment of the present application;

1B is a schematic diagram of a frequency-domain spectrum spreading provided by an embodiment of the present application;

1C is a schematic diagram of a system architecture provided by an embodiment of the present application;

2 is a schematic flowchart of an audio processing method provided by an embodiment of the present application;

3 is a schematic diagram of a playback interface of a target audio provided by an embodiment of the present application;

4 is a schematic diagram of a playback interface of another target audio provided by an embodiment of the present application;

5 is a schematic diagram of a playback interface of another target audio provided by an embodiment of the present application;

6A is a schematic diagram of an audio setting interface corresponding to a target audio provided by an embodiment of the present application;

6B is a schematic diagram of a first audio frequency provided by an embodiment of the present application;

6C is a schematic diagram of another first audio frequency provided by an embodiment of the present application;

6D is a schematic diagram of still another first audio frequency provided by an embodiment of the present application;

6E is a schematic diagram of still another first audio frequency provided by an embodiment of the present application;

7A is a schematic diagram of a post-processing process flow of a high-frequency mode provided by an embodiment of the present application;

7B is a schematic flowchart of an improved Griffinlim algorithm provided by an embodiment of the present application;

8A is a schematic diagram of displaying a first audio spectrogram and a third audio spectrogram provided by an embodiment of the present application;

FIG. 8B is a schematic diagram showing the before and after comparison spectrum of the audio segment 1 provided by an embodiment of the present application;

FIG. 8C is another schematic diagram showing the before-and-after comparison spectrum of the audio segment 1 provided by an embodiment of the present application;

9 is a schematic flowchart of another method for generating an audio enhancement model provided by an embodiment of the present application;

10 is a schematic diagram of the internal structure of an encoding-decoding architecture provided by an embodiment of the present application;

FIG. 11 is a schematic flowchart of training through a generative adversarial network provided by an embodiment of the present application;

12 is a schematic diagram of a low-frequency band and a high-frequency band frequency point provided by an embodiment of the present application;

FIG. 13 is a schematic flowchart of a prediction by a generative adversarial network model provided by an embodiment of the present application;

14 is a schematic structural diagram of an audio processing apparatus provided by an embodiment of the present application;

FIG. 15 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

For ease of understanding, terms involved in this application are first introduced.

1. Super quality (super quality, SQ) sound quality

SQ sound quality can also be called lossless sound quality. Wherein, if the audio source is a compact disc (compact disc, CD) quality and audio with CD quality or higher as audio A, then audio A can be of SQ quality. Alternatively, if the audio obtained by audio A through a lossless encoder is denoted as audio B, then audio B can be of SQ quality. Alternatively, if audio A or audio B can still be stored in a lossless format after being encoded by a higher-quality lossy encoder, the encoded audio may be recorded as audio C, and audio C may be of SQ quality.

2. High quality (high quality, HQ) sound quality

HQ quality refers to audio in standard mp3 format (bit rate 320kbps) that has not been up-converted with a low bit rate. The main technical indicators for judging whether it is HQ sound quality are: (1) the highest tangent of the spectrum reaches more than 20K; (2) if there is attenuation near the 20K spectrum, the proportion of the spectrum higher than 20K must be greater than 25%; (3) if the spectrum is near 18K If there is attenuation, the proportion of spectrum higher than 18K should be greater than 45%. Among them, the spectrum of standard mp3 format (320kbps) can reach 20K.

3. Low quality (LQ) sound quality

If the audio obtained by sampling at a lower sampling rate (eg, below 28 kHz) is recorded as audio D; then audio D can be of LQ quality. Or, if the audio whose spectral height does not reach 14 kHz after being encoded by a lower bit rate encoder (such as LAME 64bits version 3.99.5 encoder at a bit rate of 80kbps and below) is recorded as audio E, then Audio E can be LQ sound quality.

4. Non-super high-quality sound quality

Non-Ultra High Quality sound quality can also be referred to as non-lossless sound quality. The non-ultra-high quality sound quality can be HQ sound quality or LQ sound quality. The main technical indicators for judging whether it is non-lossless sound quality can be summarized as: (1) the file format is compressed by a lossy encoder; (2) the file format is compressed by a lossless encoder, or uncompressed, but it can be found that after lossy compression If the secondary storage is in a lossless format, that is, there are obvious traces of lossy compression; among them, the above traces can be mainly described by the spectrum height and density: the spectrum reaches more than 20k (excluding 20k), and meets a certain ratio, such as More than 10% have effective energy.

5. Music spectrum spread

The music spectrum extension (Music Bandwidth Extension) technology may also be referred to as a music super-resolution (Music Super Resolution) technology. From the time domain, as shown in Figure 1A, deep neural network (DNN) technology can be used for time domain interpolation to introduce high-frequency details; from the frequency domain, as shown in Figure 1B, DNN can be used Repair technology to restore the lost high-frequency components.

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

In order to enhance the rhythm and somatosensory feedback of music and improve user experience, an embodiment of the present application provides an audio processing method. In order to better understand the audio processing method provided by the embodiments of the present application, a system architecture to which the audio processing method is applied is first introduced below.

Please refer to FIG. 1C , which is a schematic diagram of a system architecture provided by an embodiment of the present application, and the audio processing method proposed by the present application can be executed through the system architecture. The system architecture includes a terminal device 101 and a server 102 . The terminal device 101 and the server 102 are connected through wired or wireless communication. It should be noted that, the above-mentioned terminal device 101 may set an audio enhancement model, so as to process the audio being played through the audio enhancement model, so as to obtain ultra-high-quality audio.

The terminal device 101 is a device with wireless communication function, which can be a smart phone, a tablet computer, a smart wearable device, a personal computer, etc., in which an application client, such as audio playback software, can be run. In some embodiments of the present application, the terminal device may also be a device with a transceiving function, such as a chip system. Wherein, the chip system may include a chip, and may also include other discrete devices, which is not limited in this embodiment of the present application.

The terminal device 101 may acquire data, such as video, audio, text, and the like, from the server 102 . The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or may provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud Cloud servers for basic cloud computing services such as communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The database in the server 102 may be a local database of the server 102, or may be a cloud database accessible by the server 102, which is not limited in this application. It should be noted that, the above system architecture is described by taking the example of including one terminal device 101 and one server 102 , and the number of terminal devices and the number of servers does not constitute a limitation on this application.

The audio processing method provided by the embodiment of the present application is further described in detail below:

Please refer to FIG. 2, which is a schematic flowchart of an audio processing method provided by an embodiment of the present application. The audio processing method can be executed by a terminal device (such as the terminal device 101 shown in FIG. 1C above), the method can also be executed by a server, or executed jointly by the server and the terminal (for example, the server obtains the target audio played by the terminal, and the target After the first audio in the audio is enhanced in sound quality, a second audio is obtained, and a third audio with super high quality is determined according to the first audio and the second audio, so that the third audio is returned to the terminal for playback). For ease of understanding, the embodiment in FIG. 2 is described by taking the method being executed by a terminal device as an example. Wherein, the audio processing method may include at least the following steps S201-S204:

S201. Acquire a first audio to be played from the target audio in response to an audio enhancement enabling instruction for the target audio.

Wherein, the target audio is the audio in the playing state. The target audio may be non-ultra-high-quality audio, that is, non-lossless audio, such as HQ-quality or LQ-quality audio. The target audio can be songs, recordings, etc., which is not limited in this application. The above-mentioned target audio is in the playing state can be understood as the terminal device is playing the target audio, for example, a music application or a playback module (such as a radio or a recorder) of the terminal device is playing the target audio, and the present application uses the music application on the terminal device is playing the target audio. Audio is used as an example to illustrate.

The above audio enhancement enable instruction can be used to enable the audio processing flow for the target audio, that is, to perform steps S201 to S204 for the target audio. It should be noted that the user may input an audio enhancement enabling instruction for the target audio in the above music application, and correspondingly, the terminal device receives the enhancement enabling instruction.

The terminal device may receive an audio enhancement enable instruction for the target audio input on one of the following two interfaces. specific:

Interface 1: The playback interface of the target audio. That is, the terminal device may receive an audio enhancement enable instruction for input of the target audio on the playback interface of the target audio.

Wherein, the playing interface of the target audio may be an interface when the music application on the terminal device plays the target audio; as shown in FIG. 3 , FIG. 3 shows the playing interface of the target audio. In the playback interface shown in Figure 3, different sound quality and the corresponding file size of each sound quality can be listed below the target audio, such as: standard quality 2.0M, HQ quality 7.1M, SQ quality 18.5M, etc.; in each sound quality option An audio enhancement on button can be provided below the , which is used to start the audio enhancement processing process. Optionally, the user can select the above three different sound quality, so that the terminal device can play the target audio according to the sound quality selected by the user. Optionally, the user can also set to automatically select the sound quality, so that the terminal device can automatically select the currently applicable sound quality according to the current network conditions and other factors. As shown in Figure 3, before the audio enhancement button is turned on, the target audio can be played in HQ sound quality. Optionally, the sound quality option and the audio enhancement enable button are located below the target audio only for example, and do not constitute a limitation to the present application.

It should be noted that, in the target audio playback interface as shown in FIG. 3 , the user can click the audio enhancement activation button for the target audio to activate the audio enhancement processing flow for the target audio, that is, perform steps S201 to S204.

Interface 2: The audio setting interface corresponding to the target audio. That is, the terminal device may receive an audio enhancement enabling instruction for the target audio input on the audio setting interface corresponding to the target audio.

It should be noted that, in the case where there is no audio enhancement enable button on the playback interface of the target audio, the terminal device can jump to the audio setting interface corresponding to the target audio by performing a click operation on the playback interface of the target audio. As shown in FIG. 4, there is no audio enhancement enable button in the playback interface of the target audio in FIG. 4, then the terminal device can jump to the audio setting interface corresponding to the target audio after receiving the click operation for the audio enhancement button Or, as shown in Figure 5, the playback interface of the target audio in Figure 5 does not have an audio enhancement open button, then the terminal device can be through receiving the audio setting operation for the target audio input (such as by clicking the right button, from the menu Select the audio setting button) to jump to the audio setting interface corresponding to the target audio, which is not limited in this application.

As shown in FIG. 6A , FIG. 6A shows an audio setting interface corresponding to the target audio. In the audio setting interface corresponding to the target audio shown in FIG. 6A , the user can click the audio enhancement enable button to start the audio enhancement process for the target audio.

When playing the target audio, the terminal device may receive an audio enhancement start instruction input by the user for the target audio. That is, when playing the target audio, the terminal device may receive an audio enhancement enable instruction input by the user for the target audio, so as to perform spectrum expansion on the target audio, thereby improving the audio quality and further improving the user experience.

Wherein, in order to distinguish it from the audio after the sound quality enhancement, the audio before the sound quality enhancement may be called the first audio. The first audio may be the audio obtained by the terminal device from the target audio. As shown in FIG. 6B , the first audio may be a short duration of the audio obtained from the target audio (the first audio is obtained according to the preset duration). ); Optionally, as shown in Figure 6C, this first audio frequency can also be a longer audio frequency (from the time point when the audio enhancement on instruction is received to the end time point of the target audio frequency) obtained from the target audio frequency Obtain the first audio), which is not limited in this application.

It should be noted that when receiving the audio enhancement turn-on instruction, the terminal device may take the time point at which the audio enhancement turn on instruction is received as the start time point, so as to obtain the first audio frequency of the preset duration from the start time point. . As shown in FIG. 6B or FIG. 6C above, if the target audio has already played 30 seconds of audio data when the terminal device receives the audio enhancement enable instruction, the terminal device can use the 30th second as the starting time point to start from the 30th second. Seconds start to acquire the above-mentioned first audio.

In an optional implementation manner, the start time point of acquiring the first audio is the time point when the audio enhancement on instruction is received, and the end time point is the end time point of the target audio. In this case, if the preset duration for acquiring the first audio is shorter than the remaining playing duration of the target audio, multiple first audios may be acquired from the target audio. In addition, since the durations of different audios may be different, the number of first audios acquired by the terminal device from different audios may also be different. For example: Assuming that the total duration of audio 1 is 3 minutes, if the duration of the first audio is 5 seconds, the terminal device can obtain up to 36 first audios from the audio 1; assuming that the total duration of audio 2 is 1 minute, If the duration of the first audio is 5 seconds, the terminal device can acquire at most 12 first audios from the audio 2 .

Optionally, for different audios, the duration of the first audio acquired by the terminal device may also be different. For example: assuming that the total duration of audio 3 is 3 minutes, for audio 3, if the duration of the first audio acquired by the terminal device is 5 seconds, then a maximum of 36 first audios can be acquired from audio 3; assuming audio 4 The total duration is 3 minutes. For this audio 4, if the duration of the first audio acquired by the terminal device is 10 seconds, a maximum of 18 first audios can be acquired from the audio 4. This application does not limit the number of first audios acquired by the terminal device.

In an implementation manner, the above-mentioned first audio may include one or more audio segments. As shown in FIG. 6D, the first audio may include one audio segment; optionally, as shown in FIG. 6E, the first audio may also include multiple audio segments, which is not limited in this application.

It should be noted that, in the case where the first audio includes one audio segment, the number of the first audio acquired by the terminal device may be the same as the number of audio segments. As shown in FIG. 6D, the total duration of the target audio is 3 minutes, and the first audio (for example, called first audio 1) includes an audio clip, if the duration of the audio clip is 5 seconds, the target audio has been played When the audio data is 30 seconds, the terminal device can obtain 30 audio segments (ie, 30 first audio 1s) from the target audio.

Optionally, when the first audio includes multiple audio segments, the number of the first audio acquired by the terminal device may be different from the number of audio segments. As shown in FIG. 6E , the total duration of the target audio is 3 minutes, the first audio (for example, called first audio 2 ) includes all audio data after the 30th second of the target audio, and the first audio 2 includes multiple audios segment, if the duration of the audio segment is 5 seconds, when the target audio has played 30 seconds of audio data, the terminal device can obtain a first audio 2 from the target audio, and the first audio 2 includes 30 audio clips.

It should be noted that an audio segment may include the minimum audio data amount required by the terminal device to perform audio enhancement processing, so that the terminal device can perform audio enhancement processing after acquiring the audio segment. For example, if the minimum amount of audio data that can be performed by the terminal device for audio enhancement processing is audio with a duration of 5 milliseconds, one audio segment may include at least audio with a duration of 5 milliseconds. Wherein, the audio with a minimum audio data volume of 5 milliseconds in audio enhancement processing is only used as an example, and does not constitute a limitation to the present application.

The above audio segment may also be referred to as audio data of unit duration, that is, the first audio may include one or more audio data of unit duration. Optionally, the above-mentioned one audio segment may also be referred to as one batch-sized audio data. In the embodiments of the present application, audio clips are used as examples for description, which does not constitute a limitation on the present application. Optionally, the above-mentioned audio clip may also be used as a unit of frame, which is not limited in this application.

After receiving the audio enhancement turn-on instruction, the terminal device may obtain one or more audio clips included in the first audio from the target audio according to the audio enhancement turn-on instruction, so that the target audio with a longer duration being played is changed to It is divided into one or more audio clips with a short duration, so that after an audio clip is acquired, the audio enhancement processing can be started quickly, and the processed audio clip can be obtained more quickly.

S202. Input the first audio into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model.

Wherein, the above-mentioned second audio may be the audio output by the audio enhancement model. The second audio may be audio obtained by subjecting the first audio to audio enhancement processing, and the second audio may contain more signal components than the first audio. It should be noted that the second audio may correspond to the first audio, that is, if the first audio includes an audio segment, the second audio also includes an audio segment; if the first audio includes multiple audio segments, Then the second audio also includes a plurality of audio segments.

In an implementation manner, the above audio enhancement model is a model obtained by training audio samples through a generative adversarial network (Generative Adversarial Networks, GAN), and the audio samples are ultra-high-quality audio. How to generate an audio enhancement model will be described in detail in the embodiment shown in FIG. 9 , which will not be repeated here.

S203, processing the mode of the second audio according to the high frequency band mode of the first audio and modifying the phase of the second audio according to the low frequency band phase of the first audio, so as to process the second audio into a third audio; Tri-audio is super high quality audio.

Wherein, the above-mentioned third audio frequency may be an audio frequency obtained by performing high-frequency modulo post-processing and phase correction based on the above-mentioned first audio frequency and the above-mentioned second audio frequency. That is to say, after the modulo and phase optimization of the second audio output from the audio enhancement model is performed, the third audio can be obtained.

In an implementation manner, based on the high frequency band mode of the first audio frequency and the high frequency band mode of the second audio frequency, the mode of the second audio frequency is subjected to high frequency mode post-processing to obtain the full frequency band mode of the second audio frequency; The phase of the second audio frequency is phase-corrected to obtain the full-band phase of the second audio frequency; the second audio frequency is processed based on the full-band mode and the full-band phase to determine the third audio frequency.

For the post-processing flow of the high-frequency mode, reference may be made to the schematic diagram shown in FIG. 7A . Specifically, the terminal device can compare the high frequency band mode of the first audio frequency with the high frequency band mode of the second audio frequency to determine the high frequency band mode with more high frequency frequency point energy, so as to use the high frequency band mode with more energy. The band mode is post-processed as the high frequency band mode of the high frequency mode. For example, if the energy of the high frequency band mode of the first audio frequency is higher than the energy of the high frequency band mode of the second audio frequency, the high frequency band mode of the first audio frequency is used as the post-processing high frequency band mode of the high frequency mode. Through the post-processing process of the high-frequency mode, the original high-frequency frequency points can be preserved as much as possible when the energy of the original high-frequency frequency points is larger, so as not to damage the characteristics of the original audio. Optionally, the above-mentioned post-processing flow of the high-frequency mode is similar to the post-processing flow of the image mode, and details are not repeated here.

In an implementation manner, the low-band phase of the first audio frequency is subjected to image processing to obtain the image phase; the image phase is calculated by using a speech signal reconstruction algorithm to obtain the calculated phase; and the second audio frequency is calculated according to the calculated phase. Phase correction is performed on the phase of the second audio frequency to obtain the full-band phase of the second audio frequency.

The above speech signal reconstruction algorithm may be an improved Griffinlim algorithm, and a specific process of the improved Griffinlim algorithm may be shown in FIG. 7B . Specifically, in this embodiment of the present application, the phase of the low frequency band of the first audio frequency can be mirrored, so that the phase after mirroring can be used as the initial phase in the improved Griffinlim algorithm, so that the calculated phase is more accurate .

It should be noted that the number of iterations of the phase correction processing for the second audio frequency can be set manually. Although the number of iterations is more, the result can be more accurate, but in order to reduce the calculation amount of the terminal device, it can generally be set to 1 iteration or 1. 2 times is sufficient, which is not limited in this application. Optionally, for other steps corresponding to the improved Griffinlim algorithm, reference may be made to the relevant steps of the Griffinlim algorithm, which will not be repeated in this application.

The terminal device can determine the third audio frequency according to the second audio frequency, the full-band mode after post-processing of the high-frequency mode, and the full-band phase after phase correction by the improved Griffinlim algorithm. Specifically, the terminal device can combine the above-mentioned second audio frequency, full-band mode and full-band phase based on Euler's formula, and recover the time-domain signal through Inverse Short-time Fourier Transform (ISTFT). There are no restrictions on the application. By optimizing the modulo and phase of the second audio, details in the processed third audio can be made more accurate, thereby improving the user's listening experience.

It should be noted that, after obtaining the third audio, the terminal device may buffer the third audio for the terminal device to obtain and play the third audio. Optionally, the terminal device may cache the third audio in the memory of the terminal device, or may cache the third audio in another storage space, which is not limited in this application.

In an implementation manner, the terminal device may display the frequency spectrum of the first audio frequency and the third audio frequency on the audio setting interface.

It should be noted that, in order to simplify the description, “the third audio is super high-quality audio after audio enhancement processing” is used as an example for description, and the “audio enhancement processing” can mean “audio enhancement processing and modulus and phase optimization.” processing", which does not constitute a limitation on this application.

As can be seen from the foregoing content, the first audio may be audio before audio enhancement processing, and the third audio may be ultra-high-quality audio after audio enhancement processing. As shown in FIG. 8A , after the enhancement processing button is turned on in the audio setting interface, the terminal device can also set the spectrogram of the audio before audio enhancement (that is, the above-mentioned first audio) and the audio after the audio enhancement (that is, the above-mentioned audio) on the audio setting interface. The spectrogram of the third audio) is displayed to show the user the before and after contrast effect of turning on the audio enhancement. Wherein, the placement positions of the first audio frequency and the third audio frequency spectrogram in FIG. 8A are only used for example, and do not constitute a limitation to the present application.

Optionally, the above-mentioned before-and-after comparison effect diagram of audio enhancement may be exemplary, for example, the spectrum height before audio enhancement is enabled may be a spectrum of about 12K, and the spectrum height after audio enhancement is enabled may be a spectrum of about 22K. It can be understood that the legends corresponding to different audios may be the same or different, so as to show a general effect to the user, and do not represent the actual frequency spectra of the first audio and the third audio.

Optionally, the above-mentioned before-and-after comparison effect diagram of audio enhancement may be a legend actually generated during audio enhancement processing. For example, the spectrogram before audio enhancement is turned on may be the frequency spectrum of the first audio being played, and the frequency spectrum after audio enhancement is turned on. The graph may be the frequency spectrum of the third audio that has undergone audio enhancement processing. It can be understood that the before-and-after comparison effect graphs corresponding to different audios may be different, so as to show the frequency spectrum of the first audio and the third audio in the actual processing process to the user.

In an implementation manner, the terminal device may display the first frequency spectrum and the second frequency spectrum of the target audio segment for the target audio segment currently being processed by the audio enhancement model; the first frequency spectrum is the frequency spectrum of the target audio segment before the audio enhancement model is input , the second frequency spectrum is the frequency spectrum of the target segment after the audio enhancement model is input, and the target audio segment is the audio segment being processed among the multiple audio segments.

Wherein, the above-mentioned first frequency spectrum may be the frequency spectrum of the target audio clip before the audio enhancement processing (that is, before the above-mentioned input audio enhancement model), and the above-mentioned second frequency spectrum may be the target audio clip after the audio enhancement processing (i.e. after the above-mentioned input audio enhancement model). ), which is not limited in this application.

It should be noted that, when the first audio includes multiple audio segments (ie, multiple audio segments), the above-mentioned before-and-after comparison effect diagram may be used to show the frequency spectrum of the audio segment being processed. For example, as shown in FIG. 8B , assuming that the first audio includes audio segment 1, audio segment 2 and audio segment 3, and the terminal device is performing audio enhancement processing on audio segment 1 (that is, the above-mentioned target audio segment), then compare the effect diagrams before and after A before and after spectrum when audio segment 1 is processed can be shown. Optionally, if the terminal device starts to process audio segment 2 (ie, the above-mentioned target audio segment), the before-and-after comparison effect diagram may show the before-and-after comparison spectrum when audio segment 2 is processed, which is not limited in this application.

Optionally, assuming that the first audio includes audio clip 1, audio clip 2, and audio clip 3, the before-and-after comparison effect diagram may first show the before-and-after comparison spectrum of audio clip 1, and then show the before-and-after comparison spectrum of audio clip 2, and finally show the before and after comparison spectrum of audio clip 2. The before-and-after comparison spectrums of the audio segment 3 are output, so as to display the before-and-after comparison spectrums of all audio segments included in the first audio. As shown in FIG. 8C , the figure exemplarily shows the before-and-after comparison spectrum of the audio segment 1 .

S204. Replace the playback target audio with playback of the third audio.

After acquiring the third audio, the terminal device can play the third audio for the user. Optionally, the terminal device may suspend the playback of the first audio when acquiring the first audio, and buffer and play the third audio in time when acquiring the third audio, so as to improve the user's listening experience. Wherein, the interval from pausing the first audio to playing the third audio is shorter, that is, the time for inputting the first audio into the audio enhancement model for processing is shorter, and the audio enhancement model can quickly process one or more audios included in the first audio Fragments to reduce user wait time.

In an implementation manner, the terminal device deletes the buffered third audio when the third audio is played.

As can be seen from the foregoing content, the third audio may include one or more audio clips. When playing the third audio, the terminal device may play the audio clips in sequence, and delete the buffer of the audio clip after playing one audio clip. Exemplarily, assuming that the third audio includes multiple audio clips, such as audio clip 4, audio clip 5, and audio clip 6, and audio clip 4 is played, the terminal device may delete the buffered audio clip 4. Optionally, the terminal device may also delete the cached second audio after the second audio is played, that is, after the audio segment 4, the audio segment 5 and the audio segment 6 are all played, which is not limited in this application. By deleting the audio clips that have been played, the cache can be emptied, thereby freeing up storage space to store audio clips that the terminal device performs subsequent audio enhancement processing.

In an implementation manner, the terminal device deletes the buffered third audio when receiving a playback progress bar dragging instruction or an audio switching instruction input for the target audio.

Among them, the playback progress bar drag command can be used to drag the playback progress of the audio, such as dragging the audio to 1 minute and 30 seconds; the audio switching command can be used to switch the audio being played, such as switching the audio being played 1 to Audio 3.

Since the terminal device performs audio enhancement processing on the currently playing audio in real time, when the terminal device receives the playback progress bar dragging instruction or audio switching instruction input for the target audio, the terminal device needs to perform audio enhancement processing. The audio has changed; that is, the audio that has been previously processed and cached will no longer be played, so the terminal device can delete the cached audio to clear the cache, and then use the cache to store the playback progress bar drag command or audio After switching the command, and after audio enhancement processing.

In an implementation manner, the terminal device stops inputting the audio in the target audio into the audio enhancement model when receiving the audio enhancement closing instruction for the target audio input.

The audio enhancement close instruction may be used to close the audio enhancement processing flow. When the terminal device receives the audio enhancement closing instruction for the target audio input, it can stop the audio enhancement processing for the target audio, that is, stop inputting the audio in the target audio (such as the first audio) into the audio enhancement model, and then end the above-mentioned audio enhancement. process.

It should be noted that, in the case where the audio enhancement processing effect for the target audio is not ideal, or in the case that the audio enhancement processing for the target audio fails, the terminal device can receive the above-mentioned music enhancement close instruction to reduce the energy consumption of the terminal device. . It can be understood that, when the terminal device receives the instruction to turn off the audio enhancement, the target audio will be played in the original sound quality.

It should also be noted that the above-mentioned target audio may also be ultra-high-quality audio, that is, lossless-quality audio, such as SQ-quality audio. By performing audio enhancement processing on the ultra-high-quality target audio, the terminal device can make up for the lack of high frequency of the target audio, so that the audio after audio enhancement has a louder sound and richer details.

In the embodiment of the present application, the terminal device may obtain the first audio from the target audio according to the audio enhancement enabling instruction by receiving the audio enhancement enabling instruction input for the audio in the playing state (such as the target audio), so as to input the first audio The audio enhancement model obtains the audio output by the audio enhancement model (such as the above-mentioned second audio), and determines the third audio after modulo and phase optimization according to the first audio and the second audio, to buffer and play the third audio, Therefore, the terminal device can perform audio enhancement processing on the audio in the real-time playback state of the audio, thereby obtaining ultra-high-quality audio, so as to improve the user's listening experience.

Please refer to FIG. 9. FIG. 9 is a schematic flowchart of a method for generating an audio enhancement model provided by an embodiment of the present application. The method for generating an audio enhancement model may be executed by a terminal device (the terminal device may be the terminal device that executes the embodiment shown in FIG. 2, or may be other terminal devices) or a server. Assuming that the embodiment shown in FIG. 2 is executed by the terminal device 1 , the audio enhancement model generation method is executed by the terminal device 2 or the server, and the audio enhancement model generated by the terminal device 2 or the server can be deployed in the terminal device 1 . For ease of understanding and differentiation, the embodiment shown in FIG. 9 is described by taking the method being executed by the terminal device 2 as an example. The method for generating an audio enhancement model may include at least the following steps S901-S902:

S901. Generate an initial architecture of an audio enhancement model.

The initial architecture of the audio enhancement model may include one or more algorithms required by the audio enhancement model. Since the audio enhancement model can learn the mapping relationship of the high frequency band from the low frequency band, an encoder-decoder network model can be used as the initial architecture of the audio enhancement model.

Optionally, FIG. 10 is a schematic diagram of the internal structure of the encoder-decoder architecture. As shown in Figure 10, the encoder-decoder architecture can include lightweight algorithms such as Depthwise Separable Convolution (DWconv2D), SplitSRBlock and Sub-pixel convolution (SubPixel2D). , the specific calculation process of several lightweight algorithms will be briefly described below.

1. DWconv2D

Among them, DWconv2D is a combination of depthwise (Depthwise, DW) convolution and pointwise (Pointwise, PW) convolution. Depthwise convolution is different from conventional convolution operations. One convolution kernel of depthwise convolution is responsible for one channel, and one channel is only convolved by one convolution kernel; while each convolution kernel of conventional convolution operates on each input simultaneously. aisle. Taking image processing as an example, for a 5 × 5 pixel, three-channel color input image (that is, the size of the input image is 5 × 5 × 3), the depth convolution can be operated in a two-dimensional plane, and its volume The number of product kernels is the same as the number of channels in the previous layer (one-to-one correspondence between channels and convolution kernels); therefore, a three-channel image can generate three feature images after operations.

The operation of point-by-point convolution is very similar to the regular convolution operation, the difference is that the size of the convolution kernel of point-by-point convolution is 1×1×M, and M is the depth of the previous layer. Therefore, through the point-by-point convolution operation, the images of the previous step can be weighted in the depth direction to generate a new feature image. In pointwise convolution, the number of feature images is the same as the number of filters.

2. SplitSRBlock

Further optimized on the basis of DWconv2D, a new end-to-end mobile phone-side super-resolution system, SplitSR, is proposed. The split convolution divides the input features along the depth channel according to a certain ratio (the ratio is adjustable) to reduce the amount of computation and memory consumption. , thereby speeding up the inference process. Specifically, the input features can be separated along the depth channel first, some of them participate in the DWconv2D calculation, and some do not participate in any calculation (ie, feature retention); then the features of the two parts can be combined and spliced according to the depth channel. Through this algorithm, the amount of computation can be reduced, and some features can also be retained in the next layer, so that the decoding layer can also obtain more primary features.

3. SubPixel2D

SubPixel2D is an algorithm that combines upsampling (upsample) and convolution (consvolution) operations. The algorithm can act on low-resolution features, so that the low-resolution features can obtain high-resolution features through the algorithm. This algorithm reduces the risk of introducing too many factors when using deconvolution as an upsampling method. SubPixel2D can rearrange each channel of each pixel into an r*r region to correspond to a sub-block of size r*r in the high-resolution image, that is, a feature image of size 1*H*W can be Rearranged into high-resolution images of size 1*rH*rW. If the four-dimensional vector reorganizes the feature size, it can be rearranged from [B, H, W, r*r*C] to [B, rH, rW, C]. Although this algorithm is called sub-pixel convolution, it does not actually have a convolution operation.

In the embodiment of the present application, by using a simplified encoder-decoder network model to generate the initial architecture of the audio enhancement model, the number of network layers and the number of input frames can be reduced; wherein, the encoder-decoder network model uses a special encoding (encoder) network Structural unit to reduce the amount of calculation and model size; and use a special decoder module for upsampling to reduce the memory usage unit, so that when the terminal device 1 runs the audio enhancement model, it can run in real time and occupy extremely low Central Processing Unit (CPU) resource and memory consumption.

S902 , training audio samples through a generative confrontation network GAN to obtain an audio enhancement model. The above audio samples may be ultra-high-quality audio.

It should be noted that the above-mentioned GAN training can include a generator (Generator) and a discriminator (Discriminator), and through GAN training, the predicted audio generated by the generator can not be distinguished from true and false. Among them, the generator can use the lightweight encoder-decoder architecture mentioned in the above steps, and the discriminator can use the two-class network model structure (such as VGG-like).

As shown in Figure 11, Figure 11 shows a schematic diagram of the flow of training through GAN. Specifically, the terminal device 2 can obtain a fourth audio from the audio sample, and the fourth audio is a low-band audio of the audio sample; and input the fourth audio into the encoder-decoder architecture to obtain a fifth audio; the fifth audio The audio can be the audio obtained by GAN training.

It should be noted that when acquiring the audio sample, the terminal device 2 can extract the Short-time Fourier Transform (STFT) feature of the audio sample, take the modulo of the STFT feature, and then take the logarithm, to get the logarithm of the audio sample; where fft_length=2048, hop_length=256. The logarithmic size of the audio sample can be expressed as [T, 1024], where T can be the length of a feature sequence (ie, the above-mentioned first audio), and the size of the T value is related to the duration of the audio sample. Optionally, if the modulo logarithm is converted into a fixed frame length of 32 frames, the modulo logarithmic size of the audio sample can be expressed as [X, 32, 1024], where X=T/32.

Since the lightweight DWconv2D is used as the convolution operation unit in the encoder-decoder architecture, a four-dimensional input and output can be constructed. For example, if the second dimension is extended to 1 by default, the logarithmic size of the above audio sample can be expressed as [ X, 1, 32, 464], that is, the input to the model during the training process can be [X, 1, 32, 464], and each value can be expressed as [batch size, fixed value 1, frame length, low-band points]. Wherein, the number of low-band points 464 may correspond to a spectrum height of 10K, and the calculation method may be: 464=2048*10K (spectrum height)/44.1K (sampling rate). It can be understood that the output of the model during the training process can be [X, 1, 32, 580], and each value can be expressed as [batch size, fixed value 1, frame length, high frequency band points]. Among them, the calculation method of the number of high-frequency frequency points 580 may be: 580=(1024-464)+20; wherein, 1024 may correspond to the full frequency band of 44.1K sampling rate and 22.05K spectral height, and 20 may correspond to the overlap area (overlap) The number of frequency points. This value is not fixed and can be adjusted to prevent sudden changes in the calculation process. Referring to FIG. 12 , FIG. 12 shows a schematic diagram of the frequency points of the low frequency band and the high frequency band.

Optionally, the loss functions of Generator and Discriminator in GAN can be:

可以表示判别器的损失函数；E可以表示交叉熵损失函数；x可以表示音频样本，D可以表示判别器，D(x)可以表示将音频样本置于判别器进行判别；z可以表示音频增强模型的输入，该输入为音频样本的低频带音频；G可以表示生成器，G(z)可以表示将输入音频增强模型的低频带音频进行预测生成，该生成结果为全频带音频；D(G(z))可以表示将生成的全频带音频至于判别器进行判别；

可以表示生成器的损失函数；可以先训练D(判别器)，再训练G (生成器)，两者相互对抗，直至收敛。需要说明的是，除了使用上述损失函数外，还可以额外引入两个损失函数以加强低频带特征到高频带特征的预测，如下所示，总的L_G损失函数可以为：in,

It can represent the loss function of the discriminator; E can represent the cross entropy loss function; x can represent the audio sample, D can represent the discriminator, D(x) can represent the audio sample placed in the discriminator for discrimination; z can represent the audio enhancement model The input is the low-band audio of the audio sample; G can represent the generator, and G(z) can represent the prediction and generation of the low-band audio of the input audio enhancement model, and the generation result is the full-band audio; D(G( z)) can indicate that the generated full-band audio is judged by the discriminator;

The loss function of the generator can be represented; D (discriminator) can be trained first, then G (generator) can be trained, and the two will fight against each other until convergence. It should be noted that, in addition to the above loss functions, two additional loss functions can be introduced to enhance the prediction of low-band features to high-band features. As shown below, the total _LG loss function can be:

Among them, L _G can represent the total loss function, L _LSD can represent the Least Significant Difference (LSD) loss function, which can also be called L2 loss; L _l1pixcel can represent L1 loss, which can also be called L1 norm loss ; the above-mentioned λ ₁ and λ ₂ can represent the weight hyperparameters, which can generally be set to 10 and 0.1 respectively; L can represent time (that is, the horizontal axis in the spectrogram), and K can represent the frequency point (that is, the vertical axis in the spectrogram). axis), X ^HR can represent the high frequency band, X ^HR (l, k) can represent the location of a specific frequency point in the high frequency band, X ^SR can represent the generated high frequency band, and X ^SR (l, k) can represent the generated high frequency band. The location of the specific frequency point in the high frequency band. It can be seen that the L _LSD loss function can obtain the error by taking the square of the difference between the target value (high frequency band) and the model output value (the generated high frequency band) and then taking the square root; the L _l1pixcel loss function can be obtained by taking the target value ( The difference between the high frequency band) and the model output value (the generated high frequency band) is calculated by the absolute value calculation to obtain the error.

In the embodiment of the present application, by using the GAN training model instead of directly using the general generative model (such as an auto encoder) training method, the model can be trained more fully, so that more high frequencies can be learned from the low frequency band The details of the high-frequency bands can be generated to obtain more high-frequency details, thereby restoring the real high-frequency features more realistically.

Optionally, when the terminal device 2 performs prediction generation by using the trained audio enhancement model, it may perform post-processing on the modulus and phase of the audio-enhanced audio. Specifically, as shown in Figure 13, input 44.1K sampling rate audio (such as the above audio samples), then calculate the STFT feature to obtain the corresponding modulus and phase, take the logarithm of the modulus to obtain the logarithmic modulus, and intercept the logarithmic modulus to the spectrum The height is 10K (such as the fourth audio frequency above), input the trained generator and the high-frequency modulo post-processing module, and a full-band modulo can be obtained. The generated full-band modulo and the low-band phase mirror are used to obtain the full-band phase, and the Euler formula is used to obtain the full-band phase. , and using ISTFT, the 44.1K sampling rate time-domain waveform can be obtained, which is further sent to the improved Griffinlim algorithm to correct the phase, and the time-domain waveform after modulo and phase post-processing (ie, the predicted audio in Figure 13) can be obtained.

It should be noted that when the prediction generation is performed by the trained audio enhancement model, the input audio sample can be ultra-high-quality audio (that is, the audio with a sampling rate of 44.1K in Figure 13), and the audio sample is processed into audio After enhancement processing (including modulo and phase optimization processing), the resulting audio can still be predicted ultra-high-quality audio (ie, audio with a sampling rate of 44.1K). The predicted audio can make up for the lack of high frequencies of the audio samples in the frequency domain, which can make the audio samples fluctuate faster in the time domain, thereby making the predicted audio sound louder and more detailed.

Optionally, when performing prediction generation through the audio enhancement model, for the high-frequency modulo post-processing flow and the phase correction processing flow, please refer to the detailed description of the high-frequency modulo post-processing and phase correction corresponding to S203 in the corresponding embodiment of FIG. 2 . The application will not be repeated here.

In the embodiment of the present application, by deploying a lightweight algorithm when generating the initial architecture of the audio enhancement model, the amount of calculation and the size of the model can be reduced; and the audio enhancement model can be obtained by training ultra-high-quality audio samples through GAN, thereby Learn from the low frequency band to learn more details of the high frequency band; then optimize the modulo and phase of the audio predicted by the audio enhancement model, which can make the prediction result more accurate, thereby improving the audio playback effect.

Based on the above audio processing method, an embodiment of the present invention provides an audio processing apparatus. Please refer to FIG. 14 , which is a schematic structural diagram of an audio processing apparatus provided by an embodiment of the present invention. The audio processing apparatus 1400 may run the following units:

Obtaining unit 1401 is used to respond to the audio enhancement opening instruction for the target audio, and obtain the first audio to be played from the target audio; the target audio is the audio in the playing state;

processing unit 1402, for inputting the first audio into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model;

The processing unit 1402 is further configured to process the modulo of the second audio frequency according to the high frequency band mode of the first audio frequency and modify the phase of the second audio frequency according to the low frequency band phase of the first audio frequency, so as to process the second audio frequency as tertiary audio; the tertiary audio is super high quality audio;

The playing unit 1403 is configured to replace the playing target audio with playing the third audio.

In one embodiment, the above audio processing apparatus further includes a determining unit 1404 . The above-mentioned processing unit 1402 is further configured to perform high-frequency post-processing on the second audio frequency mode based on the high frequency band mode of the first audio frequency and the high frequency frequency band mode of the second audio frequency to obtain the full frequency band mode of the second audio frequency; the above processing unit 1402, also used for performing phase correction on the phase of the second audio frequency according to the low frequency band phase of the first audio frequency to obtain the full frequency band phase of the second audio frequency; the determining unit 1404 is used for all frequency bands based on the full frequency band mode and the full frequency band phase. The second audio is processed to determine the third audio.

In one embodiment, the above-mentioned processing unit 1402 is further configured to perform mirror image processing on the low-band phase of the first audio frequency to obtain the mirror image phase; the above-mentioned processing unit 1402 is also used to perform an operation on the mirror image phase by using a speech signal reconstruction algorithm to obtain: Calculated phase; the above-mentioned processing unit 1402 is further configured to perform phase correction on the phase of the second audio frequency according to the calculated phase to obtain the full-band phase of the second audio frequency.

In one embodiment, the above-mentioned processing unit 1402 is further configured to delete the buffered third audio when receiving a drag instruction or a switching instruction input for the target audio.

In one embodiment, the above-mentioned processing unit 1402 is further configured to delete the buffered third audio when the third audio is played.

In one embodiment, the above-mentioned processing unit 1402 is further configured to stop inputting the audio in the target audio into the audio enhancement model when receiving the audio enhancement closing instruction for the target audio input.

In one embodiment, the above audio processing apparatus further includes a communication unit 1405 . The communication unit 1405 is configured to receive, on the target audio playback interface, an audio enhancement enabling instruction for the target audio input.

In one embodiment, the above-mentioned communication unit 1405 is further configured to receive, on the audio setting interface corresponding to the target audio, an audio enhancement enabling instruction for input of the target audio.

In one embodiment, the above audio processing apparatus further includes a display unit 1406 . The display unit 1406 is used for displaying the frequency spectrum of the first audio and the third audio in the audio setting interface.

In one embodiment, the above-mentioned display unit 1406 is further configured to display the first frequency spectrum and the second frequency spectrum of the target audio segment for the target audio segment currently being processed by the audio enhancement model; the first frequency spectrum is the input audio enhancement for the target audio segment. The frequency spectrum before the model, the second frequency spectrum is the frequency spectrum after the target segment is input to the audio enhancement model, and the target audio segment is the audio segment being processed among the multiple audio segments.

In one embodiment, the above-mentioned audio enhancement model is a model obtained by training audio samples through a generative adversarial network, and the audio samples are ultra-high-quality audio.

According to an embodiment of the present invention, each step involved in the audio processing method shown in FIG. 2 may be performed by each unit in the audio processing apparatus shown in FIG. 14 . For example, step S201 in FIG. 2 can be performed by the acquiring unit 1401 in the audio processing apparatus 1400 shown in FIG. 14 , step S202 can be performed by the processing unit 1402 in the audio processing apparatus 1400 shown in FIG. 14 , and step S204 can be performed by FIG. 14 The playback unit 1403 in the shown audio processing apparatus 1400 performs and so on.

According to another embodiment of the present invention, each unit in the audio processing apparatus shown in FIG. 14 may be respectively or all combined into one or several other units to form, or some of the unit(s) may be disassembled. It is divided into a plurality of units with smaller functions, which can realize the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit may also be implemented by multiple units, or the functions of multiple units may be implemented by one unit. In other embodiments of the present invention, the audio-based processing apparatus may also include other units. In practical applications, these functions may also be implemented with the assistance of other units, and may be implemented by cooperation of multiple units.

According to another embodiment of the present invention, a general-purpose computing device, such as a computer, may be implemented using processing elements such as a central processing unit (CPU), random access storage medium (RAM), read-only storage medium (ROM), etc., and storage elements. The computer program (including program code) capable of executing the steps involved in the corresponding method as shown in FIG. 2 is executed to construct the audio processing apparatus as shown in FIG. 14 , and to implement the audio processing method of the embodiment of the present invention. The computer program can be recorded on, for example, a computer storage medium, loaded into the above-mentioned computing device through the computer storage medium, and executed therein.

To sum up, the terminal device can respond to the audio enhancement enabling instruction by receiving the audio enhancement enabling instruction input for the audio in the playing state (such as the target audio), and obtain the first audio to be played from the target audio, so as to convert the first audio The audio is input to the pre-trained audio enhancement model, the audio output by the audio enhancement model (such as the second audio) is obtained, and according to the first audio and the second audio, an ultra-high-quality third audio is determined to play the third audio, Therefore, the terminal device can perform audio enhancement processing on the audio in the real-time playback state of the audio, thereby obtaining ultra-high-quality audio, so as to improve the user's listening experience.

Based on the above embodiments of the audio processing method and the audio processing apparatus, an embodiment of the present invention further provides an electronic device, where the electronic device may correspond to the aforementioned terminal device. Please refer to FIG. 15, which is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. The electronic device 1500 may at least include: a processor 1501, an input interface 1502, an output interface 1503, and a computer storage medium 1504. connect.

The computer storage medium 1504 may be stored in the memory 1505 of the electronic device 1500, the computer storage medium 1501 for storing a computer program including program instructions, and the processor 1501 for executing the computer storage medium 1504 storage program instructions. The processor 1501 (or called CPU (Central Processing Unit, central processing unit)) is the computing core and the control core of the electronic device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing:

In response to the audio enhancement opening instruction for the target audio, the first audio to be played is obtained from the target audio; the target audio is the audio in a playing state; the first audio is input into the pre-trained audio enhancement model, and the output of the audio enhancement model is obtained the second audio frequency; process the mode of the second audio frequency according to the high frequency band mode of the first audio frequency and modify the phase of the second audio frequency according to the low frequency band phase of the first audio frequency, so as to process the second audio frequency into the third audio frequency ; The third audio is super high quality audio; replace the playback target audio with the playback third audio.

In one embodiment, the above-mentioned processor 1501 is further configured to perform post-processing on the high frequency mode of the second audio frequency based on the high frequency band mode of the first audio frequency and the high frequency band mode of the second audio frequency, so as to obtain the full range of the second audio frequency. frequency band mode; the above-mentioned processor 1501 is also used to perform phase correction on the phase of the second audio frequency according to the low frequency band phase of the first audio frequency to obtain the full frequency band phase of the second audio frequency; the above processor 1501 is also used for the full frequency band mode and the full-band phase to process the second tones to determine the third tones.

In one embodiment, the above-mentioned processor 1501 is further configured to perform mirror image processing on the low frequency band phase of the first audio frequency to obtain the mirror image phase; the above-mentioned processor 1501 is also used to perform an operation on the mirror image phase by using a speech signal reconstruction algorithm to obtain Calculated phase; the processor 1501 is further configured to perform phase correction on the phase of the second audio frequency according to the calculated phase to obtain the full-band phase of the second audio frequency.

In one embodiment, the processor 1501 deletes the buffered third audio when receiving a drag instruction or a switching instruction input for the target audio.

In one embodiment, the processor 1501 deletes the buffered third audio when the third audio is played.

In one embodiment, the processor 1501, when receiving an audio enhancement closing instruction for the target audio input, stops inputting the audio in the target audio into the audio enhancement model.

In one embodiment, the processor 1501 receives an audio enhancement enable instruction for the target audio input on the target audio playback interface.

In one embodiment, the processor 1501 receives, on the audio setting interface corresponding to the target audio, an audio enhancement enable instruction for the input of the target audio.

In one embodiment, the processor 1501 displays the frequency spectrum of the first audio and the third audio on the audio setting interface.

In one embodiment, the processor 1501 displays a first frequency spectrum and a second frequency spectrum of the target audio segment for the target audio segment currently being processed by the audio enhancement model; the first frequency spectrum is the frequency spectrum of the target audio segment before the audio enhancement model is input. , the second frequency spectrum is the frequency spectrum of the target segment after the audio enhancement model is input, and the target audio segment is the audio segment being processed among the multiple audio segments.

In one embodiment, the above-mentioned audio enhancement model is a model obtained by performing phase correction on a generative adversarial network model, and the generative adversarial network model is a model obtained by training an audio sample through a generative adversarial network. High quality audio.

To sum up, the electronic device receives an audio enhancement turn-on instruction input for a target audio, and the target audio is the audio in the playing state; and in response to the audio enhancement turn-on command, obtains the first audio to be played from the target audio; An audio is input to the pre-trained audio enhancement model, the second audio output by the audio enhancement model is obtained, and the third audio of the ultra-high-quality audio is determined according to the first audio and the second audio; thus the third audio is played. It should be understood that, by inputting the first audio into the audio enhancement model, the electronic device can obtain the third audio of super high quality, thereby improving the user experience.

In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. The technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art, or all or part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions for So that a computer device (which may be a personal computer, a server, or a network device, etc., specifically a processor in the computer device) executes all or part of the steps of the foregoing methods in the various embodiments of the present application. Wherein, the aforementioned storage medium may include: U disk, mobile hard disk, magnetic disk, optical disk, read-only memory (English: Read-Only Memory, abbreviation: ROM) or random access memory (English: Random Access Memory, abbreviation: RAM) ) and other media that can store program codes.

Those of ordinary skill in the art can realize that the units and steps of each example described in conjunction with the embodiments disclosed in this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. A computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. Computer instructions may be stored in, or transmitted over, computer storage media. Computer instructions may be sent from one website site, computer, server, or data center to another website site by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) , computer, server or data center. A computer storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, or the like that contains an integration of one or more available media. Useful media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the technical field disclosed in the present invention can easily think of changes or substitutions. should be covered within the scope of protection of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (10)

1. An audio processing method, wherein the method is applied to a terminal device, and the method comprises: In response to the audio enhancement opening instruction for the target audio, obtain the first audio to be played from the target audio; the target audio is the audio in the playing state; Inputting the first audio into the pre-trained audio enhancement model to obtain the second audio output by the audio enhancement model; The modulo of the second audio is processed according to the high-band modulo of the first audio and the phase of the second audio is modified according to the low-band phase of the first audio to convert the second audio Processed as third audio; the third audio is ultra-high-quality audio; Playing the target audio is replaced with playing the third audio. 2. The method according to claim 1, wherein the determining a third audio according to the first audio and the second audio comprises: Based on the high frequency band mode of the first audio frequency and the high frequency band mode of the second audio frequency, perform high frequency mode post-processing on the mode of the second audio frequency to obtain the full frequency band mode of the second audio frequency; performing phase correction on the phase of the second audio frequency according to the low frequency band phase of the first audio frequency to obtain the full frequency band phase of the second audio frequency; The second audio is processed based on the full-band mode and the full-band phase to determine the third audio. 3 . The method according to claim 2 , wherein, according to the low frequency band phase of the first audio frequency, phase correction is performed on the phase of the second audio frequency to obtain the full frequency band phase of the second audio frequency. 4 . ,include: Performing mirror image processing on the low frequency band phase of the first audio frequency to obtain a mirror image phase; Utilize the speech signal reconstruction algorithm to operate the mirror phase to obtain the phase after operation; Phase correction is performed on the phase of the second audio frequency according to the calculated phase to obtain the full-band phase of the second audio frequency. 4 . The method according to claim 2 , wherein the audio enhancement model is a model obtained by training audio samples through a generative confrontation network, and the audio samples are ultra-high-quality audio. 5 . 5. The method according to any one of claims 1-4, wherein the method further comprises: The buffered third audio is deleted when a drag instruction or a switching instruction input for the target audio is received or when the third audio is finished playing. 6. The method according to any one of claims 1-4, wherein the receiving an audio enhancement enabling instruction for target audio input comprises: On the playback interface of the target audio or the audio setting interface corresponding to the target audio, an audio enhancement enabling instruction input for the target audio is received. 7. The method according to claim 6, wherein the method further comprises: On the audio setting interface, the frequency spectrum of the first audio and the third audio is displayed. 8. The method according to claim 7, wherein the first audio comprises a plurality of audio segments, the second audio comprises audio segments respectively enhanced by the audio segments, and the audio enhancement model processing each of the audio clips in turn to obtain audio clips after audio enhancement; The displaying the frequency spectrum of the first audio frequency and the third audio frequency includes: For the target audio segment currently being processed by the audio enhancement model, display the first frequency spectrum and the second frequency spectrum of the target audio segment; wherein the first frequency spectrum is the frequency spectrum of the target audio segment, and the second frequency spectrum is the frequency spectrum of the enhanced audio segment corresponding to the target audio segment. 9. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to invoke the program instructions, A method as claimed in any one of claims 1-8 is performed. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program comprising program instructions that, when executed by a processor, cause the processor to execute The method of any one of claims 1-8.

Images