CN114827886A - Audio generation method and device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure provides an audio generation method, an audio generation device, an electronic device and a storage medium. The audio generation method may include: acquiring an audio signal to be processed; obtaining a plurality of soundtrack signals for a plurality of sound sources by soundtrack separation of an audio signal to be processed; determining user orientation information in a three-dimensional space and generating three-dimensional space metadata corresponding to each of a plurality of soundtrack signals; and generating a three-dimensional audio signal corresponding to the audio signal to be processed based on the user orientation information, the separated plurality of audio track signals and the three-dimensional spatial metadata corresponding to each audio track signal. The method and the device can generate the three-dimensional audio which is closer to the real three-dimensional audio, increase the immersion feeling of the three-dimensional audio and improve the user experience.

Description

Audio generation method, apparatus, electronic device and storage medium

technical field

The present disclosure relates to the technical field of audio processing, and in particular, to an audio generation method, an audio generation apparatus, an electronic device, a storage medium, and a program product.

Background technique

Three-dimensional 3D audio refers to the audio signal in which the audio signal is distributed in the three-dimensional space and played dynamically. For example, as the music is played, the human voice moves from far to near in the three-dimensional space, and the drums, bass and other musical instruments are in different places in the three-dimensional space. Jump back and forth.

However, the current mainstream audio is basically a two-channel stereo format or a single-channel format, and the audio content of multi-channel and 3D formats is relatively scarce. Therefore, the automatic conversion of traditional stereo audio into 3D audio is an important direction for the development of 3D audio.

SUMMARY OF THE INVENTION

The present disclosure provides an audio generation method, an audio generation apparatus, an electronic device, and a storage medium to solve at least the above-mentioned problems.

According to a first aspect of the embodiments of the present disclosure, an audio generation method is provided, the method may include: acquiring an audio signal to be processed; an audio track signal; determining user orientation information in a three-dimensional space, and generating three-dimensional spatial metadata corresponding to each of the plurality of audio track signals, wherein the three-dimensional spatial metadata includes a corresponding sound source orientation information and sound width information in three-dimensional space; based on the user orientation information, the plurality of separated audio track signals, and the three-dimensional spatial metadata corresponding to each of the audio track signals, generating the three-dimensional audio signal corresponding to the audio signal to be processed.

As an example, generating the three-dimensional spatial metadata corresponding to each audio track signal in the plurality of audio track signals may include: acquiring feature information of the audio signal to be processed, wherein the feature information includes all at least one of beat information and structure information of the audio signal to be processed, wherein the structure information includes type information of each audio segment of the audio signal to be processed; The three-dimensional spatial metadata corresponding to the track signal.

As an example, generating the three-dimensional spatial metadata corresponding to each of the audio track signals based on the feature information may include: for the vocal signals in the plurality of audio track signals, according to the structure information determine the position adjustment information of the sound source corresponding to the vocal signal relative to the user orientation information in the three-dimensional space, and determine the three-dimensional spatial metadata of the vocal signal based on the position adjustment information .

As an example, generating three-dimensional spatial metadata corresponding to each of the audio track signals based on the feature information may include: for the musical instrument signals in the plurality of audio track signals, according to the tempo information and the The type of each audio segment in the structure information is determined, the movement information of the sound source corresponding to the musical instrument signal in the three-dimensional space is determined, and the three-dimensional space metadata of the musical instrument signal is determined based on the movement information.

As an example, generating the three-dimensional spatial metadata corresponding to each audio track signal in the plurality of audio track signals may include: determining from a plurality of preset templates respectively corresponding to each audio track signal A preset template, wherein the preset template includes at least one of the movement track information, movement speed information and sound width change information of the corresponding sound source in the three-dimensional space; 3D spatial metadata for the audio track signal.

As an example, generating the three-dimensional spatial metadata corresponding to each of the plurality of audio track signals may include: acquiring setting information input by a user, wherein the setting information includes the setting information corresponding to the plurality of audio track signals. at least one of the movement trajectory, movement speed, and sound width variation value of each sound source corresponding to the audio track signal in the three-dimensional space; and three-dimensional space metadata for each audio track signal is respectively generated based on the setting information.

As an example, generating an audio signal corresponding to the to-be-processed audio signal based on the user orientation information, the plurality of separated audio track signals, and the three-dimensional spatial metadata corresponding to each audio track signal The 3D audio signal may include: identifying the type of playback device used to play the 3D audio signal; acquiring a rendering strategy corresponding to the type of the playback device, and based on the user orientation information, the plurality of separated audio A track signal and the three-dimensional spatial metadata corresponding to each audio track signal, and a three-dimensional audio signal corresponding to the to-be-processed audio signal is generated through the rendering strategy.

As an example, generating the three-dimensional audio signal corresponding to the to-be-processed audio signal by the rendering strategy may include: when the playback device is an in-ear playback device, for each audio track signal, based on the The orientation information corresponding to each audio frame of the audio track signal and the user orientation information generate a three-dimensional audio signal corresponding to the audio track signal; The sound width of the 3D audio signal corresponding to the audio track signal.

As an example, generating a three-dimensional audio signal corresponding to the to-be-processed audio signal by using the rendering strategy may include: when the playback device is an external playback device, for each audio track signal, based on The position information of the sound source corresponding to the sound track signal and the position information of the plurality of speakers, the sound track signal is rendered to generate a three-dimensional audio signal corresponding to the sound track signal; sound width information to adjust the sound width of the three-dimensional audio signal corresponding to the track signal.

According to a second aspect of the embodiments of the present disclosure, there is provided an audio generation apparatus, the apparatus may include: an acquisition module configured to acquire an audio signal to be processed; an audio track separation module configured to The signal is track-separated to obtain a plurality of track signals for the plurality of sound sources; the metadata generation module is configured to determine user orientation information in the three-dimensional space, and generate a signal corresponding to each of the plurality of track signals The three-dimensional space metadata corresponding to the audio track signal, wherein the three-dimensional space metadata includes the position information and sound width information of the corresponding sound source in the three-dimensional space; the rendering module is configured to, based on the user position information, separate the The plurality of audio track signals and the three-dimensional spatial metadata corresponding to each of the audio track signals generate a three-dimensional audio signal corresponding to the to-be-processed audio signal.

As an example, the metadata generation module may be configured to: acquire feature information of the audio signal to be processed, wherein the feature information includes at least one of beat information and structure information of the audio signal to be processed, wherein , the structure information includes type information of each audio segment of the to-be-processed audio signal; and three-dimensional spatial metadata corresponding to each audio track signal is respectively generated based on the feature information.

As an example, the metadata generation module may be configured to: for the human voice signal in the plurality of audio track signals, according to the type of each audio segment in the structure information, determine the voice signal corresponding to the human voice signal. The source is relative to the position adjustment information of the user orientation information in the three-dimensional space, and the three-dimensional space metadata of the human voice signal is determined based on the position adjustment information.

As an example, the metadata generation module may be configured to: for an instrument signal in the plurality of track signals, determine the instrument signal according to the tempo information and the type of each audio segment in the structure information The movement information of the corresponding sound source in the three-dimensional space, and the three-dimensional space metadata of the musical instrument signal is determined based on the movement information.

As an example, the metadata generation module may be configured to: determine a preset template corresponding to each of the audio track signals from a plurality of preset templates, wherein the preset template includes the corresponding sound source in three-dimensional at least one of movement track information, movement speed information, and sound width variation information in space; using the determined preset template to generate three-dimensional spatial metadata for each of the audio track signals respectively.

As an example, the metadata generation module may be configured to: acquire setting information input by a user, wherein the setting information includes movement trajectories, movement trajectories, and movement trajectories of each sound source corresponding to the plurality of sound track signals in the three-dimensional space. at least one of a velocity and a sound width variation value; and generating three-dimensional spatial metadata for each of the audio track signals based on the setting information, respectively.

As an example, the rendering module may be configured to: identify the type of playback device used to play the 3D audio signal; acquire a rendering strategy corresponding to the type of the playback device, and based on the user orientation information, the separated For the plurality of audio track signals and the three-dimensional spatial metadata corresponding to each audio track signal, a three-dimensional audio signal corresponding to the to-be-processed audio signal is generated through the rendering strategy.

As an example, the rendering module may be configured to: when the playback device is an in-ear playback device, for each audio track signal, based on the orientation information corresponding to each audio frame of the audio track signal and Using the user orientation information, a three-dimensional audio signal corresponding to the audio track signal is generated; based on the sound width information of each audio frame, the sound width of the three-dimensional audio signal corresponding to the audio track signal is adjusted.

As an example, the rendering module may be configured to: when the playback device is an external playback device, for each audio track signal, based on the orientation information of the sound source corresponding to the audio track signal and the multiple The orientation information of each speaker is used to render the audio track signal to generate a 3D audio signal corresponding to the audio track signal; the 3D audio signal corresponding to the audio track signal is adjusted based on the sound width information of the sound source. The sound width of the signal.

According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, the electronic device may include: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions are stored by the At least one processor, when run, causes the at least one processor to perform the audio generation method as described above.

According to a fourth aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the audio generation method as described above.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product whose instructions are executed by at least one processor in an electronic device to perform the audio generation method as described above.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

By separating the audio track signals of multiple single sound sources from the audio to be processed and giving these sound sources various 3D spatial trajectories, the audio to be processed is converted into 3D audio, so as to obtain 3D audio that is closer to the real 3D audio , which increases the immersion of 3D audio and improves the user experience.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

1 is a flowchart of an audio generation method according to an embodiment of the present disclosure;

2 and 3 illustrate schematic diagrams of a virtual three-dimensional space according to an embodiment of the present disclosure;

4 is a schematic flowchart of an audio generation method according to another embodiment of the present disclosure;

5 is a schematic structural diagram of an audio generation device according to an embodiment of the present disclosure;

6 is a block diagram of an audio generation apparatus according to an embodiment of the present disclosure;

7 is a block diagram of an electronic device according to an embodiment of the present disclosure.

Throughout the drawings, it should be noted that the same reference numerals are used to refer to the same or similar elements, features and structures.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid in that understanding, but are to be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

The terms and words used in the following description and claims are not limited to the bibliographical meanings, but, are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purpose only and not for the purpose of limiting the present disclosure as defined by the claims and their equivalents.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely exemplary of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

In the related art, a conventional signal processing scheme is used to separate the direct sound and the background sound from the stereo signal, and then perform different processing on the direct sound and the background sound to generate a 3D audio signal. However, the track separation algorithm based on traditional signal processing has limited effect, and cannot separate the track signal of a single sound source such as human voice and various musical instruments in a targeted manner. The resulting 3D audio is more immersive than the real 3D audio. There is a noticeable difference in audio. In addition, a 3D audio effect can also be constructed by rotating the audio signal to a certain extent in the three-dimensional space based on the detection result of the audio card point. However, this scheme does not perform targeted processing of each sound element based on the track separation algorithm. For example, when the drum sound rotates, the vocal signal must also rotate together, which results in a single 3D audio effect.

The present disclosure combines a series of technologies such as audio card point detection, audio track separation, and music structure analysis to endow audio tracks such as vocals, drums, bass, etc. with different three-dimensional spatial trajectories, and convert traditional stereo audio into 3D audio. signal, and finally generate immersive 3D audio content through audio space rendering technology.

Hereinafter, according to various embodiments of the present disclosure, the method and apparatus of the present disclosure will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart of an audio generation method according to an embodiment of the present disclosure. The audio generation method of FIG. 1 can be used to convert conventional stereo audio, single channel audio, etc. into three-dimensional audio.

The audio generation method according to the present disclosure may be performed by any electronic device. The electronic device may be at least one of a smartphone, a tablet computer, a portable computer, a desktop computer, and the like. The electronic device may be installed with a target application for implementing the three-dimensional audio generation method of the present disclosure.

Referring to FIG. 1, in step S101, an audio signal to be processed is acquired. Here, the audio signal to be processed may be, for example, a stereo music signal, a monophonic music signal, or the like.

In step S102, multiple track signals for multiple sound sources are obtained by performing track separation on the audio signal to be processed.

The audio signal to be processed can be track separated using a deep learning based track separation system. The sound track separation technology based on deep learning can separate the vocal and individual instrument signals with a higher degree of restoration. For example, a deep learning-based audio track separation system can be composed of three parts: an encoder, a separation module, and a decoder. Can first perform short-time Fourier transform on the audio signal to be processed to obtain the spectral data of the audio signal, input the spectral data into the encoder to obtain the coding feature of the audio signal, and use the separation module to further extract the target track feature from the coding feature. , and finally obtain the target masking matrix corresponding to the target audio track signal through the decoder based on the target audio track feature. After multiplying the spectral data by the target masking matrix, and then performing short-time inverse Fourier transform on the multiplication result, the target audio track signal can be obtained. Here, the target track signal may include one or more track signals. For example, after performing the above process on a conventional stereo music signal, multiple track signals such as vocals, drums, bass, and other instruments can be obtained.

In step S103, the user orientation information in the three-dimensional space is determined, and three-dimensional space metadata associated with each of the separated multiple audio track signals is generated. The user orientation information may include at least a part of the three-dimensional position coordinates, direction, speed, moving path, etc. of the virtual user at each time point or each time period in the three-dimensional space. For example, the user orientation information may be fixed to a central position in a three-dimensional space. The three-dimensional spatial metadata of each sound source can be determined with reference to the user orientation information. The 3D space metadata of a track signal can be used to indicate how the corresponding sound source changes in the 3D space, for example, how the sound source moves in the 3D space, the sound width of the sound source changes in the 3D space, and so on.

The three-dimensional space metadata may include position information and sound width information of the corresponding sound source in the three-dimensional space. Here, the position information of the sound source may include part or all of the three-dimensional position coordinates, direction, speed, moving path, etc. of the corresponding sound source at each time point or each time period in the three-dimensional space.

2 and 3 show schematic diagrams of a virtual three-dimensional space according to an embodiment of the present disclosure.

Assuming that there is a virtual three-dimensional space, the three-dimensional coordinate axes of the space are X-axis, Y-axis, Z-axis, where Z-axis represents height, and the range of each coordinate axis is set to [0,1], as shown in Figure 2 .

The central position of the user/listener in the three-dimensional space, such as (0.5, 0.5, 0.5), the white dots in Figure 3 can represent various sound sources, such as vocals and musical instruments, and the coordinates of the dots can represent the sound source. The three-dimensional spatial location of the source, the size of the dots can represent the width of the sound. However, the above-described examples are merely exemplary, and the present disclosure is not limited thereto. Furthermore, the sound source orientation in the three-dimensional spatial metadata may be an orientation relative to the user orientation.

According to an embodiment of the present disclosure, three-dimensional spatial metadata for each track signal included in the audio signal to be processed may be generated based on feature information of the audio signal to be processed. The feature information may include at least one of tempo information and structure information of the audio signal to be processed.

The beat information can be obtained by performing audio jam point detection on the audio signal to be processed. For example, audio jam point detection can be implemented by a deep learning-based model, which can mainly include a feature extraction module, a deep learning-based probability prediction module, and a global beat position estimation module. Based on deep learning detection, the beat position of the audio signal can be accurately determined.

Feature extraction usually uses frequency domain features. For example, the Mel spectrum of the audio signal to be processed and its first-order difference can be used as the input features of the feature extraction module, and then the extracted features are input to the probability prediction module. The probabilistic prediction module can be implemented using deep networks such as CRNN to learn the local and time-series features of the audio signal to be processed. Through the probability prediction module, the probability of whether it is a beat point can be calculated for each frame of audio data. Finally, based on the predicted probability, the global optimal beat position is obtained by using the global beat position estimation module. The global beat position estimation module can be implemented by a dynamic programming algorithm. The generated beat bits can include normal beats and rebeats. The above process of audio jam point detection is only exemplary, and the present disclosure is not limited thereto.

The structure information of the audio signal can be obtained by performing audio structure analysis on the audio signal to be processed. The structure information may include type information and time information of each audio segment of the audio signal. For example, audio structure analysis technology refers to segmenting an audio signal into different types of segments through algorithms, such as several different types of segments such as overture, verse, chorus, and transition. For example, the audio structure analysis process mainly includes several steps such as segmentation, clustering and identification. Taking a music signal as an example, the input music signal is first processed into frames, and the spectral features (such as Mel frequency cepstral coefficients, etc.) of the speech frame are extracted. By calculating the feature correlation between frames, the correlation matrix of the music signal can be obtained. According to the correlation matrix, the speech signal can be segmented into segments, and the segments can be clustered according to the correlation between the segments. After going through the segmentation and clustering process, for the input music signal, a music structure such as a-b-c-b-c form and the time points of the corresponding segments can be obtained. Finally, based on the number of repetitions of each segment and the acoustic characteristics such as volume and brightness, the verse part and the chorus part in the music signal can be determined. The above process of audio structure analysis is only exemplary, and the present disclosure is not limited thereto.

After obtaining the tempo information of the audio signal to be processed and the structure information of each audio segment, corresponding three-dimensional spatial metadata can be generated for different audio track signals.

For the human voice signal in the separated multiple audio track signals, the position adjustment information of the sound source corresponding to the human voice signal relative to the user orientation information in the three-dimensional space can be determined according to the type of each audio segment in the structure information. The position adjustment information determines the three-dimensional spatial metadata of the vocal signal.

As an example, for the vocal signal in the audio track signal, the sound source corresponding to the vocal signal may be set to move in the three-dimensional space toward the user's position during the audio segment of the first preset type, and the sound source corresponding to the vocal signal may be set to move toward the user's position in the three-dimensional space, and the sound source corresponding to the vocal signal may be set to move in the three-dimensional space during the audio segment of the first preset type. The relevant information is used as the three-dimensional spatial metadata of the human voice signal.

Taking a music signal as an example, for a human voice signal, the distance between the sound source corresponding to the human voice signal and the listener can be changed from far to near in the three-dimensional space in the verse part. For example, the sound source changes from coordinates (0.5, 0, 0.5) to (0.5, 0.3, 0.5) in the three-dimensional space of FIG. 2 and gradually approaches the listener, thereby increasing the sense of substitution of music.

As another example, for the vocal signal in the audio track signal, the height coordinates of the sound source in the three-dimensional space corresponding to the vocal signal may be increased to a predetermined height during the audio segment of the second preset type and the sound source may be increased to a predetermined height. The sound width of the source is increased to a predetermined sound width, and the information related to the increase to the predetermined height and the predetermined sound width is taken as three-dimensional spatial metadata of the vocal signal.

Taking the music signal as an example, for the vocal signal, the height in the three-dimensional coordinates of the sound source corresponding to the vocal signal and the width of the entire sound can be gradually increased in the three-dimensional space in the transition section between the verse and the chorus, so that The sound source finally reaches a certain height and sound width value in the chorus part, thereby enhancing the shocking effect of the chorus. For example, the position coordinates of the sound source change from (0.5, 0, 0.5) to (0.5, 0, 1) in the three-dimensional space of FIG. 2, and the sound width value changes from 0.05 to 0.5. Changes in height and width values can follow a preset linear or non-linear function.

For the musical instrument signals in the separated multiple track signals, the movement information of the sound source corresponding to the musical instrument signal in the three-dimensional space can be determined according to the type of each audio segment in the beat information and the structure information, and the musical instrument signal can be determined based on the movement information. 3D spatial metadata.

As an example, for the musical instrument signal in the audio track signal, the sound source corresponding to the musical instrument signal may be set to periodically move according to a predetermined trajectory in the three-dimensional space according to the tempo information, and the information related to the movement may be used as the three-dimensional signal of the musical instrument signal. Spatial metadata. Taking music signals as an example, sound sources corresponding to musical instruments such as drums and bass can periodically change with a specific trajectory in a three-dimensional space according to the beat. For example, the sound source of the musical instrument can be rotated according to a specific trajectory in three-dimensional space, and the sound source can be positioned in the middle of the listener's head when replaying, so as to further enhance the listening experience of the listener.

As yet another example, for the musical instrument signal in the audio track signal, the rotational speed of the sound source in the three-dimensional space corresponding to the musical instrument signal may be increased to a predetermined rotational speed during the audio segment of the second preset type, and the same as the increase Information related to a predetermined rotational speed is used as three-dimensional spatial metadata of the musical instrument signal. Taking a music signal as an example, for sound sources corresponding to instruments such as drums and bass, the spatial rotation speed of the instrument sound source in the chorus can be increased according to the characteristics of the verse and chorus, thereby enhancing the dynamic of the chorus.

According to another embodiment of the present disclosure, a preset template corresponding to each of the separated audio track signals may be determined from a plurality of preset templates, and the determined preset template may be used to generate a three-dimensional model for each audio track signal, respectively. Spatial metadata.

Each preset template may include at least one of movement track information, movement speed information, and sound width variation information of the sound source in the three-dimensional space. The preset template can preset the change process of the sound source in the three-dimensional space, such as how the sound source moves in the three-dimensional space, and the sound width of the sound source in the three-dimensional space changes. After the to-be-processed audio signal is separated into a plurality of track signals, a preset template may be allocated for each separated track signal, so that the track signal changes according to the information in the corresponding template. For example, a preset template matching the audio track signal may be determined from a plurality of preset templates according to the characteristics of each audio track signal.

As an example, the preset template may include moving the distance between the vocal sound source and the listener from far to near in the verse part, gradually increasing the height of the vocal sound source in three-dimensional coordinates in the transition section of the main chorus, and the whole Templates such as the width of the sound, the periodic change of the instrument sound source in a specific trajectory in the three-dimensional space according to the beat, and the improvement of the spatial rotation speed of the instrument sound source in the chorus. However, the above examples are only exemplary, and the present disclosure is not limited thereto. By applying corresponding preset templates to different audio track signals, three-dimensional spatial metadata that is more in line with the properties of each audio track signal is generated, so that a more realistic three-dimensional audio signal can be generated.

According to another embodiment of the present disclosure, setting information input by a user may be acquired, and the setting information may include at least one of a movement trajectory, movement speed, and sound width variation value of each sound source corresponding to a plurality of sound track signals in a three-dimensional space. One, and then generate three-dimensional spatial metadata for each track signal separately based on the setting information.

As an example, three-dimensional spatial metadata for each separated audio track signal may be generated separately based on user input. The user input may be used to set at least one of a movement trajectory, a movement speed, and a sound width variation value of each sound source corresponding to the plurality of track signals in the three-dimensional space. The user can customize the signal changes for each track according to his understanding of the audio content to be processed.

Taking the music signal as an example, the user can customize the movement track of the bass sound source in the chorus in the three-dimensional space according to his own understanding of the music content. When generating the three-dimensional music signal, the user can A custom movement track is applied to this bass source. However, the above examples are only exemplary, and the present disclosure is not limited thereto. Users can set the changes of each sound source in the three-dimensional space according to their own preferences and understanding of the music content, so as to obtain the desired music effect, which satisfies the user's needs and improves the user experience.

According to yet another example of the present disclosure, a preset template may be used to generate three-dimensional spatial metadata for the separated multiple audio track signals based on beat information of the audio signal to be processed and each audio segment. That is to say, the sound source corresponding to each audio track signal can be changed in the corresponding beat position and segment part according to the preset template in the three-dimensional space.

In step S104, a three-dimensional audio signal corresponding to the to-be-processed audio signal is generated based on the user orientation information, the separated multiple audio track signals, and the three-dimensional spatial metadata corresponding to each audio track signal. The generated 3D audio signal may include separated audio track signals and 3D spatial metadata corresponding to each audio track signal, and the final 3D audio effect may be obtained through spatial audio rendering technology.

According to an embodiment of the present disclosure, when generating a 3D audio signal, the type of playback device used to play the 3D audio signal may be considered, and different ways are used to render the 3D audio signal for different playback devices.

Specifically, the type of playback device used to play the 3D audio signal can be identified, the rendering strategy corresponding to the type of the playback device can be obtained, and then based on the user orientation information, the separated multiple audio track signals and the corresponding audio track signals The 3D spatial metadata of the 3D space is generated, and the 3D audio signal corresponding to the to-be-processed audio signal is generated through the acquired rendering strategy.

For the in-ear playback device, a three-dimensional audio signal corresponding to the audio track signal may be generated based on the orientation information and user orientation information corresponding to each audio frame of the audio track signal for each of the separated audio track signals, and based on each audio track signal The sound width information of each audio frame is used to adjust the sound width of the three-dimensional audio signal corresponding to the audio track signal. For example, each audio frame of the audio track signal can be convolved with its HRTF corresponding to its three-dimensional coordinates based on a head-related transfer function (HRTF) to obtain a corresponding binaural audio signal.

For an external playback device, the audio track signal can be rendered based on the orientation information of the sound source corresponding to the audio track signal and the orientation information of the plurality of speakers for each audio track signal to generate a corresponding audio track signal. The three-dimensional audio signal is adjusted, and the sound width of the three-dimensional audio signal corresponding to the track signal is adjusted based on the sound width information of the sound source. For example, based on the Vector Basis Amplitude Phase Shift (VBAP) technique, the unit directional vector of the sound can be expressed as a linear combination of the unit directional vectors of the speakers closest to the direction of the sound source, and the gain factor of each speaker can be calculated, so as to Multiple speakers are rendered for 3D music effects.

Considering the type of playback device, a three-dimensional audio signal can be generated to make the playback effect of each type of playback device better.

The 3D audio generation method proposed in the present disclosure can automatically convert traditional two-channel stereo music into 3D music, increasing the immersion of the music.

FIG. 4 is a schematic flowchart of an audio generation method according to another embodiment of the present disclosure. In FIG. 4 , the description is made by taking the conversion of stereo music/mono-channel music into 3D music as an example. However, the system shown in Figure 4 can also be used to convert any form of audio to 3D audio.

The input single-channel or stereo music is extracted through the track separation module to extract a variety of preset track signals, such as the audio signals of sound sources such as vocals, drums, and bass. At the same time, the input single-channel or stereo music extracts the beat information of the music through the audio card point detection module, and extracts the structure information of the verse, chorus and other types of music clips through the audio structure analysis module. Based on the card point information and music structure information, according to some custom templates or user editing, the three-dimensional metadata of each track signal is determined through the three-dimensional metadata generation module. The spatial audio rendering module can use the spatial audio rendering technology to obtain the final 3D music signal based on the separated audio track signal and the corresponding three-dimensional spatial metadata.

The audio track separation module can be implemented based on deep learning, for example, it can include an encoder, a decoder, and a separator. The input time-domain music signal passes through the STFT module to obtain the corresponding music spectrum signal. After passing through the encoder composed of multi-layer convolutional layers, the separator is used to further extract the track features, and finally the target mask corresponding to the target track signal is obtained through the decoder. matrix. After multiplying the music spectrum signal with the target masking matrix, and then passing through the ISTFT module, the target audio track signal, such as vocals, drums, bass and other instruments, can be obtained.

The audio jam point detection module can be implemented based on deep learning, for example, it can include a feature extraction module, a probability prediction module based on a deep model, and a global beat position estimation module. First, feature extraction usually uses frequency domain features. In one implementation, the Mel spectrum and its first-order difference are usually used as input features. The probability prediction module is usually implemented by deep networks such as CRNN to learn local features and time series features. Through the probability prediction module, the probability of whether it is a beat point can be calculated for each frame of audio data. Finally, based on the predicted probability, the global beat position estimation module uses the dynamic programming algorithm to obtain the global optimal beat position. The generated beat bits can include normal beats and rebeats.

The music structure analysis module can divide the music signal into different segments by algorithm, such as several different parts such as overture, verse, chorus and transition. The music structure analysis process mainly includes several steps such as segmentation, clustering and identification. Firstly, the music signal is divided into frames, and the spectral features such as the Mel frequency cepstral coefficient MFCC of the speech frame are extracted. By calculating the feature correlation between frames, the correlation matrix of the music signal can be obtained. The speech signal can be segmented into segments according to the correlation matrix, and the segments can be clustered according to the correlation between the segments. After the segmentation and clustering process, for the music signal to be processed, a music structure similar to a-b-c-b-c form and corresponding segment time points can be obtained. Finally, based on the number of repetitions of the segment, as well as acoustic characteristics such as volume and brightness, the verse and chorus in the music can be judged.

The three-dimensional metadata generation module can generate corresponding three-dimensional spatial metadata for different track signals based on the music tempo information and the music structure information. The three-dimensional space metadata corresponds to the information of the audio track signal in the three-dimensional space, and specifically mainly includes the three-dimensional position coordinates and the width of the sound.

The template of each sound source can be customized by the user according to his own understanding of the music content, or the three-dimensional spatial metadata of each sound source can be automatically generated through some preset templates.

For example, the preset template may include moving the distance between the vocal source and the listener from far to near in the verse, gradually increasing the height of the vocal source in three-dimensional coordinates in the transitional section of the chorus, and Templates such as the width of the entire sound, the periodic change of the instrument sound source in a specific trajectory in the three-dimensional space according to the beat, and the improvement of the spatial rotation speed of the instrument sound source in the chorus. However, the above examples are only exemplary, and the present disclosure is not limited thereto.

The 3D music signal generated by the spatial audio rendering module may include separate audio track signals and three-dimensional metadata corresponding to each audio track, and the final 3D music effect can be obtained through the spatial audio rendering technology. For example, for a headphone playback device, each audio frame of an input audio track can be convolved with its HRTF corresponding to its three-dimensional coordinates based on a head-related transfer function (HRTF) to obtain a corresponding binaural audio signal. For multiple speaker playback devices, the unit direction vector of the sound can be expressed as a linear combination of the unit direction vectors of multiple speakers closest to the sound source direction based on the vector magnitude translation technique (VBAP), and the gain factor of each speaker can be calculated, Thereby rendering a 3D music effect for multiple speakers.

FIG. 5 is a schematic structural diagram of an audio generation device of a hardware operating environment according to an embodiment of the present disclosure.

As shown in FIG. 5 , the audio generation device 500 may include: a processing component 501 , a communication bus 502 , a network interface 503 , an input and output interface 504 , a memory 505 , and a power supply component 506 . Among them, the communication bus 502 is used to realize the connection and communication between these components. The input-output interface 504 may include a video display (such as a liquid crystal display), a microphone and speakers, and a user interaction interface (such as a keyboard, mouse, touch input device, etc.), and optionally, the input-output interface 504 may also include a standard wired interface , wireless interface. Optionally, the network interface 503 may include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 505 may be a high-speed random access memory or a stable non-volatile memory. Optionally, the memory 505 may also be a storage device independent of the aforementioned processing component 501 .

Those skilled in the art can understand that the structure shown in FIG. 5 does not constitute a limitation on the audio generation device 500, and may include more or less components than those shown, or combine some components, or arrange different components.

As shown in FIG. 5 , the memory 505 as a storage medium may include an operating system (such as a MAC operating system), a data storage module, a network communication module, a user interface module, a program, and a database.

In the audio generation device 500 shown in FIG. 5 , the network interface 503 is mainly used for data communication with external electronic devices/terminals; the input and output interface 504 is mainly used for data interaction with the user; the processing component 501 in the audio generation device 500 The memory 505 may be set in the audio generation device 500, and the audio generation device 500 invokes the program stored in the memory 505 and various APIs provided by the operating system through the processing component 501 to execute the audio generation method provided by the embodiments of the present disclosure.

The processing component 501 may include at least one processor, and a computer-executable instruction set is stored in the memory 505, and when the computer-executable instruction set is executed by the at least one processor, performs the audio generation method according to an embodiment of the present disclosure. However, the above-described examples are merely exemplary, and the present disclosure is not limited thereto.

The processing component 501 can realize the control of the components included in the audio generation apparatus 500 by executing a program.

As an example, the audio generation device 500 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, or any other device capable of executing the above set of instructions. Here, the audio generating device 500 does not have to be a single electronic device, but can also be any set of devices or circuits capable of individually or jointly executing the above-mentioned instructions (or instruction sets). Audio generation device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces locally or remotely (eg, via wireless transmission).

In the audio generation apparatus 500, the processing component 501 may comprise a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller or a microprocessor. By way of example and not limitation, processing components 501 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

Processing component 501 can execute instructions or code stored in memory, where memory 505 can also store data. Instructions and data may also be sent and received over a network via network interface 503, which may employ any known transport protocol.

The memory 505 may be integrated with the processing component 501, eg, RAM or flash memory arranged within an integrated circuit microprocessor or the like. Additionally, memory 505 may comprise a separate device, such as an external disk drive, storage array, or any other storage device that may be used by a database system. The memory and processing component 501 may be operatively coupled, or may communicate with each other, such as through I/O ports, network connections, etc., to enable processing component 501 to read data stored in memory 505

FIG. 6 is a block diagram of an audio generation apparatus according to an embodiment of the present disclosure.

6 , the audio generation apparatus 600 may include an acquisition module 601 , an audio track separation module 602 , a metadata generation module 603 and a rendering module 604 . Each module in the audio generating apparatus 600 may be implemented by one or more modules, and the name of the corresponding module may vary according to the type of the module. In various embodiments, some modules in the audio generation apparatus 600 may be omitted, or additional modules may also be included. Furthermore, modules/elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules/elements prior to combination.

The obtaining module 601 can obtain the audio signal to be processed.

The track separation module 602 can obtain multiple track signals for multiple sound sources by performing track separation on the audio signal to be processed.

The metadata generation module 603 may determine the user orientation information in the three-dimensional space, and generate three-dimensional spatial metadata corresponding to each of the separated multiple audio track signals, wherein the three-dimensional spatial metadata may include corresponding The position information and sound width information of the sound source in three-dimensional space.

Optionally, the metadata generation module 603 may acquire feature information of the audio signal to be processed, wherein the feature information may include at least one of beat information of the audio signal to be processed and structure information of each audio segment; 3D spatial metadata for each track signal.

Optionally, the metadata generation module 603 may obtain beat information by performing audio jam point detection on the audio signal to be processed; and obtain the structure information of each audio segment by performing audio structure analysis on the audio signal to be processed.

Optionally, the metadata generation module 603 can determine the sound source corresponding to the vocal signal relative to the user orientation information in the three-dimensional space according to the type of each audio segment in the structure information for the vocal signal in the multiple audio track signals. Based on the position adjustment information, the three-dimensional spatial metadata of the human voice signal is determined based on the position adjustment information.

For example, the metadata generation module 603 may set the sound source corresponding to the vocal signal to the position of the user in the three-dimensional space during the audio segment of the first preset type for the vocal signal in the plurality of audio track signals For movement, the information related to the movement is used as the three-dimensional spatial metadata of the human voice signal.

For another example, the metadata generation module 603 may increase the height coordinates of the sound source in the three-dimensional space corresponding to the vocal signal to a predetermined value during the audio segment of the second preset type for the vocal signal in the plurality of audio track signals. height and increase the sound width of the sound source to a predetermined sound width, and use information related to the increase to the predetermined height and the predetermined sound width as three-dimensional spatial metadata of the human voice signal.

Optionally, the metadata generation module 603 can determine the movement information of the sound source corresponding to the musical instrument signal in the three-dimensional space according to the type of each audio segment in the beat information and the structure information for the musical instrument signal in the multiple audio track signals, Three-dimensional spatial metadata of the instrument signal is determined based on the movement information.

For example, the metadata generation module 603 can set the sound source corresponding to the musical instrument signal to periodically move according to a predetermined trajectory in the three-dimensional space according to the beat information for the musical instrument signal in the plurality of audio track signals, and convert the information related to the movement to the sound source corresponding to the musical instrument signal. 3D spatial metadata as an instrument signal.

For another example, the metadata generation module 603 may increase the rotational speed of the sound source corresponding to the musical instrument signal in the three-dimensional space to a predetermined rotational speed during the audio segment of the second preset type for the musical instrument signal in the plurality of audio track signals , the information related to the increase to a predetermined rotational speed is used as the three-dimensional spatial metadata of the musical instrument signal.

Optionally, the metadata generation module 603 may determine a preset template corresponding to each audio track signal from a plurality of preset templates, wherein the preset template includes the movement track information and movement speed of the sound source in the three-dimensional space. at least one of information and sound width variation information; using the determined preset template to generate three-dimensional spatial metadata for each audio track signal, respectively.

Optionally, the metadata generation module 603 may respectively generate three-dimensional spatial metadata for multiple audio track signals according to the determined preset template based on the feature information of the audio signal to be processed.

Optionally, the metadata generation module 603 can obtain setting information input by the user, wherein the setting information includes the moving track, moving speed and sound width variation value of each sound source corresponding to the multiple sound track signals in the three-dimensional space. At least one, and three-dimensional spatial metadata for each track signal is generated separately based on the setting information.

For example, the metadata generation module 603 may receive user input, wherein the user input is used to set at least one of the movement trajectory, movement speed and sound width variation value of each sound source corresponding to the plurality of sound track signals in the three-dimensional space; The three-dimensional spatial metadata for the plurality of audio track signals are respectively generated based on the user input.

The rendering module 604 may generate a three-dimensional audio signal corresponding to the to-be-processed audio signal based on the user orientation information, the separated multiple audio track signals, and the three-dimensional spatial metadata corresponding to each audio track signal.

The rendering module 604 can identify the type of the playback device used to play the three-dimensional audio signal, obtain the rendering strategy corresponding to the type of the playback device, based on the user orientation information, the separated multiple audio track signals and the corresponding audio track signals. The three-dimensional spatial metadata is used to generate a three-dimensional audio signal corresponding to the to-be-processed audio signal through the acquired rendering strategy.

For an in-ear playback device, the rendering module 604 may, for each audio track signal in the plurality of audio track signals, generate a corresponding audio track signal based on the orientation information and user orientation information corresponding to each audio frame of the audio track signal. 3D audio signal; adjusts the sound width of the 3D audio signal corresponding to the track signal based on the sound width information of each audio frame.

For an external playback device, the rendering module 604 may render the audio track signal based on the position information of the sound source corresponding to the audio track signal and the position information of the plurality of speakers for each audio track signal in the plurality of audio track signals , to generate a three-dimensional audio signal corresponding to the audio track signal; and adjust the sound width of the three-dimensional audio signal corresponding to the audio track signal based on the sound width information of the sound source.

The method of converting a conventional stereo signal into a three-dimensional audio signal has been described in detail above according to FIG. 1 to FIG. 4 , and will not be described here again.

According to an embodiment of the present disclosure, an electronic device can be provided. 7 is a block diagram of an electronic device according to an embodiment of the present disclosure, the electronic device 700 may include at least one memory 702 and at least one processor 701, the at least one memory 702 stores a set of computer-executable instructions, when the computer-executable instructions When the collection is executed by the at least one processor 701, the audio generation method according to the embodiment of the present disclosure is executed.

The processor 701 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special purpose processor system, a microcontroller or a microprocessor. By way of example and not limitation, processor 701 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, and the like.

The memory 702 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, a program for executing the audio generation method of the present disclosure, and a database.

The memory 702 may be integrated with the processor 701, eg, RAM or flash memory may be arranged within an integrated circuit microprocessor or the like. Additionally, memory 702 may comprise a separate device, such as an external disk drive, storage array, or any other storage device that may be used by a database system. The memory 702 and the processor 701 may be operatively coupled, or may communicate with each other, eg, through I/O ports, network connections, etc., to enable the processor 701 to read files stored in the memory 702 .

Additionally, the electronic device 700 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 700 may be connected to each other via a bus and/or network.

Those skilled in the art can understand that the structures shown in FIG. 7 do not constitute a limitation, and may include more or less components than those shown, or combine some components, or arrange different components.

According to embodiments of the present disclosure, there may also be provided a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the audio generation method according to the present disclosure. Examples of the computer-readable storage medium herein include: Read Only Memory (ROM), Random Access Programmable Read Only Memory (PROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Random Access Memory (RAM) , dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM , DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or Optical Disc Storage, Hard Disk Drive (HDD), Solid State Hard disk (SSD), card memory (such as a multimedia card, Secure Digital (SD) card, or Extreme Digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other apparatuses configured to store, in a non-transitory manner, a computer program and any associated data, data files and data structures and to provide said computer program and any associated data, data files and data structures The computer program is given to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer readable storage medium can be executed in an environment deployed in computer equipment such as client, host, proxy device, server, etc. Furthermore, in one example, the computer program and any associated data, data files and data structures are distributed over networked computer systems so that the computer programs and any associated data, data files and data structures are stored, accessed and executed in a distributed fashion by one or more processors or computers.

According to an embodiment of the present disclosure, a computer program product can also be provided, wherein instructions in the computer program product can be executed by a processor of a computer device to complete the above audio generation method.

Other embodiments of the present disclosure will readily suggest themselves to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A method of audio generation, the method comprising:

acquiring an audio signal to be processed;

obtaining a plurality of soundtrack signals for a plurality of sound sources by soundtrack separation of the audio signal to be processed;

determining user orientation information in a three-dimensional space, and generating three-dimensional space metadata corresponding to each of the plurality of soundtrack signals, wherein the three-dimensional space metadata includes orientation information and sound width information of a corresponding sound source in the three-dimensional space;

generating a three-dimensional audio signal corresponding to the audio signal to be processed based on the user orientation information, the separated plurality of audio track signals, and the three-dimensional spatial metadata corresponding to each of the audio track signals.

2. The method of claim 1, wherein generating three-dimensional spatial metadata corresponding to each soundtrack signal of the plurality of soundtrack signals comprises:

acquiring feature information of the audio signal to be processed, wherein the feature information includes at least one of beat information and structure information of the audio signal to be processed, and the structure information includes type information of each audio clip of the audio signal to be processed;

and respectively generating three-dimensional space metadata corresponding to each audio track signal based on the characteristic information.

3. The method according to claim 2, wherein generating three-dimensional spatial metadata corresponding to each of the audio track signals respectively based on the feature information comprises:

for a human voice signal in the plurality of audio track signals, according to the type of each audio clip in the structure information, determining position adjustment information of a sound source corresponding to the human voice signal relative to the user orientation information in a three-dimensional space, and determining three-dimensional space metadata of the human voice signal based on the position adjustment information.

4. The method according to claim 2, wherein generating three-dimensional spatial metadata corresponding to each of the audio track signals respectively based on the feature information comprises:

and for the musical instrument signals in the plurality of music track signals, determining the movement information of the sound source corresponding to the musical instrument signals in the three-dimensional space according to the beat information and the types of the audio fragments in the structure information, and determining the three-dimensional space metadata of the musical instrument signals based on the movement information.

5. The method of claim 1, wherein generating three-dimensional spatial metadata corresponding to each soundtrack signal of the plurality of soundtrack signals comprises:

determining a preset template corresponding to each audio track signal from a plurality of preset templates, wherein the preset template comprises at least one of movement track information, movement speed information and sound width change information of a corresponding sound source in a three-dimensional space;

and respectively generating three-dimensional space metadata for each audio track signal by using the determined preset template.

6. The method of claim 1, wherein generating three-dimensional spatial metadata corresponding to each soundtrack signal of the plurality of soundtrack signals comprises:

acquiring setting information input by a user, wherein the setting information comprises at least one of a moving track, a moving speed and a sound width variation value of each sound source corresponding to the plurality of sound track signals in a three-dimensional space;

generating three-dimensional spatial metadata for each of the track signals, respectively, based on the setting information.

7. An apparatus for audio generation, the apparatus comprising:

an acquisition module configured to acquire an audio signal to be processed;

a sound track separation module configured to obtain a plurality of sound track signals for a plurality of sound sources by performing sound track separation on the audio signal to be processed;

a metadata generation module configured to determine user orientation information in a three-dimensional space and generate three-dimensional space metadata corresponding to each of the plurality of soundtrack signals, wherein the three-dimensional space metadata includes orientation information and sound width information of a corresponding sound source in the three-dimensional space;

a rendering module configured to generate a three-dimensional audio signal corresponding to the audio signal to be processed based on the user orientation information, the separated plurality of audio track signals, and the three-dimensional spatial metadata corresponding to each of the audio track signals.

8. An electronic device, comprising:

at least one processor;

at least one memory storing computer-executable instructions,

wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the audio generation method of any of claims 1 to 6.

9. A computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the audio generation method of any of claims 1 to 6.

10. A computer program product in which instructions are executed by at least one processor in an electronic device to perform the audio generation method of any of claims 1 to 6.

Images