CN105979469B - A recording processing method and terminal - Google Patents

Abstract

The invention provides a sound recording processing method and a terminal, wherein the sound recording processing method comprises the steps of obtaining scene image information collected by a camera and sound information collected by a microphone during sound recording, obtaining position information of each sound source in a scene image according to the scene image information, generating sound channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of sound channels required to be adopted during sound recording playing, and synthesizing the sound information collected by the microphone into multi-channel audio data according to the sound channel coefficient information.

Description

A recording processing method and terminal

technical field

The present invention relates to the technical field of terminals, and in particular, to a recording processing method and a terminal.

Background technique

With the rapid development of mobile terminal technology, people often use mobile terminals to record and record life events.

Stereo audio recording and video recording can improve the degree of scene restoration, and play more three-dimensionally in the dual speakers or headphones of the mobile terminal, improving the user experience. A common method for stereo audio recording and video recording is to use multiple microphones in the mobile terminal for sound collection, and the positioning effect of multiple microphones will be better.

However, many mobile terminals only have a single microphone configuration, and the configuration of multiple microphones is generally limited by the size of the terminal. If the size of the mobile terminal is small, the relative positions of the multiple microphones are relatively close, the sound localization is poor, and the audio and video recording The effect is not good.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a recording processing method and terminal, and the existing mobile terminal uses a single microphone to record the problem that it is difficult to synthesize multi-channel sound.

In order to solve the above technical problems, on the one hand, the present invention provides a recording processing method, which is applied to a terminal, and the method includes:

Obtain the scene image information collected by the camera and the sound information collected by the microphone during recording and video recording;

Obtain position information of each sound source in the scene image according to the scene image information;

Generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and the multiple channels that need to be used when playing the recording;

According to the channel coefficient information, the sound information collected by the microphone is synthesized into multi-channel audio data.

On the other hand, the present invention also provides a terminal, comprising:

The acquisition module is used to acquire the scene image information collected by the camera and the sound information collected by the microphone during recording and recording;

a location information acquisition module, configured to acquire location information of each sound source in the scene image according to the scene image information;

A channel coefficient determination module, configured to generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of channels that need to be used when playing the recording;

The synthesis module is used for synthesizing the sound information collected by the microphone into multi-channel audio data according to the channel coefficient information.

The beneficial effects of the above-mentioned technical solutions of the present invention are as follows:

Multi-channel audio data can be synthesized only by using the monophonic sound information collected by one microphone, so the terminal used for recording does not need to set up multiple microphones, which reduces the cost of the terminal used for recording.

Description of drawings

1 is a flowchart of a recording processing method according to Embodiment 1 of the present invention;

2 is a flowchart of a recording processing method according to Embodiment 2 of the present invention;

3 is a flowchart of a recording processing method according to Embodiment 3 of the present invention;

FIG. 4 is a structural block diagram of a terminal according to an embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments. The following examples are intended to illustrate the present invention, but not to limit the scope of the present invention.

Please refer to FIG. 1. FIG. 1 is a flowchart of a recording processing method according to Embodiment 1 of the present invention. The method is applied to a terminal and includes the following steps:

Step S11: Acquire scene image information collected by the camera and sound information collected by the microphone during audio recording and video recording.

The sound information collected by the microphone is monophonic sound information.

Step S12: Acquire position information of each sound source in the scene image according to the scene image information.

There can be one or more than one sound source in the scene image.

The position information of the sound source refers to the position information of the sound source in the scene image, for example, it may include: whether the sound source is located in the left half of the scene image, or is located in the right half of the scene image part. Or, the scale factor of the sound source in the horizontal direction of the scene image, for example, when the sound source is at the leftmost end of the scene image in the horizontal direction, the scale factor is [1,0], and at the rightmost end of the scene image in the horizontal direction When the scale factor is [0, 1], when the center point in the horizontal direction of the scene image, the scale factor is [0.5, 0.5]. Alternatively, it may also include information about the front and rear of the sound source in the scene image, that is, information about the relative distance between the sound source and the camera. Of course, other types of location information can also be used.

Step S13: Generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of channels to be used when playing the recording.

The channel coefficient information refers to the proportional coefficient occupied by each channel when playing the recording.

For example, the multiple channels used when playing the recording include a left channel and a right channel, and the position information of the sound source in the scene image is: the sound source is located in the left half of the scene image, Also the position information located in the right half of the scene image. When the sound source is located in the left half of the scene image, the channel coefficient information corresponding to the position information of the sound source may be [1,0], that is, the sound information of the sound source is played by the left channel, and the sound information of the sound source is played by the left channel. The channel has no sound. When the sound source is located in the right half of the scene image, the channel coefficient information corresponding to the position information of the sound source may be [0, 1], that is, the sound information of the sound source is played by the right channel, and the sound information of the sound source is played by the right channel. The channel has no sound.

Another example may be that the multiple channels used when playing the recording include a left channel and a right channel, and the position information of the sound source in the scene image is: the scale coefficient of the sound source in the lateral direction of the scene image. Assuming that the scale factor of the sound source in the horizontal direction of the scene image is [0.2, 0.8], the left channel plays the sound of 20% of the sound source, and the left channel plays the sound of 80% of the sound source.

Of course, in addition to two channels, more channels can also be used when playing the recording, for example, three channels, whose channel coefficient information can be similar to [0.2, 0.4, 0.4].

Step S14: Synthesize the sound information collected by the microphone into multi-channel audio data according to the channel coefficient information.

In this embodiment of the present invention, multi-channel audio data can be synthesized by using only the monophonic sound information collected by one microphone. Therefore, multiple microphones do not need to be set in the terminal used for recording, which reduces the cost of the terminal used for recording. .

In the above embodiment, the terminal that executes the recording processing method may be a recording terminal at the same time, or may not be a recording terminal, and is only used for processing recording. For example, the terminal may be a computer, and the recording terminal may be a video camera. , the camera transmits the recorded audio and video to the computer, and the computer performs multi-channel sound synthesis.

That is to say, the time for synthesizing the multi-channel sound may be at the same time as the audio and video recording, or it may be the multi-channel sound synthesis after the audio and video recording, for example, the multi-channel sound synthesis is performed when the audio and video recording is played.

In addition, the above-mentioned terminal for executing the recording processing method may also be used to play the synthesized multi-channel information, that is, the multi-channel for playing the recording is the multi-channel on the terminal for executing the recording processing method, After the step of synthesizing the sound information collected by the microphone into multi-channel audio data, the method may further include: playing the multi-channel audio data.

Of course, the multi-channel used for playing the recording may not be the multi-channel on the terminal used for executing the recording processing method, but the multi-channel on other playback devices. After the step of synthesizing the sound information of the device into multi-channel audio data, the method may further include: transmitting the multi-channel audio data to a playback device for playback, where the multi-channel is the multi-channel on the playback device. That is to say, the terminal is only responsible for synthesizing the recording into multi-channel sound, and is not responsible for playing.

Please refer to FIG. 2. FIG. 2 is a flowchart of a recording processing method according to Embodiment 2 of the present invention. The method is applied to a terminal. The terminal in this embodiment of the present invention includes a camera and a microphone, and the method includes the following steps:

Step S21 : when a request for enabling the recording and recording function is received, the camera is turned on to collect scene image information, and the microphone is turned on to collect sound information.

The recording and recording function may be the recording and recording function in the camera application software in the terminal, or may be the recording and recording function in the real-time communication application software in the terminal, such as the video chat function of WeChat.

Step S22: Acquire scene image information collected by the camera and sound information collected by the microphone during audio recording and video recording.

Step S23: Acquire position information of each sound source in the scene image according to the scene image information.

Step S24: Generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of channels to be used when playing the recording.

Step S25: Synthesize the sound information collected by the microphone into multi-channel audio data according to the channel coefficient information.

In this embodiment of the present invention, the terminal that executes the recording processing method is also a terminal that performs recording and video recording. And it can synthesize multi-channel audio data while recording and recording.

In the embodiment of the present invention, an image recognition technology may be used to obtain the position information of each sound source in the scene image, which will be described with an example below.

Please refer to FIG. 3. FIG. 3 is a flowchart of a recording processing method according to Embodiment 3 of the present invention. The method is applied to a terminal and includes the following steps:

Step S31: Acquire scene image information collected by the camera and sound information collected by the microphone during audio recording and video recording.

Step S32: Identify the vocalized organism in the scene image according to the scene image information.

The organisms include humans and animals.

Step S33: Perform facial recognition on the vocalized organism in the scene image, and determine each sound source.

For example, from continuous images, the lips and faces of the sound source are identified, and then the sound source is identified.

Step S34: Acquire location information of each sound source.

Step S35: Generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of channels to be used when playing the recording.

Step S36: Synthesize the sound information collected by the microphone into multi-channel audio data according to the channel coefficient information.

In the embodiment of the present invention, the position information of the sound source is determined through the facial recognition technology, and the implementation manner is simple.

Of course, in some other embodiments of the present invention, the position information of the sound source may also be determined by other methods.

As mentioned in the above embodiment, the position information of the sound source may be whether the sound source is located in the left half of the scene image or in the right half of the scene image. When the position information of the sound source indicates that the sound source is located in the left half of the scene image, the channel coefficient information corresponding to the position information of the sound source is configured to use the left channel to play the sound of the sound source information, there is no sound from the right channel. For example, channel coefficient information may be represented as [1, 0]. When the position information of the sound source indicates that the sound source is located in the right half of the scene image, the channel coefficient information corresponding to the position information of the sound source is configured to use the right channel to play the sound of the sound source information, there is no sound from the left channel. For example, channel coefficient information may be represented as [0,1].

When a period of time includes multiple sound sources at the same time, the channel coefficient information can be expressed in the form of a matrix. For example, if the same period includes two sound sources, the channel coefficient information of the two sound sources can be expressed as .

As mentioned in the above embodiment, the position information of the sound source may also be the scale coefficient of the sound source in the horizontal direction of the scene image. For example, when the sound source is at the leftmost end of the horizontal direction of the scene image, the scale coefficient is [ 1,0], the scale factor is [0,1] when it is at the far right end of the scene image in the horizontal direction, and the scale factor is [0.5, 0.5] when it is at the center point in the horizontal direction of the scene image. At this time, the step of generating channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and the multiple channels that need to be used when playing the recording includes: according to the position information of the sound source, calculate the scale coefficient of the sound source in the horizontal direction of the scene image; calculate the coefficient occupied by the left channel and the right channel according to the scale coefficient of the sound source in the horizontal direction of the scene image information to obtain channel coefficient information corresponding to the position information of the sound source. For example, the scale coefficient of the sound source in the lateral direction of the scene image is [1,0], and at this time, the channel coefficient information is also [1,0].

As mentioned in the above embodiment, the position information of the sound source may also be information about the front and rear of the sound source in the scene image, that is, the relative distance information between the sound source and the camera. In this case, the terminal used for playing the recording can be equipped with a scene where multiple channels are set before and after.

Of course, the location information of the sound source may also be of other types, and will not be illustrated one by one again.

Referring to FIG. 4 , an embodiment of the present invention further provides a terminal, including:

The acquisition module is used to acquire the scene image information collected by the camera and the sound information collected by the microphone during recording and recording;

a location information acquisition module, configured to acquire location information of each sound source in the scene image according to the scene image information;

A channel coefficient determination module, configured to generate channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of channels that need to be used when playing the recording;

The synthesis module is used for synthesizing the sound information collected by the microphone into multi-channel audio data according to the channel coefficient information.

The above-mentioned terminal may be a terminal such as a mobile phone, a tablet computer, a camera, or a desktop computer.

Preferably, the terminal further includes:

A playing module is used to play the multi-channel audio data.

Preferably, the terminal further includes:

the camera and the microphone; and

The control module is configured to control the camera to turn on and collect scene image information, and control the microphone to turn on and collect sound information when a request to turn on the recording and video recording function is received.

In an embodiment of the present invention, the location information acquisition module includes:

a first identification unit, configured to identify the vocalized organism in the scene image according to the scene image information;

a second recognition unit, configured to perform facial recognition on the vocalized organism in the scene image, and determine each sound source;

an acquiring unit, configured to acquire the position information of each sound source.

The above are the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, without departing from the principles of the present invention, several improvements and modifications can be made. It should be regarded as the protection scope of the present invention.

Claims (11)

1, recording processing method, applied to terminal, characterized in that the method includes:

acquiring scene image information acquired by a camera and sound information acquired by a microphone during recording; the sound information collected by the microphone is single sound channel sound information;

acquiring position information of each sound source in the scene image according to the scene image information;

generating sound channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of sound channels required to be adopted when the sound record is played;

synthesizing sound information collected by the microphone into multi-channel audio data according to the sound channel coefficient information;

the multiple sound channels required to be adopted when the recording is played comprise a left sound channel and a right sound channel of a terminal used for playing the recording, and the step of generating sound channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and the multiple sound channels required to be adopted when the recording is played comprises the following steps:

calculating a proportionality coefficient of the sound source in the transverse direction of the scene image according to the position information of the sound source;

calculating coefficient information occupied by a left sound channel and a right sound channel according to the proportional coefficient of the sound source in the transverse direction of the scene image to obtain sound channel coefficient information corresponding to the position information of the sound source;

a plurality of sound channels that need adopt when playing the recording still include a plurality of sound channels that are used for playing the terminal of recording and set up from beginning to end, the positional information of sound source still includes: the sound source is information before and after the scene image.

2. The recording processing method according to claim 1, wherein after the step of synthesizing the sound information collected by the microphones into multi-channel audio data, the method further comprises:

and playing the multi-channel audio data.

3. The audio recording processing method according to claim 1, wherein the terminal includes the camera and the microphone, and before the step of acquiring the scene image information collected by the camera and the sound information collected by the microphone during audio recording, the method further includes:

when a request for opening a recording function is received, the camera is started to collect scene image information, and the microphone is started to collect sound information.

4. The audio recording method according to claim 3, wherein the audio recording function is an audio recording function in a camera application in a terminal or an audio recording function in a real-time communication application in the terminal.

5. The audio recording method according to claim 1, wherein the step of obtaining location information of each sound source in the scene image according to the scene image information comprises:

identifying a sounding organism in the scene image according to the scene image information;

carrying out facial recognition on the sounding organisms in the scene images, and determining each sound source;

the position information of each sound source is acquired.

6. The recording processing method of claim 1, wherein the step of generating channel coefficient information corresponding to the position information of each of the sound sources based on the position information of each of the sound sources and a plurality of channels required to be used when the recording is played comprises:

when the position information of the sound source indicates that the sound source is positioned in the left half part of the scene image, the sound channel coefficient information corresponding to the position information of the sound source is configured to play the sound information of the sound source by using a left sound channel;

when the position information of the sound source indicates that the sound source is located in the right-half part of the scene image, the channel coefficient information corresponding to the position information of the sound source is configured to play the sound information of the sound source in the right channel.

7. The audio recording processing method according to claim 1, wherein the audio recording processing method is executed by the terminal at the time of audio recording; or the terminal executes the playing of the audio and video recording.

A terminal of the type , comprising:

the acquisition module is used for acquiring scene image information acquired by the camera and sound information acquired by the microphone during recording; the sound information collected by the microphone is single sound channel sound information;

the position information acquisition module is used for acquiring the position information of each sound source in the scene image according to the scene image information;

a sound channel coefficient determining module, configured to generate sound channel coefficient information corresponding to the position information of each sound source according to the position information of each sound source and a plurality of sound channels required to be used when a recording is played;

the synthesis module is used for synthesizing the sound information collected by the microphone into multi-channel audio data according to the sound channel coefficient information;

the plurality of sound channels required to be adopted when the recording is played comprise a left sound channel and a right sound channel of a terminal for playing the recording;

the sound channel coefficient determining module is used for calculating a proportionality coefficient of the sound source in the transverse direction of the scene image according to the position information of the sound source; calculating coefficient information occupied by a left sound channel and a right sound channel according to the proportional coefficient of the sound source in the transverse direction of the scene image to obtain sound channel coefficient information corresponding to the position information of the sound source;

a plurality of sound channels that need adopt when playing the recording still include a plurality of sound channels that are used for playing the terminal of recording and set up from beginning to end, the positional information of sound source still includes: the sound source is information before and after the scene image.

9. The terminal of claim 8, further comprising:

and the playing module is used for playing the multi-channel audio data.

10. The terminal of claim 8, further comprising:

the camera and the microphone; and

and the control module is used for controlling the camera to start and collect scene image information and controlling the microphone to start and collect sound information when receiving a request for starting the recording and video recording function.

11. The terminal of claim 8, wherein the location information obtaining module comprises:

an recognition unit for recognizing a biological body that utters in a scene image based on the scene image information;

a second recognition unit configured to perform face recognition on the biological object that uttered sound in the scene image, and identify each sound source;

an obtaining unit configured to obtain the position information of each sound source.

Images