CN100556151C - A kind of video terminal and a kind of audio code stream processing method - Google Patents

Abstract

Embodiments of the invention disclose the method that a kind of audio code stream is handled, and this method is: compressed video stream is decoded, obtain to comprise the image of source of sound, detect the positional information of described source of sound in described image; Compressed audio bitstream stream is decoded, obtain voice messaging; Positional information according to described source of sound is handled described voice messaging, and the sound bearing of playback and the position of described source of sound are complementary.Like this, receiving terminal does not need to depend on the sound source position information that transmitting terminal provides, and the positional information of source of sound and the sound bearing information of playback are complementary.Embodiments of the invention also disclose a kind of video terminal simultaneously.

Description

A video terminal and an audio stream processing method

technical field

The invention relates to communication technology, in particular to a video terminal and an audio code stream processing method.

Background technique

With the popularization of broadband, video communication occupies an increasingly important position in our social life, and the video era of communication has begun. However, at present, the screens of televisions are getting bigger and bigger, and some video communication systems use projectors or video wall displays, which cause the participants to move on the screen. The position of the person changes, that is, the sound has no orientation information, resulting in a lack of realism in video communication.

The prior art discloses a method for solving the above-mentioned problem: place a strip-shaped device on the top of the TV set, and there are multiple microphones, multiple speakers, and cameras in the device. After the sound signals collected by the multiple microphones are processed, a voice signal and a speaker's orientation information relative to the elongated device can be obtained. The sending end of the video communication system transmits the obtained voice signal and speaker's orientation information to the receiving end through the network, and the receiving end selects one or more speakers to play according to the received orientation information, so that the speaker can be reproduced at the receiving end. location information.

In the above solution, the speaker's orientation information collected by the sending end is relative to the elongated device, not relative to the camera lens. When the camera lens is turned, the speaker directly in front of the strip device is on the side of the screen, or even not in the screen, but the collected sound orientation information is still directly in front, which leads to the position of the speaker in the screen and the collected information. Sound orientation information does not match.

In addition, the sending end needs to send the orientation information to the receiving end through the network. If the sending end and the receiving end are devices of different manufacturers, there will be intercommunication problems, which means that the receiving end cannot correctly process the orientation information of the sending end.

Contents of the invention

Embodiments of the present invention provide a video terminal and an audio stream processing method, so that the sending end does not need to send the location information of the sound source to the receiving end through the network, and the replayed sound can also accurately match the location of the sound source.

A method for processing an audio code stream, the method comprising:

Decode the video compression code stream to obtain the image containing the audio source;

If the sound source is a speaker, detecting the lip position of the speaker from the image according to the lip feature;

According to the detected lip position, detect the lip movement position;

If the lip movement position has been detected in the previous frame image obtained by decoding the video compression code stream, then the current frame detects whether there is a lip near the lip movement position of the previous frame, if not, then in the entire Detect the lip movement position within the image range, if there is, further judge whether the lips are moving; if they are moving, use the moving lip position as the lip movement position;

Detecting position information of the sound source according to the detected lip movement position;

Decode the audio compression code stream corresponding to the video compression code stream to obtain voice information;

The voice information is processed according to the location information of the sound source, so that the replayed sound orientation matches the location information of the sound source.

A video terminal, comprising:

The video decoding module is used to decode the received video compression code stream, and output the decoded image;

The audio decoding module is used to decode the audio compression code stream corresponding to the received video compression code stream, and output the decoded voice information;

The sound source position detection module is used to receive the image sent by the video decoding module, and extract the sound source as the speaker's lip feature, and detect the speaker's lip position from the image according to the lip feature;

According to the detected lip position, detect the lip movement position;

If the lip movement position has been detected in the previous frame image obtained by decoding the video compression code stream, then the current frame detects whether there is a lip near the lip movement position of the previous frame, if not, then in the entire Detect the lip movement position within the image range, if there is, further judge whether the lips are moving; if they are moving, use the moving lip position as the lip movement position;

According to the detected lip movement position, the sound source position information is detected;

The sound orientation processing module is used to receive the voice information sent by the audio decoding module and the sound source location information sent by the sound source location detection module, and match the sound location and the location of the sound source.

The embodiments of the present invention process the replayed sound by detecting the position information of the sound source in the image, so that the orientation of the replayed sound in the speaker can match the position of the sound source in the image; at the same time, the receiving terminal does not have to rely on the sending terminal to provide the sound source location information.

Description of drawings

Fig. 1 is the method flowchart of the embodiment of the present invention;

Fig. 2 is an application scene of the embodiment of the present invention;

Fig. 3 is the flowchart of lip movement detection in the embodiment of the present invention;

Fig. 4 is a structural diagram of a video terminal in an embodiment of the present invention.

Detailed ways

An embodiment of the present invention provides a method for processing an audio code stream. As shown in Figure 1, the method consists of the following steps:

Decoding the video compression code stream to obtain an image containing a sound source, and detecting the position information of the sound source in the image;

Decode the audio compression code stream corresponding to the video compression code stream to obtain voice information;

The voice information is processed according to the position information of the sound source, so that the position of the played sound matches the position of the sound source.

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The present invention will be described in detail below by taking a video conference as an application scenario of the embodiment of the present invention. But this application scenario is not used to limit the present invention.

Fig. 2 is a schematic diagram of a video communication system. In FIG. 2 , 10 is a site at the sending end, 11 is a site at the receiving end, and 12 is a communication network, which may be an IP network, a PSTN network, or a wireless network. In the venue 10, 101 is a camera, 102 is a video communication terminal, 103 is a TV, 104 is a participant, and 105 and 106 are speakers. The terminal 102 has a built-in microphone, or it can be placed outside independently, and connected to the terminal 112 through a transmission line. In the venue 11, 111 is a camera, 112 is a video communication terminal, 113 is a television, 104a is an image of a participant 104, and 115 and 116 are speakers. The terminal 112 has a built-in microphone, or it can be placed outside independently, and connected to the terminal 102 through a transmission line. After the camera 101 in the venue 10 at the sending end captures the image, it transmits it to the terminal 102. After the terminal 102 encodes the image, it transmits it to the terminal 112 through the network 12. The terminal 112 decodes the received image code stream and decodes the The subsequent images are transmitted to the TV 113 for display. After the microphone in the venue 10 captures the sound signal, it is transmitted to the terminal 102, and the terminal 102 performs audio encoding, and transmits the encoded audio stream to the terminal 112 through the network 12, and after the terminal 112 decodes the received audio stream, transmits it to Loudspeakers 115, 116 reproduce.

In the 11 venues in FIG. 2 , in order to make the sound have a sense of presence, it is necessary to match the sound reproduced by the speakers 115 and 116 to the position of the speaker 104a.

Below we take the speaker in the conference as the sound source in the video conference as an example to illustrate the method of the present invention:

Step1: Decode the compressed video stream sent by the sender to obtain the image of the sender, and then detect the position information of the speaker in the image.

Video decoding is performed on the video compression code stream to obtain multi-frame images, and then the images in the frame sequence are detected to obtain the position information of the speaker.

Among them, there are many ways to detect the position of the speaker. For example, image recognition technology is used to detect the position of the speaker in the image using certain characteristics of the speaker as features. The features that can be used for detection include face, eyes, lips, etc. , below we take the speaker's lips as an example to illustrate how to determine the position information of the speaker by detecting the position of the speaker's lip movement.

Please refer to the lip movement detection processing flow in FIG. 3 .

S11: Detect the lip movement position in the current frame, if there is lip movement in the current frame, execute step S12; otherwise, execute step S14;

S12: further judge whether there are multiple lip movement positions, if there are multiple lip movement positions, select a lip movement position among the multiple lip movement positions, or calculate the center position of the multiple lip movement positions and use this center position as Lip movement position, execute step S13; otherwise, directly execute step S13;

S13: output the position of lip movement;

S14: Do not output the lip movement position.

The lip movement position is the position of the speaker's lips. The detection method in the prior art may be used to detect the position of the lip movement. A simple and effective method is based on lip color, lip color search can be done in YIQ or YUV color space. For example, in the YIQ space, after statistical and experimental results, the optimal thresholds for each component of lip color are Y∈[80,220], I∈[12,78], Q∈[7,25]. According to these thresholds, it is relatively easy to search for the position of the lips. If the search is only based on the lip color, it will inevitably lead to some misjudgments. Therefore, after the lip position is searched based on the lip color, it can be further judged based on the skin color around the lips. Skin color also has a relatively concentrated threshold range, which can be used to determine whether the color around the lips is skin color. If so, it means that the judgment of the lip position is correct, otherwise it is incorrect. In addition, features that can be used include eye features, etc.

After judging the position of the lips, it is also necessary to judge whether the lips are in a moving state, which can be easily judged according to the size and speed of the lips at the same position in several frames of images before and after. Since the lip movement position is continuous, it is not necessary to detect the lip movement position in the entire range of the image in each frame. The specific method is that if the lip movement position has been detected in the previous frame, then the lip movement position of the current frame can be detected. Detect whether there are lips near the lip movement position in the previous frame. If not, search for the lip movement position in the entire image range. If there is, further judge whether the lips are moving; if they are moving, move the position of the lips As the lip movement position, otherwise, set a predetermined number of frames, and keep the lip movement position unchanged within the predetermined number of frames after the current frame. If there is no movement of the lips beyond the predetermined number of frames, start searching for lips in the entire image range again. move position. Using this method can greatly reduce the amount of calculation, and can ensure the continuity of sound orientation.

In video communication, especially in the application of video conferencing, there may be multiple participants in the same venue. Choose an appropriate lip movement position from the three lip movement positions. As mentioned before, if there is a lip movement position in the previous frame, the lip movement position is only detected in the vicinity of the lip movement position in the previous frame, so if multiple lip movement positions are detected, the lip movement position is also searched in the entire image range just happened. There are many strategies for selecting a lip movement position from multiple lip movement positions, such as selecting the front lip movement position and filtering out the side lip movement positions; lip position. In the venue, sometimes there may be multiple speakers at the same time. If the above methods cannot be used to select a suitable lip movement position, at this time, the center position of the lip movement positions of these multiple speakers can be calculated, and this center position can be used as The position of the output lip movement.

Step2: Decode the audio compression code stream sent by the sender to obtain voice information;

The decoding of the audio compressed code stream and the video compressed code stream described in Step1 and Step2 can be performed simultaneously or separately, and there is no sequence.

Step3: Process the received voice information according to the speaker's position information, so that the speaker's voice direction matches its position.

Processing the speech information according to the position of the speaker can be realized by using the method in the prior art. An example is given below. For the application scenario in Figure 2, if two speakers are played back, and the two speakers are on the left and right sides of the TV, a sound processing solution is to adjust the sound amplitude of the left and right channels to achieve the sound level and picture The purpose of matching the position of the speaker is to match the position of the speaker with the direction of the voice. The specific adjustment method can be described by the following two formulas:

D=(g1-g2)/(g1+g2)

C=g1*g1+g2*g2

In the above two formulas, C is a fixed value, g1 is the amplitude gain of the left channel, g2 is the amplitude gain of the right channel, and D is the relative distance of the speaker in the horizontal direction on the screen calculated according to the lip movement position information, so that The distance between the lip movement position and the vertical line in the middle of the screen is D' (the lip movement position is a positive value on the left side of the screen, and a negative value on the right side), and the horizontal width of the TV screen is W, then D is calculated according to the following formula:

D=D'/(W/2)

The method of processing sound according to the position information of the sound source can also adopt HRTF (Head Related Transfer Functions, Head Related Transfer Function). The technology of using HRTF to virtualize a sound source has been disclosed in existing technical documents, and will not be described in detail in the present invention.

In the method provided by the embodiment of the present invention, by detecting and obtaining the speaker's position information in the place of sound reproduction, the receiving terminal does not need to rely on the sending terminal to provide the speaker's position information; The replayed voice information is processed, so that the replayed sound and the position of the speaker in the image can be accurately matched.

It should be noted that the audio code stream processing method provided by the present invention is not limited to processing the audio code stream received from the sending end, and is also suitable for processing locally stored video and audio code streams.

The embodiment of the invention also provides a video terminal. As shown in Figure 4, there are modules such as video decoding, audio decoding, sound source position detection, and sound orientation processing in the video communication terminal. After the video compression code stream is decoded by the video decoding module, it is output to the TV for display on the one hand, and is output to the sound source position detection module on the other hand. The sound source position detection module receives the image output by the video decoding module, detects the image, extracts the feature of the sound source, thereby obtains the sound source position information, and outputs the sound source position information to the sound orientation processing module. After the audio compression code stream is decoded by the audio decoding module, it is output to the sound orientation processing module. The sound orientation processing module processes the received audio code stream according to the location information of the sound source, so that the processed sound orientation is consistent with the location of the sound source, and generates two audio outputs on the left and right, which are respectively sent to the left and right speakers for playback. In order to have a better sound playback effect, the video communication terminal can be connected with three or more speakers externally, and at this time, the sound orientation processing module outputs three or more audio streams correspondingly.

The purpose of the sound source position detection module in the video terminal is to detect the image output by the video decoding module, and obtain the position information of the sound source therein. Therefore, if the audio source is a speaker in a video terminal, position detection can be realized by extracting the speaker’s lip features, or by detecting features such as the speaker’s face, as long as the module can detect The position of the speaker is sufficient.

If the speaker's position is detected based on the speaker's lips, the sound source position detection module includes:

The first receiving module is used to receive the image containing the speaker sent by the video decoding module;

A feature extraction module, configured to extract the lip features of the speaker in the image received by the first receiving module;

The position detection module is used to determine the position of the speaker according to the lip features of the speaker extracted by the feature extraction module.

Wherein, the lip movement detection method described above may be used to detect the lip movement position.

The sound orientation processing module includes:

The second receiving module is used to receive the voice information sent by the audio decoding module and the speaker's location information sent by the location detection module;

A matching module, configured to match the position of the replayed sound with the position of the speaker according to the voice information received by the second receiving module and the position information of the speaker.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. An audio code stream processing method, characterized in that, comprising: Decode the video compression code stream to obtain the image containing the audio source; If the sound source is a speaker, detecting the lip position of the speaker from the image according to the lip feature; According to the detected lip position, detect the lip movement position; If the lip movement position has been detected in the previous frame image obtained by decoding the video compression code stream, then the current frame detects whether there is a lip near the lip movement position of the previous frame, if not, then in the entire Detect the lip movement position within the image range, if there is, further judge whether the lips are moving; if they are moving, use the moving lip position as the lip movement position; Detecting position information of the sound source according to the detected lip movement position; Decode the audio compression code stream corresponding to the video compression code stream to obtain voice information; The voice information is processed according to the location information of the sound source, so that the replayed sound orientation matches the location information of the sound source. 2. The method according to claim 1, characterized in that the lip movement position is detected according to the size and the speed of change of the lips at the same position in several frames of images before and after. 3. The method according to claim 1, wherein when at least two speakers are used to replay the voice, the processing of the voice information according to the position information of the sound source is specifically: Adjust the amplitude of the sound of the left and right channels of the speaker, so that the horizontal orientation of the sound matches the position of the speaker. 4. The method according to claim 1, wherein said detecting the location information of said sound source in said image further comprises: When there are multiple lip movement positions in the image, calculate the central position of the multiple lip movement positions, and use this central position as the output speaker's position. 5. The method of claim 1, wherein said lip characteristics include lip color. 6. The method according to claim 5, characterized in that after determining the position of the lips according to the color of the lips, it is further judged whether the color around the lips is the color of the skin. 7. The method according to claim 5 or 6, after detecting the position of the lips, it is further judged whether the lips are moving; if they are moving, the position of the moving lips is taken as the lip movement position, otherwise, a predetermined number of frames is set, The lip movement position is kept unchanged within the predetermined number of frames after the current frame, and if there is no movement of the lips beyond the predetermined number of frames, the search for the lip movement position in the entire image range is restarted. 8. A video terminal, characterized in that, The video decoding module is used to decode the received video compression code stream, and output the decoded image; The audio decoding module is used to decode the audio compression code stream corresponding to the received video compression code stream, and output the decoded voice information; The sound source position detection module is used to receive the image sent by the video decoding module, and extract the sound source as the speaker's lip feature, and detect the speaker's lip position from the image according to the lip feature; According to the detected lip position, detect the lip movement position; If the lip movement position has been detected in the previous frame image obtained by decoding the video compression code stream, then the current frame detects whether there is a lip near the lip movement position of the previous frame, if not, then in the entire Detect the lip movement position within the image range, and if so, further judge whether the lips are moving; if they are moving, use the moving lip position as the lip movement position; according to the detected lip movement position, the sound source position information is detected; The sound orientation processing module is used to receive the voice information sent by the audio decoding module and the sound source location information sent by the sound source location detection module, and match the sound location and the location of the sound source. 9. The video terminal according to claim 8, wherein the sound source position detection module comprises: The first receiving module is used to receive the image containing the speaker sent by the video decoding module; A feature extraction module, configured to extract the lip features of the speaker in the image received by the first receiving module; The position detection module is used to determine the position of the speaker according to the lip features extracted by the feature extraction module. 10. The video terminal according to claim 8, wherein the sound orientation processing module comprises: The second receiving module is used to receive the voice information sent by the audio decoding module and the speaker's location information sent by the location detection module; A matching module, configured to match the position of the replayed sound with the position of the speaker according to the voice information received by the second receiving module and the position information of the speaker.

Images