CN107205131A - A kind of methods, devices and systems for realizing video calling - Google Patents

Abstract

A kind of methods, devices and systems for realizing video calling, including：First terminal gathers digital audio and video signals and digital video signal respectively；Digital audio and video signals are converted to text message by first terminal, and text message is packaged into text bag, digital audio and video signals are packaged into audio pack, digital video signal is packaged into video bag；Text bag, audio pack and video bag are sent to second terminal by first terminal respectively.

Description

A method, device and system for implementing video calls

technical field

This article relates to but not limited to the field of video calls, especially a method, device and system for realizing video calls.

Background technique

With the rapid development of mobile and Internet broadband technology, the visual communication value-added service has been rapidly promoted among home users. Through the technology based on this service, value-added services such as face-to-face communication and online video teaching can be obtained. Adding synchronous subtitles to the audio of video communication services can not only provide better services to users with poor hearing, but also make a useful supplement to the actual audio effect when the network is not good.

In related technologies, methods for adding voice subtitles in a video call roughly include:

The first terminal separately collects digital audio signals and digital video signals; performs voice coding processing on the collected digital audio signals, and encapsulates the digital audio signals after voice coding processing into audio packets; converts the collected digital audio signals through voice recognition technology For text information, the text information and the collected digital video signal are superimposed and synthesized, and then the video encoding process is performed, and the digital video signal after the video encoding process is packaged into a video packet; the audio packet and the video packet are respectively sent to the second terminal;

The second terminal receives the audio packet and the video packet, performs speech decoding on the digital audio signal after the speech coding processing in the audio packet to obtain a digital audio signal and plays it, and performs video decoding on the digital video signal after the intermediate frequency coding processing of the video packet to obtain a digital video signal and display.

In the above method, when the network condition is not good, since the video packet is relatively large, the probability of packet loss and jitter in the video packet will be greater. In this way, the text information will be lost together with the video packet, resulting in Information is lost.

Contents of the invention

Embodiments of the present invention provide a method, device and system for implementing video calls, which can reduce information loss during video calls when the network condition is poor.

The embodiment of the present invention proposes a method for implementing a video call, including:

The first terminal respectively collects digital audio signals and digital video signals;

The first terminal converts the digital audio signal into text information, encapsulates the text information into a text packet, encapsulates the digital audio signal into an audio packet, and encapsulates the digital video signal into a video packet;

The first terminal respectively sends the text packet, the audio packet and the video packet to the second terminal.

Optionally, before encapsulating the digital audio signal into an audio packet, the method further includes: performing speech coding processing on the digital audio signal by the first terminal;

The encapsulating the digital audio signal into an audio packet includes: the first terminal encapsulating the speech encoded digital audio signal into the audio packet.

Optionally, before encapsulating the digital video signal into a video packet, the method further includes: performing video encoding processing on the digital video signal by the first terminal;

The encapsulating the digital video signal into a video packet includes: the first terminal encapsulating the video encoded digital video signal into the video packet.

The embodiment of the present invention also proposes a method for implementing a video call, including:

The second terminal receives the text packet from the first terminal;

The second terminal judges that the time corresponding to the timestamp in the received text packet is less than or equal to the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, and displays the received text packet and the buffered text packet. , the text information in the text packet whose time corresponding to the timestamp field is less than or equal to the time corresponding to the timestamp field of the audio packet being played or the video packet being displayed.

Optionally, when the second terminal determines that the time corresponding to the timestamp in the received text packet is greater than the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, the method further includes :

The second terminal buffers the received text package.

Optionally, when the second terminal does not receive the audio package and the video package within the preset time after receiving the text package, the method also includes:

The second terminal displays the text information in the cached text package.

Optionally, after the second terminal receives the text packet from the first terminal, the second terminal determines that the time corresponding to the time stamp in the received text packet is less than or equal to the audio packet being played or the audio packet being played. Before the time corresponding to the timestamp of the displayed video package also includes:

The second terminal determines that the subtitle display function has been turned on.

The embodiment of the present invention also proposes a first terminal, including:

Acquisition module, used for respectively collecting digital audio signal and digital video signal;

The first processing module is used to convert the digital audio signal into text information, encapsulate the text information into a text packet, encapsulate the digital audio signal into an audio packet, and encapsulate the digital video signal into a video packet;

A sending module, configured to respectively send the text packet, the audio packet and the video packet to the second terminal.

Optionally, the first processing module is specifically used for:

converting the digital audio signal into text information, performing speech encoding processing on the digital audio signal, encapsulating the text information into a text packet, encapsulating the speech encoded digital audio signal into the audio packet, and processing the digital video signal Perform video encoding processing, and encapsulate the digital video signal after video encoding processing into the video packet.

The embodiment of the present invention also proposes a second terminal, including:

a receiving module, configured to receive a text packet from the first terminal;

The second processing module is used to judge that the time corresponding to the timestamp in the received text packet is less than or equal to the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, and display the received text packet and cache In the text packet, the time corresponding to the timestamp field is less than or equal to the text information in the text packet of the time corresponding to the timestamp field of the audio packet being played or the video packet being displayed.

Optionally, the second processing module is also used for:

When it is determined that the time corresponding to the time stamp in the received text package is greater than the time corresponding to the time stamp of the audio package being played or the video package being displayed, buffer the received text package.

Optionally, the second processing module is also used for:

When the audio package and the video package are not received within a preset time after the text package is received, the text information in the cached text package is displayed.

The embodiment of the present invention also proposes a system for realizing video calls, including:

The first terminal is used to separately collect digital audio signals and digital video signals; convert the digital audio signals into text information, encapsulate the text information into text packets, encapsulate the digital audio signals into audio packets, and encapsulate the digital video signals into video packets ; Send the text packet, audio packet and video packet to the second terminal respectively;

The second terminal is used to receive the text packet from the first terminal; it is determined that the time corresponding to the timestamp in the received text packet is less than or equal to the time corresponding to the timestamp of the audio packet being played or the video packet being displayed , to display the text information in the text packets whose time corresponding to the timestamp field in the received text packet and cached text packet is less than or equal to the time corresponding to the timestamp field of the audio packet being played or the video packet being displayed.

Optionally, the second terminal is also used for:

When it is determined that the time corresponding to the time stamp in the received text package is greater than the time corresponding to the time stamp of the audio package being played or the video package being displayed, buffer the received text package.

Optionally, the second terminal is also used for:

When the audio package and the video package are not received within a preset time after the text package is received, the text information in the cached text package is displayed.

Compared with related technologies, the technical solutions of the embodiments of the present invention include: the first terminal separately collects digital audio signals and digital video signals; the first terminal converts the digital audio signals into text information, encapsulates the text information into text packets, and converts the digital The audio signal is encapsulated into an audio packet, and the digital video signal is encapsulated into a video packet; the first terminal respectively sends the text packet, the audio packet and the video packet to the second terminal. Through the scheme of the embodiment of the present invention, the first terminal sends the text packet, audio packet and video packet to the second terminal respectively, so that when the network condition is not good, the text will not be lost when the video packet is lost, thereby reducing the Information lost during a call.

Description of drawings

The accompanying drawings in the embodiments of the present invention are described below. The accompanying drawings in the embodiments are used for further understanding of the present invention and are used together with the description to explain the present invention, and do not constitute a limitation to the protection scope of the present invention.

FIG. 1 is a flow chart of a method for implementing a video call at a sending end according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method for implementing a video call at a receiving end according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of the structural composition of a first terminal according to an embodiment of the present invention;

FIG. 4 is a schematic structural diagram of a second terminal according to an embodiment of the present invention;

FIG. 5 is a schematic structural composition diagram of a system for implementing a video call according to an embodiment of the present invention.

detailed description

In order to facilitate the understanding of those skilled in the art, the present invention will be further described below in conjunction with the accompanying drawings, which cannot be used to limit the protection scope of the present invention. It should be noted that, in the case of no conflict, the embodiments in the present application and various manners in the embodiments can be combined with each other.

Referring to Fig. 1, the embodiment of the present invention proposes a method for implementing a video call, including:

Step 100, the first terminal separately collects digital audio signals and digital video signals.

In this step, the first terminal may collect digital audio signals at the collection time specified in G.711 (an audio coding method formulated by the International Telecommunication Union), and collect digital video signals at a preset video frame rate. For example, a digital audio signal is collected every 10 milliseconds (ms), and a digital video signal is collected every 40 ms.

Step 101. The first terminal converts digital audio signals into text information, encapsulates the text information into text packets, encapsulates digital audio signals into audio packets, and encapsulates digital video signals into video packets.

In this step, the first terminal may use voice recognition technology to convert the digital audio signal into text information.

In this step, the text packet, or audio packet, or video packet may be encapsulated according to the specification of the Real-time Transport Protocol (RTP, Real-time Transport Protocol) packet protocol.

The format of the header of the RTP packet is shown in Table 1.

Table 1

In Table 1, V represents the protocol version, 2 bits (bit),

P represents a padding bit, 1 bit, when P is set, the header and tail of the RTP packet contain additional padding bytes.

X is an extension bit, 1 bit, when X is set, it means that a header is extended after the header of the RTP packet.

CC represents the number of Contributing Source Identifiers identifiers.

M is a marker bit, 1 bit.

PT is a payload type (Payload Type), 7 bits, and for a text packet, it can be represented by a type that is not used in the related art, for example, 20.

Sequence number, 16 bits, every time an RTP packet is sent, the sequence number increases by 1. In the embodiment of the present invention, the serial numbers of the text package, the audio package and the video package are numbered independently.

Timestamp, 32 bits, records the sampling moment of the first byte in the RTP packet. For audio and video packets, the time stamp is the time when the collection starts, and for the text package, the time stamp is the time when the corresponding audio package starts to be collected.

Synchronization Source Identifier (SSRC, Synchronization Source Identifier), 32 bits, indicates the source of the RTP packet, and there cannot be two identical SSRC values in the same RTP session.

CSRC, 0~15 items, 32 bits each, this field is not required for the header of the RTP packet.

The timestamp field in the text packet is the time when the digital audio signal or digital video signal is collected, and the Payload Type is the voice text information type (undefined values can be used, such as 20, etc.).

Step 102, the first terminal respectively sends the text packet, the audio packet and the video packet to the second terminal.

In this step, different types of packets may be sent separately according to different strategies. For example, audio packets are sent according to the audio coding sampling frequency, video packets are sent according to the agreed frame rate interval, and text packets are sent according to the audio coding sampling frequency.

Optionally, before encapsulating the digital audio signal into an audio packet, it also includes: the first terminal performs speech coding processing on the digital audio signal; correspondingly,

Encapsulating the digital audio signal into an audio packet includes: the first terminal encapsulating the speech encoded digital audio signal into an audio packet.

Optionally, before encapsulating the digital video signal into a video packet, the method further includes: performing video encoding processing on the digital video signal by the first terminal; correspondingly,

The encapsulating the digital video signal into a video packet includes: the first terminal encapsulating the video encoded digital video signal into the video packet.

Through the scheme of the embodiment of the present invention, the first terminal sends the text packet, audio packet and video packet to the second terminal respectively, so that when the network condition is not good, the text will not be lost when the video packet is lost, thereby reducing the Information lost during a call.

Referring to Fig. 2, the embodiment of the present invention also proposes a method for implementing a video call, including:

Step 200, the second terminal receives the text packet from the first terminal.

Step 201, the second terminal determines that the time corresponding to the timestamp in the received text packet is less than or equal to the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, and displays the received text packet and cached In the text package, the text information in the text package whose time corresponding to the time stamp is less than or equal to the time corresponding to the time stamp of the audio package being played or the video package being displayed.

In this step, the text information can be displayed according to the preset display area and/or font size. Specifically, the number of words that can be displayed on the screen at one time can be determined according to the display area and/or font size, the number of times the text information of a text package needs to be displayed can be calculated, and the dwell time for displaying once can be determined according to the acquisition frequency of the audio package corresponding to a text package , displayed according to dwell time.

For example, the audio packet corresponding to a text packet is collected once every 20ms. The text packet has 100 words in total, and the number of words that can be displayed at one time is 10 words, so it needs to be displayed 10 times, and the dwell time of each display is 2ms.

In this step, the text information can be displayed on the graphics layer of the screen, that is, superimposed on the video layer for displaying digital video signals for display.

Optionally, between step 200 and step 201:

The second terminal determines that the subtitle display function has been turned on.

When the second terminal determines that the subtitle display function is off, this process ends.

The method also includes:

The second terminal determines that the time corresponding to the timestamp in the received text packet is greater than the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, and buffers the received text packet.

The method also includes:

The second terminal does not receive the audio package and the video package within the preset time, and displays the text information in the cached text package.

In the above method, after the second terminal receives the audio packet and/or video packet, it can play or display according to the rules stipulated in the audio and video decoding protocol standard.

Wherein, after the second terminal receives the audio packet, it can play according to the rules agreed in the audio decoding protocol standard (such as G711); after receiving the video packet, the second terminal can play the to display.

Referring to Fig. 3, the embodiment of the present invention also proposes a first terminal, including:

Acquisition module, used for respectively collecting digital audio signal and digital video signal;

The first processing module is used to convert the digital audio signal into text information, encapsulate the text information into a text packet, encapsulate the digital audio signal into an audio packet, and encapsulate the digital video signal into a video packet;

A sending module, configured to respectively send the text packet, the audio packet and the video packet to the second terminal.

In the first terminal of the embodiment of the present invention, the first processing module is specifically used for:

Convert the digital audio signal into text information, perform speech coding processing on the digital audio signal, encapsulate the text information into a text package, package the digital audio signal after the speech coding process into an audio package, and perform video coding processing on the digital video signal. The digital video signal processed by video encoding is encapsulated into a video packet.

Referring to Figure 4, the embodiment of the present invention also proposes a second terminal, including:

a receiving module, configured to receive a text packet from the first terminal;

The second processing module is used to judge that the time corresponding to the timestamp in the received text packet is less than or equal to the time corresponding to the timestamp of the audio packet being played or the video packet being displayed, and display the received text packet and cache In the text packet, the time corresponding to the timestamp field is less than or equal to the text information in the text packet of the time corresponding to the timestamp field of the audio packet being played or the video packet being displayed.

In the second terminal in the embodiment of the present invention, the second processing module is also used for:

When it is determined that the time corresponding to the time stamp in the received text package is greater than the time corresponding to the time stamp of the audio package being played or the video package being displayed, the received text package is cached.

In the second terminal in the embodiment of the present invention, the second processing module is also used for:

When the audio package and the video package are not received within the preset time after receiving the text package, the text information in the cached text package is displayed.

Referring to Fig. 5, the embodiment of the present invention also proposes a system for implementing video calls, including:

The first terminal is used to separately collect digital audio signals and digital video signals; convert the digital audio signals into text information, encapsulate the text information into text packets, encapsulate the digital audio signals into audio packets, and encapsulate the digital video signals into video packets ; Send the text packet, audio packet and video packet to the second terminal respectively;

The second terminal is used to receive the text package from the first terminal; it is determined that the time corresponding to the time stamp in the received text package is less than or equal to the time corresponding to the time stamp of the audio package being played or the video package being displayed Time, displaying the text information in the text packets whose time corresponding to the timestamp field is less than or equal to the time corresponding to the timestamp field of the audio packet being played or the video packet being displayed in the received text packet and cached text packet.

In the system of the embodiment of the present invention, the second terminal is also used for:

When it is determined that the time corresponding to the time stamp in the received text package is greater than the time corresponding to the time stamp of the audio package being played or the video package being displayed, the received text package is cached.

In the system of the embodiment of the present invention, the second terminal is also used for:

When the audio package and the video package are not received within the preset time after receiving the text package, the text information in the cached text package is displayed.

It should be noted that the above-described embodiments are only for the convenience of those skilled in the art to understand, and are not intended to limit the protection scope of the present invention. Any obvious replacements and improvements made by the invention are within the protection scope of the present invention.

Claims (15)

1. a kind of method for realizing video calling, it is characterised in that including：

First terminal gathers digital audio and video signals and digital video signal respectively；

Digital audio and video signals are converted to text message by first terminal, and text message is packaged into text bag, Digital audio and video signals are packaged into audio pack, digital video signal is packaged into video bag；

Text bag, audio pack and video bag are sent to second terminal by first terminal respectively.

2. according to the method described in claim 1, it is characterised in that described to encapsulate digital audio and video signals Also include before into audio pack：The first terminal carries out voice coding processing to the digital audio and video signals；

It is described digital audio and video signals are packaged into audio pack to include：The first terminal is to voice coding processing Digital audio and video signals afterwards are packaged into the audio pack.

3. according to the method described in claim 1, it is characterised in that described to encapsulate digital video signal Also include before into video bag：The first terminal carries out Video coding processing to the digital video signal；

It is described digital video signal is packaged into video bag to include：The first terminal is to Video coding processing Digital video signal afterwards is packaged into the video bag.

4. a kind of method for realizing video calling, it is characterised in that including：

Second terminal receives the text bag from first terminal；

Second terminal judges that the timestamp corresponding time in the text bag received is less than or equal to The timestamp corresponding time of the audio pack of broadcasting or the video bag shown, show the text received In the text bag of bag and caching, the timestamp field corresponding time is less than or equal to the audio pack played Or the text message in the text bag of the timestamp field corresponding time of the video bag shown.

5. method according to claim 4, it is characterised in that when the second terminal judges institute The timestamp corresponding time in the text bag received is stated more than the audio pack played or is shown Video bag timestamp corresponding time when, this method also includes：

The text bag received described in the second terminal caching.

6. method according to claim 5, it is characterised in that when second terminal receive it is described When not receiving audio pack and video bag in the preset time after text bag, this method also includes：

Text message in the text bag of the second terminal display caching.

7. method according to claim 4, it is characterised in that the second terminal, which is received, to be come from After the text bag of first terminal, the timestamp pair in the second terminal judges the text bag that receives The time answered is less than or equal to the audio pack played or the timestamp of the video bag shown is corresponding Also include before time：

The second terminal judges that caption display function has been opened.

8. a kind of first terminal, it is characterised in that including：

Acquisition module, for gathering digital audio and video signals and digital video signal respectively；

First processing module, for digital audio and video signals to be converted into text message, text message is encapsulated Into text bag, digital audio and video signals are packaged into audio pack, digital video signal is packaged into video bag；

Sending module, for text bag, audio pack and video bag to be sent into second terminal respectively.

9. first terminal according to claim 8, it is characterised in that the first processing module tool Body is used for：

Digital audio and video signals are converted into text message, the digital audio and video signals are carried out at voice coding Reason, text bag is packaged into by text message, and institute is packaged into the digital audio and video signals after voice coding processing Audio pack is stated, Video coding processing is carried out to the digital video signal, to the number after Video coding processing Word vision signal is packaged into the video bag.

10. a kind of second terminal, it is characterised in that including：

Receiving module, for receiving the text bag from first terminal；

Second processing module, is less than for the timestamp corresponding time in the text bag judging to receive Or equal to the timestamp corresponding time of the audio pack played or the video bag shown, display connects In the text bag and the text bag of caching that receive, the timestamp field corresponding time, which is less than or equal to, to be broadcast Text in the text bag of the timestamp field corresponding time of the audio pack put or the video bag shown Information.

11. second terminal according to claim 10, it is characterised in that the Second processing module It is additionally operable to：

It is more than the sound played when the timestamp corresponding time in the text bag received of judging Frequency was wrapped or during the timestamp of video bag that shows corresponding time, the text bag received described in caching.

12. second terminal according to claim 11, it is characterised in that the Second processing module It is additionally operable to：

When not receiving audio pack and video bag in the preset time after receiving the text bag, show Show the text message in the text bag of caching.

13. a kind of system for realizing video calling, it is characterised in that including：

First terminal, for gathering digital audio and video signals and digital video signal respectively；DAB is believed Number text message is converted to, text message is packaged into text bag, digital audio and video signals are packaged into audio Bag, video bag is packaged into by digital video signal；Text bag, audio pack and video bag are sent to respectively Second terminal；

Second terminal, for receiving the text bag from first terminal；Judge the text bag received In the timestamp corresponding time be less than or equal to the audio pack played or the video bag that shows In timestamp corresponding time, the text bag for showing the text bag received and caching, timestamp field pair The time answered is less than or equal to the timestamp field pair of the audio pack played or the video bag shown Text message in the text bag for the time answered.

14. system according to claim 13, it is characterised in that the second terminal is additionally operable to：

It is more than the sound played when the timestamp corresponding time in the text bag received of judging Frequency was wrapped or during the timestamp of video bag that shows corresponding time, the text bag received described in caching.

15. system according to claim 14, it is characterised in that the second terminal is additionally operable to：

When not receiving audio pack and video bag in the preset time after receiving the text bag, show Show the text message in the text bag of caching.