CN116233563A - Method and system for transmitting audio and video data - Google Patents

Abstract

The embodiment of the present application provides a method and system for transmitting audio and video data. A connection based on the QUIC protocol is established between the sending end and the receiving end. The connection based on the QUIC protocol includes multiple transmission streams. The method includes: Decapsulate the audio and video stream data to obtain multiple types of source data; encapsulate according to the type of each type of source data to obtain a target data message supporting a preset protocol; determine from the multiple transport streams and The type of the target data packet matches the target transport stream, and transmits the target data packet to the receiving end through the target transport stream. Based on QUIC as the transmission protocol, after the first 1-RTT handshake based on the QUIC protocol to establish a connection, the subsequent handshake reconnection only needs 0-RTT; the RTMP protocol can be replaced by a custom preset protocol, which can realize the connection based on the QUIC protocol. Send audio and video data directly, saving the time of RTMP handshake and greatly reducing the delay of the first frame.

Description

Method and system for audio and video data transmission

technical field

The embodiments of the present application relate to the technical field of data transmission, and in particular to a method, system, computer equipment, and computer-readable storage medium for audio and video data transmission.

Background technique

In the prior art, if the scheme of TCP+RTMP is adopted to transmit audio and video stream data, three handshakes of TCP and multiple handshakes of RTMP must be performed before transmitting audio and video data, and these handshakes will greatly increase the transmission delay of the network; and , if the client changes the network, the TCP connection will be disconnected. If the client needs to continue receiving audio and video data, it needs to re-initiate the TCP and RTMP handshake. viewing experience. In addition, when using the same TCP connection to transmit audio and video data, in order to ensure the orderly transmission of data, TCP will have the problem of queue head blocking, that is, when transmitting audio and video data, if the data in the front is lost, it will cause Even if the data is received, it cannot be used by the application layer, causing audio and video data transmission to interfere with each other, and the video watched by the user freezes, affecting the viewing experience.

Contents of the invention

The purpose of the embodiments of the present application is to provide an audio and video data transmission method, system, computer equipment, and computer-readable storage medium, which are used to solve the following problem: when transmitting audio and video data, audio and video data transmission interfere with each other, causing users to The watched video freezes, which affects the viewing experience.

An aspect of the embodiment of the present application provides an audio and video data transmission method, which is applied to the sending end, and a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol includes multiple transmission streams , the method includes:

Decapsulate the audio and video stream data to be transmitted to obtain various types of source data;

Encapsulate according to the type of the source data of each type to obtain a target data message supporting a preset protocol;

Determining a target transport stream matching the type of the target data packet from the plurality of transport streams, and transmitting the target data packet to the receiving end through the target transport stream.

Optionally, encapsulating according to the type of each type of source data to obtain a target data message supporting a preset protocol, including:

Encapsulating header data based on the preset protocol in the source data; wherein the header data based on the preset protocol includes a type field;

The value of the type field is configured according to the type corresponding to the source data to obtain a target data packet supporting a preset protocol.

Optionally, the header data based on the preset protocol further includes a variable-length format field, a variable-length field and a timestamp field;

The encapsulation according to the type of each type of source data to obtain the target data message supporting the preset protocol also includes:

determining the length of the source data, and configuring the value of the variable-length format field and the value of the variable-length field according to the length of the source data; and,

Determine the time stamp corresponding to the source data in the audio and video stream data, and configure the value of the time stamp field according to the time stamp.

Optionally, the sum of the byte lengths of the variable-length format field and the type field is one byte, the byte length of the timestamp is four bytes, and the variable-length format field is used for Indicates the number of bytes corresponding to the variable-length field.

Optionally, configuring the value of the variable-length format field and the value of the variable-length field according to the length of the source data includes:

configuring the value of the variable length field according to the length of the source data, and determining the number of bytes corresponding to the variable length field according to the value of the variable length field;

The value of the variable-length format field is configured according to the number of bytes corresponding to the variable-length field.

Optionally, the encapsulating according to the type of each type of source data to obtain a target data message supporting a preset protocol further includes:

The header data based on the UDP protocol and the header data based on the QUIC protocol are encapsulated in the source data.

Optionally, the type of the source data includes metadata type, video frame type and audio frame type.

Optionally, the multiple transport streams are independent of each other, and each of the transport streams is used to transmit a type of data packet.

An aspect of the embodiment of the present application provides a method for audio and video data transmission, which is applied to the receiving end, and a connection based on the QUIC protocol is established between the receiving end and the sending end, and the connection based on the QUIC protocol includes multiple transport stream, the method comprising:

receiving the target data packet transmitted by the sender through the multiple transmission streams;

Analyzing the target data packet to obtain source data and header data based on a preset protocol;

Determine the type corresponding to the source data according to the header data based on the preset protocol;

Encapsulate the source data according to the type corresponding to the source data to obtain audio and video stream data.

Optionally, the header data based on the preset protocol includes a type field;

The determining the type corresponding to the source data according to the header data based on the preset protocol includes:

Determine the type corresponding to the source data according to the value of the type field.

Optionally, the header data based on the preset protocol also includes a variable-length format field, a variable-length field, and a timestamp field;

After the step of analyzing the target data message to obtain the source data and the header data based on the preset protocol, it also includes:

determining the number of bytes of the variable-length field according to the value of the variable-length format field;

According to the number of bytes of the variable length field, determine the value of the variable length field from the header data based on the preset protocol, and determine the source data according to the value of the variable length field the corresponding length;

According to the value of the timestamp field, determine the corresponding timestamp of the source data in the audio and video stream data.

Optionally, the encapsulation of the source data according to the type corresponding to the source data to obtain audio and video stream data includes:

According to the type, length and time stamp corresponding to the source data, the source data is encapsulated to obtain audio and video stream data.

An aspect of the embodiment of the present application provides an audio and video data transmission system, including a sending end and a receiving end, wherein a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol Contains multiple transport streams;

The sending end is used to obtain the audio and video stream data to be transmitted; decapsulate the audio and video stream data to obtain multiple types of source data; perform the source data according to each type of source data Encapsulate to obtain a target data message supporting a preset protocol; determine a target transport stream matching the type of the target data message from the multiple transport streams, and transmit the target data message through the target transport stream The text is transmitted to the receiving end;

The receiving end is used to receive the target data message transmitted by the sending end through the multiple transmission streams; parse the target data message to obtain source data and header data based on a preset protocol; according to the The header data of the preset protocol determines the type corresponding to the source data; the source data is encapsulated according to the type corresponding to the source data to obtain audio and video stream data.

An aspect of the embodiments of the present application provides a computer device, the computer device includes a memory, a processor, and a computer program stored in the memory and operable on the processor, when the processor executes the computer program The steps of the above-mentioned audio and video data transmission method are realized.

An aspect of the embodiments of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and the computer program can be executed by at least one processor, so that the at least one When the processor executes the computer program, the steps of the above audio and video data transmission method are realized.

The audio and video data transmission method, system, equipment, and computer-readable storage medium provided by the embodiments of the present application use QUIC as the transmission protocol. Compared with TCP, which requires 1.5-RTT for each connection establishment, this application is based on the QUIC protocol for the first time. After the 1-RTT handshake is used to establish a connection, the subsequent handshake reconnection only needs 0-RTT; the custom preset protocol is used to replace the RTMP protocol, and the audio and video data can be sent directly after the connection is established based on the QUIC protocol, eliminating the need for RTMP handshake Time, greatly reducing the delay of the first frame. Even if the client is disconnected or the network is reconnected, QUIC's connection migration and 0-RTT characteristics make the network seamlessly connect and send data, improving the smooth experience of users watching videos; using the multiplexing transmission characteristics of the QUIC protocol to realize Low-latency transmission of audio and video data.

In addition, this application divides a piece of audio and video stream data into different types of multiple types of data, and after these types of data are packaged through a custom preset protocol, multiple transmission streams through a connection based on the QUIC protocol are ordered Reliable and safe transmission, one type of data is sent using one transmission stream in the same connection, so that the transmission speed of various types of data does not affect each other, which can further reduce the delay of audio and video data in network transmission, and achieve low-latency transmission of audio Effects on video data.

Description of drawings

Fig. 1 schematically shows an application environment diagram of an audio and video data transmission method according to an embodiment of the present application;

FIG. 2 schematically shows a flow chart of an audio and video data transmission method according to Embodiment 1 of the present application;

Fig. 3 schematically shows a schematic diagram of a Connection for a QUIC under the C/S software architecture according to an embodiment of the present application;

FIG. 4 schematically shows a schematic diagram of a data structure of a data packet according to Embodiment 1 of the present application;

Fig. 5 schematically shows a schematic diagram of content contained in header data based on a preset protocol according to Embodiment 1 of the present application;

FIG. 6 schematically shows a flowchart of an audio and video data transmission method according to Embodiment 2 of the present application;

FIG. 7 schematically shows a block diagram of an audio and video data transmission device according to Embodiment 3 of the present application;

FIG. 8 schematically shows a block diagram of an audio and video data transmission device according to Embodiment 4 of the present application;

FIG. 9 schematically shows a block diagram of an audio and video data transmission system according to Embodiment 5 of the present application; and

FIG. 10 schematically shows a schematic diagram of a hardware architecture of a computer device suitable for implementing a method for transmitting audio and video data according to Embodiment 6 of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes, and should not be understood as indicating or implying their relative importance or implicitly indicating the indicated technical features quantity. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In addition, the technical solutions of the various embodiments can be combined with each other, but it must be based on the realization of those skilled in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that the combination of technical solutions does not exist , nor within the scope of protection required by the present application.

In the prior art, if the scheme of TCP+RTMP is adopted to transmit audio and video stream data, three handshakes of TCP and multiple handshakes of RTMP must be performed before transmitting audio and video data, and these handshakes will greatly increase the transmission delay of the network; and , if the client changes the network, the TCP connection will be disconnected. If the client needs to continue receiving audio and video data, it needs to re-initiate the TCP and RTMP handshake. viewing experience.

In addition, when using the same TCP connection to transmit audio and video data, in order to ensure the orderly transmission of data, TCP will have the problem of queue head blocking, that is, when transmitting audio and video data, if the data in the front is lost, it will cause Even if the data is received, it cannot be used by the application layer, causing audio and video data transmission to interfere with each other, and the video watched by the user freezes, affecting the viewing experience.

In view of this, the present application aims to propose an audio and video data transmission scheme based on the QUIC protocol, including: by establishing a connection based on the QUIC protocol between the sending end and the receiving end, the connection based on the QUIC protocol includes multiple transmission stream, the method includes: decapsulating the audio and video stream data to be transmitted by the sending end to obtain multiple types of source data; encapsulating according to the type of each type of source data to obtain a target datagram supporting a preset protocol determining a target transport stream matching the type of the target data packet from the plurality of transport streams, and transmitting the target data packet to the receiving end through the target transport stream.

By using QUIC as the transport protocol, compared to TCP which requires 1.5-RTT for each connection establishment, this application only needs 0-RTT for subsequent handshakes and reconnections after the first 1-RTT handshake based on the QUIC protocol. The preset protocol replaces the RTMP protocol, which can directly send audio and video data after establishing a connection based on the QUIC protocol, which saves the time of RTMP handshake and greatly reduces the delay of the first frame. Even if the client is disconnected or the network is reconnected, QUIC's connection migration and 0-RTT characteristics make the network seamlessly connect and send data, improving the smooth experience of users watching videos; using the multiplexing transmission characteristics of the QUIC protocol to realize Low-latency transmission of audio and video data.

In addition, this application divides a piece of audio and video stream data into different types of multiple types of data, and after these types of data are packaged through a custom preset protocol, multiple transmission streams through a connection based on the QUIC protocol are ordered Reliable and safe transmission, one type of data is sent using one transmission stream in the same connection, so that the transmission speed of various types of data does not affect each other, which can further reduce the delay of audio and video data in network transmission, and achieve low-latency transmission of audio Effects on video data.

The present application provides multiple embodiments to further introduce audio and video data transmission schemes, for details, refer to the following.

In the description of the present application, it should be understood that the numerals before the steps do not indicate the order in which the steps are performed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore cannot be construed as limiting the present application.

The following is an explanation of the terms used in this application:

QUIC: Quick UDP Internet Connection, a low-latency Internet transport layer protocol based on UDP (UserDatagram Protocol, User Datagram Protocol) developed by Google in the early days. At present, IETF (InternetEngineering Task Force, Internet Engineering Task Force) has officially Published standardized version RFC 9000.

RTMP: Real Time Messaging Protocol, real-time message transmission protocol, a network protocol designed for real-time data communication.

RTT: Round Trip Time, representing the round-trip delay of network data. 0-RTT means that application data can be sent during the handshake phase.

Audio and video stream: a complete audio and video stream, including audio, video and basic metadata. Common video files such as mp4, mov, flv, avi, rmvb and other video files are a container package that contains audio and video Two parts, and are generated by encoding and compressing through some specific encoding algorithms.

Fig. 1 schematically shows a schematic diagram of an environment application according to an embodiment of the present application. As shown in Figure 1:

The computer device 10000 can connect to the client 30000 via the network 20000 .

The computer device 10000 can provide services, such as performing network debugging, or returning audio and video data transmission result data to the client 30000 and so on.

Computer equipment 10000 may be located in a data center, such as a single site, or distributed among different geographic locations (eg, across multiple sites). The computer device 10000 may provide services via one or more networks 20000 . Network 20000 includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices and/or the like. Network 20000 may include physical links, such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, and the like. Network 20000 may include wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

Computer device 10000 may be implemented by one or more computing nodes. One or more compute nodes may include virtualized compute instances. Virtualized computing instances may include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. The compute node may load the virtual machine by the compute node based on the virtual image and/or other data defining the specific software (eg, operating system, application-specific, server) used for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and/or terminated on one or more computing nodes. A hypervisor can be implemented to manage the use of different virtual machines on the same compute node.

Client 30000 may be configured to access computer device 10000 content and services. Client 30000 may include any type of electronic device, such as mobile device, tablet device, laptop computer, workstation, virtual reality device, game device, set-top box, digital streaming device, vehicle terminal, smart TV, set-top box, etc.

The client 30000 can output (for example, display, render, and present) audio and video data transmission result data to the user.

The network debugging solution will be introduced through multiple embodiments below. This scheme can be implemented by the computer device 10000 .

Embodiment one

Fig. 2 schematically shows a flowchart of an audio and video data transmission method according to Embodiment 1 of the present application. Applied to the sending end, a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol includes multiple transmission streams, and the method includes steps S202-S206, wherein,

Step S202, decapsulating the audio and video stream data to be transmitted to obtain various types of source data;

The audio and video data transmission scheme of this embodiment can be used under the C/S software architecture, not limited to the encapsulation format of audio and video streams, wherein, the sending end is responsible for decapsulating and sending audio and video data, and the receiving end is responsible for receiving and repackaging audio and video data. video data. In specific implementation, the sending end may be a server or a client, and similarly, the receiving end may be a server or a client, which is not limited in this embodiment.

In this embodiment, when the sending end needs to transmit audio and video stream data, it first decapsulates the audio and video stream data to be transmitted to obtain various types of source data. The specific protocol used to decapsulate the audio and video stream data may be a published technology, which will not be repeated in this embodiment of the present application.

In this embodiment, the source data obtained after decapsulation may include three types, namely metadata type, video frame type and audio frame type. Among them, metadata is data describing data, mainly information used to describe data attributes. In addition, when a type of extended metadata is required, other types may also be defined, which is not specifically limited in this embodiment.

Step S204, encapsulating according to the type of each type of source data to obtain a target data message supporting a preset protocol;

Among them, the preset protocol is a self-defined protocol, which is mainly used to identify the relevant information of the data message between the sender and the receiver, including information such as message type, message length, and video time stamp, so that the sender and receiver Audio and video data packets can be directly transmitted between terminals without unnecessary handshake interaction.

In this embodiment, for each type of source data, a header based on a preset protocol may be encapsulated according to the type of the source data to obtain a target data packet supporting the preset protocol. Specifically, the Type field can be included in the header based on the preset protocol. When encapsulating the data packet, the value of the Type field can be configured according to the type of the source data to obtain the target data packet supporting the preset protocol. Therefore, The target data message carries data type information, and the receiving end can determine the data type by analyzing the target data message.

Step S206, determining a target transport stream matching the type of the target data packet from the plurality of transport streams, and transmitting the target data packet to the receiving end through the target transport stream.

In this embodiment, a connection based on the QUIC protocol is established between the sending end and the receiving end. A QUIC Connection includes multiple transport streams, and the multiple transport streams are independent of each other. Each transport stream is used for transmission A type of data packet, so that the transmission speed of various types of audio and video data does not affect each other. After the target data message is generated, a target transport stream matching the type of the target data message is determined from multiple transport streams, and then the target data message is transmitted to the receiving end through the target transport stream.

In one example, as shown in Figure 3, assuming that the server is the sending end and the client is the receiving end, a QUIC Connection is established between the server and the client. This QUIC Connection contains three streams, respectively stream1, stream2 and stream3, among them, stream1 is used to transmit metadata, stream2 is used to transmit video frame data, and stream3 is used to transmit audio frame data, so that the transmission speed of these three types of data does not affect each other, and the speed of audio and video data transmission is accelerated.

Several optional embodiments are provided below to optimize the audio and video data transmission method, specifically as follows:

In a preferred embodiment of the present application, the step S204 may include the following steps:

Encapsulate header data based on the preset protocol in the source data; wherein, the header data based on the preset protocol includes a type field; configure the type field according to the type corresponding to the source data Value, get the target data packet that supports the preset protocol.

In this embodiment, the header data based on the preset protocol includes a type field, which is used to define the packet type. When encapsulating the source data, the header data based on the preset protocol is encapsulated in the source data; then, the value of the type field is configured according to the type corresponding to the source data to obtain a target data packet supporting the preset protocol.

In a preferred embodiment of the present application, the header data based on the preset protocol further includes a variable-length format field, a variable-length field and a timestamp field;

The step S204 may also include the following steps:

Determine the length of the source data, and configure the value of the variable-length format field and the value of the variable-length field according to the length of the source data; and, determine that the source data is in the audio and video stream data corresponding timestamp, and configure the value of the timestamp field according to the timestamp.

In this embodiment, the header data based on the preset protocol may also include a variable-length format field, a variable-length field and a timestamp field, wherein the variable-length format field is used to indicate the corresponding The number of bytes; the variable length field is used to describe the length of the message; the timestamp field is used to describe the corresponding timestamp of the message in the audio and video stream data.

In a preferred embodiment of the present application, the sum of the byte lengths of the variable-length format field and the type field is one byte, and the byte length of the time stamp is four bytes. Specifically, the variable-length format field may be 2 bits, and the type field may be 6 bits. A variable-length field has a variable length in bytes.

In a specific implementation, the variable-length format field can be recorded as Fmt, and the variable-length field can be recorded as Length. The correspondence between the byte lengths of Fmt and Length can be shown in Table 1 below:

Table 1

Wherein, when the Fmt field is 0, 1, 2, and 3, the Length field occupies 1, 2, 4, and 8 bytes respectively, and the value of the Length depends on the length of the above-mentioned source data.

In a preferred embodiment of the present application, configuring the value of the variable-length format field and the value of the variable-length field according to the length of the source data may include the following steps:

Configure the value of the variable-length field according to the length of the source data, and determine the number of bytes corresponding to the variable-length field according to the value of the variable-length field; The number of sections configures the value of the variable-length format field.

In this embodiment, the value of the variable-length field is configured according to the length of the source data, and the number of bytes corresponding to the variable-length field is determined according to the value of the variable-length field; then according to the number of bytes corresponding to the variable-length field Configure the value of the variable-length format field. The variable-length format field is used to describe the number of bytes of the variable-length resource, so that the receiving end can parse the target data packet.

In a preferred embodiment of the present application, the step S204 may also include the following steps:

The header data based on the UDP protocol and the header data based on the QUIC protocol are encapsulated in the source data.

In this embodiment, the encapsulated target data packet is transmitted based on the QUIC protocol, which may be achieved by encapsulating header data based on the UDP protocol and header data based on the QUIC protocol in the source data.

As shown in FIG. 4 , a data structure diagram of a data message includes header data UDPHeader based on the UDP protocol, header data QUIC Header based on the QUIC protocol, a preset protocol Header, and message content data payload. Among them, the QUIC Header conforms to the RFC 9000 standard. The header part of the preset protocol can define various fields according to the needs. It is mainly used for identifying the payload message type, message length, audio and video timestamp and other information at both ends of the Client and Server, so that both ends do not need to The redundant handshake interaction directly transmits audio and video data information; among them, the payload message part carries metadata decapsulated by the sender, audio frames and video frames and other types of data.

As shown in Figure 5, it is a schematic diagram of the content contained in the header data based on the preset protocol; it includes a variable-length format field fmt, a type field type, a variable-length field length, and a timestamp field timestamp. Among them, the fmt field occupies 2 bits and is used to indicate the number of bytes of the variable-length length field; the type field occupies 6 bits; the fmt and type fields occupy 1 byte; the timestamp field occupies 4 bytes. The length of Length is variable and depends on the size of the datagram.

Embodiment two

FIG. 6 schematically shows a flowchart of an audio and video data transmission method according to Embodiment 2 of the present application. Applied to the receiving end, a connection based on the QUIC protocol is established between the receiving end and the sending end, and the connection based on the QUIC protocol includes multiple transmission streams, and the method includes steps S602-S608, wherein,

Step S602, receiving the target data message transmitted by the sender through the multiple transport streams;

In this embodiment, a connection based on the QUIC protocol is established between the sending end and the receiving end. A QUIC Connection includes multiple transport streams, and the multiple transport streams are independent of each other. Each transport stream is used for One type of data message is transmitted, so that the transmission speed of various types of audio and video data does not affect each other. Specifically, when the sending end transmits audio and video data, the receiving end can receive the target data packet transmitted by the sending end through multiple transport streams connected by the QUIC protocol.

Step S604, analyzing the target data message to obtain source data and header data based on a preset protocol;

Step S606, according to the header data based on the preset protocol, determine the type corresponding to the source data;

In this embodiment, after receiving the target data message, the source data and the header data based on the preset protocol are obtained by parsing the target data message, and then the source data corresponding to the source data is determined according to the header data based on the preset protocol. type. Specifically, the header data based on the preset protocol contains a type field, and the type corresponding to the source data can be determined through the value of the type field.

Step S608: Encapsulate the source data according to the type corresponding to the source data to obtain audio and video stream data.

In this embodiment, the receiving end can obtain audio and video stream data by encapsulating the source data according to the type corresponding to the source data.

Several optional embodiments are provided below to optimize the audio and video data transmission method, specifically as follows:

In a preferred embodiment of the present application, the header data based on the preset protocol includes a type field; the step S606 may include the following steps:

Determine the type corresponding to the source data according to the value of the type field.

In this embodiment, the receiving end can obtain the value of the type field by analyzing the header data based on the preset protocol, and determine the type corresponding to the source data.

In a preferred embodiment of the present application, the header data based on the preset protocol further includes a variable-length format field, a variable-length field, and a timestamp field;

After the step S604, the following steps may also be included:

Determine the number of bytes of the variable-length field according to the value of the variable-length format field; determine the number of bytes from the header data based on the preset protocol according to the number of bytes of the variable-length field The value of the variable length field, and according to the value of the variable length field, determine the length corresponding to the source data; according to the value of the timestamp field, determine that the source data corresponds to the audio and video stream data the timestamp of .

In this embodiment, the header data based on the preset protocol may also include a variable-length format field, a variable-length field and a timestamp field, wherein the variable-length format field is used to indicate the corresponding The number of bytes; the variable length field is used to describe the length of the message; the timestamp field is used to describe the corresponding timestamp of the message in the audio and video stream data.

When the receiving end parses the data message, it determines the number of bytes of the variable-length field according to the value of the variable-length format field; then, according to the number of bytes of the variable-length field, from the header data based on the preset protocol Determine the value of the variable length field in the variable length field, and determine the length corresponding to the source data according to the value of the variable length field; determine the corresponding time stamp of the source data in the audio and video stream data according to the value of the time stamp field.

In a preferred embodiment of the present application, the source data is encapsulated according to the type corresponding to the source data to obtain audio and video stream data, including:

According to the type, length and time stamp corresponding to the source data, the source data is encapsulated to obtain audio and video stream data.

In this embodiment, when the receiving end repackages audio and video data, it can package various source data according to the type, length and time stamp corresponding to the source data to obtain the required audio and video stream data.

Embodiment Three

Figure 7 schematically shows a block diagram of an audio-video data transmission device according to Embodiment 3 of the present application, the audio-video data transmission device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of accomplishing specific functions. The following description will specifically introduce the functions of the program modules in the embodiments of the present application.

As shown in Figure 7, the audio and video data transmission device 700 is applied to the sending end, and a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol includes a plurality of transmission streams, which may include The following modules:

A data decapsulation module 710, configured to decapsulate audio and video stream data to be transmitted to obtain various types of source data;

A source data encapsulation module 720, configured to encapsulate according to the type of each type of source data, to obtain a target data message supporting a preset protocol;

A data message transmission module 730, configured to determine a target transport stream matching the type of the target data message from the multiple transport streams, and transmit the target data message to the target transport stream through the target transport stream the receiving end.

In a preferred embodiment of the present application, the source data encapsulation module 720 includes:

The first header data encapsulation submodule is configured to encapsulate header data based on the preset protocol in the source data; wherein, the header data based on the preset protocol includes a type field; according to the The type corresponding to the source data configures the value of the type field to obtain a target data packet supporting a preset protocol.

In a preferred embodiment of the present application, the header data based on the preset protocol further includes a variable-length format field, a variable-length field and a timestamp field;

The first header data encapsulation submodule also includes:

a variable-length field configuration unit, configured to determine the length of the source data, and configure the value of the variable-length format field and the value of the variable-length field according to the length of the source data; and,

A timestamp field configuration unit, configured to determine a timestamp corresponding to the source data in the audio and video stream data, and configure a value of the timestamp field according to the timestamp.

In a preferred embodiment of the present application, the sum of the byte lengths of the variable-length format field and the type field is one byte, the byte length of the timestamp is four bytes, and the variable-length The variable-length format field is used to indicate the number of bytes corresponding to the variable-length field.

In a preferred embodiment of the present application, the variable length field configuration unit includes:

The byte number determination subunit of the variable-length field is configured to configure the value of the variable-length field according to the length of the source data, and determine the corresponding variable-length field according to the value of the variable-length field. number of bytes;

The variable-length format field configuration subunit is configured to configure the value of the variable-length format field according to the number of bytes corresponding to the variable-length field.

In a preferred embodiment of the present application, the source data encapsulation module 720 further includes:

The second header data encapsulation submodule is used to encapsulate the header data based on UDP protocol and the header data based on QUIC protocol in the source data.

In a preferred embodiment of the present application, the type of source data includes metadata type, video frame type and audio frame type.

In a preferred embodiment of the present application, the multiple transport streams are independent of each other, and each transport stream is used to transmit a type of data packet.

Embodiment Four

Figure 8 schematically shows a block diagram of an audio-video data transmission device according to Embodiment 4 of the present application, the audio-video data transmission device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of accomplishing specific functions. The following description will specifically introduce the functions of the program modules in the embodiments of the present application.

As shown in Figure 8, the audio and video data transmission device 800 is applied to the receiving end, and a connection based on the QUIC protocol is established between the receiving end and the sending end, and the connection based on the QUIC protocol includes a plurality of transmission streams, which may include The following modules:

A data message receiving module 810, configured to receive the target data message transmitted by the sender through the multiple transmission streams;

A data packet parsing module 820, configured to parse the target data packet to obtain source data and header data based on a preset protocol;

A data type determination module 830, configured to determine the type corresponding to the source data according to the header data based on the preset protocol;

The data encapsulation module 840 is configured to encapsulate the source data according to the type corresponding to the source data to obtain audio and video stream data.

In a preferred embodiment of the present application, the header data based on the preset protocol includes a type field;

The data type determining module 830 includes:

The type determination submodule is configured to determine the type corresponding to the source data according to the value of the type field.

In a preferred embodiment of the present application, the header data based on the preset protocol further includes a variable-length format field, a variable-length field, and a timestamp field;

The data message parsing module 820 also includes:

A submodule for determining the number of bytes of the variable-length field is used to determine the number of bytes of the variable-length field according to the value of the variable-length format field;

A data length determination submodule, configured to determine the value of the variable-length field from the header data based on the preset protocol according to the byte number of the variable-length field, and determine the value of the variable-length field according to the number of bytes of the variable-length field value to determine the corresponding length of the source data;

The timestamp determination submodule is configured to determine the corresponding timestamp of the source data in the audio and video stream data according to the value of the timestamp field.

In a preferred embodiment of the present application, the data encapsulation module 840 includes:

The data encapsulation sub-module is used for encapsulating the source data according to the type, length and time stamp corresponding to the source data to obtain audio and video stream data.

Embodiment five

Figure 9 schematically shows a block diagram of an audio-video data transmission system according to Embodiment 5 of the present application, the audio-video data transmission device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of accomplishing specific functions. The following description will specifically introduce the functions of the program modules in the embodiments of the present application.

As shown in Figure 9, the audio and video data transmission system 900 includes a sending end 910 and a receiving end 920, wherein,

The sending end 910 is used to obtain audio and video stream data to be transmitted; decapsulate the audio and video stream data to obtain various types of source data; Encapsulate to obtain a target data message supporting a preset protocol; determine a target transport stream matching the type of the target data message from the multiple transport streams, and transmit the target data message through the target transport stream The message is transmitted to the receiving end;

The receiving end 920 is configured to receive the target data message transmitted by the sending end through the multiple transport streams; parse the target data message to obtain source data and header data based on a preset protocol; according to the Based on the header data of the preset protocol, the type corresponding to the source data is determined; and the source data is encapsulated according to the type corresponding to the source data to obtain audio and video stream data.

Embodiment six

FIG. 10 schematically shows a schematic diagram of a hardware architecture of a computer device 10000 suitable for implementing a method for transmitting audio and video data according to Embodiment 6 of the present application. In this embodiment, the computer device 10000 is a device capable of automatically performing numerical calculation and/or information processing according to preset or stored instructions. For example, it can be a smartphone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including FEN independent server, or a server cluster composed of multiple servers), etc. . As shown in FIG. 10 , the computer device 10000 at least includes but is not limited to: a memory 10010 , a processor 10020 , and a network interface 10030 that can communicate with each other through a system bus. in:

The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000 , such as a hard disk or memory of the computer device 10000 . In some other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, referred to as SMC), a secure digital (Secure Digital (referred to as SD) card, flash memory card (Flash Card) and so on. Certainly, the memory 10010 may also include both an internal storage module of the computer device 10000 and an external storage device thereof. In this embodiment, the memory 10010 is generally used to store the operating system installed in the computer device 10000 and various application software, such as program codes of audio and video data transmission methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

The processor 10020 may be a central processing unit (Central Processing Unit, CPU for short), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000 , such as performing control and processing related to data interaction or communication with the computer device 10000 . In this embodiment, the processor 10020 is configured to run program codes stored in the memory 10010 or process data.

The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, and establish a data transmission channel and a communication link between the computer device 10000 and an external terminal. The network can be Intranet, Internet, Global System of Mobile Communication (GSM for short), Wideband Code Division Multiple Access (WCDMA for short), 4G network , 5G network, Bluetooth (Bluetooth), Wi-Fi and other wireless or wired networks.

It should be noted that FIG. 10 only shows a computer device having components 10010-10030, but it should be understood that implementing all of the illustrated components is not required and that more or fewer components may instead be implemented.

In this embodiment, the audio and video data transmission method stored in the memory 10010 can also be divided into one or more program modules, and executed by one or more processors (the processor 10020 in this embodiment) to Complete the embodiment of this application.

Embodiment seven

The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for transmitting audio and video data in the embodiment are implemented.

In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory ( ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In some other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, referred to as SMC), a secure digital ( Secure Digital (referred to as SD) card, flash memory card (Flash Card) and so on. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is usually used to store the operating system and various application software installed on the computer device, such as the program code of the audio and video data transmission method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computing devices, and they can be concentrated on a single computing device, or distributed among multiple computing devices. Optionally, they may be implemented in program code executable by a computing device, thereby, they may be stored in a storage device to be executed by a computing device, and in some cases, may be implemented in a code different from that described herein The steps shown or described are executed in sequence, or they are fabricated into individual integrated circuit modules, or multiple modules or steps among them are fabricated into a single integrated circuit module for implementation. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structures or equivalent process transformations made by using the description of the application and the accompanying drawings are directly or indirectly used in other related technical fields. , are all included in the patent protection scope of the present application in the same way.

Claims (15)

1. A method for transmitting audio and video data, characterized in that it is applied to a sending end, and a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol includes a plurality of transport streams, and the Methods include: Decapsulate the audio and video stream data to be transmitted to obtain various types of source data; Encapsulate according to the type of the source data of each type to obtain a target data message supporting a preset protocol; Determining a target transport stream matching the type of the target data packet from the plurality of transport streams, and transmitting the target data packet to the receiving end through the target transport stream. 2. audio-video data transmission method according to claim 1, is characterized in that, described according to the type of each class described source data is encapsulated, obtains the target data message that supports preset agreement, comprises: Encapsulating header data based on the preset protocol in the source data; wherein the header data based on the preset protocol includes a type field; The value of the type field is configured according to the type corresponding to the source data to obtain a target data packet supporting a preset protocol. 3. The audio-video data transmission method according to claim 2, wherein the header data based on the preset protocol also includes a variable-length format field, a variable-length field and a timestamp field; The encapsulation according to the type of each type of source data to obtain the target data message supporting the preset protocol also includes: determining the length of the source data, and configuring the value of the variable-length format field and the value of the variable-length field according to the length of the source data; and, Determine the time stamp corresponding to the source data in the audio and video stream data, and configure the value of the time stamp field according to the time stamp. 4. audio-video data transmission method according to claim 3, is characterized in that, the byte length sum of the format field of described variable length and type field is a byte, and the byte length of described time stamp is Four bytes, the variable-length format field is used to indicate the number of bytes corresponding to the variable-length field. 5. The audio-video data transmission method according to claim 4, wherein the value of the format field of the variable length and the value of the variable length field configured according to the length of the source data include: configuring the value of the variable length field according to the length of the source data, and determining the number of bytes corresponding to the variable length field according to the value of the variable length field; The value of the variable-length format field is configured according to the number of bytes corresponding to the variable-length field. 6. audio-video data transmission method according to claim 2, is characterized in that, described according to the type of described source data of each class is encapsulated, obtains the target data message that supports preset agreement, also comprises: The header data based on the UDP protocol and the header data based on the QUIC protocol are encapsulated in the source data. 7. The audio and video data transmission method according to claim 1, wherein the type of the source data includes metadata type, video frame type and audio frame type. 8. The audio and video data transmission method according to claim 1, wherein the plurality of transmission streams are independent of each other, and each of the transmission streams is used to transmit one type of data message. 9. A method for audio and video data transmission, characterized in that it is applied to a receiving end, and a connection based on the QUIC protocol is established between the receiving end and the sending end, and the connection based on the QUIC protocol includes a plurality of transmission streams, so The methods described include: receiving the target data packet transmitted by the sender through the multiple transmission streams; Analyzing the target data packet to obtain source data and header data based on a preset protocol; Determine the type corresponding to the source data according to the header data based on the preset protocol; Encapsulate the source data according to the type corresponding to the source data to obtain audio and video stream data. 10. The audio and video data transmission method according to claim 9, wherein the header data based on the preset protocol includes a type field; The determining the type corresponding to the source data according to the header data based on the preset protocol includes: Determine the type corresponding to the source data according to the value of the type field. 11. The method for transmitting audio and video data according to claim 10, wherein the header data based on the preset protocol also includes a variable-length format field, a variable-length field, and a timestamp field; After the step of analyzing the target data message to obtain the source data and the header data based on the preset protocol, it also includes: determining the number of bytes of the variable-length field according to the value of the variable-length format field; According to the number of bytes of the variable length field, determine the value of the variable length field from the header data based on the preset protocol, and determine the source data according to the value of the variable length field the corresponding length; According to the value of the timestamp field, determine the corresponding timestamp of the source data in the audio and video stream data. 12. The audio and video data transmission method according to claim 11, wherein said source data is encapsulated according to the type corresponding to said source data to obtain audio and video stream data, comprising: According to the type, length and time stamp corresponding to the source data, the source data is encapsulated to obtain audio and video stream data. 13. An audio and video data transmission system, characterized in that it includes a sending end and a receiving end, wherein a connection based on the QUIC protocol is established between the sending end and the receiving end, and the connection based on the QUIC protocol includes a plurality of transmission flow; The sending end is used to obtain the audio and video stream data to be transmitted; decapsulate the audio and video stream data to obtain multiple types of source data; perform the source data according to each type of source data Encapsulate to obtain a target data message supporting a preset protocol; determine a target transport stream matching the type of the target data message from the multiple transport streams, and transmit the target data message through the target transport stream The text is transmitted to the receiving end; The receiving end is used to receive the target data message transmitted by the sending end through the multiple transmission streams; parse the target data message to obtain source data and header data based on a preset protocol; according to the The header data of the preset protocol determines the type corresponding to the source data; the source data is encapsulated according to the type corresponding to the source data to obtain audio and video stream data. 14. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that the processor executes the computer program to realize the rights The steps of the audio and video data transmission method described in any one of 1 to 8 or 9 to 12 are required. 15. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer program can be executed by at least one processor, so that the at least one processor executes the rights The steps of the audio and video data transmission method described in any one of 1 to 8 or 9 to 12 are required.

Images