CN105898353A - Real-time audio and video transmission method and device, transmission stream packaging method and multiplexer - Google Patents

Abstract

An embodiment of the present invention provides a method for real-time transmission of audio and video, including: packaging the original video stream and the original audio stream into grouped video original streams and grouped audio original streams; The original video stream and the original audio stream of the group are multiplexed and stored as program slices, each of which includes a program association table and a program mapping table; receiving the request, and transmitting the program according to the request Program slices. The embodiment of the present invention also provides a transport stream packaging method and a multiplexer. The embodiment of the present invention can reduce redundant data during real-time transmission of audio and video.

Description

Audio and video real-time transmission method and device, transport stream packaging method and multiplexer

technical field

Embodiments of the present invention relate to the field of audio and video technologies, and in particular, to a method and device for real-time audio and video transmission, a transport stream packaging method, and a multiplexer.

Background technique

The HLS (HTTP Live Streaming) protocol is an extended HTTP protocol developed by Apple and used in webcast and on-demand scenarios. The protocol uses Transport Stream (TS) to encapsulate data. Each transport stream carries data information such as video (Video), audio (Audio), and stream index information such as program association table (Program Association Table, PAT) and program mapping table (Program Map Table, PMT).

In live broadcasting, in order to allow users to search for correct video and audio data through the program association table and program mapping table at any time, it is necessary to periodically insert the program association table and program mapping table into the transport stream.

Due to the periodic insertion of the program association table and the program mapping table, the slices in HLS will be too large, which not only takes up download time, but also causes more time delays and occupies more server storage space during live broadcast.

Contents of the invention

Embodiments of the present invention provide a method and device for real-time audio and video transmission, a transport stream packaging method and a multiplexer, to solve the technical problem of excessive redundant data in real-time audio and video transmission in the prior art.

An embodiment of the present invention provides a method for real-time transmission of audio and video, including:

Packing the original video stream and the original audio stream into packetized video original streams and packetized audio original streams, respectively;

The program association table, the program mapping table, the grouped video original stream, and the grouped audio original stream are multiplexed and stored as program slices, and each program slice includes a program association table and a program map surface;

A request is received and the program slice is transmitted according to the request.

Further, before the step of packaging the original video stream and the original audio stream into packetized video original streams and grouped audio original streams, respectively, the steps include: respectively encoding video data and audio data to form the video original stream and the audio original stream .

An embodiment of the present invention provides a transport stream packaging method, including:

Receive packetized video original stream, packetized audio original stream, a program association table, and a program mapping table;

Multiplexing the one program association table, the one program mapping table, the grouped video original stream, and the grouped audio original stream into a program slice;

The section of program slice is stored.

Further, before the step of receiving the original video stream of the packet, the original audio stream of the packet, a program association table, and a program mapping table, it also includes;

A step of packaging the original video stream and the original audio stream into packetized video original streams and packetized audio original streams respectively, and outputting.

An embodiment of the present invention provides an audio and video real-time transmission device, including:

A video packetizer, used to package the video original stream into a grouped video original stream;

an audio packetizer for packetizing audio primitive streams into grouped audio primitive streams;

a multiplexer, configured to store the program association table, the program mapping table, the original video stream of the group, and the original audio stream of the group into program slices after multiplexing, and each of the program slices includes a program association table and a program mapping table;

The multiplexer is further configured to receive a request, and transmit the program slice according to the request.

Further, the above-mentioned device also includes:

A video encoder, configured to encode video data to form the original video stream.

Further, the above-mentioned device also includes:

An audio encoder, configured to encode audio data respectively to form the audio original stream.

An embodiment of the present invention provides a multiplexer, including:

The transceiver module is used to receive the grouped original video stream, the grouped audio original stream, a program association table, and a program mapping table, and output them;

A program slice multiplexing module, configured to multiplex the one program association table, the one program mapping table, the grouped video original stream, and the grouped audio original stream into a program slice;

A storage module, configured to store the segment of the program.

Further, the above-mentioned transceiving module is connected with a video grouper and an audio grouper, and the video grouper and audio grouper output grouped video original streams and grouped audio original streams to the transceiver module.

The audio and video real-time transmission method and device, transmission stream packaging method and multiplexer provided by the embodiments of the present invention do not need to periodically insert the program association table and the program mapping table, but only need to add a The program association table and a program mapping table can achieve the effect of not only maintaining compatibility with the transport stream in the prior art, but also reducing redundant data in the transport stream.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic diagram of the format of a slice of a transport stream in the prior art;

2 is a schematic diagram of the format of a slice of a transport stream of an audio and video real-time transmission method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of the composition of a sliced packet of a transport stream of a method for real-time audio and video transmission according to an embodiment of the present invention;

4 is a flowchart of a method for real-time audio and video transmission according to an embodiment of the present invention;

FIG. 5 is a flowchart of a transport stream packaging method according to an embodiment of the present invention;

FIG. 6 is a flow chart of parsing a transport stream slice according to an embodiment of the present invention;

7 is a schematic structural diagram of an audio and video real-time transmission device according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a multiplexer according to an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

In order to better illustrate the technical solutions of the embodiments of the present invention and the beneficial effects produced by them, the present invention will first introduce the format of slices of transport streams in the prior art.

FIG. 1 is a schematic diagram of the format of a transport stream slice in the prior art.

The transport stream is a hierarchical encapsulation format. It is necessary to find the program association table first to obtain the number of program mapping tables in the transport stream, so as to obtain the number of programs in the transport stream. Each program mapping table corresponds to a program, and then The video packet identifier (Packet Identification, PID) and audio packet identifier of the program are acquired according to the program mapping table.

According to the HLS protocol, a long program is usually divided into multiple 10-second slices. Therefore, in the slice of the transport stream in the prior art, in order to facilitate the user to receive the transport stream at any time, the program association table and the program mapping table are periodically inserted into the slice of the transport stream, so that the slice of the transport stream in the prior art is at least It includes more than two program association tables, and more than two program association tables.

For example, in Figure 1, a program is cut into n slices, user A can receive it from program association table 1, and user B can receive it from program association table n to find the nearest program mapping table to facilitate live broadcast. Therefore, the prior art adds redundant data in the transport stream.

The inventors of the present invention have found after a long period of research that in each slice, it is not necessary to periodically insert the program association table and the program mapping table, but only need to add a program association table and program mapping table at the beginning of each slice. A program mapping table can achieve the effect of not only maintaining compatibility with the transport stream in the prior art, but also reducing redundant data in the transport stream.

The format of the slice of the transport stream of the present invention will be described below.

FIG. 2 is a schematic diagram of a format of a transport stream slice in a method for real-time audio and video transmission according to an embodiment of the present invention.

The transport stream in the embodiment of the present invention is a packet-based stream, and each group of transport streams is divided into several slices. It should be noted that the format of only one slice is shown in FIG. 2 , and the formats of other slices are actually the same as The slice format in Figure 2 is similar.

The slices in the embodiment of the present invention include package 1, package 2 ... package n-1, package n, and each slice includes only one program association table and one program mapping table.

FIG. 3 is a schematic diagram of a packet composition of a transport stream slice in a real-time audio and video transmission method according to an embodiment of the present invention.

The sliced packet in the embodiment of the present invention is composed of packet header and data, and each packet is 188 bytes or 204 bytes, wherein the packet of 204 bytes is added with 16 bytes after the packet of 188 bytes The CRC check data is formed.

The packet header may also include an extended self-use area, the length of the packet header occupies 4 bytes, and the self-use area and packet data occupy a total of 184 bytes. The packet header consists of 8 parts: synchronization byte, transmission error code indicator, payload unit start indicator, transmission priority, packet identifier, transmission scrambling control, adaptive area control and continuous counter.

Among them, the packet restriction in the data stream can be detected through the automatic correlation characteristic of the bit string of the synchronization byte, and packet synchronization can be established; the transmission error indicator means that when there is an error code that cannot be eliminated, the error correction decoder can be used Indicates a 1-bit error, but the error cannot be corrected; the payload unit start indicator indicates whether the packet has definite start information; the transmission priority is used to assign priority to the packet; the packet identifier value is determined by the user Determined, the decoder distinguishes the packets of different elementary streams (ElementaryStream, ES) according to the packet identifier to reconstruct the original stream; the transmission scrambling control can be used to indicate whether the packet content is scrambled; the adaptive area control is used to indicate whether there is Adaptive area, usually represented by 2 bits, for example, "01" indicates that there is useful information but no adaptive area, "10" indicates that there is no useful information and there is adaptive area, "11" indicates that there is useful information and there is automatic Adaptation area, "00" means no definition; the continuous counter can count the transmission sequence of the packet identifier, and according to the reading of the continuous counter, the receiving end can judge whether there is a packet loss or a packet transmission sequence error. All in all, the packet header has the functions of synchronization, identification, error detection and encryption for the transport stream.

In addition, the packet identifier is a unique identifier in the transport stream, and the data content in the packet is determined by the packet identifier. If the packet identifier in the header of a packet in a transport stream is 0, then the data content of the packet is the program association table, if not 0, it is video data, audio data, program mapping table, or other types of data.

Fig. 4 is a flow chart of an audio and video real-time transmission method according to an embodiment of the present invention.

The real-time transmission of audio and video includes real-time transmission of video data and real-time transmission of audio data, so the video original stream and audio original stream need to be packaged into grouped video original stream and grouped audio original stream respectively; then the program association table , program mapping table, grouped video original stream, and grouped audio original stream are multiplexed and stored as program slices. Each program slice includes a program association table and a program mapping table, as follows:

The specific steps of packaging the original video stream of the packet are as follows:

S101: Encode the video data to form an original video stream (Elementary Stream, ES);

The original video stream is the original basic bit stream output by the encoder, which only contains the information necessary for the decoder and is close to the original image. The encoded original video stream may be in MPEG-2 or MPEG-4 format, and may also be in H.264 format or other formats, which is not limited in the present invention.

S102: Pack the original video stream into a packetized original video stream (Packet Elemental Stream, PES);

The grouped original video stream exists in the form of divided data packets, mainly adding time stamps and other description information to the data frame on the original video stream. The grouped original video stream provides standard packet headers and packaging methods, and provides decoding time sign.

The specific steps of packaging the packetized audio original stream are as follows:

S201: Encode the audio data to form an audio original stream;

If the digital audio signal is directly transmitted without compression, it will occupy a huge bandwidth. Therefore, it is necessary to use audio compression technology to process the audio data in order to effectively transmit the audio data. The audio coding method in this step may be MPEG-1 audio coding, or MPEG-2AAC (Advanced Audio Coding) coding, which is not limited in the present invention.

S202: Pack the original video stream into packetized audio original streams.

The audio packetizer 202 divides the audio original stream into segments or packs them into groups, and adds corresponding header files to pack into grouped audio original streams. The packets of the audio original stream may be discontinuous.

After packing the original video stream and the original audio stream of the packet, enter step S300;

S300: Store the grouped original video stream, grouped audio original stream, program association table and program mapping table as a program slice after multiplexing, and the program slice includes a program association table and a program mapping table.

Since the HLS protocol stipulates that a complete program is divided into multiple slices with a duration of 10 seconds, and the multiple slices constitute a complete program, the process from the establishment of the link to the disconnection of the link (that is, the entire process of watching the program ), continuously link and download multiple slices of the program, the download process takes a byte as the minimum unit, so it is necessary to periodically provide the program association table and the program mapping table to ensure that the user can download slices from any point.

However, in this embodiment, the slice containing the program association table and the program mapping table is used as the smallest unit of the transport stream, which can ensure that the user can download slices from any point, so that the compatibility with the existing HLS protocol can be maintained and periodicity is not required. Insert program association table and program mapping table.

S400: Receive a request, and transmit program slices according to the request.

Fig. 5 is a flowchart of a transport stream packaging method according to an embodiment of the present invention.

S301: Receive a grouped video original stream, a grouped audio original stream, a program association table, and a program mapping table;

S302: Multiplex a program association table, a program mapping table, grouped video original streams, and grouped audio original streams into a program slice;

S303: Store a program slice.

Fig. 6 is a flow chart of parsing a transport stream slice according to an embodiment of the present invention:

501: Search the program association table;

In this embodiment, the program association table whose packet identifier is 0 is searched in units of 188 bytes. Since the packet identifier of the program association table is fixed at 0, and the program association table is also transmitted in the form of packets, searching the program association table is essentially searching for packets whose packet identifier is 0 in the packet header.

502: Parse the program association table to obtain the package identifier of the program mapping table;

In this embodiment, the program association table whose packet identifier is 0 is first analyzed to obtain the packet identifier of the program mapping table.

503: Search the program mapping table according to the packet identifier of the mapping table;

In this embodiment, the number of program mapping tables is equal to the number of programs.

504: Search for the packet identifier of the video data and the packet identifier of the audio data according to the program mapping table;

In this embodiment, the packet identifier of the video data and the packet identifier of the audio data are searched in order to obtain the video data and the audio data.

505: Analyze the video data and the audio data according to the packet identifier of the video data and the packet identifier of the audio data.

The above parsing process is completed by the receiving end, which is usually a consumer electronic product, such as a smart TV, a set-top box, a projector, a video player, a personal computer, a smart phone, a tablet computer, etc.

FIG. 7 is a schematic structural diagram of an audio and video real-time transmission device according to an embodiment of the present invention.

An audio and video real-time transmission device 10 according to an embodiment of the present invention includes: a video encoder 101 , an audio encoder 201 , a video packetizer 102 , an audio packetizer 202 , and a multiplexer 300 .

Since the collected video data and audio data are all analog data, sampling, quantization and compression encoding are required to form the original video stream and the original audio stream. The original video stream and the original audio stream are unsegmented continuous code streams.

The video encoder 101 is used to encode video data to form an original video stream. In this embodiment, the original video stream includes multiple access units, and each access unit is the encoded data of an image. The video coding method of the present invention may adopt transform coding, entropy coding, motion estimation and motion compensation, or a mixed coding method for coding, which is not limited in the present invention.

The audio encoder 201 is used for respectively encoding audio data to form an audio original stream. In this embodiment, if the digital audio signal is directly transmitted without compression, it will occupy a huge bandwidth. Therefore, it is necessary to use audio compression technology to process the audio data in order to effectively transmit the audio data. Digital audio compression coding compresses the audio data signal as much as possible on the premise of ensuring that the signal does not produce distortion in the auditory sense. Digital audio compression coding is realized by removing redundant components in the sound signal. The audio coding method of the present invention may be waveform coding, parameter coding, or mixed coding, which is not limited in the present invention.

The video packetizer 102 is used for packetizing the video original stream into grouped video original streams. In detail, the video packetizer 102 divides the original video stream into segments or packs it into groups, and adds corresponding header files to pack the original video stream into groups, and the packets of the original video stream may be discontinuous of.

The audio packetizer 202 is used to packetize the audio original stream into grouped audio original streams. In detail, the audio packetizer 202 divides the audio original stream into segments or packs them into groups, and adds corresponding header files to pack into grouped audio original streams. The packets of the audio original stream may be discontinuous of.

The multiplexer 300 is used to multiplex and store the program association table, the program mapping table, the grouped original video stream, and the grouped audio original stream as program slices, each of which includes a program association table and a program map. In this embodiment, the multiplexer 300 packs the program association table, program mapping table, packetized video original stream, and packetized audio original stream into slices with a fixed length of 188 bytes during transmission.

The multiplexer 300 is also configured to receive requests and transmit program slices according to the requests.

The audio and video real-time transmission device 100 provided by the embodiment of the present invention can be specifically applied to electronic equipment in practical applications, and related functional modules can be implemented by hardware processors. The electronic equipment is usually various types of video servers.

FIG. 8 is a schematic structural diagram of a multiplexer according to an embodiment of the present invention.

According to an embodiment of the present invention, a multiplexer 300 includes a transceiver module 302 , a program slice multiplexer module 303 , and a storage module 301 .

The transceiver module 302 is used to receive the original video stream of the packet, the original audio stream of the packet, a program association table, and a program mapping table, and output them to the program slice multiplexing module 303;

The program slice multiplexing module 303 is used to multiplex a program association table, a program mapping table, a grouped video original stream, and a grouped audio original stream into a program slice;

The storage module 301 is used for storing a segment of a program.

Through the above-mentioned technical solution of the embodiment of the present invention, the present invention does not need to periodically insert the program association table and the program mapping table, but only needs to add a program association table and a program mapping table at the beginning of each slice, then The effect of not only maintaining compatibility with the transport stream in the prior art, but also reducing redundant data in the transport stream can be achieved.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative effort.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (9)

1. an audio frequency and video live transmission method, it is characterised in that including:

Video primary flow and audio frequency primary flow are packaged as respectively video primary flow and the audio frequency of packet of packet Primary flow；

By Program Association Table, Program Map Table, the video primary flow of described packet, and described packet Being stored as program section after audio frequency primary flow multiplexing, the section of each described program includes a programming association Table and a Program Map Table；

Receive request, and transmit the section of described program according to described request.

Method the most according to claim 1, it is characterised in that include described by video primary flow and Audio frequency primary flow wraps before being packaged as the video primary flow of packet and the audio frequency primary flow step of packet respectively Include: video data and voice data are separately encoded the described video primary flow of formation and described audio frequency primary flow.

3. a transmission stream packaging method, it is characterised in that including:

Receive the video primary flow of packet, the audio frequency primary flow of packet, Program Association Table, a joint Mesh mapping table；

By a described Program Association Table, a described Program Map Table, the video primary flow of described packet, And the audio frequency primary flow of described packet is multiplexed with one section of program section；

Store described one section of program section.

4. method as claimed in claim 3, it is characterised in that original at the video of described reception packet Also include before stream, the audio frequency primary flow of packet, Program Association Table, a Program Map Table step；

Video primary flow and audio frequency primary flow are packaged as respectively video primary flow and the audio frequency of packet of packet Primary flow, and the step exported.

5. an audio frequency and video real-time Transmission device, it is characterised in that including:

Video packetizer, for being packaged as the video primary flow of packet by video primary flow；

Audio packet device, for being packaged as the audio frequency primary flow of packet by audio frequency primary flow；

Multiplexer, for by Program Association Table, Program Map Table, the video primary flow of described packet, with And after the audio frequency primary flow multiplexing of described packet, it being stored as program section, the section of each described program includes One Program Association Table and a Program Map Table；

Described multiplexer, is additionally operable to receive request, and transmits the section of described program according to described request.

Device the most according to claim 5, it is characterised in that also include:

Video encoder, for forming described video primary flow by video data encoding.

Device the most according to claim 5, it is characterised in that also include:

Audio coder, for being separately encoded the described audio frequency primary flow of formation by voice data.

8. a multiplexer, it is characterised in that including:

Transceiver module, for receiving the video primary flow of packet, the audio frequency primary flow of packet, a program Contingency table, a Program Map Table, and export；

Program section Multiplexing module, for by a described Program Association Table, a described Program Map Table, The video primary flow of described packet, and the audio frequency primary flow of described packet is multiplexed with the section of one section of program；

Memory module, is used for storing described one section of program section.

9. multiplexer as claimed in claim 3, it is characterised in that described transceiver module and video packets Device and audio packet device connect, and described video packetizer and audio packet device are to the output point of described transceiver module The video primary flow of group and the audio frequency primary flow of packet.