CN115866258A - Method and device for generating and processing transport stream and program stream - Google Patents

Abstract

The invention discloses a method and a device for generating and processing a transport stream and a program stream. The processing method and the processing device can analyze the dependency identification information in the transport stream or the program stream to obtain the dependency between the main bit stream and the knowledge bit stream, obtain the main bit stream and the knowledge bit stream referred to by the main bit stream according to the dependency, and decode the image in the main bit stream by taking the knowledge image in the knowledge bit stream as the reference image. The generation method and the device can put the main bit stream and the knowledge bit stream which depends on the main bit stream into a transport stream packet or a program stream packet as an elementary stream packet, and put dependency identification information into the transport stream packet or the program stream packet to describe the dependency between the elementary stream to which the main bit stream belongs and the elementary stream to which the knowledge bit stream belongs. The invention can conveniently and compatibly realize the description of the dependency relationship between the main bit stream and the knowledge bit stream, and simultaneously can not waste data volume.

Description

Method and device for generating and processing transport stream and program stream

technical field

The invention belongs to the technical field of image or video compression, storage and transmission, and specifically relates to methods for generating and processing transport streams and program streams.

Background technique

1. Traditional video coding scheme

In the existing video sequence processing, in order to make the encoded video sequence support the random access function, the video sequence is divided into multiple segments with random access function (referred to as random access segments), as shown in Figure 1, a video sequence It includes at least one random access segment, each random access segment corresponds to a display period and includes a random access image and multiple non-random access images, each image has its own display time to describe the time when the image is displayed or played. Pictures in a random access segment can be intra-coded or coded using inter-frame prediction with reference to other pictures in the random access segment, where the referenced picture can be a picture to be displayed, or a composite picture that cannot be displayed wait. However, in the prior art, a picture (not including leading pictures) whose display order is after the random access picture can only refer to other pictures in the random access segment to which the picture belongs, but cannot refer to the pictures before or after the random access segment to which the picture belongs Randomly access images in fragments, as shown in Figure 1. Specifically, there are several ways to describe the dependency between the current image and candidate reference images:

In existing video coding schemes (such as H.264\AVC or H.265\HEVC), the dependency relationship between the current picture and candidate reference pictures is described by the reference picture configuration set of the video compression layer, where the reference picture configuration set Describes the numbered difference between the reference image and the current image. The reason why only the number difference is described in the reference picture configuration set is because in the existing video coding schemes, the candidate reference picture and the current picture belong to the same independently decodable random access segment, and the candidate reference picture and the current picture can only be The same numbering rule is used, such as numbering in time order, so the candidate reference image can be accurately located according to the difference between the current image number and the candidate reference image number. If the reference image and the current image use different numbering rules, since the existing video coding scheme does not provide a method for describing different numbering rules in the code stream, the same numbering difference will point to different candidate reference images, resulting in codecs that cannot be used The correct candidate reference image.

In the scalable video coding scheme (Scalable Video Coding, SVC) and the multiview video coding scheme (Multiview Video Coding, MVC), as shown in Figure 2, in the existing inter-frame prediction (only using the Candidate reference picture), SVC/MVC uses inter-layer/inter-view prediction to extend the range of candidate reference pictures for the current picture, where the extended candidate reference picture has the same number as the current picture (for example, the same time stamp) and Not belonging to the same hierarchy as an independently decodable segment. In the video compression layer, SVC/MVC uses layer identification to describe the dependency relationship of code streams of different layers/viewpoints, and jointly uses the same number of images to describe the dependency relationship of images between layers/viewpoints.

In the background frame technology of AVS2, as shown in FIG. 3 , the dependency relationship between the coded image and the scene image is described by the identifier of the reference image type in the video compression layer. Specifically, AVS2 uses the logo to describe the special scene image type (ie, G image and GB image), and uses a specific reference cache (ie scene image cache) to manage G/GB images, and at the same time, uses the logo to describe whether the current image refers to G /GB image, and use a specific reference image queue construction method (that is, put the G/GB image into the last reference image bit of the reference image queue by default), and finally, enable the current image numbered according to the rule to refer to the image that is not numbered according to the rule Candidate reference image (i.e. GB image), or a candidate reference image (i.e. G image) that uses the same rule number as the current image but the number difference exceeds the constraint range. However, this technology limits that only one candidate reference image can exist in the scene image buffer at any time, and the candidate reference image still belongs to the same independently decodable segment as the current image.

2. Video coding scheme based on knowledge base

The above-mentioned mechanism in the prior art will limit the number of available reference images of the image to be encoded, and cannot effectively improve the efficiency of image encoding and image decoding.

In order to mine and utilize the mutual reference information of images between multiple random access segments during encoding, when encoding (or decoding) an image, the encoder (or decoder) can select from the database that is compatible with the current encoding image (or Decoded image) An image with similar texture content is used as a reference image. This reference image is called a knowledge base image, and the database storing the collection of the above reference images is called a knowledge base. At least one image in this video refers to at least one knowledge base image. The method of encoding and decoding is called knowledge base-based video coding (English: library-based video coding), also known as video coding based on information related to a large time span (referred to as long-span coding). Encoding a video sequence by using the video coding based on the knowledge base will generate a knowledge code stream including the coded code stream of the knowledge base image and a main code stream including the code stream obtained by encoding each frame image of the video sequence with reference to the knowledge base image. These two code streams are similar to the basic layer code stream and enhancement layer code stream generated by scalable video coding (English: scalable video coding, SVC), that is, the main code stream depends on the knowledge code stream, and the knowledge code stream can be interspersed The main code stream is used to form a spliced code stream. However, the dependency relationship between the dual-stream organization method of knowledge base-based video coding and the hierarchical code-stream organization method of SVC is different. The alignment time segment is dependent, while the video layer in the knowledge base-based video coding dual-code stream depends on the knowledge layer according to the non-alignment time segment.

In the codec technology using knowledge image, the knowledge image is acquired and used to provide additional candidate reference images for the codec of the image. Figure 4 shows the dependency between the sequence image and the knowledge image in the codec technology using knowledge image relation. The knowledge image enables sequence images to utilize large-span related information and improves the efficiency of encoding and decoding. However, the existing technical solutions cannot effectively support the description of the dependency relationship between the sequence image and the knowledge image and efficiently manage the knowledge image.

3. Generation and processing methods of existing transport streams and program streams

The generation and processing method of transport stream and program stream synthesizes one or more audio, video bit streams or other basic data streams into a single or multiplexed stream for storage and transmission, and the syntax information in the data stream is used for Synchronously decode and display audio and video information, while ensuring that the data buffer in the decoder does not overflow or underflow.

A transport stream is a stream defined mainly for program transmission and storage in environments where significant errors may occur (often manifested as bit errors or packet loss), a single stream consisting of one or more programs, and these A program can contain one or more different clock references. Multiple PES (Packetized Elementary Stream) packets composed of multiple elementary streams in the same program share a clock reference, and the length of the transport stream packet is 184 bytes. The packet header of the transport stream packet defines the predetermined time for each byte sent from the channel to enter the program stream decoder, as a reference for clock correction and buffer management. Its data structure is shown in Figure 5.

A program stream is a stream defined for the transmission and storage of a program in an environment where the error rate is very low and the system encoding process is the main consideration. It is composed of one or more PES packets with a common time reference. The resulting single stream contains only one program, and the length of the program stream packet is variable. The packet header of the program stream packet specifies the predetermined time for each byte sent from the channel to enter the program stream decoder, as a reference for clock correction and buffer management, and its data structure is shown in Figure 6.

Each transport stream and program stream contains some information to identify the relevant characteristics of the elementary streams that make up a program and the relationship between elementary streams. This information can include the language used in the soundtrack, and the relationship between layers when implementing multi-layer video coding. For example, the transport stream packet may carry a Program-Specific Information (PSI, Program-Specific Information) table specifying data content included in the transport stream packet. The PSI table is carried in the transport stream, including the following four: a) Program Association Table (PAT, Program Association Table); b) Program Mapping Table (PMT, Program Map Table); c) Conditional Access Table (CAT, Conditional Access Table ; d) Network Information Table (NIT, Network Information Table). The program stream packet can carry a program stream map (PSM, Program Stream Map) and a program stream directory (PSD, Program Stream Directory). These tables containing description information can record the attributes of each transport stream/program stream, or each program, or each elementary stream, which can be realized by using different program element descriptors (descriptors). For example, hierarchy_descriptor can describe the attributes of elementary streams with hierarchical relationship in the program, and identify the dependency relationship between elementary streams.

An elementary stream generally refers to a coded video bit stream, a coded audio bit stream, or some other coded bit stream in a PES packet. An elementary stream has one and only one sequence of PES packets with stream_id to transmit.

The PES packet header starts with a 32-bit start code, which also identifies the flow or flow type to which the packet data belongs, and its data structure is shown in FIG. 7 . The PES packet header may contain decoding and presentation time stamps (DTS and PTS). The PES packet header also contains other optional fields. The PES packet data field contains a variable number of consecutive bytes from an elementary stream.

For a program containing only a single video elementary stream, the data structure of the transport stream is shown in Figure 8. Among them, the first row is a series of TS packets arranged according to the data stream, and the content in the second row of squares is part of the data contained in the corresponding TS packet header or payload. In this example, the first TS packet contains PAT, the second TS packet contains PMT, and subsequent TS packets contain PES packets.

For a video containing only a single bit stream, the data structure of the program stream is shown in Figure 9. Among them, the first row is a series of PS packets arranged according to the data stream, and the content in the second row of squares is part of the data contained in the corresponding PS packet header or payload. In this example, the first PS packet PSM is included, and subsequent PS packets include PES packets.

Contents of the invention

Although the existing methods for generating and processing transport streams and program streams can support the processing of video streams with hierarchical coding (such as SVC and MVC, or time-domain layered coding of HEVC and VVC), due to the multiple The dependencies between the multi-layer video streams are not consistent, and it is necessary to carry out a special design for each layer coding to accurately describe the dependencies between the multi-layer video streams in the transport stream and the program stream, so that the receiving end can quickly Process the data to decode correctly. Therefore, the present invention makes an innovative design aiming at the characteristics of the dependency relationship between the main bit stream and the knowledge bit stream using long-span coding to solve the above problems. At the same time, due to the use of long-span coding, the dependency description between the main bit stream and the knowledge bit stream is not realized through time stamp synchronization, but through dependency identification (this is because a part of the knowledge bit stream Knowledge images can be shared reference by images in multiple random access segments in the main bit stream, so they cannot be synchronized using a single time stamp), existing transport streams and program streams can only be synchronized using time stamps, so the present invention An innovative design is also made to realize the dependency description between the main bit stream and the knowledge bit stream conveniently and compatiblely.

For this reason, the first object of the present invention provides a method for processing transport streams/program streams, including:

Analyzing the dependency identification information of the target program in the transport stream/program stream to obtain the dependency between the main bit stream and the knowledge bit stream in the target program;

acquiring a main bit stream and a knowledge bit stream in the target program, wherein at least one knowledge image in the knowledge bit stream is referenced by at least two temporally non-adjacent images in the main bit stream;

The pictures in the primary bitstream are decoded using the knowledge pictures in the knowledge bitstream as reference pictures.

Preferably, the dependency identification information is recorded in the program mapping table of the transport stream or the program stream mapping of the program stream, the program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet, The number information and attribute information of the elementary streams contained in the program are recorded.

Preferably, the dependency identification information is recorded in the program element descriptor in the program mapping table of the transport stream or the program stream mapping of the program stream, and the descriptor is used to identify whether the corresponding elementary stream is a reference knowledge bit stream If yes, this descriptor also identifies the index numbers of all knowledge bitstreams referenced by the main bitstream.

Preferably, the decoding time is recorded in the elementary stream packet to which the knowledge image in the knowledge bitstream and the main bitstream image belong to in the main bitstream, and the decoding time recorded in the elementary stream packet to which the knowledge image in the knowledge bitstream belongs It should be earlier than the decoding time recorded in the elementary stream packet to which the primary bitstream image belongs in the primary bitstream, wherein the primary bitstream image is the first one in the primary bitstream that is decoded with reference to the knowledge image image.

The second object of the present invention is to provide a method for generating a transport stream/program stream, including:

obtaining a knowledge bitstream and a main bitstream of a program, wherein at least one knowledge picture in the knowledge bitstream is referenced by at least two temporally non-adjacent pictures in the main bitstream;

subpacketizing the knowledge bit stream and main bit stream as elementary streams to obtain elementary stream packets, and put them into transport stream packets or program stream packets;

Putting dependency identification information in the transport stream packet or program stream packet, the information describes the dependency between the elementary stream to which the primary bitstream belongs and the elementary stream to which the knowledge bitstream belongs;

Output the resulting Transport Stream or Program Stream.

Preferably, said putting dependency identification information in said transport stream packet or program stream packet includes the following operations: putting said dependency identification information into a program mapping table of a transport stream or a program stream of a program stream In the mapping, the program mapping table is located in the transport stream packet, and the program stream mapping is located in the program stream packet, both of which record the number information and attribute information of the elementary stream contained in the program.

Preferably, putting the dependency identification information into the transport stream packet or the program stream packet includes the following operations: putting the dependency identification information into the program mapping table of the transport stream or the program stream of the program stream In the program element descriptor in the mapping, this descriptor is used to identify whether the corresponding elementary stream is the main bit stream that refers to the knowledge bit stream, and if so, this descriptor also identifies all The index number of the knowledge bitstream.

Preferably, the sub-packaging of the knowledge bit stream and the main bit stream as elementary streams to obtain elementary stream packets further includes the following operation: according to the time when the knowledge image contained in the elementary stream packet is used, set Determine the decoding time recorded in the elementary stream packet, and the decoding time should be earlier than the decoding time recorded in the elementary stream packet to which the main bit stream image belongs in the main bit stream, wherein the main bit stream image is the The first picture in the primary bitstream to be decoded with reference to the included knowledge picture.

The third object of the present invention is to provide a transport stream/program stream processing device, including:

Parsing module: parsing the dependency identification information of the target program in the transport stream/program stream to obtain the dependency between the main bit stream and the knowledge bit stream in the target program;

Acquisition module: acquire the knowledge bit stream and main bit stream in the target program, wherein at least one knowledge image in the knowledge bit stream is referenced by at least two images in the main bit stream that are not adjacent in time;

Decoding module: use the knowledge image in the knowledge bit stream as a reference image to decode the image in the main bit stream.

The fourth object of the present invention is to provide a transport stream/program stream generation device, comprising:

Acquisition module: acquire the knowledge bit stream and main bit stream of the target program, wherein at least one knowledge image in the knowledge bit stream is referenced by at least two temporally non-adjacent images in the main bit stream;

Subpackaging module: Subpackage the knowledge bit stream and main bit stream as elementary streams to obtain elementary stream packets, and put them into transport stream packets or program stream packets;

Putting module: Putting dependency identification information into the transport stream packet or program stream packet, the information describes the dependency between the elementary stream to which the primary bit stream belongs and the elementary stream to which the knowledge bit stream belongs relation;

Output module: output the generated transport stream or program stream.

The beneficial effects of the present invention are as follows: firstly, by increasing the dependency identification information between elementary streams, the function of storing and transmitting video programs including main bit streams and knowledge bit streams through transport streams or program streams is realized, so that the main bit streams The better compression effect obtained by stream and knowledge bit stream can be effectively used for data storage and transmission, reducing bandwidth consumption in practical applications; in addition, the dependency identification information designed by the present invention can be fully compatible with existing standard technologies and will not affect The update of existing equipment brings additional overhead. At the same time, due to the use of flexible program element descriptors, the flexible use of dependencies can be realized, and the syntax elements that may be brought about by the existing methods when processing the main bit stream and the knowledge bit stream are avoided. waste.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a schematic diagram of the image dependency relationship of a video sequence divided into random access segments using prior art 1;

Fig. 2 is a schematic diagram of the image dependency relationship of the video sequence divided into random access segments using the second prior art;

Fig. 3 is a schematic diagram of the image dependency relationship of the video sequence divided into random access segments using the prior art 3;

Fig. 4 is a schematic diagram of the image dependency of the video sequence divided into random access segments using prior art 4;

FIG. 5 is a schematic diagram of a data structure of a transport stream packet in the prior art;

FIG. 6 is a schematic diagram of a data structure of a program stream packet in the prior art;

Fig. 7 is a schematic diagram of a data structure of a PES packet in the prior art;

FIG. 8 is a schematic diagram of a data structure of a transport stream in the prior art;

FIG. 9 is a schematic diagram of a data structure of a program stream in the prior art;

FIG. 10 is a schematic diagram of a transport stream/program stream processing method disclosed in the present invention;

FIG. 11 is a schematic diagram of a method for generating a transport stream/program stream disclosed in the present invention;

FIG. 12 is a schematic diagram of a data structure of a transport stream in an embodiment disclosed in the present invention;

Fig. 13 is a schematic diagram of a data structure of a program stream in an embodiment disclosed by the present invention;

FIG. 14 is a schematic diagram of a transport stream/program stream processing device disclosed in the present invention;

FIG. 15 is a schematic diagram of an apparatus for generating a transport stream/program stream disclosed in the present invention.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the drawings and specific embodiments are only exemplary, and should not be construed as limiting the protection scope of the present invention. Definition of Terms:

Before describing the embodiment, declare the necessary noun definitions earlier:

Knowledge image: also known as knowledge base image, knowledge image is an image outside the image set that needs to be displayed in the random access segment to which the current image belongs and the nearest random access segment before it, knowledge image is a kind of reference image, used It is used to provide a reference for an image to be encoded or an image to be decoded.

RL (Reference to Library) image: An image that is encoded or decoded only with reference to the knowledge image. In one implementation, the RL image immediately follows the sequence header data, and the random access function is realized by first decoding the external knowledge image and then decoding the RL image; in one implementation, the RL image immediately follows the sequence header data, After the knowledge image data, the random access function is realized by first decoding the knowledge image and then decoding the RL image; in one implementation, the RL image follows the sequence header data, and there is data such as auxiliary enhancement information or extended information or user information in between. The random access function is realized by first decoding the knowledge image and then decoding the RL image.

Knowledge bit stream: also known as knowledge code stream, refers to a bit stream that only contains knowledge images. At least one knowledge picture in the knowledge bitstream is referenced by at least two temporally non-adjacent pictures in the main bitstream.

Main bit stream: also called main code stream, refers to the bit stream to which the image referring to the knowledge image belongs, and the bit stream does not contain the data of the knowledge image. In an implementation manner, the knowledge stream and the mainstream are interspersed and spliced into one code stream.

Example

In this embodiment, a transport stream/program stream processing method is provided. As shown in FIG. The bitstream and the master bitstream that references the knowledge bitstream, if present, locates the knowledge bitstream it depends on for the master bitstream for the decoding process. Specifically, the processing operations are as follows:

1. Obtain transport stream or program stream;

2. Analyzing the dependency identification information in the transport stream or program stream to obtain the dependency between the main bit stream and the knowledge bit stream;

3. According to the dependency relationship, process the transport stream or the program stream to obtain the main bit stream and its referenced knowledge bit stream;

4. Using the knowledge picture in the knowledge bit stream as a reference picture to decode the picture in the main bit stream.

Another embodiment corresponding to this embodiment provides a transport stream/program stream generation method at the sending end, as shown in FIG. 11 , the method is used to generate the stream processed by the above processing method. Specifically, the generation operation is as follows:

1. Obtain the main bit stream and its dependent knowledge bit stream;

2. subpacket the main bit stream and the knowledge bit stream respectively as elementary streams to obtain elementary stream packets, and put them into transport stream packets or program stream packets;

3. Putting dependency identification information in the transport stream packet or program stream packet, the information describes the dependency between the elementary stream to which the main bitstream belongs and the elementary stream to which the knowledge bitstream belongs;

4. Output the generated Transport Stream or Program Stream.

The methods disclosed in the above embodiments provide key operations for generating and processing transport streams and program streams. In different implementations and application scenarios, the above operations will be further expanded.

In an implementation manner, the above operation of "processing the transport stream or program stream to obtain the main bit stream and its reference knowledge bit stream according to the dependency relationship" further includes: parsing the header of the packet in the transport stream or program stream Information, according to the packet header information to obtain the information of the program contained in the transport stream or program stream and the information of the elementary stream in the program.

Specifically, for the transport stream, according to the header information of the transport stream packet, the program association table PAT and program mapping table PMT carried in the transport stream packet, as well as the conditional acceptance table CAT and network information table NIT that may exist, are obtained from these The serial number information of the program, the information of the elementary stream contained in the program, the attribute information of the elementary stream, etc. are obtained by parsing the table.

Specifically, for the program stream, the program stream mapping segment PSM and the program stream address segment PSD carried in the program stream packet are obtained by parsing the header information of the program stream packet, and the number information of the program, the Basic stream information, basic stream attribute information, etc.

Obtain the dependency identification information from the PAT or PMT of the transport stream or the PSM of the program stream to identify whether the elementary stream in the program is a knowledge bit stream or a main bit stream, and obtain the dependency between the main bit stream and the knowledge bit stream relation.

In another implementation, the above operation of "processing the transport stream or program stream according to the dependency relationship to obtain the main bit stream and its referenced knowledge bit stream" further includes: The PES packet to which the target program belongs is obtained from the packet, and the data of the elementary stream in the target program is extracted from the PES packet. Then, according to the dependency identification information obtained through the above analysis, the elementary stream including the main bitstream and the elementary stream including the knowledge bitstream are identified and acquired.

In yet another implementation, the above-mentioned operation of "using the knowledge image in the knowledge bitstream as a reference image to decode the image in the main bitstream" further includes: when the image to be decoded in the main bitstream is an RL image, Only use the knowledge picture in the knowledge bitstream as a reference picture for decoding; when the picture to be decoded in the main bitstream is a P/B picture, use its original short-term reference picture and (if the picture refers to the knowledge picture) the knowledge picture The composed reference image set is decoded.

In an implementation manner, the method for generating the transport stream and the program stream further includes: the main bit stream and the knowledge bit stream are respectively divided into PES packets as different elementary streams. The ordering of these PES packets can be to first generate PES packets containing the knowledge bit stream, and then generate PES packets containing the main bit stream; or first generate PES packets containing the main bit stream, and then generate PES packets containing the knowledge bit stream The PES packet; it can also be alternately interspersed with the two. In yet another implementation, the method for generating the above-mentioned transport stream and program stream further includes: the main bit stream and the knowledge bit stream are combined into one elementary stream and subpackaged into PES packets. At this time, the data of the knowledge bit stream and the main bit Stream data cannot exist simultaneously in the same PES packet.

In yet another implementation manner, the above-mentioned operation of "putting dependency identification information into the transport stream packet or program stream packet" further includes:

For the transport stream, put the dependency identification information into the program association table PAT or program mapping table PMT carried in the transport stream packet, as well as the conditional acceptance table CAT and network information table NIT that may exist.

For the program stream, the dependency identification information is put into the program stream mapping segment PSM or the program stream address segment PSD carried in the program stream packet.

In the foregoing embodiments and implementations thereof, there are many specific implementation methods for generating and processing the dependency identification information, which are described in detail as follows.

In one implementation, considering the compatibility with the existing standard methods, an attempt is made to use existing syntax elements and add new semantics to distinguish the main bit stream and the knowledge bit stream and describe the dependencies between them.

Specifically, in the adaptation field (adaptation field) of the transport stream, there is a syntax element elementary_stream_priority_indicator to describe the importance of the connection stream contained in the transport stream packet to which the adaptation field belongs. This syntax element can be reused and a new semantics to distinguish master bitstream and knowledge bitstream and describe the dependencies between them. For example, in the adaptation field, the syntax element elementary stream priority indicator field elementary_stream_priority_indicator adds the following semantics in italics (other semantics are the original semantics of this field/grammatical element):

"This field is 1 bit and indicates the priority of the elementary stream data carried in the payload data of the transport stream packet with the same PEID. A value of '1' indicates that the payload data has priority over other transport stream packets High. For video, this field may only be '1' if the payload data contains one or more bytes from an intra-coded picture. A value of '0' indicates that the payload data is different from other values of this field. Transport stream packets with '1' have the same priority. For AVS3 encoded video, when there are knowledge bit streams and main bit streams, the value '0' indicates that the payload data belongs to the main bit stream, and the value is '1' 'Indicates that the payload data belongs to the knowledge bit stream, and the data of the knowledge bit stream needs to be decoded first when decoding the data of the main bit stream."

The advantage of this implementation is that there is no need to add new syntax elements, but since the adaptation field only exists in the transport stream packet, and the adaptation field mainly contains some unimportant information, it is an optional field, so this scheme may It will introduce other redundant grammatical elements, and cannot solve the problem of describing dependencies in program streams.

In yet another implementation, considering that the elementary streams contained in each program are recorded in the program mapping table PMT of the transport stream and the program stream mapping PSM of the program stream, the program mapping table of the transport stream and the program stream Additional flags are added to the program stream map of the program to describe the dependency relationship between the main bit stream and the knowledge bit stream. As shown in the following table, the syntax elements in italics are newly added syntax elements used to identify the dependency relationship between the main bit stream and the knowledge bit stream.

Table 1 Syntax elements of the transport stream program mapping table

Among them, the existing syntax elements and their definitions in the program mapping table can be accurately described in the existing method of transport stream, and will not be repeated here. The semantics of the newly added syntax element identifier are as follows:

Knowledge bit stream identifier is_library_stream

This field is 1 bit. It defines whether the elementary stream corresponding to this identifier is a knowledge bit stream in the program. A value of '1' indicates that the elementary stream corresponding to this flag is a knowledge bit stream, which is dependent on the main bit stream in the same program; a value of '0' indicates that the elementary stream corresponding to this flag is a main bit stream, which is dependent on The knowledge bitstream of the main bitstream should be processed before decoding the main bitstream.

The purpose of adding the conditional judgment "if(stream_type==0xD4)" in the program mapping table is to reduce the data waste of the syntax element is_library_stream when processing non-long-span coded video bit streams, because only when the processed video stream uses The syntax element is_library_stream is only necessary when the AVS3 encoding method (stream_type is defined as 0xD) and the main bit stream and knowledge bit stream are generated.

The advantage of this implementation is that the dependency relationship between the main bit stream and the knowledge bit stream can be quickly obtained when parsing PMT or PSM, but this flag will be wasted when only using the AVS3 encoding method to generate only the main bit stream At the same time, it is not compatible with the original standard method.

In another implementation manner, for the generation and processing of the program stream, a new syntax element is added in the program stream mapping of the program stream, as shown in the following table.

Table 2 Syntax of Program Stream Mapping of Program Stream

Among them, the existing syntax elements and their definitions in the program mapping table can be accurately described in the existing method of transport stream, and will not be repeated here. The semantics of the newly added syntax element identifier are as follows:

Knowledge bit stream identifier is_library_stream

This field is 1 bit. It defines whether the elementary stream corresponding to this identifier is a knowledge bit stream in the program. A value of '1' indicates that the elementary stream corresponding to this flag is a knowledge bit stream, which is dependent on the main bit stream in the same program; a value of '0' indicates that the elementary stream corresponding to this flag is a main bit stream, which is dependent on The knowledge bitstream of the main bitstream should be processed before decoding the main bitstream.

This implementation has the same advantages and disadvantages as the previous implementation.

In another implementation, considering that there are program element descriptors in the transport stream and the program stream that can describe the attributes of the elementary stream, a new descriptor can be added to describe the dependency between the primary bit stream and the knowledge bit stream. In this embodiment, the AVS3 long-span coded video stream descriptor is designed to represent the program elements including the video coded by using the coding method based on the large time-span related information in AVS3, including the main bit stream and the knowledge bit stream. These program elements are multiplexed in multiple streams.

Table 3 AVS3 long-span coded video stream descriptor

Among them, the semantics of the syntax elements are as follows, and the defined number of digits and values are for example only, which can be any available number of digits and values:

Description sub-tag field descriptor_tag

This field has a length of 8 bits and is used to identify each descriptor.

Descriptor length field descriptor_length

This field is 8 bits long. Specifies the number of bytes of the descriptor immediately following this field.

Knowledge bit stream identifier is_library_stream

This field is 1 bit. It defines whether the elementary stream corresponding to the descriptor is a knowledge bit stream in the program. A value of '1' indicates that the elementary stream corresponding to this descriptor is a knowledge bit stream, which is dependent on the main bit stream in the same program; a value of '0' indicates that the elementary stream corresponding to this descriptor is a main bit stream, which depends on the same For the knowledge bitstream in the program, the knowledge bitstream it depends on should be processed before decoding the main bitstream.

The above method uses program element descriptors, which are compatible with the original standards, and the descriptors can be freely selected by the generating end, thus avoiding data waste. This is a compatible and data-saving implementation. However, the above implementation method only uses 1 bit identifier to judge whether the basic stream in the program is a knowledge bit stream or a main bit stream. By default, the main bit stream must refer to the knowledge bit stream, which cannot cope with various application situations.

Therefore, in another implementation, the syntax elements in the program element descriptor are extended, and on the basis of identifying the main bit stream and the knowledge bit stream, the index number of the knowledge bit stream referenced by the main stream is further identified, and at the same time it supports Scenarios where one master bitstream can refer to multiple knowledge bitstreams. The syntax and semantics of this extended descriptor are as follows.

Table 4 AVS3 long-span coded video stream descriptor

Among them, the semantics of the syntax elements are as follows, and the defined number of digits and values are for example only, which can be any available number of digits and values:

Description sub-tag field descriptor_tag

This field has a length of 8 bits and is used to identify each descriptor.

Descriptor length field descriptor_length

This field is 8 bits long. Specifies the number of bytes of the descriptor immediately following this field.

Knowledge bit stream identifier is_library_stream

This field is 1 bit. It defines whether the elementary stream corresponding to the descriptor is a knowledge bit stream in the program. A value of '1' indicates that the elementary stream corresponding to this descriptor is a knowledge bit stream; a value of '0' indicates that the elementary stream corresponding to this descriptor is a main bit stream, which depends on the knowledge bit stream in the same program and is decoded when decoding the main bit stream. Bitstreams should be preceded by methods of the knowledge bitstreams on which they depend.

Referenced knowledge bit stream number field num_ref_library_stream

This field is 8 bits long. When the elementary stream corresponding to the descriptor is defined as the main bit stream, the number of elementary bit streams including the knowledge bit stream that it refers to. The value of num_ref_library_stream can be 0, at this time, the main bit stream does not refer to the knowledge bit stream.

Referenced knowledge bit stream index number library_stream_id

This field is 8 bits long. When the elementary stream corresponding to this descriptor is defined as the main bit stream, it refers to the index number of the elementary bit stream including the knowledge bit stream.

In one embodiment, all the above-mentioned embodiments and their implementations should also meet the constraints: when the knowledge bit stream and the main bit stream are subpackaged into PES packets, the PES to which the knowledge image belongs in the referenced knowledge bit stream The decoding moment DTS recorded in the packet shall be earlier than the decoding moment recorded in the packet of the elementary stream to which the primary bitstream picture belongs in the primary bitstream referencing the knowledge bitstream, wherein the primary bitstream image is the primary bitstream image of the primary bitstream The first image in the bitstream to be decoded with reference to the knowledge image.

In one embodiment, the above constraints can be satisfied by other methods. One method of realization is that the decoding moment DTS of the PES packet to which the knowledge image belongs in the knowledge bitstream is expanded into multiple DTSs, and these DTS are related to the PES packets to which all the main bitstream images that refer to the knowledge image in the main bitstream belong. One-to-one correspondence between decoding moments, that is, PES packets containing knowledge images are decoded at multiple moments for reference. In another implementation method, the PES packet to which the knowledge image belongs does not use DTS to establish a dependency relationship with the main bitstream image, but can add new image-level dependency identification information to the PES packet to which the main bitstream belongs. The identification information can be indexed to the PES packet where all the knowledge images referenced by the main bitstream image in the PES packet are located. Compared with the previous implementation, this is an explicit expression method of dependency relationship.

In yet another embodiment, when the main bitstream and its dependent knowledge bitstream exist in the program, and the transport stream is generated using the aforementioned method, its data structure should be as shown in FIG. 12 . Wherein, a video program in the TS stream includes two elementary streams, namely the knowledge bit stream (the PID of the TS packet to which it belongs=0x11) and the main bit stream (the PID of the TS packet to which it belongs=0x12). In PMT, avs3crr_descriptor() is added for the two elementary streams of the program, the value of is_library_stream in the descriptor corresponding to the elementary stream containing the knowledge bit stream is 1, and the descriptor corresponding to the elementary stream containing the main bit stream The value of the is_library_stream flag in is 0.

Similarly, in another embodiment, when the above method is used to generate the program stream, its data structure should be as shown in FIG. 13 . Wherein, a video program in the PS stream includes two elementary streams, which are the knowledge bit stream (the PID of the PS packet to which it belongs=0x11) and the primary bit stream (the PID of the PS packet to which it belongs=0x12). In PSM, avs3crr_descriptor() is added for the two elementary streams of the program, the value of is_library_stream in the descriptor corresponding to the elementary stream containing the knowledge bit stream is 1, and the descriptor corresponding to the elementary stream containing the main bit stream The value of the is_library_stream flag in is 0.

In yet another embodiment, when processing the received transport stream or program stream, the receiving end needs to go through a series of buffering and demultiplexing operations, and then obtain the elementary stream data in the transport stream or program stream for use in decoding. This embodiment describes the operations that the receiving end should implement, so as to realize correct buffering and decoding of elementary stream data, and avoid overflow or underflow of elementary stream data in the data cache.

The methods for generating and processing transport streams and program streams are provided above. Correspondingly, the operations in the above embodiments and implementations can be implemented as corresponding devices and modules.

In one embodiment, a transport stream/program stream processing device is provided. As shown in FIG. 14 , the device includes a processor, storage and a set of modules capable of implementing the following operations, including:

1. Obtaining module: used to obtain transport stream or program stream;

2. Parsing module: parsing the dependency identification information in the transport stream or program stream to obtain the dependency between the main bit stream and the knowledge bit stream;

3. Processing module: according to the dependency relationship, process the transport stream or the program stream to obtain the main bit stream and its referenced knowledge bit stream;

4. Decoding module: use the knowledge image in the knowledge bit stream as a reference image to decode the image in the main bit stream. The programs of the modules are all stored and executed by the processor.

Another embodiment corresponding to this embodiment provides a transport stream/program stream generation device at the sending end, as shown in Figure 15, the device includes a processor, storage and a set of modules capable of implementing the following operations ,include:

1. Acquisition module: acquire the main bit stream and its dependent knowledge bit stream;

2. Packetization module: respectively put the main bit stream and the knowledge bit stream into transport stream packets or program stream packets as elementary stream packets;

3. Inserting module: put dependency identification information in the transport stream packet or program stream packet, this information describes the relationship between the elementary stream to which the main bit stream belongs and the elementary stream to which the knowledge bit stream belongs dependencies;

4. Output module: output the generated transport stream or program stream.

The programs of the modules are all stored and executed by the processor.

The above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the above preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced All should not deviate from the spirit and scope of the technical solution of the present invention.

Claims (10)

1. A method for processing a transport/program stream, comprising:

analyzing the dependency relationship identification information of the target program in the transport stream/program stream to acquire the dependency relationship between the main bit stream and the knowledge bit stream in the target program;

acquiring a main bit stream and a knowledge bit stream in the target program, wherein at least one knowledge image in the knowledge bit stream is referred to by at least two temporally non-adjacent images in the main bit stream;

the pictures in the main bitstream are decoded using the knowledge pictures in the knowledge bitstream as reference pictures.

2. The method of claim 1, wherein:

the dependency identification information is recorded in a program mapping table of the transport stream or a program stream map of the program stream, the program mapping table is located in a transport stream packet, the program stream map is located in a program stream packet, and the number information and attribute information of elementary streams contained in the program are recorded.

3. The method of claim 2, further comprising:

the dependency identification information is recorded in a program element descriptor in a program mapping table of a transport stream or a program stream map of a program stream, and the descriptor is used for identifying whether the corresponding elementary stream is a main bitstream of a reference knowledge bitstream, and if so, the descriptor also identifies index numbers of all knowledge bitstreams referenced by the main bitstream.

4. The method according to claim 1, wherein the decoding time recorded in the elementary stream packet to which the knowledge image in the knowledge bit stream belongs and the main bit stream image in the main bit stream, is earlier than the decoding time recorded in the elementary stream packet to which the main bit stream image in the main bit stream belongs, wherein the main bit stream image is the first image in the main bit stream to be decoded by referring to the knowledge image.

5. A method for generating a transport stream/program stream, comprising:

acquiring a knowledge bit stream and a main bit stream of a program, wherein at least one knowledge image in the knowledge bit stream is referred to by at least two temporally non-adjacent images in the main bit stream;

the knowledge bit stream and the main bit stream are used as basic streams to be packaged to obtain basic stream packet packets, and the basic stream packet packets are put into a transmission stream packet or a program stream packet;

putting dependency relationship identification information into the transport stream packet or the program stream packet, wherein the dependency relationship identification information describes the dependency relationship between the elementary stream to which the main bit stream belongs and the elementary stream to which the knowledge bit stream belongs;

and outputting the generated transport stream or program stream.

6. The method according to claim 5, wherein said placing dependency identification information in said transport stream packet or program stream packet comprises the following operations:

and putting the dependency relationship identification information into a program mapping table of a transport stream or a program stream map of a program stream, wherein the program mapping table is positioned in a transport stream packet, the program stream map is positioned in the program stream packet, and both the program mapping table and the program stream map record the number information and the attribute information of the elementary streams contained in the program.

7. The method according to claim 6, wherein said placing dependency identification information in said transport stream packet or program stream packet comprises the following operations:

and putting the dependency relationship identification information into a program element descriptor in a program mapping table of a transport stream or a program stream map of a program stream, wherein the descriptor is used for identifying whether the corresponding elementary stream is a main bit stream of a reference knowledge bit stream, and if so, the descriptor also identifies index numbers of all knowledge bit streams referred by the main bit stream.

8. The method according to claim 5, wherein packetizing the knowledge bit stream and the main bit stream as elementary streams to obtain elementary stream packet further comprises:

and setting the decoding time recorded in the basic flow packet according to the time when the knowledge image contained in the basic flow packet is used, wherein the decoding time is earlier than the decoding time recorded in the basic flow packet to which the main bit stream image belongs in the main bit stream, and the main bit stream image is the first image which is decoded by referring to the contained knowledge image in the main bit stream.

9. A transport/program stream processing apparatus, comprising:

an analysis module: analyzing the dependency relationship identification information of the target program in the transport stream/program stream to acquire the dependency relationship between the main bit stream and the knowledge bit stream in the target program;

an acquisition module: acquiring a knowledge bit stream and a main bit stream in the target program, wherein at least one knowledge image in the knowledge bit stream is referred to by at least two temporally non-adjacent images in the main bit stream;

a decoding module: the pictures in the main bitstream are decoded using the knowledge pictures in the knowledge bitstream as reference pictures.

10. An apparatus for generating a transport/program stream, comprising:

an acquisition module: acquiring a knowledge bit stream and a main bit stream of a target program, wherein at least one knowledge image in the knowledge bit stream is referred to by at least two temporally non-adjacent images in the main bit stream;

a sub-packaging module: the knowledge bit stream and the main bit stream are used as basic streams to be packaged to obtain basic stream packet packets, and the basic stream packet packets are put into a transmission stream packet or a program stream packet;

putting in a module: putting dependency relationship identification information into the transport stream packet or the program stream packet, wherein the dependency relationship identification information describes the dependency relationship between the elementary stream to which the main bit stream belongs and the elementary stream to which the knowledge bit stream belongs;

an output module: and outputting the generated transport stream or program stream.

Images