CN104253996A - Video data sending and receiving methods, video data sending and receiving devices and video data transmission system - Google Patents

Abstract

The invention discloses a method for sending and receiving video data, its device, and a transmission system. The sending method includes: determining the original code of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame The background frame in the code stream; each determined background frame is divided into at least two data packets, which are respectively added to the auxiliary code stream identifier and then encapsulated to form the auxiliary code stream; each non-background frame in the original coded code stream is added to the main After the code stream is identified, it is encapsulated to form a main code stream; and the auxiliary code stream and the main code stream are sent. Since the background frame is divided into at least two data packets, which are added to the sub-code stream identifier and encapsulated to form a sub-code stream, the sub-code stream and the main code stream are sent later, which avoids the direct transmission of the background frame and reduces the number of images in the transmission due to the code stream. The instantaneous bit rate is too high, resulting in delay or loss of subsequent non-background frames, and at the same time, it does not reduce the quality of the high-quality background image used as the background frame, taking into account the transmission bandwidth and image quality.

Description

Video data sending and receiving method and its device and transmission system

technical field

The present invention relates to video coding and decoding technology, in particular to a method for sending and receiving video data, a device thereof, and a transmission system.

Background technique

At present, access terminals for digital video have developed from traditional personal computers (PCs) to smart phones, personal digital assistants (Personal Digital Assistant, PDA), tablets and other terminals. Digital video signals are also faced with different transmission environments, including wireless networks, third-generation mobile communication (3G) networks, and so on. Different receiving terminals have different processing capabilities and storage capabilities, and different network environments have different transmission bandwidths, which pose a great challenge to digital video transmission.

Based on the above reasons, more and more attention has been paid to the video compression technology that can reduce the requirement on the transmission bandwidth of the network. Usually, digital video usually contains a lot of redundant information, such as: spatial redundancy, time redundancy and information entropy redundancy. The main purpose of video compression is to remove redundant information in digital video. Because the video signal has a large amount of data and requires a large transmission bandwidth, it is necessary to compress the video signal before transmitting the digital video signal. In addition, video compression has two requirements: a) a certain compression ratio; b) guarantee a certain quality.

Predictive coding is one of the important technical means to eliminate redundant information in digital video. Predictive coding is to use the correlation between signals, use one or more previous signals as the predicted value of the current signal, and encode the difference between the actual value and the predicted value of the current signal. In inter-frame predictive coding, the image used as the predictive value of the current signal is called a reference image, and the reference image usually has higher quality and bit rate.

Usually, the background frame will be used as a reference image for subsequent images. However, for fixed bandwidth and low bandwidth network transmission environments, the background frame becomes the bottleneck of transmission. This is because the background frame is a high-quality background image modeled by the image in the sequence. If the modeling is ideal, an image without any foreground will be obtained. For efficient compression of subsequent images (refer to the background frame), usually the background The frame adopts a high-quality encoding method, and its encoding process adopts the process of encoding a key frame (I frame). After encoding, the amount of data is large and the quality is relatively high. If the background frame is directly transmitted, it is easy to cause the instantaneous bit rate of the image in the bit stream transmission to be too high, resulting in delay or loss of the subsequent non-background frame; if the quality of the high-quality background image used as the background frame is reduced, it will affect the subsequent need to refer to the high-quality The compression effect of the non-background frames encoded by the background image, resulting in a decrease in overall performance. Therefore, when transmitting video data containing background frames in the prior art, there is a problem that both transmission bandwidth and image quality cannot be considered.

Contents of the invention

In view of this, the embodiments of the present invention provide a method for sending and receiving video data, its device, and a transmission system, so as to solve the problem in the prior art that transmission bandwidth and image quality cannot be balanced.

Embodiments of the present invention provide a method for sending and receiving video data, a device thereof, and a transmission system, specifically as follows:

In a first aspect, a method for sending video data, the method includes:

Determine the background frame in the original coded stream of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame;

Divide each determined background frame into at least two data packets, add sub-code stream identifiers respectively, and then encapsulate to form a sub-code stream;

Add each non-background frame in the original encoded code stream to the main code stream mark and then encapsulate it to form the main code stream;

Send the auxiliary code stream and the main code stream.

Through this possible implementation, the background frame is divided into at least two data packets, which are respectively added with the sub-code stream identifier and then encapsulated to form a sub-code stream, and the sub-code stream and the main code stream are sent later, avoiding the direct transmission of the background frame. Reduce the delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of the image in the bit stream transmission, and at the same time, it does not reduce the quality of the high-quality background image used as the background frame, and will not affect the subsequent need to refer to the high-quality background image The compression effect of non-high-quality background images for encoding, taking into account transmission bandwidth and image quality.

With reference to the first aspect, in a first possible implementation manner, the sending the auxiliary code stream and the main code stream includes:

The auxiliary code stream and the main code stream are sent in time division, wherein the time point of completion of sending the auxiliary code stream is no later than the time point of completion of sending of the main code stream that requires the auxiliary code stream to be decoded.

In combination with the first possible implementation of the first aspect, in the second possible implementation, each of the determined background frames is divided into at least two data packets, which are respectively added with sub-stream identifiers and then encapsulated, Form a secondary code stream, including:

According to the data size of the background frame, the channel capacity under the current bandwidth, and the requirements for data packets in the network protocol, each determined background frame is divided into at least two data packets, which are respectively added to the sub-code stream identifier and then encapsulated to form Secondary stream.

With reference to the first aspect, the first possible implementation of the first aspect or the second possible implementation of the first aspect, in the third possible implementation, the original encoded code stream includes The image serial number of the playback sequence of each frame image, the method also includes:

determining that the maximum value of the image sequence number in the original coded stream is less than the number of image frames contained in the original coded stream;

Generate position offsets of each main code stream and each auxiliary code stream in the original code stream, where the position offset represents an offset relative to the position of the sequence header in the original code stream;

The sending the main code stream and the auxiliary code stream includes:

When sending the main stream, send the position offset of the main stream along with the main stream;

When sending the sub-code stream, the position offset of the sub-code stream is sent along with the sub-code stream.

With reference to the first aspect, the first possible implementation of the first aspect or the second possible implementation of the first aspect, in a fourth possible implementation, the original encoded code stream includes The image serial number of the playback sequence of each frame image, the method also includes:

Determining that the maximum value of the image sequence numbers in the original coded stream is equal to the number of image frames contained in the original coded stream;

After said dividing each determined background frame into at least two data packets, before encapsulating after respectively adding auxiliary code stream identification, said method also includes:

Numbering the data packets obtained by segmentation, adding the numbers to the corresponding data packets, the numbering is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame .

In a second aspect, a method for receiving video data, the method includes:

Receive video data code stream, described video data code stream comprises main code stream that contains main code stream identification and auxiliary code stream that contains auxiliary code stream identification, and described auxiliary code stream is that each determined background frame is divided into at least two data packets, which are formed by encapsulation after adding the sub-code stream identifier respectively, and the main code stream is formed by adding each non-background frame in the original code stream to the main code stream identifier and then encapsulating;

Determine the main code stream and auxiliary code stream in the received code stream according to the identifier contained in the code stream;

Restore the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder.

Through this possible implementation, since the received background frame is divided into at least two data packets, which are respectively added with the auxiliary code stream identifier and then encapsulated to form the auxiliary code stream and the main code stream, reducing the image transient code caused by the code stream transmission. If the rate is too high, the delay or loss of subsequent non-background frames will not be reduced, and the quality of the high-quality background image used as the background frame will not be reduced, and the quality of the subsequent non-high-quality background image that needs to be coded with reference to the high-quality background image will not be affected. The compression effect takes into account the transmission bandwidth and image quality.

With reference to the second aspect, in the first possible implementation manner, each received main code stream or auxiliary code stream is accompanied by a position offset of the main code stream or auxiliary code stream in the original encoded code stream, The position offset is an offset relative to the position of the sequence header in the original encoded code stream;

Said restoring the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder includes:

Restore the determined main code stream and auxiliary code stream to a video image by using the position offset of each received main code stream or auxiliary code stream in the original encoded code stream along with the main code stream or auxiliary code stream The original encoded code stream for the decoder to decode.

With reference to the second aspect, in a second possible implementation manner, the auxiliary code stream includes numbers, the main code stream includes image sequence numbers, and the image sequence numbers are used to indicate the playback order of each frame of images, The number is used to determine the image sequence number of the background frame to which the data packet contained in the sub-code stream belongs and the position of the data in the data packet in the background frame;

Said restoring the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder includes:

Using the serial number to restore the determined secondary stream to a background frame, and determine the image sequence number of the restored background frame;

Using the image serial number of the restored background frame and the image serial number included in the main code stream, the background frame and the main code stream are restored to the original encoded code stream for decoding by the decoder.

In a third aspect, the device includes:

The first determining unit is used to determine the background frame in the original coded stream of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame;

A sub-code stream forming unit, configured to divide each determined background frame into at least two data packets, add sub-code stream identifiers and then encapsulate to form a sub-code stream;

The main code stream forming unit is used for encapsulating each non-background frame in the original encoded code stream after adding the main code stream identifier to form the main code stream;

A sending unit, configured to send the auxiliary code stream and the main code stream.

Through this possible implementation, the background frame is divided into at least two data packets, which are respectively added with the sub-code stream identifier and then encapsulated to form a sub-code stream, and the sub-code stream and the main code stream are sent later, avoiding the direct transmission of the background frame. Reduce the delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of the image in the bit stream transmission, and at the same time, it does not reduce the quality of the high-quality background image used as the background frame, and will not affect the subsequent need to refer to the high-quality background image The compression effect of non-high-quality background images for encoding, taking into account transmission bandwidth and image quality.

With reference to the third aspect, in a first possible implementation manner, the sending unit is further configured to time-divisionally send the auxiliary code stream and the main code stream, wherein the sending completion time of the auxiliary code stream is not later At the point in time when the main code stream that requires the auxiliary code stream to be decoded is sent.

In conjunction with the third aspect, in a second possible implementation, the auxiliary code stream forming unit is further configured to: Each determined background frame is divided into at least two data packets, which are respectively added with sub-code stream identifiers and then encapsulated to form sub-code streams.

With reference to the third aspect, the first possible implementation of the third aspect, or the second possible implementation of the third aspect, in the third possible implementation, the device further includes:

The second determination unit is configured to determine that the maximum value of the image sequence number in the original encoded code stream is less than the number of image frames contained in the original encoded code stream;

The position offset generation unit is used to generate the position offset of each main code stream and each auxiliary code stream in the original code stream, and the position offset represents the offset relative to the position of the sequence header in the original code stream. displacement.

The sending unit is further configured to send the position offset of the main code stream along with the main code stream when sending the main code stream; when sending the auxiliary code stream, send the position offset of the auxiliary code stream along with the auxiliary code stream Stream sending.

With reference to the third aspect, the first possible implementation of the third aspect, or the second possible implementation of the third aspect, in a fourth possible implementation, the device further includes:

A third determining unit, configured to determine that the maximum value of the image sequence number in the original encoded code stream is equal to the number of image frames contained in the original encoded code stream;

The numbering unit is used for numbering the divided data packets after dividing each determined background frame into at least two data packets, respectively adding the sub-stream identifier and encapsulating, and adding the numbers to the corresponding The number is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame.

In a fourth aspect, a device for receiving video data, the device includes:

The receiving unit is used to receive the video data code stream, the video data code stream includes the main code stream containing the main code stream identification and the auxiliary code stream containing the auxiliary code stream identification, and the auxiliary code stream is each background to be determined The frame is divided into at least two data packets, which are formed by encapsulation after adding the auxiliary code stream identifier respectively, and the main code stream is formed by adding each non-background frame in the original code stream to the main code stream identifier and then encapsulating;

A code stream type determining unit, configured to determine the main code stream and the auxiliary code stream in the received code stream according to the identifier contained in the code stream;

The restoration unit is used to restore the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder.

Through this possible implementation, since the received background frame is divided into at least two data packets, which are respectively added with the auxiliary code stream identifier and then encapsulated to form the auxiliary code stream and the main code stream, reducing the image transient code caused by the code stream transmission. If the rate is too high, the delay or loss of subsequent non-background frames will not be reduced, and the quality of the high-quality background image used as the background frame will not be reduced, and the quality of the subsequent non-high-quality background image that needs to be coded with reference to the high-quality background image will not be affected. The compression effect takes into account the transmission bandwidth and image quality.

With reference to the fourth aspect, in the first possible implementation manner, each received main code stream or auxiliary code stream is accompanied by a position offset of the main code stream or auxiliary code stream in the original encoded code stream, The position offset is an offset relative to the position of the sequence header in the original encoded code stream;

The restoration unit is further used to restore the determined main code stream and auxiliary code stream to the original code stream of the video image by using the position offset of each received code stream accompanied by the code stream in the original encoded code stream Encoded code stream for decoder to decode.

With reference to the fourth aspect, in a second possible implementation manner, the auxiliary code stream includes numbers, the main code stream includes image sequence numbers, and the image sequence numbers are used to indicate the playback order of each frame of images, The number is used to determine the image sequence number of the background frame to which the data packet contained in the sub-code stream belongs and the position of the data in the data packet in the background frame;

The restoration unit is further used to restore the determined secondary code stream to a background frame by using the serial number, and determine the image sequence number of the restored background frame;

Using the image serial number of the restored background frame and the image serial number included in the main code stream, the background frame and the main code stream are restored to the original encoded code stream for decoding by the decoder.

According to a fifth aspect, a video data transmission system, the system includes: any video data sending device mentioned above and any video data receiving device mentioned above.

Through this possible implementation, the background frame is divided into at least two data packets, which are respectively added with the sub-code stream identifier and then encapsulated to form a sub-code stream, and the sub-code stream and the main code stream are sent later, avoiding the direct transmission of the background frame. Reduce the delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of the image in the bit stream transmission, and at the same time, it does not reduce the quality of the high-quality background image used as the background frame, and will not affect the subsequent need to refer to the high-quality background image The compression effect of non-high-quality background images for encoding, taking into account transmission bandwidth and image quality.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings without making creative efforts.

FIG. 1 is a flow chart of a method for sending video data in Embodiment 1 of the present invention;

Fig. 2 is a schematic diagram of repackaging and forming a main code stream and an auxiliary code stream after the original code stream data is extracted and added with an identifier in Embodiment 1 of the present invention;

FIG. 3 is a schematic diagram of coding levels of a video sequence in Embodiment 1 of the present invention;

FIG. 4 is a schematic diagram of a layered stream structure obtained after encoding the video sequence shown in FIG. 3 in Embodiment 1 of the present invention;

FIG. 5 is a flow chart of another method for sending video data in Embodiment 1 of the present invention;

FIG. 6 is a flow chart of a method for receiving video data in Embodiment 2 of the present invention;

7 is a schematic diagram of determining the main code stream and the auxiliary code stream by using the received code stream in Embodiment 2 of the present invention and restoring them to the original coded code stream;

FIG. 8 is one of the structural schematic diagrams of a device for sending video data in Embodiment 3 of the present invention;

FIG. 9 is the second structural schematic diagram of a device for sending video data in Embodiment 3 of the present invention;

FIG. 10 is a third structural schematic diagram of a device for sending video data in Embodiment 3 of the present invention;

FIG. 11 is a schematic structural diagram of a device for receiving video data in Embodiment 4 of the present invention;

FIG. 12 is a schematic structural diagram of a video data transmission system in Embodiment 5 of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments . Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In order to solve the problem in the prior art that transmission bandwidth and image quality cannot be balanced, an embodiment of the present invention provides a method for sending and receiving video data, a device thereof, and a transmission system.

The solution of the present invention will be described in detail below through specific examples. Of course, the present invention is not limited to the following examples.

Embodiment one

Embodiment 1 of the present invention provides a method for sending video data, the flow chart of which is shown in Figure 1, specifically comprising the following steps:

Step 101: Determine the background frame in the original coded stream of the video image according to the identification information indicating that the current frame is a background frame and/or a non-background frame;

Usually, there are five image types in video coding, which are Intra picture (I frame, also known as key frame), forward predictive coding frame (Predictive-frame, P frame), bidirectional predictive interpolation coding Frame (Bi-directional interpolated prediction frame, B frame), background frame (backGround frame, G frame) and switching frame (Switched, S frame). Among them, I frame, P frame, B frame and S frame may be called non-background frames. Of course, it is not limited to the above image types, and other image types may also be defined according to actual needs. After the video is encoded using the frame encoding method, the original encoded code stream of the video image is obtained; there will be flag bits in the code stream to identify what kind of frame the current frame is, such as flag bits or start code information.

Take the Audio Video coding Standard (AVS) in which the background frame is a high-quality code stream and other frames are high-compression code streams as an example. The background frame and the non-background frame are in the same code stream. AVS There will be a flag in the code stream to indicate whether the current frame is a background frame. More specifically, if there is a background in the code stream, the background picture flag (background_picture_flag) is 1, and the background picture output flag (background_picture_output_flag) is 0, then it is determined that the current frame is a background frame.

Step 102: Divide each determined background frame into at least two data packets, add sub-code stream identifiers respectively, and then encapsulate to form a sub-code stream;

Preferably, this step 102 includes:

According to the data size of the background frame, the channel capacity under the current bandwidth, and the requirements for data packets in the network protocol, each determined background frame is divided into at least two data packets, which are respectively added to the sub-code stream identifier and then encapsulated to form Secondary stream.

The above-mentioned step 102 specifically includes: each code stream obtained after encoding the image of the background frame to obtain the original code stream and dividing it into the application layer Protocol Data Unit (Protocol Data Unit, PDU), the PDU of the transport layer, and the network layer respectively in turn. The PDU of the data link layer and the PDU of the data link layer, that is, the data packet obtained by dividing the background frame, and finally adding the sub-code stream identifier for encapsulation, and converting it into a sub-code stream from the physical layer.

Furthermore, the preferred steps of the above step 102 can also be further specifically the following steps A to D:

Step A: Judging whether the data volume of the determined background frame is greater than the channel capacity under the current bandwidth; if the judgment result is yes, then execute step B, if the judgment result is no, then execute step C;

Step B: Divide the determined background frame into at least two data packets according to the channel capacity under the current bandwidth and the data volume requirements of the network protocol, and add the at least two data packets to the sub-stream identifier respectively Afterwards, encapsulation is carried out to form a sub-code stream, and the size of any data packet in the above-mentioned at least two data packets is not greater than the channel capacity under the current bandwidth;

Step C: judging whether the data volume of the above-mentioned determined background frame is greater than the limitation of the maximum data volume of the data packet in the network protocol, if so, then perform step D; if not, then perform step E;

Step D: Divide the determined background frame into at least two data packets according to the data volume requirements of the data packets in the network protocol, add the sub-code stream identifiers respectively, and then encapsulate to form a sub-code stream.

Executing step E: directly adding the determined background frame into the sub-stream identifier and then encapsulating it to form a sub-stream.

Preferably, in order to obtain the position information of the divided data packets in the original code stream, so that the subsequent receiver can restore the background frame, after the above-mentioned division of each determined background frame into at least two data packets, add Before encapsulation after the secondary code stream is identified, the above method also includes:

The divided data packets are numbered, and the number is added to the corresponding data packet, and the number is used to determine the background frame to which the data packet belongs and the position of the data in the data packet in the background frame.

Step 103: adding each non-background frame in the original encoded code stream to the main code stream identifier and then encapsulating to form the main code stream;

During specific implementation, after the background frames in the original coded stream are determined, the remaining coded streams are non-background frames.

Considering that the background frame is a high-quality background image modeled by the image in the video sequence, the amount of data after encoding is large (for example: the background frame in AVS is usually several times that of the I frame), and the large amount of data means that it is relatively large. High instantaneous bit rate, if the background frame and non-background frame are transmitted in the same bit stream, in the case of low bandwidth, the background frame will not be transmitted in time, which will cause the need for the background image as a reference The decoding delay of the subsequent frame after the frame is decoded, therefore, in the above step 102 and step 103, the background frame and the non-background frame are reorganized into different code streams, and the background frame is divided into at least two data packets, It avoids the direct transmission of background frames, and reduces the occurrence of delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of images in bit stream transmission.

The auxiliary code stream identification and the main code stream identification in step 102 and step 103 can be an identification bit, for example: identify the main code stream with 0, and identify the auxiliary code stream with 1;

It should be noted that, in the above, step 102 is executed first and then step 103 is executed, step 102 and step 103 may also be executed simultaneously, or step 103 may be executed first and then step 102 is executed.

Figure 2 provides a schematic diagram of extracting the original code stream data and adding the logo to repackage to form the main code stream and auxiliary code stream; from Figure 2, it can be known that the original code stream includes sequence header, image data (that is, non-background frame ), high-quality image data (that is, the background frame); after being extracted and added to the logo, it becomes the main code stream composed of image data and the auxiliary code stream composed of high-quality image data. It should be noted that the auxiliary code stream in Figure 2 The high-quality image data in is the high-quality image data obtained by dividing the high-quality image data in the original code stream.

Step 104: Send the above-mentioned auxiliary code stream and main code stream.

Preferably, this step 104 includes:

The auxiliary code stream and the main code stream are transmitted in time division, wherein the time point of completion of sending the above auxiliary code stream is no later than the time point of completion of transmission of the main code stream that requires the auxiliary code stream to be decoded.

Specifically, the sub-code streams are distributed among the main code streams before being sent, so that time-division transmission can be realized.

Since the number of background frames in the video sequence is usually small, and it is located before the non-background frames that need to be decoded in the code stream, therefore, the auxiliary code converted from the data packet obtained by dividing the background frame is sent in time division stream and the main code stream converted from non-background frames, and ensure that the sending completion time point of the auxiliary code stream is not later than the sending completion time point of the main code stream that needs to be decoded by the auxiliary code stream, so that the data constituting the background frame The packet is sent first to ensure that the background frame has been transmitted when the non-background frame needs to be decoded by the background frame, which can avoid the decoding delay of the non-background frame that needs the background frame as a reference frame to be decoded due to the untimely transmission of the background frame .

Furthermore, although the essence of the auxiliary code stream is the data packet obtained by segmenting the background frame, since the data volume of the data packet obtained by segmentation is still relatively large, sending two or more auxiliary code streams together may still cause The sudden increase of the instantaneous transmission rate of the code stream, in order to avoid the sudden increase of the instantaneous transmission rate of the code stream, the auxiliary code streams are distributed between the main code streams and then sent. Among them, the code streams adjacent to each auxiliary code stream are the main code streams Code stream, which can realize time-sharing transmission.

In addition, after the encoder encodes the video image, the coded code stream of each frame of image contains a picture sequence number (Picture Order Count, POC) used to indicate the playback order of the frame of image; The position of the image serial number in the encoded code stream is explained:

FIG. 3 is a schematic diagram of coding layers of a video sequence; wherein, a video sequence is composed of images; images are composed of slices; and slices are composed of coding blocks.

Figure 4 is a schematic diagram of the hierarchical code stream structure obtained after encoding the video sequence shown in Figure 3. It can be seen from Figure 4 that the original coded code stream of the video is composed of sequences; the sequence includes sequence headers and image data; the image data includes image Header and slice data; slice data includes slice header and macroblock data; wherein, the image header includes image sequence number.

Since the original encoded code stream of each frame of image contains the image sequence number, which is used to indicate the playback order of each frame of image, therefore, after the main code stream and auxiliary code stream are sent out, the image sequence number can usually be used However, considering that the number of image serial numbers encoded by some encoders has a small number of digits (for example, only 2 bytes), the maximum value of the image serial number is 255, and restarts from 0 after exceeding 255 number; in the embodiment of the present invention, since the auxiliary code stream is distributed to the main code stream and sent, the sending order of the image frames is not carried out in accordance with the order of the image sequence numbers. Therefore, the image sequence of the dispersed auxiliary code stream The number may be repeated with the image sequence numbers of other subsequent code streams, which may cause errors in the decoding of the receiver. In order to ensure that the receiver correctly restores the original code stream, and then decodes correctly, preferably, the above The method for sending video data also includes the following steps, the flow of which is shown in FIG. 5 .

Step 501: Determine whether the maximum value of the image sequence number in the original encoded code stream is smaller than the number of image frames contained in the original encoded code stream; if yes, execute step 502; if not, execute step 503.

This step 501 may be performed after or before any of the above steps 101-103, or may be performed simultaneously with any of the steps 101-103.

Step 502: Generate position offsets of each main code stream and each auxiliary code stream in the original encoded code stream, where the position offset represents an offset relative to the position of the sequence header in the original encoded code stream.

Specifically, the above position offset can be represented by the number of units offset from the sequence head, where each unit represents a main code stream or a secondary code stream.

This step 502 may be performed after the above step 103 and before step 104 .

After completing step 502 here, when sending the main code stream, send the position offset of the main code stream along with the main code stream; when sending the auxiliary code stream, send the position offset of the auxiliary code stream with the auxiliary code stream Stream sending.

Step 503: Determine whether the maximum value of the image sequence number in the original coded stream is equal to the number of image frames contained in the original coded stream; if yes, perform step 504; if not, prompt that the coded stream is wrong.

Step 504: After dividing each determined background frame into at least two data packets, adding sub-stream identifiers respectively before encapsulation, numbering the divided data packets, and adding the numbers to corresponding data packets, the The number is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame.

The above method for sending video data will be described in more detail below through a specific example.

Suppose the original code stream is: I1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15G16 P17 P18.

Among them, the I in I1 represents an intra-frame coding frame, and 1 represents that the image sequence number of the I frame is 1; the P in P2 represents a forward predictive coding frame, and 2 represents that the image sequence number of the P frame is 2 , G in G16 means the background frame, and 16 means that the image sequence number of the G frame is 16, and the rest are similar and will not be repeated here.

Step 1: Determine that the background frame in the original encoded code stream is G16;

Step 2: Divide the determined background frame G16 into 10 data packets, number the divided data packets, and denote the subscript with abcdefghij, add the sub-code stream identifiers and package them respectively, and the formed sub-code streams are: G16a G16b G16c G16d G16e G16f G16g G16h G16i G16j;

Step 3: Add each non-background frame to the main stream logo and then encapsulate it. The main stream is: I1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14P15 P17 P18;

Step 4: Time-sharing sending of the auxiliary code stream in the second step and the main code stream in the third step, wherein the sending completion time of the auxiliary code stream G16a G16b G16c G16d G16e G16f G16g G16h G16i G16j is not later than the required The time point at which the primary code stream P17 and P18 for which the secondary code stream is decoded is completed.

A possible transmission that satisfies the condition that the above auxiliary code stream G16a G16b G16c G16d G16e G16f G16g G16hG16i G16j is not later than the main code stream P17, P18 that needs to be decoded by the auxiliary code stream. The result is: I1 G16a P2 G16bP3 G16c P4 G16d P5 G16e P6 G16f P7 G16g G16h P8 G16iP9 G16j P10 P11 P12 P13 P14 P15 P17 P18.

Of course, other sending results can also be used, as long as the auxiliary code stream in the second step and the main code stream in the third step are sent in time division, and it is ensured that the sending of the auxiliary code stream is not later than the time when the auxiliary code is needed The time point at which the main code stream for which the stream is decoded is completed is sufficient.

As can be seen from the above specific examples, in the solution of this embodiment, the G frame data is transmitted in advance, and the data packets constituting the G frame are not transmitted at the same time (the advance time of each data packet constituting the G frame is different), that is, the After the data is broken up, it is time-shared and transmitted.

In the technical solution of Embodiment 1 of the present invention, the background frame is divided into at least two data packets, which are respectively added to the sub-code stream identifier and then encapsulated to form a sub-code stream, and the sub-code stream and the main code stream are subsequently sent, avoiding direct transmission Background frame, reducing the delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of the image in the bit stream transmission, and at the same time, it does not reduce the quality of the high-quality background image used as the background frame, and will not affect the subsequent need to refer to the high-quality The compression effect of the non-high-quality background image encoded by the background image takes into account the transmission bandwidth and image quality.

Embodiment two

Based on the same inventive concept as in Embodiment 1, Embodiment 2 of the present invention provides a method for receiving video data, the process of which is shown in FIG. 6 , including the following steps:

Step 601: Receive the video data stream, the video data stream includes the main code stream containing the main code stream identification and the auxiliary code stream containing the auxiliary code stream identification, the auxiliary code stream is to divide each determined background frame into at least Two data packets are formed by encapsulation after adding the sub-stream identifier respectively. The main stream is formed by adding each non-background frame in the original code stream to the main stream identifier and then encapsulating;

Step 602: Determine the main code stream and auxiliary code stream in the received code stream according to the identifier contained in the code stream;

Step 603: Restore the determined primary code stream and secondary code stream to the original coded code stream of the video image for decoding by the decoder.

Since the received video data code stream usually contains the image serial number, therefore, in the solution of Embodiment 2 of the present invention, the determined main code stream and auxiliary code stream can be restored to video The original encoded code stream of the image for the decoder to decode.

Figure 7 shows a schematic diagram of determining the main code stream and the auxiliary code stream by using the received code stream and restoring them to the original coded code stream;

Preferably, the receiving completion time point of the auxiliary code stream is no later than the receiving completion time point of the main code stream that requires the auxiliary code stream to be decoded. At this time, it is ensured that the background frame has been transmitted when the non-background frame needs to be decoded by the background frame, and the decoding of the non-background frame that needs the background frame as a reference frame to be decoded due to the untimely transmission of the background frame is avoided delay.

Preferably, in order to restore the original code stream correctly, each of the above-mentioned received main code streams or auxiliary code streams is accompanied by the position offset of the main code stream or auxiliary code stream in the original encoded code stream , use each received main stream or auxiliary stream along with the position offset of the main stream or auxiliary stream in the original encoded stream to restore the determined main stream and auxiliary stream to the video The original coded stream of the image is for the decoder to decode, and the position offset is relative to the sequence header position in the original coded stream.

When the above-mentioned sub-code stream includes a number, and when the above-mentioned main code stream includes an image sequence number, use the above-mentioned number to restore the determined sub-code stream to a background frame, and determine the image sequence number of the restored background frame; use the restored background frame The image serial number included in the main code stream and the background frame and the main code stream are restored to the original encoded code stream for decoding by the decoder. The above image serial number is used to indicate the playback order of each frame image. The number is used to determine the image sequence number of the background frame to which the data packet included in the sub-code stream belongs and the position of the data in the data packet in the background frame.

The method for receiving video data in Embodiment 2 of the present invention will be described below through a specific example.

Assuming that what is received is the I1 G16a P2 G16b P3 G16c P4G16d P5 G16e P6 G16f P7 G16g G16h P8 G16i P9 G16j P10P11 P12 P13 P14 P15 P17 P18 code stream sent in the first embodiment. Numbering, the above-mentioned main code stream includes the image serial number, then utilize the receiving method of video data in the second embodiment of the present invention, the specific execution steps are as follows:

Step 1: Decapsulate the data packet. When the identification contained in the data packet is the main code stream identification, determine the main code stream; when the identification contained in the data packet is the auxiliary code stream identification, determine the The code stream is the auxiliary code stream; at this time, it is determined that the main code stream is I1 P2 P3 P4 P5 P6 P7 P8P9 P10 P11 P12 P13 P14 P15 P17 P18, and the auxiliary code stream is: G16a G16b G16cG16d G16e G16f G16g G16h G16i G16j;

Second step: utilize the coding 16a 16b 16c 16d 16e 16f 16g16h 16i 16j contained in the auxiliary code stream to restore the auxiliary code stream to the background frame G, and determine that the image sequence number of the background frame is 16;

Step 3: Using the image serial numbers of each main stream and the determined image serial number of the background frame, restore the main stream and the restored background frame G to the original encoded stream, namely: I1 P2 P3 P4 P5 P6 P7P8 P9 P10 P11 P12 P13 P14 P15 G16 P17 P18.

In the solution of the second embodiment of the present invention, since the received sub-code stream is formed by dividing each determined background frame into at least two data packets, adding sub-code stream identifiers and encapsulating them respectively, therefore, to a large extent It avoids the delay or loss of subsequent non-background frames caused by the high instantaneous bit rate of the image in the bit stream transmission, so that the received bit stream is a complete bit stream to a large extent, and it is the bit stream of the image with better compression effect , therefore, the integrity of the original coded stream obtained after restoration is relatively high, which provides a good basis for subsequent decoding of high-quality video images.

Embodiment Three

Based on the same inventive concept as that of Embodiment 1, Embodiment 3 of the present invention provides a device for sending video data. Its structural diagram is shown in FIG. Stream forming unit 83 and sending unit 84; wherein:

The first determining unit 81 is configured to determine the background frame in the original encoded code stream of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame;

The sub-code stream forming unit 82 is used to divide each determined background frame into at least two data packets, add the sub-code stream identifier and then encapsulate to form a sub-code stream;

The main code stream forming unit 83 is used for encapsulating each non-background frame in the original encoded code stream after adding the main code stream identifier to form the main code stream;

The sending unit 84 is configured to send the auxiliary code stream and the main code stream.

Preferably, the sending unit 84 is further configured to time-divisionally send the above-mentioned auxiliary code stream and the main code stream, wherein, the time point at which the above-mentioned auxiliary code stream is sent is not later than that of the main code stream that requires the auxiliary code stream to be decoded. The time when the sending is completed.

Preferably, the auxiliary stream forming unit 82 is further configured to divide each determined background frame into at least two according to the data size of the background frame, the channel capacity under the current bandwidth and the requirements for data packets in the network protocol. Each data packet is added to the sub-code stream identifier and then encapsulated to form a sub-code stream.

On the basis of the above-mentioned device in FIG. 8, Embodiment 2 of the present invention also proposes a video data sending device. The structural diagram of the device is shown in FIG. 9. In addition to the units in the above-mentioned device in FIG. include:

The second determining unit 91 is configured to determine that the maximum value of the image sequence number in the original encoded code stream is less than the number of image frames contained in the original encoded code stream;

The position offset generation unit 92 is used to generate the position offset of each main code stream and each auxiliary code stream in the original code stream, and the position offset represents the position relative to the sequence head position in the original code stream Offset.

The above-mentioned sending unit 84 is further used to send the position offset of the main code stream along with the main code stream when sending the main code stream; Stream sending.

On the basis of the above-mentioned device in FIG. 8 , Embodiment 2 of the present invention also proposes a video data sending device. The structural diagram of the device is shown in FIG. 10 . include:

The third determination unit 1001 is configured to determine that the maximum value of the image sequence number in the original encoded code stream is equal to the number of image frames contained in the original encoded code stream;

The numbering unit 1002 is used for numbering the above-mentioned data packets obtained by dividing after dividing each determined background frame into at least two data packets, respectively adding sub-stream identifiers and encapsulating them, and adding the numbers to corresponding Data packet, the number is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame.

Embodiment four

Based on the same inventive concept as that of Embodiment 2, Embodiment 4 of the present invention provides a device for receiving video data. Its structural diagram is shown in FIG. in:

The receiving unit 111 is used to receive the video data code stream, the video data code stream contains the main code stream identified by the main code stream and the auxiliary code stream containing the auxiliary code stream identification, and the above-mentioned auxiliary code stream is to divide each determined background frame At least two data packets are formed by encapsulation after adding auxiliary code stream identifiers respectively, and the above-mentioned main code stream is formed by adding each non-background frame in the original code stream to the main code stream identifier and then encapsulating;

A code stream type determination unit 112, configured to determine the main code stream and the auxiliary code stream in the received code stream according to the identifier contained in the code stream;

The restore unit 113 is configured to restore the determined main code stream and auxiliary code stream to the original coded code stream of the video image for the decoder to decode.

Preferably, each code stream received above is accompanied by a position offset of the code stream in the original encoded code stream, and the position offset is an offset relative to the position of the sequence head in the original encoded code stream quantity;

The above-mentioned restoration unit 113 is further configured to restore the determined main code stream and auxiliary code stream to the original code stream of the video image by using the position offset in the original encoded code stream accompanying the code stream in each code stream received. Encoded code stream for decoder to decode.

Preferably, the above-mentioned sub-code stream includes numbers, the above-mentioned main code stream includes image sequence numbers, the above-mentioned image sequence numbers are used to indicate the playback order of each frame of images, and the above-mentioned numbers are used to determine the data packets contained in the sub-code stream belong to The image sequence number of the background frame and the position of the data in the packet in the background frame.

The above-mentioned restoration unit 113 is further used to restore the determined auxiliary code stream to a background frame by using the above-mentioned number, and determine the image sequence number of the restored background frame; use the image sequence number of the restored background frame and the image included in the main code stream Serial number, restore the background frame and main stream to the original encoded stream for decoding by the decoder.

Embodiment five

Based on the same inventive concepts as those of Embodiment 1 to Embodiment 4 of the present invention, Embodiment 5 of the present invention provides a video data transmission system. Data receiving device 121;

Wherein, the video data sending device 120 is any video data sending device in the third embodiment, and the video data receiving device 121 is any video data receiving device in the fourth embodiment.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

While preferred embodiments of the invention have been described, additional changes and modifications to these embodiments can be made by those skilled in the art once the basic inventive concept is appreciated. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims (17)

1. A method for sending video data, characterized in that the method comprises: Determine the background frame in the original coded stream of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame; Divide each determined background frame into at least two data packets, add sub-code stream identifiers respectively, and then encapsulate to form a sub-code stream; Add each non-background frame in the original encoded code stream to the main code stream mark and then encapsulate it to form the main code stream; Send the auxiliary code stream and the main code stream. 2. The sending method of video data as claimed in claim 1, is characterized in that, described sending described sub-code stream and primary code stream, comprises: The auxiliary code stream and the main code stream are sent in time division, wherein the time point of completion of sending the auxiliary code stream is no later than the time point of completion of sending of the main code stream that requires the auxiliary code stream to be decoded. 3. The sending method of video data as claimed in claim 1, is characterized in that, described each background frame that is determined is divided into at least two data packets, adds sub-code stream mark respectively and carries out encapsulation, forms sub-code stream ,include: According to the data size of the background frame, the channel capacity under the current bandwidth, and the requirements for data packets in the network protocol, each determined background frame is divided into at least two data packets, which are respectively added to the sub-code stream identifier and then encapsulated to form Secondary stream. 4. The sending method of video data as claimed in any one of claims 1-3, wherein the original coded code stream includes an image sequence number for representing the playback order of each frame image, and the method also includes : determining that the maximum value of the image sequence number in the original coded stream is less than the number of image frames contained in the original coded stream; Generate position offsets of each main code stream and each auxiliary code stream in the original code stream, where the position offset represents an offset relative to the position of the sequence header in the original code stream; The sending the main code stream and the auxiliary code stream includes: When sending the main stream, send the position offset of the main stream along with the main stream; When sending the sub-code stream, the position offset of the sub-code stream is sent along with the sub-code stream. 5. The sending method of video data as claimed in any one of claims 1-3, wherein the original coded code stream includes an image sequence number for representing the playback order of each frame image, and the method also includes : Determining that the maximum value of the image sequence numbers in the original coded stream is equal to the number of image frames contained in the original coded stream; After said dividing each determined background frame into at least two data packets, before encapsulating after respectively adding auxiliary code stream identification, said method also includes: Numbering the data packets obtained by segmentation, adding the numbers to the corresponding data packets, the numbering is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame . 6. A method for receiving video data, characterized in that the method comprises: Receive video data code stream, described video data code stream comprises main code stream that contains main code stream identification and auxiliary code stream that contains auxiliary code stream identification, and described auxiliary code stream is that each determined background frame is divided into at least two data packets, which are formed by encapsulation after adding the sub-code stream identifier respectively, and the main code stream is formed by adding each non-background frame in the original code stream to the main code stream identifier and then encapsulating; Determine the main code stream and auxiliary code stream in the received code stream according to the identifier contained in the code stream; Restore the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder. 7. The receiving method of video data as claimed in claim 6, characterized in that, in each main code stream or auxiliary code stream received, the position in the original encoded code stream of the main code stream or auxiliary code stream is accompanied Offset, the position offset is an offset relative to the position of the sequence header in the original encoded code stream; Said restoring the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder includes: Restore the determined main code stream and auxiliary code stream to a video image by using the position offset of each received main code stream or auxiliary code stream in the original encoded code stream along with the main code stream or auxiliary code stream The original encoded code stream for the decoder to decode. 8. The method for receiving video data as claimed in claim 6, wherein the auxiliary code stream includes a serial number, and the main code stream includes an image serial number, and the image serial number is used to represent each frame image The playback sequence, the number is used to determine the image sequence number of the background frame to which the data packet contained in the auxiliary stream belongs and the position of the data in the data packet in the background frame; Said restoring the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder includes: Using the serial number to restore the determined secondary stream to a background frame, and determine the image sequence number of the restored background frame; Using the image serial number of the restored background frame and the image serial number included in the main code stream, the background frame and the main code stream are restored to the original encoded code stream for decoding by the decoder. 9. A sending device for video data, characterized in that the device comprises: The first determining unit is used to determine the background frame in the original coded stream of the video image according to the identification information used to indicate that the current frame is a background frame and/or a non-background frame; A sub-code stream forming unit, configured to divide each determined background frame into at least two data packets, add sub-code stream identifiers and then encapsulate to form a sub-code stream; The main code stream forming unit is used for encapsulating each non-background frame in the original encoded code stream after adding the main code stream identifier to form the main code stream; A sending unit, configured to send the auxiliary code stream and the main code stream. 10. The sending device of video data as claimed in claim 9, is characterized in that, described sending unit is further used for time-sharing sending described sub-code stream and main code stream, wherein, the transmission of described sub-code stream is completed The time point is no later than the time point of completion of sending the main code stream that requires the auxiliary code stream to be decoded. 11. The sending device of video data as claimed in claim 9, it is characterized in that, described sub-code stream formation unit is further used for according to the data volume size of background frame, the channel capacity under the current bandwidth and network protocol data According to the requirements of the package, each determined background frame is divided into at least two data packets, which are respectively added with sub-code stream identifiers and then encapsulated to form sub-code streams. 12. The sending device of video data according to any one of claims 9-11, characterized in that said device further comprises: The second determination unit is configured to determine that the maximum value of the image sequence number in the original encoded code stream is less than the number of image frames contained in the original encoded code stream; The position offset generation unit is used to generate the position offset of each main code stream and each auxiliary code stream in the original code stream, and the position offset represents the offset relative to the position of the sequence header in the original code stream. displacement; The sending unit is further configured to send the position offset of the main code stream along with the main code stream when sending the main code stream; when sending the auxiliary code stream, send the position offset of the auxiliary code stream along with the auxiliary code stream Stream sending. 13. The device for sending video data according to any one of claims 9-11, wherein the device further comprises: A third determining unit, configured to determine that the maximum value of the image sequence number in the original encoded code stream is equal to the number of image frames contained in the original encoded code stream; The numbering unit is used for numbering the divided data packets after dividing each determined background frame into at least two data packets, respectively adding the sub-stream identifier and encapsulating, and adding the numbers to the corresponding The number is used to determine the image sequence number of the background frame to which the data packet belongs and the position of the data in the data packet in the background frame. 14. A receiving device for video data, characterized in that the device comprises: The receiving unit is used to receive the video data code stream, the video data code stream includes the main code stream containing the main code stream identification and the auxiliary code stream containing the auxiliary code stream identification, and the auxiliary code stream is each background to be determined The frame is divided into at least two data packets, which are formed by encapsulation after adding the auxiliary code stream identifier respectively, and the main code stream is formed by adding each non-background frame in the original code stream to the main code stream identifier and then encapsulating; A code stream type determining unit, configured to determine the main code stream and the auxiliary code stream in the received code stream according to the identifier contained in the code stream; The restoration unit is used to restore the determined main code stream and auxiliary code stream to the original coded code stream of the video image for decoding by the decoder. 15. The receiving device of video data as claimed in claim 14, characterized in that, each received main code stream or auxiliary code stream is accompanied by the position of the main code stream or auxiliary code stream in the original encoded code stream Offset, the position offset is relative to the offset of the sequence header position in the original encoded code stream; The restoration unit is further used to restore the determined main code stream and auxiliary code stream to the original code stream of the video image by using the position offset of each received code stream accompanied by the code stream in the original encoded code stream Encoded code stream for decoder to decode. 16. The device for receiving video data according to claim 14, wherein the sub-code stream includes a serial number, and the main code stream includes an image sequence number, and the image sequence number is used to represent each frame image The playback sequence, the number is used to determine the image sequence number of the background frame to which the data packet contained in the auxiliary stream belongs and the position of the data in the data packet in the background frame; The restoration unit is further used to restore the determined secondary code stream to a background frame by using the serial number, and determine the image sequence number of the restored background frame; Using the image serial number of the restored background frame and the image serial number included in the main code stream, the background frame and the main code stream are restored to the original encoded code stream for decoding by the decoder. 17. A video data transmission system, characterized in that the system comprises: the video data sending device according to any one of claims 9-13 and the video data receiving device according to any one of claims 14-16 .