CN110224793A - A kind of adaptive FEC method based on media content - Google Patents

Abstract

The present invention provides an adaptive FEC method based on media content. The method classifies media content and assigns different importances, and dynamically adjusts the FEC according to the current channel status without shunting the original media data stream. The importance of the frame contained in the media data packet and the corresponding encoding scheme, the data packet is transmitted to the corresponding FEC encoder, and different degrees of protection are carried out. Finally, a source data stream will only be encoded into a FEC code stream to achieve While ensuring the quality of media content to the greatest extent, it reduces the huge amount of data caused by FEC; it does not need to split the source data stream, which reduces the complexity of FEC coding at the sending end and improves the efficiency of FEC coding; greatly reduces the The increase in the amount of data brought about by FEC coding; the coding scheme can be dynamically adjusted according to changes in the current network state, so it has stronger adaptability to time-varying networks.

Description

An Adaptive FEC Method Based on Media Content

This application is a divisional application of the parent invention application 201510673091.X with the title of invention "An Adaptive FEC Mechanism Based on Media Content" and the filing date of October 16, 2015.

technical field

The present invention relates to the technical field of multimedia transmission, and more specifically, to an adaptive FEC (forward error correction) method based on media content.

Background technique

In the heterogeneous network media service system, the content is distributed to the terminal through the Internet protocol or broadcast protocol, the Internet uses IP/TCP or UDP packets to transmit media data, and the broadcast transmits content through MPEG2-TS. UDP packets may be lost after passing through multiple network devices, and broadcast TS streams may cause bit errors due to the influence of the transmission environment, resulting in broken screens or sound pauses on the terminal side.

FEC (Forward Error Correction, Forward Error Correction) technology is a coding technology widely used in communication systems. The server side performs error correction encoding on the media data, adds redundant information and sends it together, and the terminal side performs reverse FEC decoding to restore the lost message. Taking a typical block code as an example, the basic principle is: at the sending end, by encoding the kbit information as a block, adding (n-k) bit redundancy check information to form a codeword with a length of n bits. After the code word reaches the receiving end through the channel, if the error is within the correctable range, the error bit can be checked and corrected by decoding, thereby resisting the interference caused by the channel, effectively reducing the bit error rate of the system, and improving the reliability of the communication system sex.

However, FEC processing is to reduce the bit error rate of the system at the cost of redundant overhead, and excessive FEC coding will also cause pressure on the system's real-time performance and network status.

In the ISO-23008-1/10/13 standard, the traditional FEC structure uniformly encodes all information, and has no distinction for user information. It is not suitable for mixed content or distribution under mixed networks, so they proposed a two-tier structure. Two-layer structure, as shown in Figure 1.

The first layer divides the source packet block into more small blocks for FEC protection, and the second layer is a whole block for FEC protection. The finer division of the first layer can provide a small delay, and the second layer ensures recovery performance and small redundancy.

For mixed content transmission, its content may be divided into timed and non-timed, so this two-layer structure can be adopted. Timing content uses method 1 to ensure the delay, and non-chronological content uses both methods 1 and 2 to ensure accuracy.

For users in different performance channels, users with good channel performance only need FEC1 to ensure delay and power consumption. For users with poor channel performance, FEC1 and FEC2 are performed at the same time to ensure accuracy.

This solves the problem to a certain extent. For users with poor channel performance (GroupB), the two-layer structure does improve recovery performance, but it will introduce a huge delay. For users (GroupA) with better channel performance, FEC1 is not necessarily used, and the smaller the packet, the smaller the delay. The unequal importance of the information is not taken into account. It is a complicated issue to classify information or users according to the two-layer structure, and to split a large block into small blocks.

Contents of the invention

In view of the defects in the prior art, the purpose of the present invention is to provide an adaptive FEC method based on media content, so as to solve the problem of good real-time performance but poor accuracy, good accuracy but poor real-time performance, and excessive FEC coding in the current FEC system. caused by data congestion.

To achieve the above object, the present invention adopts the following technical solutions:

An adaptive FEC method based on media content, which classifies media content and assigns different importances, and dynamically adjusts the content of media data packets according to the current channel conditions without shunting the original media data streams. The importance of the included frames and the corresponding encoding scheme, the data packets are transmitted to the corresponding FEC encoder, and different degrees of protection are performed. In the end, one source data stream will only be encoded into one FEC code stream.

Further, the method adds an unequal error protection flag bit to the signaling information transmitted together with the media processing unit (MPU).

Further, the method adds a special indication field, so that more personalized protection schemes can be implemented. Furthermore, the special indication field is added before the header of the MFU packet.

Further, the method classifies the media content and assigns different importances, and then uses signaling and indication field control to encode with different FEC coding strengths.

Furthermore, the method can dynamically change the coding scheme according to the current network conditions, and provide adaptive protection strength.

Further, the method adds an unequal error protection flag bit and a special indication field to the signaling information transmitted together with the media processing unit (MPU), so as to form more personalized protection schemes.

Preferably, the method classifies the media content and assigns different importances, and then utilizes signaling and indication field control to encode with different FEC coding strengths.

Further, the method is aimed at the MMT AL-FEC sender architecture, and the sender process is:

a) The server side generates MMTP flow and signaling according to media resources;

b) according to the importance of each frame of data in the MMT stream, the MMT packet is delivered to the FEC encoder, and for different priorities, different FEC encoding matrices are used to perform FEC encoding on the MMT packet to generate corresponding FEC codes;

c) Integrating the FEC codes formed after FEC encoding the MMT packets of different priorities of the same data stream in the previous step to generate a FEC code stream;

d) After the FEC encoding is completed, return the corresponding repair character and the FEC data payload identifier and the original data payload identifier;

e) All repaired characters are packaged into a FEC repair package and sent to the transport layer.

Furthermore, the present invention identifies the FEC flow and the FEC coding structure and FEC code adopted by the signaling. In order to support this method, the fec_flow_descriptor field in the signaling is modified: the basis of the three existing fec_coding_structures in the AL-FEC message A new fec_coding_structure is added to the above. The function of fec_coding_structure is to describe the currently used FEC coding scheme, including the selected coding algorithm, whether to use a private coding scheme, the maximum protection time window time and value, and other information. This field is located in AL -It is transmitted to the receiving end in FEC signaling; the value of the flag bit of the newly added fec_coding_structure is selected within the existing reserved range;

The original signaling only supports the repair information of multiple media resources multiplexed in one FEC stream. Now, a kind of FEC signaling control for different important parts of a media resource is added, so that the FEC method can be more refined. , the server can dynamically adjust the FEC strength of different parts of media resources according to the user's network status, and achieve a balance between network bandwidth and user experience.

Compared with the prior art, the present invention has the following beneficial effects:

Adopting the technical solution of the present invention, aiming at the data congestion caused by excessive coding in the current FEC system, by classifying the media content, giving different importance, using signaling and indicator bit control, and using different FEC coding strengths; to achieve maximum While ensuring the quality of media content to the maximum, it reduces the huge amount of data caused by FEC; since it is not necessary to split the source data stream, it reduces the complexity of FEC coding at the sending end and improves the efficiency of FEC coding; The data stream will only be encoded to generate an FEC code stream, so the increase in the amount of data caused by FEC encoding can be greatly reduced; at the same time, the encoding scheme can be dynamically adjusted according to the change of the current network state, so it is more effective for time-varying networks. Strong adaptability.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

Figure 1 is a two-layer structure of FEC for media resources in MMT;

Fig. 2a-Fig. 2b are each frame dependency graph in an image group;

Fig. 3 is a schematic diagram of the importance of a general MPU component and each part;

Fig. 4 is a structural diagram of an improved MMT AL-FEC sending end.

Detailed ways

The present invention will be described in detail below in conjunction with specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

As shown in Figure 1: the FEC two-layer structure for media resources in MMT, the first layer divides the source packet block into more small blocks for FEC protection, and the second layer is a whole block for FEC protection. The finer division of the first layer can provide a smaller delay, and the second layer ensures recovery performance and smaller redundancy, but this flexibility is not enough.

As shown in Figure 2a-Figure 2b: the dependence of each frame in an image group, this figure shows that the dependence and importance of different frames in an image group are different, the I frame is the most important, the front P frame is more important than the back P frame Important, the B frame is the least important, so hierarchical FEC can be performed according to the importance. This is also the basis for FEC.

Embodiment one:

This embodiment takes the MMT transmission protocol as an example

In the MMT scheme, the MFU packages under the MPU package have different importances and lack the protection of unequal errors, so it is impossible to set a personalized transmission scheme. In MMT packets, frames of different importance (such as I frame and B frame) can be distinguished (with indication marks) as shown in Fig. 2a-Fig. 2b, Fig. 3 . However, the protection of unequal errors is not involved in the coding. Although the Two-stage structure scheme and LA-FEC can realize unequal error protection to a certain extent, they have low flexibility and high complexity. Especially for changing channel conditions and receiving users with different characteristics (such as different buffers), the existing two methods have large redundancy, poor adaptability, can only address a single situation, and cannot be considered from the perspective of user experience.

Solution:

1 According to the packets belonging to frames with different importance levels, the importance levels are differentiated, and the coding scheme is adaptively changed in combination with channel conditions and user experience. For example, when the channel condition is very poor or the user's storage capacity is limited, the sending end will protect the I frame data more, so that the user will receive the I frame with a greater probability, and the B frame and P frame will be received to a greater extent. Abandoning, so that it is no longer discarding B and P frames in the subsequent processing of the receiving end, but solving this problem from the sending end, saving bandwidth resources, and using resources to protect more important frames. In a time-varying channel, the FEC coding scheme can be adaptively changed according to changes in the current network conditions. For example, if the network condition deteriorates, the FEC coding matrix can be changed by changing the FEC seed to increase the protection strength of the transmitted data. .

2 If no special indication field is added, it can only be coded according to the existing situation, and the unequal error protection can only be arranged according to the importance of the frame to which it belongs. The unequal error protection flag bit is added to the signaling transmitted with the MPU.

3 But if you want a more personalized scheme, change the protection scheme with the channel quality, and customize it according to user characteristics, then you need to add a special indication field. Since different MFUs have different importance, more personalized protection schemes can be implemented by adding instructions before the MFU header.

As shown in Figure 4, it is an improved MMT AL-FEC sender architecture, which mainly modifies the streamed output in the MMT protocol, and sends the MMT data packets to the FEC encoder according to the priority for FEC encoding. According to the priority of the MMT packet For different levels, select different encoding matrices, and integrate the FEC codes generated by each priority level after being processed by the FEC encoder into one FEC code stream.

The sender architecture is shown in Figure 4.

Sending process:

a) The server side generates MMTP flow and signaling according to media resources.

b) According to the importance of each frame of data in the MMT stream, the MMT packet is passed to the FEC encoder, and for different priorities, different FEC coding matrices are used to perform FEC encoding on the MMT packet to generate corresponding FEC codes.

c) Integrating the FEC codes formed after performing FEC coding on the MMT packets of different priorities of the same data stream in the previous step to generate an FEC code stream.

d) After the FEC encoding is completed, the corresponding repair character, the FEC data load identifier and the original data payload identifier are returned.

e) All repaired characters are packaged into a FEC repair package and sent to the transport layer.

The FEC flow and the adopted FEC coding structure and FEC code are identified through signaling. In order to support this method, the fec_flow_descriptor field in the signaling is modified in this embodiment.

Table 1: Description of fec_coding_structure

The original signaling only supports the repair information of multiple media resources multiplexed in one FEC stream. This embodiment adds a new fec_coding_structure. The function of fec_coding_structure is to describe the currently used FEC coding scheme, including the selected coding algorithm, whether to use Information such as the private coding scheme, the maximum protection time window time and value, etc., this field is sent to the receiving end in the AL-FEC signaling. The value of the flag bit of the newly added fec_coding_structure can be selected within the existing reserved range, and it is recommended to select 0110 in this solution.

The field semantics in the table are as follows:

seed: Random seed, which is used as the initial condition for generating pseudo-random numbers, and is used to generate pseudo-random sequences to construct the generation matrix and check matrix of FEC encoding. There are many algorithms for seed selection, such as obtaining the current time of the system, obtaining the current process ID and other methods are commonly used.

num_of_priority_for_mmtps: The number of priorities in a media resource.

priority_mapping: Priority mapping, used to indicate the mapping relationship between different data packets in media resources and resource priorities.

private_fec_flag: Indicates whether to use a private FEC coding scheme.

private_flag: Indicates whether there is a private_field used to describe the private FEC coding scheme used.

private_field_length: length field, used to describe the length of the field of the private FEC coding scheme.

private_field: used to describe the details of the private FEC scheme.

priority_id: priority id, used to indicate the priority of the MMT packet.

fec_code_id_for_repair_flow: Used to describe the FEC encoding scheme used.

repair_flow_id: 8-bit integer, used to indicate the generated FEC repair flow, corresponding to the packet id in the header of the FEC repair packet.

maximum_k_for_repair_flow: 24-bit integer, describing the maximum number of sourcesymbols in a source sysmbol block.

maximum_p_for_repair_flow: 24-bit integer, describing the maximum number of repairsymbols in a repair sysmbol block.

protection_window_time: Protection window time, indicating the maximum time difference between sending the first source or repair packet and sending the last source or repair packet in FEC encoding, in milliseconds.

protection_window_size: Protection window value, indicating the maximum count value between the payload of the first FEC packet sent and the payload of the last FEC packet sent in the FEC encoded stream.

The modified signaling uses different encoding matrices to control the FEC encoding of different priorities for different important parts of a media resource. Only one FEC code stream is finally generated for a media data stream, so that the FEC method can be more efficient. Refinement, and at the same time reduce the increase in network traffic caused by excessive FEC coding. Using this solution, the server can dynamically adjust the FEC strength of different parts of the media resource according to the user's network status. After receiving the signaling, the receiving end can parse out the seed And the priority of each resource package, generate a parity check matrix according to the seed, perform FEC decoding and restore media resources. Strike a balance between network bandwidth and user experience.

Advantages of the above solution:

1 Resource saving. If at the receiving end, artificially abandoning the received frames (such as B frames) according to the situation will cause a waste of transmission resources, the above solution solves the problem from the source end, so that unwanted packets are replaced during transmission. There is a high probability of loss, and the important package is protected to a greater extent.

2 Personalized transmission schemes. Video transmission should be based on user experience. By doing more detailed unequal error protection, the degree of FEC protection for I-frames and B-frames can be designed according to user conditions, such as visual experience and cache conditions.

3. It can adaptively change the FEC encoding method according to the current network conditions. On the basis of different strengths of FEC protection for data with different priorities, if the network condition deteriorates and the packet loss rate increases, it can be adaptively adjusted. The seed of FEC changes the generation matrix of FEC and increases the strength of FEC to resist the influence of network deterioration.

The existing AL-FEC scheme in 4MMT only supports the generation of different FEC streams for MMT packets with different priorities. The scheme proposed in this scheme supports only one FEC stream for one media resource data stream, which can greatly reduce The increased traffic brought by FEC reduces network pressure.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims (10)

1. An adaptive FEC method based on media content, characterized in that the media content is classified and given different importance, and dynamically adjusted according to the current channel condition without shunting the original media data flow The importance of the frame contained in the media packet and the corresponding encoding scheme, the method is aimed at the MMT AL-FEC sender architecture, and the sender process is: The server side generates MMT flow and signaling according to the media content; According to the importance of each frame of data in the MMT stream, the MMT packet is passed to the FEC encoder. The FEC encoding matrix is generated according to the different priorities of the MMT packet, and the FEC encoding is performed on the MMT packet to generate the corresponding FEC code; Integrate the FEC codes formed after performing FEC encoding on the MMT packets of different priorities of the same data stream in the previous step to generate a FEC code stream; After FEC encoding, return the corresponding repair character, FEC data load identification and original data load identification; All repair characters are packaged into FEC repair packets and sent to the transport layer. 2. The adaptive FEC method based on media content according to claim 1, wherein the signaling generated by the server is at least used to indicate the repair character. 3. the adaptive FEC method based on media content according to claim 1, it is characterized in that, described MMT AL-FEC sender framework, has revised the output after the streaming in the MMT agreement, MMT packet is sent according to priority Go to the FEC encoder to perform FEC encoding respectively, select different encoding matrices according to the different priorities of the MMT packets, and integrate the FEC codes generated after each priority is processed by the FEC encoder into an FEC code stream. 4. The adaptive FEC method based on media content according to claim 1, characterized in that the method adds an unequal error protection flag bit in the signaling information transmitted together with the media processing unit (MPU). 5. The adaptive FEC method based on media content according to claim 1, characterized in that, the method adds a special indication field to form more personalized protection schemes. 6. The adaptive FEC method based on media content according to claim 3, wherein the special indication field is added before the MFU header. 7. the adaptive FEC method based on media content according to claim 1 is characterized in that, described method adds unequal error protection flag bit in the signaling information that transmits together with media processing unit (MPU), adds simultaneously Special instruction fields to form more personalized protection schemes. 8. The adaptive FEC method based on media content according to claim 1, characterized in that the method can adaptively change the coding scheme according to the change of the network condition. 9. The adaptive FEC method based on media content according to claim 1, characterized in that, the method identifies the FEC flow and the FEC coding structure and FEC code adopted by signaling, and in order to support this mechanism, the signaling The fec_flow_descriptor field in the order: A new fec_coding_structure is added on the basis of the existing three fec_coding_structures in the AL-FEC message. The function of fec_coding_structure is to describe the currently used FEC coding scheme, including the selected coding algorithm, whether Using a private coding scheme, information such as the maximum protection time window time and value, this field is transmitted to the receiving end in the AL-FEC signaling; the value of the newly added flag bit of fec_coding_structure is selected within the existing reserved range; The original signaling only supports the repair information of multiple media resources multiplexed in one FEC stream. Now, a kind of FEC signaling control for different important parts of a media resource is added, so that the FEC mechanism can be more refined. , the server can dynamically adjust the FEC strength of different parts of media resources according to the user's network status, and achieve a balance between network bandwidth and user experience. 10. The adaptive FEC method based on media content according to claim 9, characterized in that the value of the flag bit of the newly added fec_coding_structure is selected as 0100.