CN118433131A

CN118433131A - Deadline-sensitive streaming method, device and equipment

Info

Publication number: CN118433131A
Application number: CN202410472729.2A
Authority: CN
Inventors: 陈恩加; 朱仕银; 孙强
Original assignee: New H3C Technologies Co Ltd
Current assignee: New H3C Technologies Co Ltd
Priority date: 2024-04-19
Filing date: 2024-04-19
Publication date: 2024-08-02

Abstract

The invention provides a method, a device and equipment for transmitting a cut-off time sensitive stream, which are used for solving the technical problem of inaccurate calculation of the real priority of a data block in a DTP. The invention obtains the real packet loss situation by differentiating the packet loss of the congestion type and the packet loss of the non-congestion type in the link congestion control, thereby avoiding the influence of the inherent packet loss/random packet loss of the link on the deadline sensitive streaming. When the real priority of the data block is calculated, influence factors of link delay fluctuation on the priority are reduced on the basis of the original DTP protocol, so that the calculation of the real priority is more accurate. Under the condition that the dependency relationship exists among the data blocks, the influence factors of the number of the dependent data blocks on the priority are introduced, and the influence degree of the dependency relationship on the priority of the current data block is adjusted through adjusting the coefficient, so that the calculation of the priority is more accurate and has application scene adaptability.

Description

Deadline-sensitive streaming method, device and equipment

Technical Field

The present invention relates to the field of data communications technologies, and in particular, to a method, an apparatus, and a device for deadline sensitive streaming.

Background

Streaming is a data transmission method used by people when watching video on the internet. This is a way to deliver a few video files at a time, typically from a remote storage location. By transmitting a file over the internet for a few seconds at a time, the client device does not have to download the entire video before starting playing. Real-time streaming refers to streaming video being sent in real-time over the internet without first recording and storing. Today, television broadcasts, video game streams, and social media videos may all be transmitted in real-time. The distinction between regular streaming and real-time streaming can be compared to the distinction between actor recitation monologues and impulse lectures. In the former, the content is pre-created, stored and forwarded to the viewer. In the latter, the viewer receives the content at the same time that the actor creates the content, as in real-time streaming.

With the popularity of a large number of emerging applications such as virtual reality VR, augmented reality AR, autopilot, ultra low latency gaming, and large-scale internet of things systems, the emerging applications exhibit several unique differences from the perspective of network traffic workload and quality of service (QoE) requirements, the most obvious difference being that the applications have certain requirements on priority and expiration date, as compared to traditional network-intensive applications such as multimedia streaming. For example, when the VR server estimates a more accurate head pose location message, or when the autopilot car updates the object's movement track, distance or speed message, frames in the video stream (frames), and messages in the online game, these messages are all transmitted by the application in the form of data blocks (blocks). For block transfer, the order in which the data arrives within a block is not critical, as long as all the data of a block arrives at the receiver before the expiration date of the block. Blocks missing an expiration date will greatly impact the user experience and when new data arrives, old blocks become useless even if they arrive later due to timeliness.

The Deadline-aware Transport Protocol (DTP) is a rapid UDP network connection based quit transmission protocol development and research protocol for real-time video streaming scenarios, which is an extension of quit protocols. Its usage scenario is to meet the need of an application to complete a block transfer before the deadline. Taking live viewing as an example, the deadlines and blocks refer to: the time delay corresponding to the actual need of completing the video picture transmission in a short time (such as 1 second) is the cut-off time, and if the time is exceeded, even if the picture is completed to be transmitted, the effect on live broadcasting is greatly influenced; while the block corresponds to a video frame, control signaling, which is a separate, controllable and individually usable data component.

When data blocks and corresponding metadata (deadlines, frame priorities) are transferred from the application layer to the transport layer, they are placed in a buffer, and the scheduler selects the data blocks to be transmitted according to a preset scheduling algorithm. The selected data block is then cut into quench formatted messages for transmission. The congestion control module is responsible for sending QUIC messages, collecting ACK and carrying out packet loss detection. If packet loss occurs, the lost QUIC message is put back into a retransmission queue RTX, and the detected network states such as bandwidth and RTT are notified to a scheduler so that the scheduler can make decisions in the next packet sending process.

The most critical part of the DTP protocol is the decision process by the scheduler as to which data block is to be transmitted: the DTP chooses to make decisions from three angles, namely the original priority of the data block, the block deadline, and the time required for the transmission of the remaining blocks to complete. Because if only the original priority of the data blocks is considered for decision, the scheduler selects the block with high priority at a certain moment, but the data block with lower priority is already very close to the deadline, sending the block with high priority will cause the block with low priority to timeout, and if the block with low priority is sent first and then the block with high priority is sent, it may be guaranteed that the former will not timeout. Further, when a block of low priority is about to be transmitted, another block of high priority is suddenly selected for transmission, the former may be overdue and have to cause retransmission, resulting in the bandwidth previously used for transmitting the block of low priority being wasted.

Based on the above analysis, the scheduler of DTP uses the following strategy: assuming that the original priority of a certain block is priority (the higher the priority is, the smaller the value of priority is), the deadline is deadline, the size of the remaining block is remaining_size, the link bandwidth is bandwidth, the link delay is RTT (Round Trip Time), and the DTP calculates the real priority of the data block by the following formula:

Real Priority represents the final calculated Real Priority of the data block. In the formula, priority indicates the original priority of the data block, RTT/2 indicates the inherent unidirectional delay of the current link, (remaining_size)/bandwidth indicates the time required for the current link bandwidth to complete transmission of the remaining data block, deadline indicates the deadline for the data block to complete transmission within a given deadline. The difference (absolute value) of deadline minus the sum of RTT/2 and (remaining_size)/bandwidth represents the difference between the time required to complete the transmission of the remaining data block and the given deadline. In the case that the self-contained priorities of two data blocks are the same, if the gap corresponding to a certain data block is smaller, the calculated true priority value is smaller, so that the true priority is higher, which means that the data block needs to be sent as soon as possible; if the difference calculated for two blocks is the same, the true priority is determined by the priority of the original block. Therefore, the real priority of the data block can be more comprehensively considered by multiplying the original priority of the data block by the difference between the time required for completing the transmission of the rest data block and the given deadline.

Although the DTP uses the QUIC as a transport layer protocol and considers the real priority of the data block from multiple aspects, the congestion control algorithm used does not consider factors such as the influence of the packet loss type on the transmission rate, so that the calculated transmission priority of the data block has a certain distortion, thereby affecting the transmission quality.

Disclosure of Invention

In view of the above, the present invention provides a method, apparatus and device for deadline-sensitive streaming to solve the technical problem of inaccurate calculation of real priority of data blocks in DTP.

Based on an aspect of an embodiment of the present invention, the present invention provides a deadline-sensitive streaming method, which is applied to a streaming device adopting a deadline-sensitive protocol DTP, and the method includes:

When path congestion control is carried out, distinguishing between congestion packet loss and non-congestion packet loss, calculating the current path sending rate (SENDING RATE) based on the real congestion packet loss rate and carrying out corresponding congestion control; the congestion packet loss refers to packet loss caused by the fullness of a transmission layer buffer area;

And calculating the Real Priority (Real Priority) of the data block to be transmitted according to the current path transmission rate (SENDING RATE) and the original Priority (Priority) of the data block to be transmitted, which are determined by the path congestion control module, and deciding whether to transmit the data block to be transmitted or not based on the Real Priority.

Further, the method for distinguishing the congestion packet loss from the non-congestion packet loss comprises the following steps:

Under the condition that the link round trip delay is measured three times in succession, if packet loss occurs in the process of measuring the link round trip delay for the third time, judging that the occurred packet loss is congestion packet loss when the following two conditions are satisfied:

The round trip delay measured for the first time is within the preset fluctuation range of the round trip delay measured for the first time, and the round trip delay measured for the second time is larger than the minimum round trip delay of the known link; and

The round trip delay measured in the second time and the third time is in the preset fluctuation range of the round trip delay measured in the second time, and the round trip delay measured in the third time is larger than the minimum round trip delay of the known link;

the preset fluctuation range of the round trip delay is determined by adding and subtracting preset fluctuation coefficients on the basis of the round trip delay measured in the previous time.

Further, the method for calculating the real priority of the data block to be sent comprises the following steps:

Subtracting the ratio of the residual block size (REMAINING SIZE) to the current transmission rate (SENDING RATE) from the cut-off time (deadline) after the data block is transmitted, subtracting the one-way time delay (RTT/2), and subtracting the time delay fluctuation influence value Obtaining a remaining transmission time (REMAINING TIME) of the data block to be transmitted;

the product of the original Priority (Priority) and the remaining transmission time (REMAINING TIME) is taken as the real Priority.

After the product of the original priority and the residual transmission time is calculated, the product is multiplied by a priority adjustment coefficient alpha, then the product of the block dependence weight adjustment coefficient beta and the reciprocal of the number of dependent blocks (DEPENDENT BLOCKS) is added, and the obtained result is used as the final real priority.

Based on another aspect of the embodiment of the present invention, the present invention further provides a deadline-sensitive streaming method, where the method is applied to a streaming device adopting a deadline-sensitive protocol DTP, and the method includes:

Subtracting the ratio of the size (REMAINING SIZE) of the residual block to the current transmission rate (SENDING RATE) from the cut-off time (deadline) after the data block is transmitted, subtracting the one-way time delay (RTT/2), and subtracting the time delay fluctuation influence value to obtain the residual transmission time of the data block to be transmitted;

Taking the product of the original priority and the residual transmission time as the real priority, and deciding whether to transmit the data block to be transmitted or not based on the real priority.

multiplying the product of the original priority and the residual transmission time by a priority adjustment coefficient alpha, adding the product of a block dependency weight adjustment coefficient beta and the reciprocal of the number of dependent blocks (DEPENDENT BLOCKS), and taking the obtained result as the real priority;

and deciding whether to send the data block to be sent or not based on the real priority.

Further, the method for calculating the residual transmission time comprises the following steps:

Subtracting the ratio of the residual block size (REMAINING SIZE) to the current transmission rate (SENDING RATE) from the cut-off time (deadline) after the data block is transmitted, subtracting the one-way time delay (RTT/2), and subtracting the time delay fluctuation influence value The remaining transmission time of the data block to be transmitted is obtained.

Based on another aspect of the embodiment of the present invention, there is also provided a deadline-sensitive streaming apparatus, which is applied to a streaming device adopting a deadline-sensitive protocol DTP, including:

The congestion control module is used for distinguishing congestion packet loss and non-congestion packet loss when path congestion control is carried out, calculating the current path sending rate (SENDING RATE) based on the real congestion packet loss rate and carrying out corresponding congestion control; the congestion packet loss refers to packet loss caused by the fullness of a transmission layer buffer area;

A decision module for subtracting the ratio of the residual block size (REMAINING SIZE) to the current transmission rate (SENDING RATE) from the cut-off time (deadline) after the data block is transmitted, subtracting the one-way time delay (RTT/2), and subtracting the time delay fluctuation influence value Obtaining a remaining transmission time (REMAINING TIME) of the data block to be transmitted; multiplying the product of the original Priority (Priority) and the residual transmission time (REMAINING TIME) by a Priority adjustment coefficient alpha, adding the product of the block dependence weight adjustment coefficient beta and the reciprocal of the number of dependent blocks (DEPENDENT BLOCKS) to obtain the final real Priority, and deciding whether to transmit the data block to be transmitted based on the real Priority.

The invention obtains the real packet loss situation by differentiating the packet loss of the congestion type and the packet loss of the non-congestion type in the link congestion control, thereby avoiding the influence of the inherent packet loss/random packet loss of the link on the deadline sensitive streaming. When the real priority of the data block is calculated, influence factors of link delay fluctuation on the priority are reduced on the basis of the original DTP protocol, so that the calculation of the real priority is more accurate. Under the condition that the dependency relationship exists among the data blocks, the influence factors of the number of the dependent data blocks on the priority are introduced, and the influence degree of the dependency relationship on the priority of the current data block is adjusted through adjusting the coefficient, so that the calculation of the priority is more accurate and has application scene adaptability.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will briefly describe the drawings required to be used in the embodiments of the present invention or the description in the prior art, and it is obvious that the drawings in the following description are only some embodiments described in the present invention, and other drawings may be obtained according to these drawings of the embodiments of the present invention for a person having ordinary skill in the art.

FIG. 1 is a flowchart illustrating a method for deadline-sensitive streaming according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of the dependency relationship between groups of images and various frames in the H.265 protocol;

Fig. 3 is a schematic structural diagram of an electronic device for implementing the deadline-sensitive streaming method according to an embodiment of the present invention.

Detailed Description

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used in this embodiment of the invention, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Although embodiments of the present invention may be described in terms of first, second, and third aspects, such descriptions are merely provided to distinguish one from another information, entity, or step, and not to describe a particular sequence or order. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of embodiments of the present invention. For another example, in some scenarios, the first information may refer to one information, or may refer to multiple information in the same class. Furthermore, the word "if" as used may be interpreted as "at … …" or "at … …" or "in response to a determination. The "and/or" in the present invention is merely an association relationship describing the association object, and indicates that three relationships may exist, for example, a and/or B may indicate: there are three cases, a alone, a and B together, and B alone, wherein a, B may be singular or plural. Also, in the description of the present invention, unless otherwise indicated, "a plurality" means two or more than two. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b, or c may represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c may be single or plural.

The inventor finds through research and analysis that although the existing DTP protocol DTP uses QUIC as a transmission layer protocol and considers the real priority of the data block from multiple aspects, the congestion control algorithm used has the following technical defects:

DTP defaults to using bottleneck bandwidth and round trip delay propagation time (Bottleneck Bandwidth and Round-trip propagation time, BBR) congestion control algorithms that do not distinguish between inherent/random packet losses (the reasons for such losses often come from below the transport layer, possibly routing problems, possibly problems with the channel quality itself) and real packet losses (due to unreasonable packet emissions at the transport layer itself), which can affect the sending rate of the data blocks in DTP and thus the real-time transmission effect (user viewing experience).

DTP calculates priority using half RTT (RTT/2) as unidirectional transmission delay, and does not consider the influence of possible delay fluctuation in the network on the block transmission completion time (for example, the link quality used when the mobile terminal is accessed to play online video is often unstable), which causes distortion of the calculated real priority to a certain extent.

DTP does not take into account the coupling between application layer data blocks, for example, in a real-time video stream encoded in the h.265 protocol, there is a codec dependency between different video frames (I-frame/P-frame/B-frame), which also affects the true priority of the data blocks.

To solve the above technical problems, the present invention proposes a deadline-sensitive video streaming scheme, and the following detailed description of the specific implementation of the present invention will be given with reference to the accompanying drawings and the specific embodiments, it should be noted that the steps shown in the flowcharts of the drawings and the embodiments may be executed in a computer system such as a set of computer executable instructions, and although a logic order is shown in the flowcharts, in some cases, the steps shown or described may be executed in an order different from that shown or described herein.

Fig. 1 is a flowchart illustrating a step of a deadline-sensitive streaming method according to an embodiment of the present invention. The cut-off time sensitive type stream transmission method provided by the invention is still realized based on the overall architecture of the DTP protocol, and the cut-off time sensitive type stream transmission process is optimized in a multi-level manner from the angles of detecting real packet loss, evaluating delay fluctuation and evaluating inter-frame coding and decoding dependency. The following embodiments of the present invention are described by way of example with respect to time-sensitive video streams, but based on the same basic idea, the technical solution of the present invention is equally applicable to any deadline-sensitive data stream with similar stream characteristics.

Step 101, when path congestion control is carried out, distinguishing congestion packet loss and non-congestion packet loss, calculating the current path sending rate (SENDING RATE) based on the real congestion packet loss rate and carrying out corresponding congestion control; the congestion packet loss refers to packet loss caused by the fullness of a transmission layer buffer area.

DTP uses BBR as a transport layer congestion control algorithm for congestion control of the path. The BBR algorithm uses bottleneck bandwidth and round trip delay propagation time (RTT) to optimize congestion control. Bottleneck bandwidth refers to the portion of the network path where the transmission bandwidth is lowest. RTT is the round trip delay time for the transmission of a data packet in a network. The BBR congestion control algorithm adjusts the congestion window according to the bottleneck bandwidth of the network and the equivalent bandwidth of the round trip delay, so as to realize more efficient data transmission. The method dynamically adjusts the sending rate by estimating the bottleneck bandwidth and RTT of the network path so as to effectively utilize the network bandwidth and avoid congestion.

The BBR congestion control module calculates the bottleneck bandwidth and RTT of the network path according to the packet loss condition of the network and reports the bottleneck bandwidth and RTT to the dispatcher of the DTP protocol, and the dispatcher dispatches and sends the data blocks of the buffer area according to a preset dispatching strategy based on the current path sending rate reported by the BBR congestion control module, so that in the transmission process of the deadline sensitive real-time video stream, the accurate distinction of the congestion packet loss and the non-congestion packet loss caused by the fullness of the buffer area of the transmission layer is very important for the BBR congestion control module to accurately estimate the current path bandwidth and adjust the sending rate, and the influence is very great. The non-congestion packet loss refers to packet loss caused by non-transmission layer reasons (such as routing problem, inherent packet loss of a link and/or random packet loss caused by the problem of channel quality).

Since packet loss based on the BBR algorithm often occurs in a buffer limited (buffer limited) stage, that is, an excessive increase in the number of flight messages (inflight) causes the intermediate device buffer to be fully occupied and packet loss occurs, the stage is characterized in that the bandwidth and the real-time round trip time RTT are both in a stable state (even if there is a fluctuation). While in the bandwidth limiting phase (bandwidth limited) where the buffer starts to be occupied but is not fully occupied, the RTT varies with the buffer occupancy, although the bandwidth is also in a steady state (the bandwidth is the bottleneck bandwidth as in the buffer limiting phase).

Based on the analysis, and considering that the BBR algorithm almost plays a role in all the two stages, in order to obtain the congestion packet loss rate capable of truly reflecting the congestion condition of the link in the link, the invention sets a fluctuation coefficient gamma for the BBR congestion control module, and distinguishes the packet loss type by judging the condition relationship between the message round trip time delay RTT _i、RTT_i+1 and the message round trip time RTT _i+2 which are measured three times continuously, thereby more accurately calculating the real congestion packet loss rate.

The judgment method adopted in this embodiment is as follows:

In the above formula, RTT _i is the first measured round trip delay, RTT _i+1 is the second measured round trip delay, and RTT _i+2 is the third measured round trip delay. RTT _i - γ and RTT _i +γ are used to define the lower and upper limits of the fluctuation range of RTT _i+1 measured for the second time, and RTT _i+1 - γ and RTT _i+1 +γ represent the lower and upper limits of the fluctuation range of RTT _i+2 measured for the third time, respectively. s.t.minRTT < RTT _i+1 means that the second measured round trip delay is greater than the known minimum link round trip delay detected by the BBR, and similarly s.t.minRTT < RTT _i+2 means that the third measured round trip delay is greater than the known minimum link delay detected by the BBR. The value of gamma needs to be determined according to the actual link condition, if the delay fluctuation of the link is larger, a larger gamma value should be selected, otherwise, a smaller gamma value should be selected. The meaning of the second row of the above formula is similar to the first row, and reference is made to the explanation of the meaning of the first row.

The BBR congestion control algorithm detects the inherent delay of the link in the delay detection phase and saves it, and if the minimum RTT detected before has not been updated for a preset period of time (e.g. 10 seconds), the BBR will force a switch to enter the delay detection phase no matter which phase was previously. After entering the delay probing phase, the BBR limits the transmission window size CWND to a very small value (e.g., only 4 packets are transmitted) and lasts for a period of time (e.g., at least 200 ms) to ensure that this stream does not cause packet accumulation on devices in the link, thereby probing the inherent round trip delay of the link.

Based on the above formula 1 and formula 2, the method for distinguishing the congestion packet loss from the non-congestion packet loss of the present invention comprises: under the condition that the link round trip delay is measured three times in succession, if packet loss occurs in the process of measuring the link round trip delay for the third time, judging whether the occurred packet loss is congestion packet loss or not by judging whether the following two conditions are met:

1) When the second round trip delay is within the preset fluctuation range of the first round trip delay, and the second round trip delay is larger than the minimum round trip delay of the known link;

2) When the third round trip delay is within the preset fluctuation range of the second round trip delay, the third round trip delay is larger than the minimum round trip delay of the known link.

The preset fluctuation range of the round trip delay is the range determined by the round trip delay addition and subtraction fluctuation coefficient gamma measured in the previous time.

The BBR congestion control module calculates the real congestion packet loss rate of the current path based on the measured real congestion packet loss number only when the two conditions are satisfied. The BBR congestion control module can calculate the current path bandwidth more accurately according to the real packet loss rate, so that congestion control can be performed more effectively, for example, the size of a sending window, the sending rate and the like are regulated.

And 102, calculating the Real Priority (Real Priority) of the data block to be transmitted according to the current path transmission rate (SENDING RATE) and the original Priority (Priority) of the data block to be transmitted, which are determined by the path congestion control module, and deciding whether to transmit the data block to be transmitted or not based on the Real Priority.

After the DTP protocol obtains the current path sending rate (SENDING RATE) reported by the congestion control module, the Real Priority (Real Priority) of the data block to be sent can be calculated based on the mode of calculating the Real Priority of the data block by the DTP protocol, and the calculation mode is as follows:

In the above formula 3, deadline is the deadline of the data block to be sent, remaining_size is the remaining block size of the data block to be sent, RTT is the link round trip delay, and priority is the original priority of the data block to be sent.

[ Optimization of remaining Transmission time calculation ]

In another embodiment of the present invention, the calculation process of calculating the remaining transmission time (REMAINING TIME) of the data block to be transmitted by the DTP protocol is also optimized, and the link delay fluctuation influence value is further reduced based on the original DTP calculated video frame remaining transmission time, so that the remaining transmission time of the data block calculated by the DTP protocol is more accurate.

The network in the real-time video stream transmission process is likely to be unstable (especially, along with the popularization of the mobile internet, many users start to get used to watch the video by using the mobile phone, and the used WIFI or the cellular network often has poor link quality), so that in the process of transmitting the deadline sensitive video stream data, the influence factors of the link delay fluctuation are taken into consideration, and the residual transmission time can be calculated more accurately.

Based on the above analysis, this embodiment needs to consider the effect of delay fluctuation in addition to the currently detected link unidirectional delay (RTT/2) when calculating the remaining transmission time of a certain video frame, so the calculation of the unidirectional delay standard deviation is added when calculating the remaining transmission time of a data block, so that the calculation of the remaining transmission time (REMAINING TIME) of a certain video frame/data block is more accurate and safer:

Wherein REMAINING TIME is the remaining transmission time of a certain video frame/data block, deadline is the deadline when the current data block is sent, REMAINING SIZE is the remaining block size, and SENDING RATE is the current sending rate. RTT is link round trip delay, RTT/2 is link unidirectional delay. stddev is the standard deviation of round trip time RTT, stddev/2 is the standard deviation of one-way time delay. stddev reflects the delay fluctuation condition of the link, k is a standard deviation adjustment coefficient of [0,1], the coefficient determines how much the unidirectional delay standard deviation should influence the calculation of the residual transmission time, on the link with larger delay fluctuation, k should take a larger value in a limited range, otherwise, takes a smaller value.

After calculating the remaining transmission time (REMAINING TIME) of the data block to be transmitted, the DTP protocol can calculate the real priority of the data block to be transmitted based on the following formula 5, i.e. the real priority of the data block to be transmitted is the product of the original priority and the remaining transmission time, so as to decide whether to transmit the data block to be transmitted according to the calculated real priority.

Real priority=priority× REMAINING TIME (formula 5)

In this embodiment, the DTP considers the influence of the time delay fluctuation possibly occurring in the network on the block transmission completion time when calculating the real priority, so that the calculated real priority is more approximate to the real value.

Optimization of real priorities based on coupling relationships between data blocks

This embodiment takes the video coding standard h.265 as an example to describe a method of optimizing the real priority based on the coupling relationship between data blocks. Fig. 2 is a schematic diagram of the dependency relationship between the image group and various frames in the h.265 protocol. Three frames are defined in the h.265 protocol: i-frames, P-frames, and B-frames. I frames are also called intra-frame coding frames, which are completely reserved for the current picture and can be finished only by the frame when decoding; p frame, also called forward predictive coding frame, which represents the difference between this frame and a previous key frame (or P frame), and the final picture must be generated based on the previously buffered picture plus the difference defined by this frame during decoding; b frames, also called bi-predictive interpolated encoded frames, simply record the difference between the present frame and the previous and subsequent frames.

That is, not only the previous buffer picture but also the subsequent picture are required for decoding the B frame, and the final picture is obtained by superimposing the previous and subsequent pictures with the present frame data. The basic unit of video stream coding and decoding is a group of pictures (Group Of Pictures, GOP), a GOP is composed of one I frame and several P and B frames, which cannot be decoded without an I frame; if there is no P frame with the front sequence number, the P frame with the rear sequence number cannot be decoded.

Based on the above analysis, in addition to the priorities of the I frame, P frame, and B frame themselves (set to 0, 1, 2), this embodiment evaluates the priority of a missing frame from the degree of impact on the decoding of other frames when the frame is missing in the GOP, and refers to the number of frames/data blocks that have a dependency on if the frame is missing and cannot be decoded by the dependency block DEPENDENT BLOCKS. On the basis, the block_info frame used in the original DTP for representing the frame data is modified, and DEPENDENT BLOCKS is added for representing other frames depending on the frame.

To sum up, this embodiment obtains the final Priority of a certain video frame by the following formula (6):

The Real Priority is the Real Priority of the optimized data block to be sent, the original Priority is the original Priority of the original frame, and the number DEPENDENT BLOCKS of dependent blocks is used for indicating the number of frames which can not be decoded and are dependent on the frame if the frame is lost, namely the number of data blocks which depend on the data block to be sent currently. REMAINING TIME is the remaining transmission time of the data block to be sent, and the remaining transmission time can be the remaining transmission time calculated by the original DTP protocol or the remaining transmission time after optimization in consideration of path delay fluctuation. Alpha is a priority adjustment coefficient, beta is a block-dependent weight adjustment coefficient, and the values of the two coefficients are required to depend on specific scenes, for example, for applications with more attention to video definition and less video fluency blocking, the value of beta should be increased appropriately to improve the transmission priority of frames with higher dependence. The value of alpha should be increased appropriately if more attention is paid to guaranteeing a high priority or a preferential transmission of data blocks closer to the deadline. The above calculation process considers the Priority of the video frame itself, the remaining transmission time and the dependency of other frames in the GOP on the current frame, and the scheduler selects the frame with the smallest Real Priority, i.e. the highest Priority, each time for transmission.

Fig. 3 is a schematic structural diagram of an electronic device for implementing the deadline-sensitive streaming method according to an embodiment of the present invention, where the device 300 includes: a processor 310 such as a Central Processing Unit (CPU), a communication bus 320, a communication interface 340, and a memory 330. Wherein the processor 310 and the memory 330 may communicate with each other via a communication bus 320. The memory 330 stores a computer program that, when executed by the processor 310, performs the functions of one or more steps of the deadline-sensitive streaming method provided by the present invention.

Memory refers to a device for storing computer programs and/or data based on some storage medium, which may be a Volatile Memory (VM) or a Non-Volatile Memory (NVM). The memory is an internal memory for directly exchanging data with the processor, and can read and write data at any time, and has high speed, and is used as a storage medium for temporary data of an operating system and other running programs. The memory may be synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The nonvolatile memory is a memory using a persistent storage medium, and has a large capacity and can store data permanently, and may be a storage class memory (Storage Class Memory, SCM), a Solid state disk (Solid STATE DISK, SSD), a NAND flash memory, a magnetic disk, or the like. SCM is a common name for new storage medium between memory and flash memory, and is a composite storage technology combining persistent storage characteristic and memory characteristic, and has access speed slower than that of DRAM and SSD hard disk.

The processor may be a general-purpose processor including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but may also be a digital signal processor (DIGITAL SIGNAL Processing, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components.

It should be appreciated that embodiments of the invention may be implemented or realized by computer hardware, a combination of hardware and software, or by computer instructions stored in non-transitory (or referred to as non-persistent) memory. The method may be implemented in a computer program using standard programming techniques, including a non-transitory storage medium configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose. Furthermore, the operations of the processes described in the present invention may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes (or variations and/or combinations thereof) described herein may be performed under control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications), by hardware, or combinations thereof, collectively executing on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

Further, the method may be implemented in any type of computing platform operatively connected to a suitable computing platform, including, but not limited to, a personal computer, mini-computer, mainframe, workstation, network or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and so forth. Aspects of the invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and/or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, which when read by a computer, is operable to configure and operate the computer to perform the processes described herein. Further, the machine readable code, or portions thereof, may be transmitted over a wired or wireless network. When such media includes instructions or programs that, in conjunction with a microprocessor or other data processor, implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media. The invention also includes the computer itself when programmed according to the methods and techniques of the present invention.

The foregoing is merely exemplary of the present invention and is not intended to limit the present invention. Various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of deadline-sensitive streaming, the method being applied to a streaming device employing a deadline-sensitive protocol DTP, the method comprising:

When path congestion control is carried out, distinguishing congestion packet loss and non-congestion packet loss, calculating the current path sending rate based on the real congestion packet loss rate and carrying out corresponding congestion control; the congestion packet loss refers to packet loss caused by the fullness of a transmission layer buffer area;

And calculating the real priority of the data block to be transmitted according to the current path transmission rate and the original priority of the data block to be transmitted, which are determined by the path congestion control module, and deciding whether to transmit the data block to be transmitted or not based on the real priority.

2. The method according to claim 1, wherein the method for distinguishing between congestion packet loss and non-congestion packet loss is:

3. The method according to claim 1, wherein the method for calculating the true priority of the data block to be transmitted is:

Subtracting the ratio of the size of the residual block to the current transmission rate from the cut-off time after the data block is transmitted, subtracting the unidirectional time delay, and subtracting the time delay fluctuation influence value to obtain the residual transmission time of the data block to be transmitted;

The product of the original priority and the remaining transmission time is taken as the real priority.

4. A method according to claim 1 or 3, characterized in that the method of calculating the true priority of the data block to be transmitted is:

After the product of the original priority and the residual transmission time is calculated, the product is multiplied by a priority adjustment coefficient alpha, then the product of a block dependence weight adjustment coefficient beta and the reciprocal of the number of dependent blocks is added, and the obtained result is used as the final real priority.

5. A method of deadline-sensitive streaming, the method being applied to a streaming device employing a deadline-sensitive protocol DTP, the method comprising:

6. The method according to claim 5, wherein the method for calculating the true priority of the data block to be transmitted is:

7. A method of deadline-sensitive streaming, the method being applied to a streaming device employing a deadline-sensitive protocol DTP, the method comprising:

Multiplying the product of the original priority and the residual transmission time by a priority adjustment coefficient alpha, adding the product of a block dependency weight adjustment coefficient beta and the reciprocal of the number of dependent blocks, and taking the obtained result as the real priority;

8. The method of claim 7, wherein the method of calculating the remaining transmission time is:

and subtracting the ratio of the size of the residual block to the current transmission rate from the cut-off time after the data block is transmitted, subtracting the one-way time delay, and subtracting the time delay fluctuation influence value to obtain the residual transmission time of the data block to be transmitted.

9. A deadline-sensitive streaming apparatus for use in a streaming device employing a deadline-sensitive protocol DTP, the apparatus comprising:

The congestion control module is used for distinguishing congestion packet loss and non-congestion packet loss when path congestion control is carried out, calculating the current path sending rate based on the real congestion packet loss rate and carrying out corresponding congestion control; the congestion packet loss refers to packet loss caused by the fullness of a transmission layer buffer area;

The decision module is used for subtracting the ratio of the size of the residual block to the current transmission rate from the cut-off time after the data block is transmitted, subtracting the unidirectional time delay, and subtracting the time delay fluctuation influence value to obtain the residual transmission time of the data block to be transmitted; multiplying the product of the original priority and the residual transmission time by a priority adjustment coefficient alpha, adding the product of a block dependence weight adjustment coefficient beta and the reciprocal of the number of dependent blocks to obtain the final real priority, and deciding whether to transmit the data block to be transmitted based on the real priority.

10. An electronic device is characterized by comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium are communicated with each other through the communication bus;

a storage medium storing a computer program;

A processor for implementing the method of any of claims 1-8 when executing a computer program stored on a storage medium.

11. A storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any of claims 1 to 8.