CN105071897A - Multipath redundant transmission method for network real-time audio conversation media data - Google Patents

Abstract

The present invention provides a network real-time audio session media data multi-path redundant transmission method, comprising: the sending end of the audio media data packs the captured audio media data according to the audio codec format and network transmission protocol negotiated by both parties of the real-time audio session, and adopts The redundant mode sends to multiple incompletely intersecting transmission paths at the same time, and performs multi-path redundant transmission of real-time audio media data. The receiving end of audio media data performs redundancy elimination and reorganization on the received audio media data packets from different transmission paths. Operate, and restore the original audio data according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session. As a result, it can effectively reduce the overall impact of packet loss, delay and jitter caused by changes in single path conditions on the end-to-end transmission of media data, thereby improving the reliability of real-time audio media data transmission and improving the user experience quality of network audio conversation services (QoE).

Description

A method for multi-path redundant transmission of network real-time audio session media data

Technical field:

The invention relates to the technical field of network communication, in particular to a method for multi-path redundant transmission of network real-time audio session media data.

Background technique:

The Internet uses a "best effort" transmission mechanism, which cannot provide quality of service (QoS) guarantees for end-to-end media transmission. Real-time audio session is a typical network communication service. Its media transmission usually occupies several kbps to hundreds of kbps of transmission bandwidth. It is a type of narrowband service. Although it occupies a small bandwidth, it has high requirements for real-time transmission. At present, audio media transmission of IP communication terminals usually adopts traditional RTP and UDP protocols for transmission control, and uses end-to-end default routing path (single path) without quality of service (QoS) guarantee for transmission, and uncertain congestion in the end-to-end path link The resulting data packet loss and delay jitter often cause audio media data to fail to be reassembled and decoded, seriously affecting the quality of real-time audio sessions. The Internet has become an important bearer network for network communications. Improving the transmission mode of real-time audio conversation media and improving the quality of service experience are important issues that need to be solved urgently.

Invention content:

Aiming at the defects of the prior art, the present invention provides a method for multi-path redundant transmission of network real-time audio session media data. The method combines the characteristics of low transmission bandwidth occupied by real-time audio media, and constructs an end-to-end incompletely intersecting transmission path. And the multi-path redundant transmission control mechanism and protocol are adopted to realize the multi-path redundant transmission of real-time audio media data. This multi-path redundant transmission method can effectively reduce the overall impact of packet loss, delay and jitter caused by changes in single path conditions on the end-to-end transmission of media data, thereby improving the reliability of real-time audio media data transmission and improving network audio sessions The user experience quality of the service.

The present invention provides a network real-time audio session media data multi-path redundant transmission method, comprising:

The sender of the audio media data packs the captured audio media data according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session, and sends them to multiple incompletely intersecting transmission paths in a redundant manner at the same time for real-time audio media data Multipath redundant transmission;

The receiving end of the audio media data performs redundancy removal and reassembly operations on the received audio media data packets from different transmission paths, and restores the original audio data according to the audio codec format and network transmission protocol negotiated by the two sides of the real-time audio session.

Optionally, the multiple incompletely intersecting transmission paths include: a default path based on a default route and one or more incompletely intersecting redundant transmission paths.

Optionally, a redundancy coefficient is set according to the real-time audio session, and the number of the multiple incompletely intersecting transmission paths is greater than the redundancy coefficient set for the real-time audio session.

Optionally, the redundant elimination and reorganization operations include:

Set up and initialize substream buffers, reassembly buffers and jitter removal windows;

The receiving end of the audio media data performs a redundancy removal operation on the received audio media data packets;

The receiver of the audio media data performs a reassembly operation on the audio media data packets after the redundancy removal operation.

Optionally, the setting and initializing the substream buffer, the reassembly buffer and the jitter removal window include:

Set the same number of sub-stream buffers as the number of incompletely intersecting transmission paths, the sub-stream buffers are used to receive audio media data packets of different transmission paths, and initialize each storage location of each of the sub-stream buffers Is empty;

The reorganization buffer is set, and the reorganization buffer is used to store the sequence number of the audio media data packet after redundant elimination, the sequence number of the substream buffer for storing the audio media data packet and the substream of the audio media data packet in the storage The storage location in the stream buffer, initialize the size of the reorganization buffer to be N _j , the sequence numbers of the audio media data packets used for storing in the reorganization buffer after the redundancy removal operation are all initialized to -1, and the storage The sequence number of the sub-stream buffer of the audio media data packet and the storage location of the audio media data packet in the stored sub-stream buffer are all initialized to be empty;

Set a jitter removal window, the jitter removal window is used to implement the jitter removal of the audio media data packet, initialize the size of the jitter removal window to W, W∈[W _min , W _max ].

Optionally, the redundant elimination operation includes:

S1, adopt polling mode to inquire about each described sub-flow buffer, obtain the latest received described audio media data packet i, extract the sequence number of described audio media data packet i, and record it as FSN _i ;

S2. Perform a modulo operation on the serial number FSN _i of the audio media data group and the size N _j of the reassembly buffer to obtain a value m, that is, m=FSN _i modN _j , and query the audio stored at position m in the reassembly buffer The sequence number of the media data packet, denoted as J_FSN _m ;

S3, if J_FSN _m =-1, then assign the value of FSN _i to J_FSN _m , store the serial number and the storage location of the sub-stream buffer where the audio media data packet i is located in the reassembly buffer m position, and perform the steps S1;

S4. If J_FSN _m ≠-1, FSN _i >J_FSN _m , then assign the value of FSN _i to J_FSN _m , and store the sequence number and storage location of the substream buffer where the audio media data packet i is located in the reassembly buffer In position m, execute step S1;

S5. If J_FSN _m ≠ -1 and FSN _i ≤ J_FSN _m , execute step S1.

Optionally, the querying each of the substream buffers in a polling manner includes:

The receiving end of the audio media data regularly counts the reception and redundancy elimination of the audio media data packets in each of the sub-stream buffers, and obtains the polling priority sequences of the sub-stream buffers corresponding to different transmission paths;

The receiving end of the audio media data queries each of the sub-stream buffers according to the polling priority sequence, and performs a redundant elimination operation.

Optionally, the reorganization operation includes:

Dynamically adjust the size of the jitter removal window;

The callback operation of the audio media data packet is performed according to the size of the jitter elimination window.

Optionally, the dynamically adjusting the size of the jitter elimination window includes:

Record the time R(i) at which the audio media data packet i arrives at the reassembly buffer, and obtain the audio media according to the timestamp in the audio media data packet i or other flags used to record the sending time of the audio media data packet The sending time S(i) of the data packet i, calculate the delay jitter J(i)=R(i)-S(i) of the audio media data packet i;

According to the audio media data packet that has arrived at the reassembly buffer, the expected delay jitter of the audio media data packet i newly arriving at the reassembly buffer is obtained Among them, N is a fixed value, i≥N, P(k) is a weighting coefficient, and $\underset{k=i-N}{\overset{i-1}{\Σ}}P\left(k\right)=1;$

Calculate the class standard error of the delay jitter of the audio media data packet i $J_{ARmS\mathrm{E.}}\left(i\right)=\sqrt{\frac{1}{N}-\underset{k=i-N+1}{\overset{i}{\Σ}}{\[J\left(k\right)-\stackrel{\‾}{J\left(k\right)}\]}^2};$

Setting thresholds g ₁ and g ₂ is used to judge the change range of the jitter elimination window size, and g ₁ <g ₂ ;

when Increase the size of the jitter removal window, if J _ARMSE (i)∈[g ₁ ,g ₂ ], the jitter removal window increases If J _ARMSE (i)＞g ₂ , the jitter removal window increases where k∈[0.5,1];

when Reduce the size of the jitter removal window, if J _ARMSE (i)∈[g ₁ ,g ₂ ], the jitter removal window is reduced If J _ARMSE (i)＞g ₂ , the jitter removal window is reduced where k∈[0.5,1];

If the adjusted jitter elimination window size W is smaller than the jitter elimination window minimum value W _min , then set the jitter elimination window size to the jitter elimination window minimum value W _min , if the adjusted jitter elimination window size W is greater than the jitter elimination window maximum value W _max , then the size of the jitter elimination window is set to the maximum value W _max of the jitter elimination window;

When J _ARMSE (i)<g ₁ , the size of the jitter elimination window does not need to be changed;

If the jitter removal window size is modified, the expected delay jitter of the changed audio media data packet i It is any integer in the interval [J(i)-4, J(i)+4], otherwise, no change is made.

Optionally, the performing the audio media data packet callback operation according to the size of the jitter elimination window includes:

Find the audio media data packet whose serial number is FSN in the jitter elimination window, perform callback decoding according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session, and find the audio media data packet serial number in the jitter elimination window It is the audio media data packet of FSN+1;

If found, then the audio media data packet serial number is FSN+1 audio media data packet callback decoding, continue to search the audio media data packet serial number is the audio media data packet of FSN+2, otherwise, judge whether the jitter elimination window has surplus, If there is no remaining, then the audio media data packet whose serial number is FSN+1 is called back and decoded again, and continues to search for the audio media data packet whose serial number is FSN+2, otherwise, wait for the preset time Continue to search after t;

If the search is successful three times in a row, reduce the jitter elimination window to the size of one data frame, and judge whether the jitter elimination window is smaller than the minimum value W _min of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the minimum value of the jitter elimination window W _min , continue to perform the search operation;

If three consecutive searches are unsuccessful, then increase the size of the jitter elimination window by one data frame, and judge whether the jitter elimination window is greater than the maximum value W _max of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the maximum value of the jitter elimination window value W _max , continue to perform the search operation.

As can be seen from the above technical solution, the multipath redundant transmission method of network real-time audio session media data of the present invention includes: the audio media data sending end of the audio media data transmits the captured audio media data according to the audio codec format negotiated by both parties of the real-time audio session and the network transmission Protocol packaging, using redundant methods to send to multiple incompletely intersecting transmission paths at the same time, for multi-path redundant transmission of real-time audio media data, the receiving end of audio media data performs redundancy on the received audio media data packets from different transmission paths The remaining culling and reorganization operations are performed, and the original audio data is restored according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session. Therefore, by using multi-path redundant transmission for real-time audio media data, the probability of packet loss compensation in data transmission can be effectively improved, and the transmission-related packet loss, delay, and jitter indicators can be improved, thereby improving the reliability of data transmission and improving Service Quality of Experience (QoE).

Description of drawings:

Fig. 1 is the schematic flow chart of the network real-time audio session media data multipath redundant transmission method that the first embodiment of the present invention provides;

2 is a structural diagram of a multipath redundant transmission system of a multipath relay transmission service system in which SIPProxy/IMSCSCF participates in session negotiation provided by the second embodiment of the present invention;

FIG. 3 is a schematic flow diagram of a method for multipath redundant transmission of network real-time audio session media data according to the second embodiment of the present invention;

Fig. 4 is the transmission schematic diagram of the audio media data packet of the multi-path relay transmission service system that SIPProxy/IMSCSCF that the second embodiment of the present invention provides participates in session negotiation;

5 is a message flow diagram of the multipath redundant transmission process of the multipath relay transmission service system in which SIPProxy/IMSCSCF participates in session negotiation provided by the second embodiment of the present invention;

Fig. 6 is the audio media data packet receiver buffer setting block diagram of the network real-time audio session media data multipath redundant transmission method that the second embodiment of the present invention provides;

Fig. 7 is the implementation design diagram of the buffer zone of the audio media data packet receiving end of the network real-time audio session media data multipath redundant transmission method provided by the second embodiment of the present invention;

8 is a schematic diagram of the encapsulation format of the audio media data packet of the multipath redundant transmission of the multipath relay transmission service system provided by the SIPProxy/IMSCSCF participating in the session negotiation provided by the second embodiment of the present invention;

FIG. 9 is a flowchart of redundant elimination of audio media data packets in the method for multi-path redundant transmission of media data in a network real-time audio session according to the second embodiment of the present invention.

Detailed ways:

The specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

FIG. 1 shows a schematic flowchart of a method for multipath redundant transmission of network real-time audio session media data provided by the first embodiment of the present invention. As shown in FIG. 1 , the method of this embodiment is as follows.

101. The sending end of audio media data packs the captured audio media data according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session, and sends them to multiple incompletely intersecting transmission paths in a redundant manner at the same time for real-time audio Multipath redundant transmission of media data.

In this step, during the establishment of the audio session, the calling party and the called party of the audio session obtain multiple incompletely intersecting transmission paths. It should be noted that the multiple incompletely intersecting transmission paths include: one based on The default path of the default route and one or more incompletely intersecting redundant transmission paths.

Specifically, during an audio session, a redundancy coefficient is set according to the real-time audio session, and the number of the multiple incompletely intersecting transmission paths is greater than the redundancy coefficient set for the real-time audio session.

102. The receiving end of the audio media data performs redundancy removal and reassembly operations on the received audio media data packets from different transmission paths, and restores the original audio data according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session.

In this step, the redundant removal and reorganization operations include:

Set up and initialize substream buffers, reassembly buffers and jitter removal windows;

The receiving end of the audio media data performs redundancy elimination on the received audio media data packets;

The receiver of the audio media data performs a reassembly operation on the audio media data packets after the redundancy removal operation.

Further, the setting and initializing the substream buffer, the reassembly buffer and the jitter elimination window include:

Set the same number of sub-stream buffers as the number of incompletely intersecting transmission paths, the sub-stream buffers are used to receive audio media data packets of different transmission paths, and initialize each storage location of each of the sub-stream buffers Is empty;

The reorganization buffer is set, and the reorganization buffer is used to store the serial number FSN of the audio media data packet after redundant elimination, the serial number of the substream buffer for storing the audio media data packet and the audio media data packet in the stored The storage location in the sub-stream buffer, initialize the size of the reorganization buffer to be N _j , and the serial numbers FSN of the audio media data packets after the redundant elimination operation for storing in the reorganization buffer are all initialized to -1, so The sub-stream buffer serial number storing the audio media data packet and the storage location of the audio media data packet stored in the stored sub-stream buffer are all initialized to be empty;

Set a jitter elimination window, the jitter elimination window is used to realize the jitter elimination of the audio media data packet, initialize the size of the jitter elimination window to W, W∈[W _min , W _max ].

Further, the redundant elimination operation of the audio media data packet includes:

S1, adopt polling mode to inquire about each described sub-flow buffer, obtain the latest received described audio media data packet i, extract the sequence number of described audio media data packet i, and record it as FSN _i ;

S2. Perform a modulo operation on the serial number FSN _i of the audio media data group and the size N _j of the reassembly buffer to obtain a value m, that is, m=FSN _i modN _j , and query the audio media stored in the m position of the reassembly buffer The sequence number of the data packet, denoted as J_FSN _m ;

S3, if J_FSN _m =-1, then assign the value of FSN _i to J_FSN _m , store the serial number and the storage location of the sub-stream buffer where the audio media data packet i is located in the reassembly buffer m position, and perform the steps S1;

S4. If J_FSN _m ≠-1, FSN _i >J_FSN _m , then assign the value of FSN _i to J_FSN _m , and store the sequence number and storage location of the substream buffer where the audio media data packet i is located in the reassembly buffer In position m, execute step S1;

S5. If J_FSN _m ≠ -1 and FSN _i ≤ J_FSN _m , execute step S1.

It should be noted that the polling method used to query each of the substream buffers includes:

The receiving end of the audio media data regularly counts the reception and elimination of the audio media data packets in each of the sub-stream buffers, and obtains the polling priority sequences of the sub-stream buffers corresponding to different transmission paths;

The receiving end of the audio media data queries each of the sub-stream buffers according to the polling priority sequence, and performs a redundant elimination operation.

Further, the reorganization operation includes:

Dynamically adjust the size of the jitter removal window;

The callback operation of the audio media data packet is performed according to the size of the jitter elimination window.

Specifically, during the multi-path real-time audio session, the receiving end of the audio data packet receives the audio media data packets transmitted by the multiple incompletely intersecting transmission paths, and according to the shared sequence number or Timestamp etc. can distinguish the mark of described audio media data grouping and carry out the redundant elimination operation of audio media data grouping; By searching the audio media data grouping in dynamic cache area or dynamic buffer window, obtain the audio media data grouping to be decoded, according to The audio codec format and the network transmission protocol negotiated by both parties of the real-time audio session perform callback decoding on the audio media data packet to restore the original audio data.

Fig. 2 shows the multipath redundant transmission system structural diagram of the multipath relay transmission service system that SIPProxy/IMSCSCF participates in the session negotiation that the second embodiment of the present invention provides, as shown in Fig. 2, the calling party 210 and the called party The parties 220 are respectively located in the user-side networks at both ends, and the SIPProxy/IMSCSCF 230, the media relay controller 240, and the media relay server 250 are all deployed in the network.

The SIPProxy/IMSCSCF 230 is a SIPProxy/IMSCSCF capable of multipath transmission session negotiation. By extending SIP and SDP messages, the signaling negotiation for multipath session transmission is realized, and a module for processing multipath session requests is added to SIPProxy/IMSCSCF230, which can effectively realize SIPProxy/IMSCSCF230 to handle multipath from the user-side network at both ends session request.

The media relay controller 240 is used to manage the network topology and behavior of the media relay server 250, and is responsible for the allocation of relay transmission paths, and the media relay server 250 participates in the construction of relay transmission paths, and is responsible for receiving data packets with forwarding. The media relay controller 240 and the media relay server 250 constitute a multi-path relay transmission service system, which provides relay transmission services for multi-path sessions. For example, two media relay servers are included in this embodiment , that is, the media relay server 250-1 and the media relay server 250-2 shown in FIG. 2 .

Specifically, the SIPProxy/IMSCSCF 230 having the capability of negotiating a multipath transmission session receives a multipath session establishment request from the user-side networks at both ends, and requests the media relay controller 240 to allocate a relay transmission path. The media relay controller 240 negotiates with the media relay server 250 to complete the allocation and establishment of the relay path, and returns the allocation information of the relay transmission path to the SIP Proxy/IMSCSCF 230 capable of negotiating multi-path transmission sessions.

The above-mentioned multipath relay transmission service system in which SIPProxy/IMSCSCF participates in session negotiation in FIG. 2 is further elaborated in conjunction with the flow diagram of the method for multipath redundant transmission of network real-time audio session media data provided by the second embodiment of the present invention shown in FIG. 3 The method of this embodiment.

301. A real-time audio session is established between the calling party and the called party, and the sending end of the audio media data packages the captured audio media data according to the audio codec format and network transmission protocol negotiated by the two parties of the real-time audio session, and adopts a redundant method at the same time Send to three incompletely intersecting transmission paths for multi-path redundant transmission of real-time audio media data.

In this step, as shown in Figure 2, the redundancy coefficient is set according to the audio session, the redundancy coefficient of the audio media data transmission in this embodiment is 2, and three transmission paths are obtained between the calling party 210 and the called party 220 260, including a default path 260-D based on a default route, a relay transmission path 260-R1 via the media relay server 250-1, and a relay transmission path via the media relay server 250-2 260-R2.

Fig. 4 shows the transmission schematic diagram of the audio media data packet of the multi-path relay transmission service system in which SIPProxy/IMSCSCF participates in the session negotiation provided by the second embodiment of the present invention. As shown in Fig. 4, each transmission path transmits the same audio Media data grouping.

Fig. 5 shows the message flow chart of the multipath redundant transmission process of the multipath relay transmission service system in which SIPProxy/IMSCSCF participates in session negotiation provided by the second embodiment of the present invention, and the specific signaling process of multipath redundant transmission as described below.

501. The calling party 210 requests the SIPProxy/IMSCSCF230 having a multi-path transmission session negotiation function to establish a multi-path real-time audio session between the calling party 210 and the called party 220, and the redundancy factor of the audio session is 2;

502. The SIPProxy/IMSCSCF 230 with multi-path transmission session negotiation capability checks whether the calling party 210 and the called party 220 have specific multi-path session capabilities. If neither party has the multi-path session capability or one party does not have the multi-path session capability, the The SIPProxy/IMSCSCF 230 that does not have the session negotiation capability of the path transmission rejects the multipath audio session request. If both the calling party 210 and the called party 220 have the multipath session capability, the SIPProxy/IMSCSCF230 with the multipath transmission session negotiation capability will request the media relay controller 240 for the multipath communication between the calling party 210 and the called party 220. Route real-time audio sessions to allocate multiple relay transmission paths;

503. Since the redundancy factor of the session is 2, the media relay controller 240 allocates two relay transmission paths for the multi-path real-time audio session between the calling party 210 and the called party 220, one of which passes through the media relay server 250 -1, one via the media relay server 250-2. The media relay controller 240 sends relay path adding requests 503-1 and 503-2 to the media relay server 250-1 and the media relay server 250-2 respectively;

504. The media relay server 250-1 and the media relay server 250-2 complete the addition of the relay path, and return relay path addition success responses 504-1 and 504-2 to the media relay controller 240 respectively;

505. The media relay controller 240 returns a relay path assignment success response to the SIPProxy/IMSCSCF 230 capable of multipath transmission session negotiation, and the response message includes information about two relay paths;

506. The SIPProxy/IMSCSCF230 having the multipath transmission session negotiation capability notifies the called party 220 of the allocation of transmission paths;

507. The called party 220 allocates and initializes a buffer for the multipath audio session, and returns a notification success response of 506 to the SIPProxy/IMSCSCF230 having the multipath transmission session negotiation capability;

508. The SIPProxy/IMSCSCF230 having the multipath transmission session negotiation capability returns a multipath session establishment success response to the calling party 210, and notifies the calling party 210 of the assignment of the two transmission paths;

509. There are three transmission paths between the calling party 210 and the called party 220: a default path 260-D based on a default route, a relay transmission path 260-R1 via the media relay server 250-1, and a The relay transmission path 260-R2 of the relay server 250-2. Using redundant transmission, the transmission mode of media data packets is shown in FIG. 4 .

302. The receiver of audio media data sets and initializes a substream buffer, a reassembly buffer, and a jitter removal window, and receives audio media data packets transmitted on three incompletely intersecting transmission paths.

In this step, the setting and initialization of the sub-stream buffer, the reassembly buffer and the jitter removal window include:

Set the same number of sub-stream buffers as the number of incompletely intersecting transmission paths, the sub-stream buffers are used to receive audio media data packets of different transmission paths, and initialize each storage location of each of the sub-stream buffers Is empty;

The reorganization buffer is set, and the reorganization buffer is used to store the sequence number of the audio media data packet after redundant elimination, the sequence number of the substream buffer for storing the audio media data packet and the substream of the audio media data packet in the storage The storage location in the stream buffer, initialize the size of the reorganization buffer to be N _j , the sequence numbers of the audio media data packets used for storing in the reorganization buffer after the redundancy removal operation are all initialized to -1, and the storage The sequence number of the sub-stream buffer of the audio media data packet and the storage location of the audio media data packet in the stored sub-stream buffer are all initialized to be empty;

Set the jitter elimination window, the jitter elimination window is used to realize the jitter elimination of the audio media data packet, initialize the size of the jitter elimination window to be W=100ms, W∈[W _min , W _max ], W _min =40ms , W _max =160 ms.

Specifically, Fig. 6 shows a block diagram of setting buffers at the receiving end of the audio media data packet of the network real-time audio session media data multipath redundant transmission method provided by the second embodiment of the present invention, as shown in Fig. 6 , including Substream buffer 610, reassembly buffer 620 and jitter removal window 630, combined with the buffer of the audio media data receiving end of the network real-time audio session media data multipath redundant transmission method provided by the second embodiment of the present invention shown in FIG. 7 Implementation design diagram, in Figure 7, the reassembly buffer is fixed, and the audio media data packets enter the sub-stream buffer 710 corresponding to each transmission path through multiple incompletely intersecting paths, wherein the sub-stream buffer 710-D corresponds to the transmission path 260- D, the sub-stream buffer 710-R1 corresponds to the transmission path 260-R1, and the sub-stream buffer 710-R2 corresponds to the transmission path 260-R2, enters the reassembly buffer 720 through the redundancy elimination operation, and then dynamically adjusts the reassembly buffer 720 The size of the jitter elimination window is 730 to realize the jitter elimination of the audio media data packet, and finally perform callback decoding on the audio media data packet according to the audio codec format and network transmission protocol negotiated by both parties of the real-time audio session, and restore the original audio data.

Fig. 8 shows a schematic diagram of the encapsulation format of the audio media data packet of the multipath redundant transmission of the multipath relay transmission service system provided by the SIPProxy/IMSCSCF participating in the session negotiation provided by the second embodiment of the present invention, and the format encapsulation adopts multipath Transport protocol (MPTP), the meanings of each flag in Figure 8 are as follows:

801: version number, 2bit, currently version 1;

802: type, 1 bit, used to describe the type of audio media data packet (media data packet or control data packet);

803: padding bit, 1bit, indicating whether there is non-payload padding data;

804: MPTP type for special applications, 4bit, indicating the specific application of this type of data packet;

805: business type, 4bit, indicating the transmission requirements of different business types;

806: Reserved field, 4bit, set to 0;

807: sub-stream serial number, 16bit, used to identify the transmission serial number of the audio media data packet in the path, in redundant transmission, set the sub-stream serial number to be the same as the audio media data packet serial number;

808: path identifier, 32bit, used to identify a transmission path, used to store audio media data packets at the receiving end of audio media data packets;

809: serial number, 32bit, used to identify the unique audio media data packet in the real-time audio session, and used for the receiver of the audio media data packet to reorganize the original audio data according to this serial number;

810: The payload is the real-time audio media data packet to be transmitted.

In a specific application, the sub-stream buffer 610 is responsible for receiving audio media data packets from multiple incompletely intersecting paths, by performing redundant elimination on the audio media data packets, and storing the storage location of the audio media data packets in the reassembly buffer 620 At the same time, according to the statistics of the received audio media data packets and the elimination of redundant audio media data packets, periodically adjust the polling priorities of the sub-stream buffers corresponding to different transmission paths of the sub-stream buffer 610 . By dynamically adjusting the size of the jitter elimination window 630, the jitter elimination of the audio media data packet can be realized, and the callback decoding process of the audio media data packet will also adjust the size of the jitter elimination window in real time according to the callback result.

303. The receiving end of the audio media data performs redundancy removal on the received audio media data packets.

In this step, FIG. 9 shows a flowchart for eliminating redundancy of audio media data packets in a method for multipath redundant transmission of network real-time audio session media data provided by the second embodiment of the present invention. The specific steps include:

901. Initialize the reassembly buffer with a size of N _j ;

902. Obtain the audio media data packet i from each path substream buffer in a polling manner;

903. Determine whether the payload of the audio media data packet i is empty, if so, execute step 911, otherwise, execute step 904;

904, extract the sequence number of the audio media data packet i, such as the sequence number 809 shown in Figure 8, denoted as FSN _i ;

905. Calculate m=FSN _i mod N _j ;

906, inquire about the sequence of the audio media data packet stored on the m position in the reorganization buffer, the sequence number 809 as shown in Figure 8 is also denoted as J_FSN _m ;

907, judge whether J_FSN _m is equal to-1, if so, then explain that the audio media data packet is newly arrived at the reassembly buffer, execute step 909, otherwise, execute step 908;

908. Determine whether J_FSN _m is less than FSN _i , if so, then explain that the audio media data packet i is a newly arrived audio media data packet, and execute step 909, otherwise, execute step 911;

909. Assign the value of FSN _i to J_FSN _m ;

910. Store the storage location of the audio media data packet i in the substream buffer into the m location in the reassembly buffer;

911. Determine whether to end receiving, if yes, end receiving audio media data packets; otherwise, execute step 902.

304. The receiving end of the audio media data reassembles the audio media data packets after the redundancy removal operation, and restores the original audio data according to the audio codec format and network transmission protocol negotiated by both parties of the real-time audio session.

In this step, the reorganization operation includes:

Dynamically adjust the size of the jitter removal window;

The callback operation of the audio media data packet is performed according to the size of the jitter elimination window.

Further, said adjusting the size of the jitter elimination window includes:

Record the time R(i) at which the audio media data packet i arrives at the reassembly buffer, and obtain the audio media according to the timestamp in the audio media data packet i or other flags used to record the sending time of the audio media data packet The sending time S(i) of the data packet i, calculate the delay jitter J(i)=R(i)-S(i) of the audio media data packet i;

According to the audio media data packet that has arrived at the reassembly buffer, the expected delay jitter of the audio media data packet i newly arriving at the reassembly buffer is obtained Among them, N is a fixed value, take N=4, i≥N, P(k) is a weighting coefficient, generally take values P(i-1)=0.5, P(i-2)=0.3, P(i-3 )=0.125, P(i-4)=0.075;

Calculate the class standard error of the delay jitter of the audio media data packet i $J_{ARmS\mathrm{E.}}\left(i\right)=\sqrt{\frac{1}{N}-\underset{k=i-N+1}{\overset{i}{\&Sigma;}}{\&lsqb;J\left(k\right)-\stackrel{\&OverBar;}{J\left(k\right)}\&rsqb;}^2};$

Set the threshold g ₁ =5ms, g ₂ =10ms to judge the change range of the jitter elimination window size;

when Increase the size of the jitter removal window, if J _ARMSE (i)∈[5,10], the increase of the jitter removal window If J _ARMSE (i) > 10, the jitter removal window increases where k=1;

when Reduce the size of the jitter removal window, if J _ARMSE (i)∈[5,10], the reduction of the jitter removal window If J _ARMSE (i) > 10, the jitter removal window is reduced where k=0.6;

If the size W of the adjusted jitter elimination window is smaller than the minimum value W _min of the jitter elimination window, then set the size of the jitter elimination window to the minimum value W _min of the jitter elimination window, if the adjusted size W of the jitter elimination window is greater than the jitter elimination window maximum value W _max , then set the size of the jitter elimination window as the maximum value W _max of the jitter elimination window;

When J _ARMSE (i)<5, the size of the jitter elimination window does not need to be changed;

It should be noted that, in order to prevent the impact of the current audio media data packet i on the prediction accuracy of the next audio media data packet i+1, if the size of the jitter removal window is modified, the time delay of the audio media data packet i is changed The jitter J(i) is any integer in the interval [J(i)-4, J(i)+4], otherwise, no change is made.

Further, the performing the callback operation of the audio media data packet according to the size of the jitter elimination window includes:

Find the audio media data packet whose serial number is FSN in the jitter elimination window, perform callback decoding according to the audio encoding solution format and network transmission protocol negotiated by both parties in the real-time audio session, and find the audio media data packet serial number in the jitter elimination window It is the audio media data packet of FSN+1;

If found, then the audio media data packet sequence number is the audio media data packet callback decoding of FSN+1, continue to search for the audio media data packet sequence number of the audio media data packet is the audio media data packet of FSN+2;

Otherwise, it is judged whether the jitter elimination window is left, if there is no remaining, then perform a callback decoding operation on the audio data packet with the audio media data packet serial number FSN+1, and continue to search for the audio media with the audio media data packet serial number FSN+2 Data grouping, otherwise, continue to search after waiting for the preset time t, t can be 10ms;

If the search is successful three times in a row, reduce the jitter elimination window to the size of one data frame, and judge whether the jitter elimination window is smaller than the minimum value W _min of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the minimum value of the jitter elimination window W _min , continue to perform the search operation;

If three consecutive searches are unsuccessful, then increase the size of the jitter elimination window by one data frame, and judge whether the jitter elimination window is greater than the maximum value W _max of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the maximum value of the jitter elimination window value W _max , continue to perform the search operation.

In summary, the method for multi-path redundant transmission of network real-time audio session media data in this embodiment can effectively improve the probability of packet loss compensation in data transmission by using multi-path redundant transmission of session audio data, and ensure the reliability of data transmission. sex. And by performing redundancy elimination and recombination operations based on dynamic buffer window jitter elimination at the receiving end of audio media data packets, the impact of delay jitter on the reception of audio media data packets is effectively reduced, thereby effectively improving user service experience quality .

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope defined by the claims of the present invention .

Claims (10)

1. A network real-time audio conversation media data multipath redundant transmission method, is characterized in that, comprises: The sender of the audio media data packs the captured audio media data according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session, and sends them to multiple incompletely intersecting transmission paths in a redundant manner at the same time for real-time audio media data Multipath redundant transmission; The receiving end of the audio media data performs redundancy removal and reassembly operations on the received audio media data packets from different transmission paths, and restores the original audio data according to the audio codec format and network transmission protocol negotiated by the two sides of the real-time audio session. 2. the network real-time audio session media data multipath redundant transmission method according to claim 1, is characterized in that, described a plurality of incompletely intersecting transmission paths comprises: a default path based on default routing and one or more Two incompletely intersecting redundant transmission paths. 3. network real-time audio session media data multipath redundant transmission method according to claim 1, it is characterized in that, according to described real-time audio session setting redundancy coefficient, the bar number of described a plurality of incompletely intersecting transmission paths Redundancy factor greater than the Realtime Audio session setting. 4. network real-time audio conversation media data multi-path redundant transmission method according to claim 1, is characterized in that, described redundant elimination and reorganization operation comprise: Set up and initialize substream buffers, reassembly buffers and jitter removal windows; The receiving end of the audio media data performs a redundancy removal operation on the received audio media data packets; The receiver of the audio media data performs a reassembly operation on the audio media data packets after the redundancy removal operation. 5. the network real-time audio session media data multipath redundant transmission method according to claim 4, is characterized in that, described setting and initialization substream buffer, reorganization buffer and jitter elimination window include: Set the same number of sub-stream buffers as the number of incompletely intersecting transmission paths, the sub-stream buffers are used to receive audio media data packets of different transmission paths, and initialize each of the sub-stream buffers storage location is empty; The reorganization buffer is set, and the reorganization buffer is used to store the sequence number of the audio media data packet after redundant elimination, the sequence number of the substream buffer for storing the audio media data packet and the substream of the audio media data packet in the storage The storage location in the stream buffer, initialize the size of the reorganization buffer to be N _j , the sequence numbers of the audio media data packets used for storing in the reorganization buffer after the redundancy removal operation are all initialized to -1, and the storage The sequence number of the sub-stream buffer of the audio media data packet and the storage location of the audio media data packet in the stored sub-stream buffer are all initialized to be empty; Set a jitter removal window, the jitter removal window is used to implement the jitter removal of the audio media data packet, initialize the size of the jitter removal window to W, W∈[W _min , W _max ]. 6. the network real-time audio conversation media data multipath redundant transmission method according to claim 4, is characterized in that, described redundant elimination operation comprises: S1, adopt polling mode to inquire about each described sub-flow buffer, obtain the latest received described audio media data packet i, extract the sequence number of described audio media data packet i, and record it as FSN _i ; S2. Perform a modulo operation on the serial number FSN _i of the audio media data packet i and the size N _j of the reassembly buffer to obtain a value m, that is, m=FSN _i modN _j , and query the value stored in the m position in the reassembly buffer The sequence number of the audio media data packet, denoted as J_FSN _m ; S3, if J_FSN _m =-1, then assign the value of FSN _i to J_FSN _m , store the serial number and the storage location of the sub-stream buffer where the audio media data packet i is located in the reassembly buffer m position, and perform the steps S1; S4. If J_FSN _m ≠-1, FSN _i >J_FSN _m , then assign the value of FSN _i to J_FSN _m , and store the sequence number and storage location of the substream buffer where the audio media data packet i is located in the reassembly buffer In position m, execute step S1; S5. If J_FSN _m ≠ -1 and FSN _i ≤ J_FSN _m , execute step S1. 7. the network real-time audio session media data multipath redundant transmission method according to claim 6, is characterized in that, described employing polling mode to inquire about each described sub-stream buffer, comprising: The receiving end of the audio media data regularly counts the reception and redundancy elimination of the audio media data packets in each of the sub-stream buffers, and obtains the polling priority sequences of the sub-stream buffers corresponding to different transmission paths; The receiving end of the audio media data queries each of the sub-stream buffers according to the polling priority sequence, and performs a redundant elimination operation. 8. the network real-time audio conversation media data multipath redundant transmission method according to claim 4, is characterized in that, described reorganization operation comprises: Dynamically adjust the size of the jitter removal window; The callback operation of the audio media data packet is performed according to the size of the jitter elimination window. 9. The network real-time audio conversation media data multipath redundant transmission method according to claim 8, is characterized in that, described dynamic adjustment shakes and eliminates window size, comprises: Record the time R(i) at which the audio media data packet i arrives at the reassembly buffer, and obtain the audio media according to the timestamp in the audio media data packet i or other flags used to record the sending time of the audio media data packet The sending time S(i) of the data packet i, calculate the delay jitter J(i)=R(i)-S(i) of the audio media data packet i; According to the audio media data packet that has arrived at the reassembly buffer, the expected delay jitter of the audio media data packet i newly arriving at the reassembly buffer is obtained Among them, N is a fixed value, i≥N, P(k) is a weighting coefficient, and $\underset{k=i-N}{\overset{i-1}{\&Sigma;}}P\left(k\right)=1;$ Calculate the class standard error of the delay jitter of the audio media data packet i $J_{ARmS\mathrm{E.}}\left(i\right)=\sqrt{\frac{1}{N}\underset{k=i-N+1}{\overset{i}{\&Sigma;}}{\&lsqb;J\left(k\right)-\stackrel{\&OverBar;}{J\left(k\right)}\&rsqb;}^2};$ Setting thresholds g ₁ and g ₂ is used to judge the change range of the jitter elimination window size, and g ₁ <g ₂ ; when Increase the size of the jitter removal window, if J _ARMSE (i)∈[g ₁ ,g ₂ ], the jitter removal window increases $2g_1+k|J\left(i\right)-\stackrel{\&OverBar;}{J\left(i\right)}|,$ If J _ARMSE (i)＞g ₂ , the jitter removal window increases $2g_2+\frac{k}{N}\underset{k=i-N+1}{\overset{i}{\&Sigma;}}|J\left(k\right)-\stackrel{\&OverBar;}{J\left(k\right)}|,$ where k∈[0.5,1]; when Reduce the size of the jitter removal window, if J _ARMSE (i)∈[g ₁ ,g ₂ ], the jitter removal window is reduced $2g_1+k|J\left(i\right)-\stackrel{\&OverBar;}{J\left(i\right)}|,$ If J _ARMSE (i)＞g ₂ , the jitter removal window is reduced $2g_2+\frac{k}{N}\underset{k=i-N+1}{\overset{i}{\&Sigma;}}|J\left(k\right)-\stackrel{\&OverBar;}{J\left(k\right)}|,$ where k∈[0.5,1]; If the adjusted jitter elimination window size W is smaller than the jitter elimination window minimum value W _min , then set the jitter elimination window size to the jitter elimination window minimum value W _min , if the adjusted jitter elimination window size W is greater than the jitter elimination window maximum value W _max , then the size of the jitter elimination window is set to the maximum value W _max of the jitter elimination window; When J _ARMSE (i)<g ₁ , the size of the jitter elimination window does not need to be changed; If the jitter removal window size is modified, the expected delay jitter of the changed audio media data packet i It is any integer in the interval [J(i)-4, J(i)+4], otherwise, no change is made. 10. The network real-time audio conversation media data multipath redundant transmission method according to claim 8, is characterized in that, described according to described jitter elimination window size, carries out described audio media data packet callback operation, comprises: Find the audio data packet whose serial number is FSN in the jitter elimination window, perform callback decoding according to the audio codec format and network transmission protocol negotiated by both parties in the real-time audio session, and find the audio media data packet serial number in the jitter elimination window. Audio media data packet of FSN+1; If found, then the audio media data packet serial number is FSN+1 audio media data packet callback decoding, continue to search the audio media data packet serial number is the audio media data packet of FSN+2, otherwise, judge whether there is surplus in the jitter elimination window, If there is no remaining, then the audio media data packet whose serial number is FSN+1 is called back and decoded again, and continues to search for the audio media data packet whose serial number is FSN+2, otherwise, wait for the preset time Continue to search after t; If the search is successful three times in a row, reduce the jitter elimination window to the size of one data frame, and judge whether the jitter elimination window is smaller than the minimum value W _min of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the minimum value of the jitter elimination window W _min , continue to perform the search operation; If three consecutive searches are unsuccessful, then increase the size of the jitter elimination window by one data frame, and judge whether the jitter elimination window is greater than the maximum value W _max of the jitter elimination window at this time, and if so, set the size of the jitter elimination window to the maximum value of the jitter elimination window value W _max , continue to perform the search operation.