CN116962613A - Data transmission method and device, computer equipment and storage medium - Google Patents

Abstract

The disclosure provides a data transmission method and device, a computer device and a storage medium, wherein the method is applied to a first device and comprises the following steps: transmitting the first media data to the second device at a preset first transmission frame rate; receiving a rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data; determining a second transmission frame rate of second media data to be transmitted according to the rendering frame rate; and transmitting the second media data to the second device based on the second transmission frame rate. According to the method and the device for displaying the media data, the sending frame rate is adjusted according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device is matched with the performance of the second device and/or the network environment, and the smoothness of media data display can be improved under the condition of balancing the device resources and/or the network environment as much as possible, and therefore user experience is improved.

Description

Data transmission methods and devices, computer equipment, storage media

Technical field

The present disclosure relates to but is not limited to the field of communications, and in particular, to a data transmission method and device, computer equipment, and storage media.

Background technique

Audio and video quality assurance is to set the audio and video transmission bit rate, target resolution and transmission frame rate within a reasonable range, and try to maintain the smoothness and high definition of audio and video communication under various terminal devices and network environments. .

In related technologies, when a terminal device receives and processes audio and video data, when the network condition changes, the first device will adjust the transmission code rate of the media data sent according to the network condition, and then adjust the transmission frame according to the transmission code rate. rate, there is no monitoring and feedback during the entire process of collecting and playing media data.

Contents of the invention

In view of this, embodiments of the present disclosure are expected to provide a data transmission method and device, computer equipment, and storage media.

In a first aspect, an embodiment of the present disclosure provides a data transmission method, applied to a first device, and the method includes:

Send the first media data to the second device at the preset first sending frame rate;

Receive the rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

Determine a second sending frame rate of the second media data to be sent according to the rendering frame rate;

Send the second media data to the second device based on the second sending frame rate.

In some embodiments, the method further includes:

Receive network parameters fed back by the second device based on the first media data;

Determine the transmission code rate and target resolution of the second media data according to the network parameters;

The sending the second media data to the second device based on the second sending frame rate includes:

Send the second media data of the target resolution to the second device based on the second sending frame rate and the sending code rate.

In some embodiments, the network parameters include: the maximum estimated code rate at the receiving end;

Determining the sending code rate and target resolution of the second media data according to the network parameters includes:

If the maximum estimated code rate of the receiving end is less than the preset transmission code rate threshold, adjust the target resolution of the second media data according to the maximum estimated code rate of the receiving end, and adjust the target resolution of the second media data according to the maximum estimated code rate of the receiving end. The value determines the sending code rate of the second media data;

If the maximum estimated code rate of the receiving end is greater than or equal to the preset transmission code rate threshold, the target resolution is maintained unchanged, and the transmission of the second media data is determined based on the maximum estimated code rate of the receiving end. Code rate.

In some embodiments, determining the second sending frame rate of the second media data to be sent based on the rendering frame rate includes:

If the rendering frame rate is not equal to the preset first sending frame rate, adjust the rendering frame rate, and use the adjusted rendering frame rate as the second sending frame rate of the second media data;

If the rendering frame rate is equal to the preset first sending frame rate, the rendering frame rate is used as the second sending frame rate of the second media data.

In some embodiments, adjusting the rendering frame rate and using the adjusted rendering frame rate as the second sending frame rate of the second media data includes:

Filter the rendering frame rate within the preset time period to obtain the filtered rendering frame rate;

Based on the filtered rendering frame rate and the preset frame rate adjustment value, the adjusted rendering frame rate is determined and the adjusted rendering frame rate is used as the second sending frame rate of the second media data; wherein, the second The sending frame rate is less than or equal to the preset first sending frame rate.

In a second aspect, embodiments of the present disclosure provide a data transmission method, applied to a second device, and the method includes:

Receive first media data sent by the first device at a preset first sending frame rate;

Determine a rendering frame rate based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data;

Feed back the rendering frame rate to the first device; wherein the rendering frame rate is used by the first device to determine the second sending frame rate of the second media data to be sent;

Receive second media data sent by the first device based on the second sending frame rate.

In some embodiments, determining the rendering frame rate based on the first media data includes:

Decode the received first media data to obtain first decoded data;

Perform post-processing on the first decoded data to obtain first post-processed data;

Render the first post-processing data to obtain a rendering frame rate.

In some embodiments, the method further includes:

Determine network parameters based on the first media data;

Feed back the network parameters to the first device; wherein the network parameters are used by the first device to determine the sending code rate and target resolution of the second media data;

Receive second media data of the target resolution sent by the first device based on the second sending frame rate and the sending code rate.

In a third aspect, an embodiment of the present disclosure provides a data transmission device, which is characterized in that it is applied to a first device, and the device includes:

A first sending module configured to send the first media data to the second device at a preset first sending frame rate;

A first receiving module configured to receive the rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

A first determination module, configured to determine the second sending frame rate of the second media data to be sent according to the rendering frame rate;

The second sending module is also configured to send the second media data to the second device based on the second sending frame rate.

In a fourth aspect, an embodiment of the present disclosure provides a data transmission device, which is characterized in that it is applied to a second device, and the device includes:

a second receiving module configured to receive the first media data sent by the first device at the preset first sending frame rate;

a second determination module, configured to determine a rendering frame rate based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

A first feedback module configured to feed back the rendering frame rate to the first device; wherein the rendering frame rate is used by the first device to determine the second sending frame rate of the second media data to be sent;

A third receiving module configured to receive second media data sent by the first device based on the second sending frame rate.

In a fifth aspect, an embodiment of the present disclosure provides a computer device, including:

processor;

Memory used to store instructions executable by the processor;

Wherein, the processor is configured to execute the method described in the first aspect or the second aspect.

In a sixth aspect, embodiments of the present disclosure provide a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect or the second aspect is implemented.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

In embodiments of the present disclosure, the first device adjusts the sending frame rate according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device adapts to the performance of the second device and/or the network environment, and can be achieved as much as possible. It is possible to improve the smoothness of media data display while balancing device resources and/or network environment, thereby improving user experience.

It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and do not limit the present disclosure.

Description of the drawings

The accompanying drawings herein are incorporated into and constitute a part of this specification. They illustrate embodiments consistent with the disclosure and, together with the description, serve to explain the technical solutions of the disclosure.

Figure 1 is a schematic diagram 1 of the implementation flow of a data transmission method provided by an embodiment of the present disclosure;

Figure 2 is a schematic diagram 2 of the implementation flow of a data transmission method provided by an embodiment of the present disclosure;

Figure 3 is an interaction diagram of a data transmission method provided by an embodiment of the present disclosure;

Figure 4 is a data transmission module relationship diagram provided by an embodiment of the present disclosure;

Figure 5A is an interactive schematic diagram 1 of a data transmission method provided by an embodiment of the present disclosure;

Figure 5B is an interactive schematic diagram 2 of a data transmission method provided by an embodiment of the present disclosure;

Figure 6 is a schematic diagram of a data transmission device applied to a first device provided by an embodiment of the present disclosure;

Figure 7 is a schematic diagram of a data transmission device applied to a second device provided by an embodiment of the present disclosure;

FIG. 8 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are further elaborated below in conjunction with the accompanying drawings and examples. The described embodiments should not be regarded as limiting the present disclosure. Those of ordinary skill in the art will All other embodiments obtained without creative efforts belong to the scope of protection of this disclosure.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or a different subset of all possible embodiments, and Can be combined with each other without conflict.

If similar descriptions of "first/second" appear in the application documents, add the following explanation: The terms "first/second/third" involved in this disclosure are only used to distinguish similar objects and do not represent the specificity of the objects. Ordering, it is understood that "first/second/third" may interchange the specific order or sequence where permitted, so that the embodiments of the disclosure described herein can be used in other ways than those illustrated or described herein. Implemented in other order.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing the disclosure only and is not intended to be limiting of the disclosure.

Figure 1 is a schematic flow chart of the implementation of the data transmission method provided by an embodiment of the present disclosure. It is applied to the first device. As shown in Figure 1, the method includes the following steps:

S101. Send the first media data to the second device at the preset first sending frame rate;

S102. Receive the rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

S103. Determine the second sending frame rate of the second media data to be sent according to the rendering frame rate;

S104. Send the second media data to the second device based on the second sending frame rate.

In this embodiment of the present disclosure, the first device may be a group including multiple terminal devices, or it may be a single terminal device; wherein, the first device may be, but is not limited to, a terminal device with coding and data transmission capabilities, such as Webcams, PCs, laptops, smartphones, tablets and portable wearable devices. The second device may be, but is not limited to, a device with a display function, such as a monitor, an electronic whiteboard, an intelligent interactive tablet, a screen, a television, a video conferencing terminal, etc. The first device and the second device may be connected in a wireless or wired manner, and the communication connection methods involved include: Internet, local area network, Bluetooth, mobile hotspot (Wi-Fi) or ZigBee protocol. The first media data includes video, audio and video, etc.

Taking the first media data as a video as an example, the first device can convert the collected analog signal into a digital signal, and the first device can encode the digital signal to obtain the first media data. The encoding standards that can be adopted include: H264, H265, vp8, vp9, av1, etc. The sending frame rate is the number of frames transmitted per second (Frames Per Second, FPS).

In the embodiment of the present disclosure, the first device may use the collection frame rate of the media data as the preset first sending frame rate, and send the first media data to the second device at the preset first sending frame rate. It should be noted that different scenarios may correspond to different preset first sending frame rates. For example, when the solution of the embodiment of the present disclosure is applied to a remote monitoring scenario, the preset first sending frame rate may be 15 FPS; When the solution of the embodiment of the present disclosure is applied to an online video scenario, the preset first sending frame rate may be 30 FPS.

In the embodiment of the present disclosure, the first device can use the real-time transport protocol (Real-time Transport Protocol, RTP) to encapsulate the first media data to obtain the RTP data, and then send the obtained RTP data to the preset first transmission frame rate. Secondary device. In the embodiment of the present disclosure, after receiving the first media data, the second device decodes the first media data, then renders and displays the decoded media data, and determines the current rendering frame rate. In this embodiment of the present disclosure, the second device may include a graphics processing unit (GPU) to render the decoded media data, and then the hardware in the second device may present the rendered content on the screen.

In this embodiment of the disclosure, the second device feeds back the rendering frame rate of the first media data to the first device. For example, the second device encapsulates the rendering frame rate using the Realtime Transport Control Protocol (RTCP). RTCP data, and then feeds back the obtained RTCP data to the first device.

In the embodiment of the present disclosure, after the first device receives the rendering frame rate fed back by the second device, it can determine the second sending frame rate of the second media data to be sent based on the rendering frame rate, and use the second sending frame rate to determine the second sending frame rate of the second media data to be sent. Send second media data. Taking the media data as audio and video as an example, the first media data and the second media data may be partial consecutive frames of the same audio and video file, or may be different audio and video files.

It should be noted that when the network is congested, the second device cannot receive the first media data in time and cannot display it in time. At this time, the rendering frame rate of the first media data within the same transmission time is less than the preset first sending frame rate. ; When the network is open, the rendering frame rate of the first media data within the same transmission time can be equal to the preset first sending frame rate. In addition, the display of media data will be affected by the performance of the decoding algorithm and rendering performance. For example, when decoding and/or rendering are time-consuming and the sending frame rate is high, it is usually difficult to play at the preset first sending frame rate. On the one hand, the rendering frame rate of the first media data is also a reflection of the performance of the second device. An excessively high sending frame rate can easily cause congestion on the receiving end display and waste resources of the first device; while when the sending frame rate is low, When the rendering performance and decoding algorithm performance of the receiving end device are good, it will cause a waste of resources on the receiving end device; therefore, adjusting the sending frame rate based on the feedback of the receiving end device can balance the device resources and/or the network environment as much as possible Improve the smoothness of media data display, thereby improving user experience.

However, the related technology only adjusts the sent media data according to the network conditions, and does not monitor the entire process of media data from collection to rendering and display, and provides feedback, that is, the performance of the receiving end device and the decoding algorithm of the receiving end device. Performance is not reflected in the entire media data adjustment process.

In this regard, in embodiments of the present disclosure, the first device adjusts the sending frame rate according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device adapts to the performance and/or network environment of the second device, It can improve the smoothness of media data display while balancing device resources and/or network environment as much as possible, thereby improving user experience.

In some embodiments, the method further includes:

Receive network parameters fed back by the second device based on the first media data;

Determine the transmission code rate and target resolution of the second media data according to the network parameters;

The sending the second media data to the second device based on the second sending frame rate includes:

Send the second media data of the target resolution to the second device based on the second sending frame rate and the sending code rate.

In this embodiment of the present disclosure, the first device sends the first media data to the second device at a preset first sending frame rate. Based on the received first media data, the second device obtains network parameters and feeds them back to the first device; wherein the network parameters include: packet loss rate (Loss Rate), receiver estimated maximum bitrate (Receiver Estimated Max Bitrate, REMB ) and round-trip delay (Round-Trip Time, RTT), etc.

In this embodiment of the present disclosure, after receiving the network parameters fed back by the second device, the first device determines the transmission code rate and target resolution of the second media data based on the network parameters; and then based on the second transmission frame rate, transmission code rate Send second media data at the target resolution to the second device.

When the network is congested, the second device may not be able to receive the media data in time and therefore cannot display it in time. Correspondingly, the media data displayed on the second device may not be smooth. That is, the network environment may also affect the display of the media data.

In this regard, the first device of the present disclosure receives the network parameters fed back by the second device based on the first media data to determine the transmission code rate and target resolution of the second media data. Among them, the code rate is the number of data bits transmitted per unit time during data transmission, and the unit is bps (bit per second). The code rate is related to the quality of media data. Correspondingly, the greater the code rate, the clearer the media data picture and the higher the picture quality. . When the network is smooth, the sending code rate value can be appropriately increased or maintained; when the network is congested, the sending code rate value can be appropriately reduced. Among them, resolution represents the number of pixels on each media data image, which affects the image size and is proportional to the image size. For example, the higher the resolution, the larger the image, and the greater the bandwidth required to transmit media data; the lower the resolution, the smaller the image, and the smaller the bandwidth required to transmit media data. When the network is smooth, the target resolution value can be appropriately increased or maintained; when the network is congested, the target resolution value can be appropriately reduced. Therefore, the embodiment of the present disclosure adjusts the code rate and resolution according to network parameters to help adjust the display of the second media data.

It can be understood that in the embodiment of the present disclosure, the first device determines the sending frame rate, sending code rate and target resolution according to the network parameters and the rendering frame rate fed back by the second device, so that the sending of the second media data can be consistent with the second The rendering capabilities of the device are adapted to the current network environment, which can improve the smoothness of media data display while balancing device resources and network resources as much as possible, thereby improving user experience.

In some embodiments, the network parameters include: the maximum estimated code rate at the receiving end;

Determining the sending code rate and target resolution of the second media data according to the network parameters includes:

If the maximum estimated code rate of the receiving end is less than the preset transmission code rate threshold, adjust the target resolution of the second media data according to the maximum estimated code rate of the receiving end, and adjust the target resolution of the second media data according to the maximum estimated code rate of the receiving end. The value determines the sending code rate of the second media data;

If the maximum estimated code rate of the receiving end is greater than or equal to the preset transmission code rate threshold, the target resolution is maintained unchanged, and the transmission of the second media data is determined based on the maximum estimated code rate of the receiving end. Code rate.

In the embodiment of the present disclosure, the network parameters include the maximum estimated code rate at the receiving end. The maximum estimated code rate at the receiving end can represent the available bandwidth of the second device and can represent whether the current network environment is congested. For example, when the bandwidth is large, the data transmission speed is faster, and the screen display of the second device is smooth; when the bandwidth is small, the screen displayed by the second device is prone to delays, mosaics, screen distortion, and other phenomena.

In this embodiment of the present disclosure, the first device compares the maximum estimated code rate of the receiving end with a preset transmission code rate threshold, where the preset transmission code rate threshold can represent the lower limit of the transmission code rate of media data at the current resolution. . If the maximum bit rate estimated by the receiving end is lower than the preset sending bit rate threshold, if it is video data, the image will be unclear and the video quality will be very poor; if it is audio data, the sound quality loss will be serious. Correspondingly, if the maximum value of the bit rate estimated by the receiving end is greater than or equal to the preset transmission bit rate threshold, it indicates that the media data quality is good. However, if the maximum value of the bit rate estimated by the receiving end is smaller, it will cause frame loss and playback of the media data. Therefore, the present disclosure can adjust the target resolution and transmission code rate according to the relationship between the maximum estimated code rate at the receiving end and the preset transmission code rate threshold to adapt to the current network environment.

In one embodiment, when the maximum value of the bit rate estimated by the receiving end is less than the preset transmission code rate threshold, it indicates that the network environment is poor. At this time, the target resolution of the second media data can be adjusted according to the maximum value of the bit rate estimated by the receiving end. , for example, reduce the target resolution of the second media data, and determine the transmission code rate of the second media data based on the maximum estimated code rate of the receiving end, for example, appropriately reduce the value of the transmission code rate based on the maximum estimated code rate of the receiving end. As the sending code rate of the second media data.

In one embodiment, when the maximum estimated code rate at the receiving end is greater than or equal to the preset transmission code rate threshold, it indicates that the network environment is good. At this time, the target resolution can be maintained unchanged. According to the estimated code rate at the receiving end, The maximum value determines the transmission code rate of the second media data. For example, based on the maximum value of the code rate estimated by the receiving end, the value of the transmission code rate is appropriately increased as the transmission code rate of the second media data.

It can be understood that in the embodiment of the present disclosure, when the maximum estimated code rate at the receiving end is less than the preset transmission code rate threshold, the target resolution of the second media data is adjusted according to the maximum estimated code rate at the receiving end to reduce, for example, Image data occupies memory, thereby saving bandwidth occupied by media data transmission, thus balancing the impact of network conditions on the display of the second media data on the second device as much as possible. In addition, in the embodiment of the present disclosure, when the maximum estimated code rate at the receiving end is greater than or equal to the preset transmission code rate threshold and the resolution is constant, the transmission code rate of the second media data is determined based on the maximum estimated code rate at the receiving end, so that The transmission of the second media data can be adapted to the current network environment, and the clarity of media data display can be improved as much as possible while balancing network resources.

In some embodiments, determining the second sending frame rate of the second media data to be sent based on the rendering frame rate includes:

If the rendering frame rate is not equal to the preset first sending frame rate, adjust the rendering frame rate, and use the adjusted rendering frame rate as the second sending frame rate of the second media data;

If the rendering frame rate is equal to the preset first sending frame rate, the rendering frame rate is used as the second sending frame rate of the second media data.

As mentioned above, the network environment is different, and the relationship between the rendering frame rate and the preset first sending frame rate is different; in addition, the rendering frame rate can reflect the performance of the second device. When the performance of the second device is poor, the first media The rendering frame rate of the data may be lower than the preset first sending frame rate. When the performance of the second device is good, the rendering frame rate of the first media data may also be equal to the preset first sending frame rate.

In this regard, in the embodiment of the present disclosure, if the rendering frame rate is not equal to the preset first sending frame rate, that is, the rendering frame rate is less than the preset first sending frame rate, it means that the performance of the second device and/or the network environment is poor, The preset first sending frame rate is no longer appropriate. At this time, the appropriate sending frame rate can be determined as much as possible under the current device performance and network environment to improve the display effect of the second media data. For example, it can be greater than the rendering frame rate. The value is used as the second sending frame rate; if the rendering frame rate is equal to the first sending frame rate, it means that the performance of the second device and the network environment are good, and the sending frame rate of the first device can be maintained unchanged.

It can be understood that in the embodiments of the present disclosure, if the rendering frame rate is not equal to the preset first sending frame rate, an appropriate sending frame rate can be determined as much as possible under the current device performance and/or network environment to improve the media Data display is smooth; if the rendering frame rate is equal to the first sending frame rate, the sending frame rate of the first device can be maintained unchanged to maintain the stability of media data display.

In some embodiments, adjusting the rendering frame rate and using the adjusted rendering frame rate as the second sending frame rate of the second media data includes:

Filter the rendering frame rate within the preset time period to obtain the filtered rendering frame rate;

Based on the filtered rendering frame rate and the preset frame rate adjustment value, the adjusted rendering frame rate is determined and the adjusted rendering frame rate is used as the second sending frame rate of the second media data; wherein, the second The sending frame rate is less than or equal to the preset first sending frame rate.

It should be noted that filtering algorithms include: Wiener filtering, Kalman filtering, etc., and the embodiments of the present disclosure do not limit the filtering algorithm used. In the embodiment of the present disclosure, the first device filters the rendering frame rate fed back by the second device, which can reduce the occurrence of abnormal values in the rendering frame rate. Then adjust the sending frame rate based on the filtered rendering frame rate and the preset frame rate adjustment value. For example, the filtered rendering frame rate and the preset frame rate adjustment value can be added, and the added value is used as the second media The second sending frame rate of the data; it may also be that the filtered rendering frame rate is multiplied by the preset frame rate adjustment value, and the multiplied value is used as the second sending frame rate of the second media data.

It should be noted that, since the rendering frame rate is not equal to the preset first sending frame rate, the rendering frame rate is usually smaller than the preset first sending frame rate, and this indicates the performance of the second device and/or the current The network environment is not good, so the second sending frame rate is also less than or equal to the preset first sending frame rate.

It can be understood that in the embodiment of the present disclosure, adjusting the sending frame rate based on the filtered rendering frame rate and the preset frame rate adjustment value can improve the accuracy of confirming the second sending frame rate, so as to improve the accuracy of the second media data. display effect.

Figure 2 is a schematic flowchart 2 of the implementation of the data transmission method provided by an embodiment of the present disclosure. It is applied to the second device. As shown in Figure 2, the method includes the following steps:

S201. Receive the first media data sent by the first device at the preset first sending frame rate;

S202. Determine the rendering frame rate based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

S203. Feed back the rendering frame rate to the first device; wherein the rendering frame rate is used by the first device to determine the second sending frame rate of the second media data to be sent;

S204. Receive second media data sent by the first device based on the second sending frame rate.

Taking the first media data as a video as an example, the first device can convert the collected analog signal into a digital signal, and the first device can encode the digital signal to obtain the first media data. In the embodiment of the present disclosure, the first device may use the collection frame rate of the media data as the preset first sending frame rate.

In this embodiment of the present disclosure, the first device may use a real-time transmission protocol to send the data obtained after encapsulating the first media data to the second device at a preset first sending frame rate. In the embodiment of the present disclosure, after receiving the first media data, the second device may decode the first media data, then render and display the decoded media data, and determine the current rendering frame rate. In this embodiment of the present disclosure, the second device may include a graphics processor to render the decoded media data, and then the hardware in the second device may present the rendered content on the screen.

In this embodiment of the present disclosure, the second device may use the real-time transmission control protocol to encapsulate the rendering frame rate of the first media data and feed it back to the first device.

In the embodiment of the present disclosure, after the first device receives the rendering frame rate fed back by the second device, it can determine the second sending frame rate of the second media data to be sent based on the rendering frame rate, and use the second sending frame rate to determine the second sending frame rate of the second media data to be sent. Send second media data.

It should be noted that the rendering frame rate of the first media data within the same transmission time is less than or equal to the preset first sending frame rate. In the embodiment of the present disclosure, the rendering frame rate can be based on the relationship between the rendering frame rate and the preset first sending frame rate. , learn the impact of device performance and/or network environment on media data display, and adjust the sending frame rate of the second media data to be sent.

It should be noted that since the display of media data is affected by the network environment, the rendering frame rate of the first media data within the same transmission time is less than or equal to the preset first sending frame rate. In addition, the display of media data will be affected by the performance of the decoding algorithm and rendering performance. For example, when decoding and/or rendering are time-consuming and the sending frame rate is high, it is usually difficult to play at the preset first sending frame rate. On the one hand, the rendering frame rate of the first media data is also a reflection of the performance of the second device. An excessively high sending frame rate can easily cause congestion on the receiving end display and waste resources of the first device; while when the sending frame rate is low, When the rendering performance and decoding algorithm performance of the receiving end device are good, it will cause a waste of resources on the receiving end device; therefore, adjusting the sending frame rate based on the feedback of the receiving end device can balance the device resources and/or the network environment as much as possible Improve the smoothness of media data display, thereby improving user experience.

In the embodiment of the present disclosure, the first device adjusts the sending frame rate according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device adapts to the performance and/or network environment of the second device, and can be as possible as possible. Improve the smoothness of media data display while balancing device resources and/or network environment, thereby improving user experience.

In some embodiments, determining the rendering frame rate based on the first media data includes:

Decode the received first media data to obtain first decoded data;

Perform post-processing on the first decoded data to obtain first post-processed data;

Render the first post-processing data to obtain a rendering frame rate.

It is understandable that there is a large amount of redundant information in the media data, and encoding is to compress the media data and remove the redundant information in the media data. In this embodiment of the present disclosure, the second device may decode the received first media data, that is, decompress the encoded data. In the embodiment of the present disclosure, the second device may perform post-processing on the decoded data, that is, processing frames in the media data that may cause quality jumps. In the embodiment of the present disclosure, the second device can render the post-processed data, display the post-processed data, and obtain the rendering frame rate.

In the embodiment of the present disclosure, the second device decodes, post-processes, and renders the received first media data, and can reduce the occurrence of flickering due to media data quality problems when displaying the media data after identifying the media data, and improve the quality of the media data. The smoothness of media data display.

In some embodiments, the method further includes:

Determine network parameters based on the first media data;

Feed back the network parameters to the first device; wherein the network parameters are used by the first device to determine the sending code rate and target resolution of the second media data;

Receive second media data of the target resolution sent by the first device based on the second sending frame rate and the sending code rate.

In this embodiment of the present disclosure, the first device sends the first media data to the second device at a preset first sending frame rate. The second device obtains network parameters based on the received first media data and feeds them back to the first device.

In this embodiment of the present disclosure, after receiving the network parameters fed back by the second device, the first device determines the transmission code rate and target resolution of the second media data based on the network parameters; and then based on the second transmission frame rate, transmission code rate Send second media data at the target resolution to the second device.

In the embodiment of the present disclosure, the first device determines the sending frame rate, sending code rate and target resolution according to the network parameters and the rendering frame rate fed back by the second device, so that the sending of the second media data can be consistent with the rendering capabilities of the second device and the Adapting to the current network environment can improve the smoothness of media data display while balancing device resources and network resources as much as possible, thereby improving user experience.

Figure 3 is an interaction diagram of a data transmission method provided by an embodiment of the present disclosure. As shown in Figure 3, it includes the following steps:

S301. The first device sends the first media data to the second device at the preset first sending frame rate;

S302. The second device determines the rendering frame rate based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data;

S303. The second device feeds back the rendering frame rate to the first device;

S304. The first device determines the second sending frame rate of the second media data to be sent according to the rendering frame rate;

S305. The first device sends the second media data to the second device based on the second sending frame rate.

In the embodiment of the present disclosure, the first device adjusts the sending frame rate according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device adapts to the performance and/or network environment of the second device, and can be as possible as possible. Improve the smoothness of media data display while balancing device resources and/or network environment, thereby improving user experience.

Figure 4 is a data transmission module relationship diagram provided by an embodiment of the present disclosure. As shown in Figure 4, the media data source uses the media data transmission adjustment module L401 based on the real-time transmission protocol sending end L402 (i.e., the first device of the embodiment of the present disclosure). ) is sent to the real-time transmission protocol receiving end L403 (ie, the second device of the embodiment of the present disclosure) using the real-time transmission protocol; wherein the media data source includes the first media data of the present disclosure, and the media data transmission adjustment module L401 belongs to the embodiment of the present disclosure. Function module in the first device. After the second device receives the first media data, the network parameter statistics module L404 of the second device obtains network parameters including the packet loss rate, the maximum estimated code rate of the receiving end, the round-trip delay, etc. based on the received first media data. ; At the same time, the decoding module L405 of the second device decodes the received first media data, and then processes the frames that may cause quality jumps in the decoded media data through the post-processing module L406, and then processes the post-processing through the rendering module L407. The processed media data is rendered and displayed, and further, the rendering frame rate statistics module L408 of the second device obtains the current rendering frame rate.

After obtaining the network parameters and the rendering frame rate of the first media data, the second device feeds back the network parameters and the rendering frame rate based on the real-time transmission control protocol through the real-time transmission control protocol sending terminal L409 (ie, the second device in the embodiment of the present disclosure). To the real-time transmission control protocol receiving end L410 (ie, the first device in the embodiment of the present disclosure).

After receiving the network parameters and rendering frame rate fed back by the second device, the sending end feedback module L411 in the first device adjusts the sending parameters of the second media data to be sent in the media data source through the media data sending adjustment module L401; The adjusted transmission parameters include the second transmission frame rate, transmission code rate, resolution, etc. according to the embodiment of the present disclosure.

In the embodiment of the present disclosure, the first device adjusts the sending frame rate according to the rendering frame rate fed back by the second device, so that the sending frame rate of the first device matches the performance of the second device and/or the network environment, and can be used as much as possible in the second device. Improve the smoothness of media data display while balancing device resources and/or network environment, thereby improving user experience.

Figure 5A is an interactive schematic diagram 1 of a data transmission method provided by an embodiment of the disclosure, focusing on the second device. As shown in Figure 5A, the receiving end (i.e., the second device of the embodiment of the disclosure) receives the data transmitted by the sending end ( That is, after the media data adjustment module L401 of the first device of the embodiment of the present disclosure sends the first media data based on the real-time transmission protocol, the network parameter statistics module L404 of the receiving end obtains network parameters based on the received first media data. At the same time, the decoding module L405 of the receiving end decodes the received first media data, and then the post-processing module L406 of the receiving end processes the frames that may cause quality jumps in the decoded media data, and then the rendering module L407 of the receiving end performs post-processing The final media data is rendered and displayed. Further, the rendering frame rate statistics module L408 of the receiving end obtains the current rendering frame rate.

The receiving end feeds back the rendering frame rate and network parameters of the first media data to the sending end through the receiving end feedback module L501 based on the real-time transmission control protocol. After the sending end feedback module L411 of the sending end receives the rendering frame rate and network parameters fed back by the receiving end, the media data transmission adjustment module L401 of the sending end determines the second sending frame of the second media data to be sent according to the rendering frame rate and network parameters. rate, transmission code rate, and target resolution, and sends the second media data of the target resolution to the receiving end at the second transmission frame rate and transmission code rate.

It should be noted that the media data transmission adjustment module L401, network parameter statistics module L404, decoding module L405, post-processing module L406, rendering module L407, rendering frame rate statistics module L408, and transmitter feedback module L411 in Figure 5A are the same as in Figure 4 Modules with the same identification in can belong to modules with the same function.

In the embodiment of the present disclosure, the receiving end decodes, post-processes, and renders the received first media data, and can reduce the occurrence of flickering due to media data quality problems when displaying the media data after identifying the media data, and improve the quality of the media data. Display fluency.

Figure 5B is an interactive schematic diagram 2 of a data transmission method provided by an embodiment of the disclosure, focusing on the first device. As shown in Figure 5B, the media data transmission control module in the sending end (ie, the first device of the embodiment of the disclosure) L502 encapsulates and sends the first media data to the second device based on the real-time transmission protocol. After the receiving terminal (i.e., the second device in the embodiment of the present disclosure) receives the first media data, the network parameter and rendering frame rate statistics module L503 of the receiving terminal determines the rendering frame rate and network parameters of the first media data, and sets the first media data to the first media data. The rendering frame rate and network parameters of a media data are fed back to the sending end by the receiving end feedback module L501 based on the real-time transmission control protocol.

After receiving the rendering frame rate and network parameters fed back by the receiving end, the sending end feedback module L411 of the sending end determines whether the rendering frame rate is not equal to the sending frame rate. If the rendering frame rate is not equal to the sending frame rate, the filtering module L504 of the sending end determines whether the rendering frame rate is not equal to the sending frame rate. The frame rate is filtered to reduce the occurrence of abnormal values in the rendering frame rate. Then the sending frame rate control module L505 of the sending end determines the second sending frame rate of the second media data based on the filtered rendering frame rate, and then determines the second sending frame rate of the second media data based on the resolution of the sending end. The rate control module L506 determines the target resolution of the second media data, and at the same time, the code rate control module L507 of the sending end determines the sending code rate of the second media data; if the rendering frame rate is equal to the sending frame rate, the resolution control module of the sending end L506 determines the target resolution of the second media data, and at the same time, the code rate control module L507 of the transmitting end determines the transmission code rate of the second media data.

Further, the media data transmission control module L502 of the sending end sends the second media data of the target resolution to the receiving end at the second sending frame rate and the sending code rate.

It should be noted that the sending end feedback module L411 and the receiving end feedback module L501 in Figure 5B may belong to modules with the same function as the modules with the same identification in Figures 4 and 5A. The media data transmission adjustment module L401 of the transmitter in Figure 5A may include the media data transmission control module L502, filtering module L504, transmission frame rate control module L505, resolution control module L506, and code rate control module L507 of the transmitter in Figure 5B. The network parameter and rendering frame rate statistics module L503 of the receiving end in Figure 5B may include the network parameter statistics module L404, decoding module L405, post-processing module L406, rendering module L407, and rendering frame rate statistics module L408 of the receiving end in Figure 5A.

In the embodiment of the present disclosure, the sending end determines the sending frame rate, sending code rate and target resolution based on the rendering frame rate and network parameters fed back by the receiving end, so that the sending of the second media data can be consistent with the performance of the second device and the current network environment. Adaptation can improve the smoothness of media data display while balancing device resources and network resources as much as possible, thereby improving user experience.

Figure 6 is a schematic diagram of a data transmission device applied to the first device provided by an embodiment of the present disclosure. As shown in Figure 6, the data transmission device 600 includes:

The first sending module 601 is used to send the first media data to the second device at the preset first sending frame rate;

The first receiving module 602 is configured to receive the rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is the display frequency at which the second device displays the first media data. ;

The first determination module 603 is configured to determine the second sending frame rate of the second media data to be sent according to the rendering frame rate;

The second sending module 604 is configured to send the second media data to the second device based on the second sending frame rate.

In some embodiments, the device further includes:

A fourth receiving module, configured to receive network parameters fed back by the second device based on the first media data;

A third determination module, configured to determine the transmission code rate and target resolution of the second media data according to the network parameters;

The second sending module 604 is further configured to send the second media data of the target resolution to the second device based on the second sending frame rate and the sending code rate.

In some embodiments, the network parameters include: the maximum estimated code rate at the receiving end;

The third determination module is also configured to adjust the target resolution of the second media data according to the maximum estimated code rate of the receiving end if the maximum estimated code rate of the receiving end is less than the preset transmission code rate threshold, And determine the transmission code rate of the second media data according to the maximum estimated code rate of the receiving end; if the maximum estimated code rate of the receiving end is greater than or equal to the preset transmission code rate threshold, maintain the target resolution If the rate remains unchanged, the sending code rate of the second media data is determined according to the maximum estimated code rate of the receiving end.

In some embodiments, the first determination module 603 is also configured to adjust the rendering frame rate if the rendering frame rate is not equal to the preset first sending frame rate, and use the adjusted rendering frame The rate is used as the second sending frame rate of the second media data; if the rendering frame rate is equal to the preset first sending frame rate, the rendering frame rate is used as the second sending frame rate of the second media data. frame rate.

In some embodiments, the first determination module 603 is also configured to filter the rendering frame rate within the preset time period if the rendering frame rate is not equal to the preset first sending frame rate to obtain the filtered based on the filtered rendering frame rate and the preset frame rate adjustment value, determine the adjusted rendering frame rate and use the adjusted rendering frame rate as the second sending frame of the second media data rate, wherein the second sending frame rate is less than or equal to the preset first sending frame rate.

Figure 7 is a schematic diagram of a data transmission device applied to a second device provided by an embodiment of the present disclosure. As shown in Figure 7, the data transmission device 700 includes:

The second receiving module 701 is used to receive the first media data sent by the first device at the preset first sending frame rate;

The second determination module 702 is used to determine the rendering frame rate based on the first media data; wherein the rendering frame rate is the display frequency of the second device displaying the first media data;

The first feedback module 703 is used to feed back the rendering frame rate to the first device; wherein the rendering frame rate is used by the first device to determine the second sending frame rate of the second media data to be sent. ;

The third receiving module 704 is configured to receive second media data sent by the first device based on the second sending frame rate.

In some embodiments, the second determination module 702 is also used to decode the received first media data to obtain the first decoded data; to post-process the first decoded data to obtain the first decoded data. Post-processing data; rendering the first post-processing data to obtain a rendering frame rate.

In some embodiments, the device further includes:

A fourth determination module, configured to determine network parameters based on the first media data;

A second feedback module, configured to feed back the network parameters to the first device; wherein the network parameters are used by the first device to determine the transmission code rate and target resolution of the second media data;

The fifth receiving module is configured to receive the second media data of the target resolution sent by the first device based on the second sending frame rate and the sending code rate.

Figure 8 is a schematic diagram of a hardware entity of a computer device in an embodiment of the present disclosure. As shown in Figure 8, the hardware entity of the computer device 800 includes: a processor 801, a communication interface 802 and a memory 803, where the processor 801 usually controls Overall operation of computer device 800. The communication interface 802 can enable the computer device to communicate with other terminals or servers through a network.

The memory 803 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the computer device 800 (for example, image data, audio data, voice communication data and Video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be carried out between the processor 801, the communication interface 802 and the memory 803 through the bus 804. The processor 801 is used to execute some or all steps in the above method.

Correspondingly, embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, some or all of the steps in the above method are implemented.

It should be pointed out here that the above description of the storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

It will be understood that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic associated with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that in various embodiments of the present disclosure, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present disclosure. The implementation process constitutes any limitation. The above serial numbers of the embodiments of the present disclosure are only for description and do not represent the advantages and disadvantages of the embodiments.

It should be noted that, in this document, the terms "comprising", "comprises" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device that includes a series of elements not only includes those elements, It also includes other elements not expressly listed or inherent in the process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article or apparatus that includes that element.

In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components may be combined, or can be integrated into another system, or some features can be ignored, or not implemented. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be electrical, mechanical, or other forms. of.

The units described above as separate components may or may not be physically separated; the components shown as units may or may not be physical units; they may be located in one place or distributed to multiple network units; Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure can be all integrated into one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above method embodiments can be completed through hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the execution includes: The steps of the above method embodiment; and the aforementioned storage media include: mobile storage devices, read-only memory (Read Only Memory, ROM), magnetic disks or optical disks and other various media that can store program codes.

Alternatively, if the above-mentioned integrated units of the present disclosure are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure can be embodied in the form of a software product in essence or that contributes to related technologies. The computer software product is stored in a storage medium and includes a number of instructions to enable a computer. A computer device (which may be a personal computer, a server, a network device, etc.) executes all or part of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: mobile storage devices, ROMs, magnetic disks or optical disks and other media that can store program codes.

The above are only embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, and should are covered by the protection scope of this disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims (12)

1. A method of data transmission, for use with a first device, the method comprising:

transmitting the first media data to the second device at a preset first transmission frame rate;

receiving a rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data;

determining a second transmission frame rate of second media data to be transmitted according to the rendering frame rate;

and transmitting the second media data to the second device based on the second transmission frame rate.

2. The method according to claim 1, wherein the method further comprises:

receiving network parameters fed back by the second device based on the first media data;

determining a sending code rate and a target resolution of the second media data according to the network parameters;

the transmitting the second media data to the second device based on the second transmission frame rate includes:

and transmitting the second media data of the target resolution to the second device based on the second transmission frame rate and the transmission code rate.

3. The method of claim 2, wherein the network parameters comprise: the receiving end estimates the maximum value of the code rate;

The determining, according to the network parameter, the sending code rate and the target resolution of the second media data includes:

if the maximum value of the estimated code rate of the receiving end is smaller than a preset sending code rate threshold value, adjusting the target resolution of the second media data according to the maximum value of the estimated code rate of the receiving end, and determining the sending code rate of the second media data according to the maximum value of the estimated code rate of the receiving end;

and if the maximum value of the estimated code rate of the receiving end is larger than or equal to the preset sending code rate threshold value, maintaining the target resolution unchanged, and determining the sending code rate of the second media data according to the maximum value of the estimated code rate of the receiving end.

4. The method of claim 1, wherein the determining a second transmission frame rate of the second media data to be transmitted according to the rendering frame rate comprises:

if the rendering frame rate is not equal to the preset first sending frame rate, adjusting the rendering frame rate, and taking the adjusted rendering frame rate as a second sending frame rate of the second media data;

and if the rendering frame rate is equal to the preset first sending frame rate, taking the rendering frame rate as a second sending frame rate of the second media data.

5. The method of claim 4, wherein the adjusting the rendering frame rate and taking the adjusted rendering frame rate as the second transmission frame rate of the second media data comprises:

filtering the rendering frame rate in the preset duration to obtain a filtered rendering frame rate;

determining an adjusted rendering frame rate based on the filtered rendering frame rate and a preset frame rate adjustment value, and taking the adjusted rendering frame rate as a second sending frame rate of the second media data; the second sending frame rate is smaller than or equal to the preset first sending frame rate.

6. A data transmission method, applied to a second device, the method comprising:

receiving first media data sent by a first device at a preset first sending frame rate;

determining a rendering frame rate based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data;

feeding back the rendering frame rate to the first device; wherein the rendering frame rate is used for the first device to determine a second transmission frame rate of second media data to be transmitted;

second media data transmitted by the first device based on the second transmission frame rate is received.

7. The method of claim 6, wherein the determining a rendering frame rate based on the first media data comprises:

decoding the received first media data to obtain first decoded data;

post-processing is carried out on the first decoded data to obtain first post-processed data;

and rendering the first post-processing data to obtain a rendering frame rate.

8. The method of claim 6, wherein the method further comprises:

determining a network parameter based on the first media data;

feeding back the network parameters to the first device; wherein the network parameter is used for the first device to determine the sending code rate and the target resolution of the second media data;

and receiving second media data of the target resolution, which is transmitted by the first device based on the second transmission frame rate and the transmission code rate.

9. A data transmission apparatus for use with a first device, the apparatus comprising:

the first sending module is used for sending the first media data to the second equipment at a preset first sending frame rate;

a first receiving module, configured to receive a rendering frame rate fed back by the second device based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data;

A first determining module, configured to determine a second transmission frame rate of second media data to be transmitted according to the rendering frame rate;

and the second sending module is further used for sending the second media data to the second device based on the second sending frame rate.

10. A data transmission apparatus for use with a second device, the apparatus comprising:

the second receiving module is used for receiving first media data sent by the first equipment at a preset first sending frame rate;

a second determination module for determining a rendering frame rate based on the first media data; wherein the rendering frame rate is a display frequency at which the second device displays the first media data;

a first feedback module configured to feed back the rendering frame rate to the first device; wherein the rendering frame rate is used for the first device to determine a second transmission frame rate of second media data to be transmitted;

and a third receiving module, configured to receive second media data sent by the first device based on the second sending frame rate.

11. A computer device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to perform the method of any one of claims 1 to 5 or claims 6 to 8.

12. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the method of any of claims 1 to 5 or of claims 6 to 8.