CN117440155A - Coding method and device for virtual desktop image and related equipment - Google Patents

Abstract

According to the method, the device and the related equipment for encoding the virtual desktop image, the available bandwidth of a network for transmitting a code stream is determined in response to receiving data to be shared, the bandwidth occupied by the data to be shared is determined based on the difference between the available bandwidth and the bandwidth occupied by target data, the target data comprises desktop transmission protocol data, and the data to be shared is encoded based on the bandwidth occupied by the data to be shared, so that the code stream is obtained. The desktop transmission protocol data is data set based on the desktop transmission protocol for transmitting the data to be shared, and it can be seen that by distinguishing the data to be shared from the target data, the bandwidth occupied by the data to be shared can be more accurately determined, so that the code rate which is more matched with the actual state of the network can be determined.

Description

Coding method and device for virtual desktop image and related equipment

Technical Field

The present disclosure relates to the field of multimedia technologies, and in particular, to a method and apparatus for encoding a virtual desktop image, and related devices.

Background

The data sharing among different electronic devices can achieve the purposes of remote office of data and the like. Remote desktop technology is a data sharing technology.

Although most remote desktop technologies have been controlled for the transmission code rate, there is still a problem in that the transmission code rate does not match the network state. A transmission code rate that does not match the network state is understood to be significantly less or greater than the network bandwidth.

Disclosure of Invention

The application provides a method, a device and related equipment for encoding a virtual desktop image, and aims to solve the problem that the transmission code rate of the virtual desktop image is not matched with the network state.

In order to achieve the above object, the present application provides the following technical solutions:

a first aspect of the present application provides a method for encoding a virtual desktop image, applied to a server, including: in response to receiving the data to be shared, determining an available bandwidth of a network for transmitting the code stream; the data to be shared represents desktop image data to be displayed on a screen in a virtual desktop running on a server side; determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data, wherein the target data comprises desktop transmission protocol data, and the desktop transmission protocol data is data set based on the desktop transmission protocol, and the desktop transmission protocol is used for transmitting the data to be shared; encoding the data to be shared based on the bandwidth occupied by the data to be shared to obtain a code stream; and sending the code stream to the client. Therefore, by distinguishing the data to be shared from the target data, the bandwidth occupied by the data to be shared can be more accurately determined, so that the code stream which is more matched with the actual state of the network is determined.

Optionally, determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data includes: the method comprises the steps of determining a first residual bandwidth as a bandwidth occupied by data to be shared based on a difference between an available bandwidth and a bandwidth occupied by desktop transport protocol data, wherein the bandwidth occupied by the desktop transport protocol data is determined based on the available bandwidth, a predetermined frame rate and a data amount of the desktop transport protocol data. The bandwidth occupied by the desktop transmission protocol data, which is determined based on the available bandwidth, the frame rate and the data amount of the desktop transmission protocol data, is higher in accuracy because of conforming to the actual transmission mechanism of the desktop transmission protocol data, and further the bandwidth occupied by the data to be shared is higher in accuracy, and the code stream obtained by encoding is more matched with the actual state of the network.

Optionally, the target data further includes: retransmitted data;

based on the difference between the available bandwidth and the bandwidth occupied by the target data, determining the bandwidth occupied by the data to be shared, further comprises: and determining the second residual bandwidth as the bandwidth occupied by the data to be shared based on the difference between the first residual bandwidth and the retransmission occupied bandwidth. The bandwidth occupied by retransmission is considered when the bandwidth occupied by the data to be shared is determined, so that the accuracy of the bandwidth occupied by the data to be shared can be further improved, and the code stream obtained by encoding is further enabled to be matched with the actual state of the network.

Optionally, the data to be shared includes: data of a desktop image, the image including a text region and a non-text region;

encoding the data to be shared based on the bandwidth occupied by the data to be shared, including: determining a third residual bandwidth as the bandwidth occupied by the data to be shared based on the difference between the bandwidth occupied by the data to be shared and the bandwidth occupied by the individual encoding of the text region; determining the code rate of the non-text region based on the third residual bandwidth; the non-literal region is encoded based on a code rate of the non-literal region. When the bandwidth occupied by the data to be shared is determined, the bandwidth occupied by the individual coding of the text area is considered, so that the accuracy of the bandwidth occupied by the data to be shared can be further improved, and the code stream obtained by coding is further matched with the actual state of the network.

Optionally, determining the code rate of the non-text region based on the third residual bandwidth includes:

the code rate of the non-text region is determined based on a maximum of a first value and a second value, the second value being a preconfigured minimum code rate, the first value being determined based on a ratio of an area of the region of interest in the desktop image to a resolution of the desktop image, and a third residual bandwidth. The configuration of the minimum code rate is beneficial to ensuring the definition of the non-text area.

Optionally, the encoding the data to be shared based on the bandwidth occupied by the data to be shared includes: determining a code rate based on the bandwidth occupied by the data to be shared; and encoding the data to be shared based on the code rate.

Optionally, the encoding the data to be shared based on the bandwidth occupied by the data to be shared includes: determining a code rate based on a maximum value of a first value and a second value, wherein the first value is determined based on the bandwidth occupied by data to be shared, and the second value is a preset minimum code rate; and encoding the data to be shared based on the code rate.

Optionally, the data to be shared includes data of a desktop image; the first value is determined based on a bandwidth occupied by data to be shared, including: the first value is determined based on a ratio of an area of the region of interest in the desktop image to a resolution of the desktop image, and a bandwidth occupied by the data to be shared.

Optionally, determining the available bandwidth of the network for transmitting the code stream includes: performing first-stage detection to determine a packet sending rate, wherein the first-stage detection comprises the steps of sending n data packets, determining the packet sending rate based on the ratio of the n data packets to a first time length, wherein the first time length is determined based on transmission parameters acquired after the n data packets are sent, and n is an integer larger than zero; based on the first-stage detection result, performing second-stage detection, wherein the second-stage detection comprises: and transmitting m data packets at a packet transmitting rate within each preset effective duration, determining an average packet transmitting rate based on the ratio of the sum of the received adjacent data packets to the interval of the effective duration, determining an available bandwidth based on the average packet transmitting rate and the packet loss rate, and determining the sum of the received adjacent data packets based on transmission parameters acquired after transmitting the m data packets. The first stage of detection provides initial data for the second stage of detection, and the first stage of detection and the second stage of detection use different detection modes, so that a more accurate bandwidth estimation result can be obtained.

Optionally, based on the first-stage detection result, performing second-stage detection includes: in the ith detection period, based on the detection result of the first stage, carrying out second stage detection to obtain a detection result of the second stage of the ith detection period;

in the ith detection period, after performing the second stage detection based on the first stage detection result, the method further includes: in the (i+1) th detection period, executing first-stage detection to obtain a first-stage detection result of the (i+1) th detection period in response to the difference between the second-stage detection result of the (i) th detection period and the second-stage detection result of the (i-1) th detection period being greater than a preset threshold, and in the (i+2) th detection period, performing second-stage detection based on the first-stage detection result of the (i+1) th detection period; and in the (i+1) th detection period, performing second-stage detection in response to a difference between a second-stage detection result of the (i) th detection period and a second-stage detection result of the (i-1) th detection period being not greater than a preset threshold. The difference between the available bandwidths estimated by two adjacent detection periods is large, and the detection result is possibly inaccurate, so that the first-stage detection is performed again, and the available bandwidths obtained by detection are guaranteed to have higher accuracy.

A second aspect of the present application provides an encoding apparatus for a virtual desktop image, including:

the first determining module is used for determining available bandwidth of a network for transmitting a code stream in response to receiving data to be shared, wherein the data to be shared represents desktop image data to be displayed on a screen in a virtual desktop running on a server side;

the second determining module is used for determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data, wherein the target data comprises desktop transmission protocol data, the desktop transmission protocol data is data set based on the desktop transmission protocol, and the desktop transmission protocol is used for transmitting the data to be shared;

the coding module is used for coding the data to be shared based on the bandwidth occupied by the data to be shared to obtain a code stream, and sending the code stream to the client.

A third aspect of the present application provides a data sharing system, comprising: an encoder for performing the method of encoding a virtual desktop image provided in the first aspect of the present application.

A fourth aspect of the present application provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing a computer program to implement the method for encoding a virtual desktop image provided in the first aspect of the present application.

A fifth aspect of the present application provides a computer readable storage medium, which when executed by a processor of an electronic device, causes the electronic device to perform the method of encoding a virtual desktop image provided in the first aspect of the present application.

A sixth aspect of the present application provides a computer program product comprising computer programs/instructions which when executed by a processor implement the method of encoding a virtual desktop image provided in the first aspect of the present application.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is an exemplary diagram of a data sharing system disclosed in an embodiment of the present application;

FIG. 2 is an exemplary diagram of yet another data sharing system disclosed in an embodiment of the present application;

FIG. 3 is a flow chart of a method of encoding a virtual desktop image disclosed in an embodiment of the present application;

fig. 4 is a flowchart of estimating a network bandwidth to obtain an available bandwidth according to an embodiment of the present application;

fig. 5 is a flowchart of acquiring a code rate according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of an encoding device for a virtual desktop image according to an embodiment of the present disclosure;

fig. 7 is an exemplary diagram of an electronic device disclosed in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include, for example, "one or more" such forms of expression, unless the context clearly indicates to the contrary. It should also be understood that in embodiments of the present application, "one or more" means one, two, or more than two; "and/or", describes an association relationship of the association object, indicating that three relationships may exist; for example, a and/or B may represent: a alone, a and B together, and B alone, wherein A, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

The plurality of the embodiments of the present application refers to greater than or equal to two. It should be noted that, in the description of the embodiments of the present application, the terms "first," "second," and the like are used for distinguishing between the descriptions and not necessarily for indicating or implying a relative importance, or alternatively, for indicating or implying a sequential order.

The inventor finds that in the technical field of remote desktops in the research process, the reason for the problem that the transmission code rate is not matched with the network state due to the traditional code rate control is as follows:

1. lack of accurate and real-time perception of network status, that is, perception of network bandwidth is not timely enough and not accurate enough.

2. Conventional rate control techniques, such as quantization parameter (Quantization Parameter, QP) techniques, employ a qualitative control manner that adjusts the rate size trend based on network conditions (e.g., bandwidth), but cannot set a specific value of the transmission rate.

Based on the above, the embodiment of the application discloses a method and a device for encoding a virtual desktop image. The method and the device for encoding the virtual desktop image can be applied to, but not limited to, a data sharing system. In some implementations, the data sharing system is a sharing system of a virtual desktop. Virtual desktops (also referred to as remote desktops or cloud desktops, etc.) can be understood as desktop virtual machines that virtualize desktops, applications, and data that were originally running locally on an electronic device (e.g., a PC) through virtualization techniques.

In addition to the data sharing system, the method and the device for encoding the virtual desktop image disclosed in the embodiments of the present application may also be applied to other systems such as a video player, which are not limited herein.

Fig. 1 and fig. 2 are diagrams illustrating a data sharing system according to an embodiment of the present application, including: the system comprises a server side and a client side.

The server side comprises a display driver, a main control module and an encoder.

According to the functional division, the main control module comprises the following modules: the device comprises a content processing module, a parameter acquisition module, a coding initiation module and a bottom layer transmission module.

The encoder comprises a coding decision module and a coding module.

Assuming that the electronic device a shares a virtual desktop in the electronic device a with the electronic device B, the client runs in an operating system of the electronic device B.

In some implementations, as shown in fig. 1, the server runs in the electronic device a, and it is understood that the server runs in the operating system of the electronic device a. In other implementations, as shown in FIG. 2, the display driver runs in the operating system of electronic device A and the main control module and encoder run in a server connected to electronic device A, which may be a background server for applications such as a remote desktop.

The data sharing system shown in fig. 1 or fig. 2 is used to implement data sharing between different electronic devices, where the data sharing includes, but is not limited to, sharing of desktop image data to be displayed to a screen in a virtual desktop (referred to as a virtual desktop image for short). A virtual desktop image may be understood as data obtained as an image of content displayed in a desktop.

Based on the data sharing system shown in fig. 1 or fig. 2, the flow of the encoding method disclosed in the embodiment of the present application is shown in fig. 3, and includes the following steps:

and S11, displaying whether the content of the virtual desktop of the monitoring electronic equipment A is updated or not, and if so, sending the content (called screen content for short) of the virtual desktop of the electronic equipment A to a content processing module.

It is understood that the content of the virtual desktop is desktop image data to be displayed to the screen in the virtual desktop.

In some implementations, the screen content is delta content, i.e., content that is different after the screen content is updated than before the screen content is updated. In other implementations, the screen content is a full volume of content, i.e., the entire display content on the updated screen.

S12, the content processing module processes the received screen content and sends a screen content processing result to the coding initiating module.

In some implementations, processing includes integration and caching, among others. Integration is understood to mean rendering and mapping of the segmented image to obtain the pre-segmented image.

S13, the parameter acquisition module acquires the target parameters and transmits the target parameters to a coding decision module in the encoder.

The target parameters include bandwidth estimation, frame rate, and packet loss rate. It will be appreciated that the target parameter is indicative of the network state.

The parameter acquisition module estimates the network bandwidth to obtain a bandwidth estimation value. In some implementations, the bandwidth estimation is obtained using the flow shown in fig. 5. In other implementations, the ACK time difference for the packet pair is used to estimate the bandwidth estimate.

The closer the bandwidth estimation value is to the actual bandwidth, the more accurate the bandwidth estimation by the monitoring statistical module is, and the code rate of the code stream output by the encoder is more matched with the actual network condition.

S14, the coding initiation module sends a screen content processing result to a coding module in the coder.

It is understood that the order of execution of S12-S14 is not limiting.

S15, a coding decision module in the encoder acquires the code rate based on the target parameter and transmits the code rate to the encoder.

The code rate obtained in this step is the code rate used for transmitting the shared data to the client.

S16, the encoder encodes the screen content processing result by using the received code rate to obtain a code stream, and transmits the code stream to the encoding initiation module.

That is, the encoder transmits a code stream obtained by encoding the screen content processing result using the code rate to the encoding initiation module.

S17, the code initiating module transmits the code stream to the bottom layer transmission module after receiving the code stream.

S18, the bottom layer transmission module transmits the code stream to the client.

The bottom layer transmission module encapsulates the code stream into a format of UDP protocol or streaming media desktop transmission protocol.

As described above, assuming that the client operates on the electronic device B, the electronic device B can decode and display the code stream after receiving the code stream, so as to display the screen content of the electronic device a on the screen, thereby realizing the sharing of the screen content.

Fig. 4 is a flow of estimating a network bandwidth by a parameter module disclosed in an embodiment of the present application to obtain an estimated bandwidth value, where bandwidth detection is preconfigured to be continuous, that is, bandwidth detection is performed in each detection period. In some implementations, the user may modify the configuration by a switch, e.g., switch on indicates that the bandwidth probe is preconfigured to be persistent and switch off indicates that the bandwidth probe is preconfigured to be non-persistent.

The following steps are included in fig. 4:

the transmission parameters are used to obtain parameters required for the subsequent first stage detection and second stage detection.

S21, performing first-stage detection.

The purpose of the first stage probing is to make a rough estimate of the network bandwidth.

In some implementations, the first stage of detection is a packet-to-Model (Probe Gap Model) detection, specifically including the steps of: transmitting n pairs of data packets to a client, and collecting transmission parameters after the n pairs of data packets are transmitted, wherein the transmission parameters include but are not limited to: round-Trip Time (RTT), acknowledgement (ACK) packet interval at the Time of packet transmission, total amount of transmitted data, total amount of ACK data, and amount of lost retransmission data. Based on the logarithm n of the data packet and the first time length t1, the packet sending rate is obtained. The calculation of the packet sending rate Ce can be seen in formula (1):

Ce＝n/t1(1)。

the first duration may be preconfigured based on experience based on the acquired RTT and ACK inter-packet interval, an example of the first duration being one of 5-30 milliseconds.

S22, performing second-stage detection.

The second stage of detection is finer detection, and a more accurate bandwidth estimation result can be obtained.

In some implementations, the second stage detection is a packet group detection mode, and the specific flow of the second stage detection is as follows:

1. m packets are sent to the client at the packet sending rate Ce (i.e. the result of the first stage probing) for each active duration. The number of the effective duration is at least two. The length of the effective duration can be preconfigured, and the configuration basis comprises network state, resources consumed by detection and accuracy required by detection results.

Calculating R using formula (2):

r=l/Σ (Ti, i=1,..m) (2), where L is the sum of received adjacent packets and can be calculated based on the ACK inter-packet interval acquired after the second phase of packet transmission, and Ti is the interval between adjacent effective durations. R can be understood as an average packet rate.

2. After the packet loss rate is acquired from the parameter acquisition module, under the condition that the packet loss rate is greater than zero, calculating the available bandwidth by using the formula (3):

a=r× (1-p) (3), where a is the available bandwidth in equation (3).

The above S21-S23 are detection flows performed in a certain detection period (e.g., the first detection period). Because the first stage detection is only a rough estimate, the first stage detection may be performed only once in the first detection period, while in other detection periods, only the second stage detection is performed based on the results of the first stage detection.

Further, before starting the detection of a certain detection period, it is determined whether the difference between the available bandwidths acquired in the two latest detection periods exceeds the available bandwidth threshold, if so, it is indicated that the available bandwidths acquired in two consecutive detection periods have a large difference, and the detection result may be inaccurate, and because the second-stage detection is calculated based on the result Ce acquired in the first-stage detection, in order to improve the accuracy of the detection result, S22 needs to be executed again.

Based on the above principle, the flow shown in fig. 4 further includes the following steps in the ith detection period (i is an integer and greater than 1):

s23, judging whether the difference value between the first available bandwidth and the second available bandwidth (the available bandwidths estimated by two adjacent detection periods) is larger than an available bandwidth threshold value.

The first available bandwidth is available bandwidth obtained in the ith-1 detection period, the second available bandwidth is available bandwidth obtained in the ith-2 detection period, if the judgment result is yes, the difference between the available bandwidths estimated in the two adjacent detection periods is larger, and the detection result is possibly inaccurate, so that S21 is executed, if the judgment result is no, the difference between the available bandwidths estimated in the two adjacent detection periods is not large, and the first-stage detection is not needed, so that S22 is executed.

The method for estimating the available bandwidth shown in fig. 4 includes two stages of detection, the first stage of detection provides initial data for the second stage of detection, and the first stage of detection and the second stage of detection use different detection modes, so that a more accurate bandwidth estimation result can be obtained.

And because the detection result in the first stage can be reused, the method also has better instantaneity on the basis of more accurate bandwidth estimation result.

In yet another implementation, in parallel with the flow shown in fig. 4, the bandwidth detection is non-persistent, such as a user setting the switch to an off state. In this case, only S21-S22 are performed in each probing cycle.

Therefore, the bandwidth estimation method provided by the embodiment of the application not only can obtain more accurate bandwidth estimation results with higher real-time performance, but also can select the estimation algorithm more flexibly so as to meet the requirements in different scenes.

Fig. 5 is a flowchart of the coding decision module acquiring a code rate based on a target parameter, including the following steps:

s31, in response to receiving the screen content processing result, available bandwidth and frame rate are acquired from the parameter acquisition module.

The screen content processing result is to-be-shared data, and the to-be-shared data is combined with S11 to represent desktop image data to be displayed on a screen in a virtual desktop running on a server side.

In connection with the flows shown in fig. 3 and 4, the parameter acquisition module estimates the available bandwidth by the probing flow shown in fig. 4. The specific manner in which the parameter obtaining module obtains the frame rate is not described herein.

S32, smoothing the available bandwidth.

Considering that background traffic interference in an actual network link may cause fluctuation of key indexes such as bandwidth estimation and the like, the fluctuation is unfavorable for code rate adjustment of an encoder, so in the embodiment, the estimated bandwidth is smoothed.

In some implementations, smoothing is performed using a window filter based on an exponentially weighted average algorithm. Specifically, a window filter is called to update the available bandwidth into a window array, and the value in the window array is subjected to exponential weighted average calculation to obtain the smoothed available bandwidth.

S33, obtaining the code rate of the code based on the bandwidth after the smoothing processing and the obtained frame rate.

The specific implementation manner of S33 comprises the following steps:

s331, the residual bandwidth (called as first residual bandwidth for short) after the occupied bandwidth of the desktop transmission protocol data is removed from the available bandwidth is calculated.

It can be understood that the data transmission needs to be implemented based on a desktop transmission protocol, that is, the data is encapsulated based on the desktop transmission protocol, and the data to be encoded includes, in addition to the screen content processing result (i.e. the data to be shared), desktop transmission protocol data such as a packet header obtained by encapsulation. That is, the desktop transport protocol data is data generated based on a desktop transport protocol used to transmit the screen content processing result.

The desktop transport protocol data also occupies a part of bandwidth overhead, so in order to accurately calculate the code rate of the processing result of the transmission screen content, the remaining bandwidth (simply referred to as the first remaining bandwidth) after the bandwidth occupied by the desktop transport protocol data is removed from the available bandwidth needs to be calculated.

In some implementations, the first residual bandwidth is calculated using equation (4):

in the formula (4), tb1 represents a first residual bandwidth, bw represents a smoothed available bandwidth, f represents a frame rate, pl represents a data amount of a transport layer packet load (i.e., a screen content processing result), htl represents a data amount of a transport layer protocol header, hal represents a data amount of an application layer protocol header, and->Representing an upward rounding. It will be appreciated that Htl and Hal constitute the data volume of the desktop transport protocol data.

And S332, calculating the residual bandwidth (called as second residual bandwidth for short) after the bandwidth occupied by retransmission is removed from the first residual bandwidth.

In some implementations, the retransmission amount is collected and updated to the window filter, and the smoothed retransmission size estimate rt is obtained from the window filter.

The second residual bandwidth is calculated based on equation (5):

tb2=tb1-rt (5), tb2 represents the second remaining bandwidth.

It can be understood that, in the case where the retransmission mechanism is turned on, in order to obtain a more accurate code rate, the retransmission bandwidth overhead due to packet loss or the like is performed S332.

The following is a flow of calculating the coding rate of the screen content processing result.

S333, judging whether the definition of the characters in the screen content processing result is preferentially ensured, if so, executing S334, and if not, executing S336.

The screen content processing results in an image that, in some implementations, includes text regions and non-text regions. In a remote office scenario, the requirement for the definition of the text is high, so in this embodiment, if a function of preferentially ensuring the definition of the text in the screen content processing result is configured, the text region is encoded separately.

In some implementations, based on the state of the switch configured in advance, it is determined whether to preferentially ensure the definition of the text, e.g., the switch is in an on state, and the explanation preferentially ensures the definition of the text.

Video coding may be understood as video coding by considering the screen content processing result as a whole, and correspondingly, hybrid coding may be understood as coding the text region and the non-text region separately, i.e. conventional lossless coding is used for the text portion and video coding is used for the non-text portion. If the result of the interpretation in S333 is yes, it is explained that hybrid encoding is adopted. The present embodiment is mainly described with respect to video coding, and the flow of coding the text region will not be described again, here, it is assumed that the code rate (bit rate) used for coding the text region is nvb.

S334, calculating the bandwidth occupied by the second residual bandwidth removal text region code to obtain a third residual bandwidth.

In some implementations, a third residual bandwidth is calculated using equation (6):

tb3=tb2-nvb × 8*f (6), nvb denotes a code rate (bit rate) used for text region coding, and tb3 denotes a third residual bandwidth.

It is understood that the third remaining bandwidth is the bandwidth available for the non-text region.

S335 is performed after S334.

S335, calculating the code rate of the non-text area based on the third residual bandwidth.

In some implementations, in order to avoid the problem that the third residual bandwidth is too small, which results in too poor definition of the non-text region, a minimum code rate is set, so that the calculated code rate is not less than the minimum code rate, that is, the code rate of the non-text region is calculated by using equation (7): vb=max (tb 3, m×vr1) (7), m denotes a minimum code rate configured in advance, vr denotes a duty ratio of the non-text region, vr1=sum (Svi, i=1, 2, … n)/res denotes a resolution of the image, svi denotes an area of the i-th non-text region, sum (Svi, i=1, 2, … n) denotes a sum of areas of all the non-text regions. It can be understood that, before this step, the information of the text area and the non-text area in the image is obtained by means of text recognition or the like.

S336, calculating the code rate based on the second residual bandwidth.

In some implementations, in order to achieve too small bandwidth allocated for encoding video, a mode of setting a minimum code rate is also adopted, as shown in equation (8), vb=max (tb 2×vr2, m×vr 2) (8), vr2 represents the duty ratio of the region of interest, and the region of interest, such as a face, is identified in advance by using an identification algorithm, so as to ensure the definition of the region of interest. The calculation mode of vr2 is similar to that of vr1, namely the ratio of the sum of the areas of the interested areas to the image resolution.

It will be appreciated that the equation vr1 may be replaced by the ratio of the sum of the areas of the regions of interest in the non-text regions to the resolution of the image.

It can be understood that the manner of setting the minimum code rate described in S335 and S336 is an alternative manner, alternatively, the minimum code rate may not be set, and the code rate may be obtained based on the bandwidth occupied by the data to be shared.

Based on the flow shown in fig. 5, the transmission code rate of the data to be shared can be obtained, so that quantitative control can be realized on the code rate of the code stream, and compared with a qualitative control mode of adjusting the size trend, fine control of the code rate can be realized.

And because bandwidth overhead is considered for various types of data in the code stream respectively, the acquired transmission code rate of the data to be shared is more accurate and is more matched with the actual condition of the network, and under the condition that the network bandwidth fluctuates, the picture blocking caused by the reduction of the network bandwidth can be relieved, the continuity and the stability of pictures are maintained, and the full utilization of the network bandwidth can be realized.

Furthermore, the method can be compatible with independent coding of the text region and the non-text region, and can still ensure the refinement and precise control of the coding of the non-text region under the condition.

The region of interest is used as one of the calculation basis of the code stream, so that the region of interest is not excessively compressed, and higher definition is ensured.

The procedure provided in the above embodiment can be summarized as the following encoding method, including the steps of:

1. in response to receiving the data to be shared, an available bandwidth of a network for transmitting the codestream is determined.

As can be seen from the above processes, the code stream carries the data to be shared for transmission. The data to be shared represents desktop image data to be displayed on a screen in a virtual desktop running on a server side. That is, the content displayed on the virtual desktop is used as image data, and the image data is transmitted to the screen of the client to be displayed, so that the purpose of sharing the virtual desktop to the client is achieved.

In some implementations, the available bandwidth is determined using the flow shown in fig. 4.

2. And determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data.

As described above, the target data includes desktop transport protocol data, which is data set based on a desktop transport protocol used to transport data to be shared. Optionally, the desktop transport protocol data further includes retransmitted data.

The specific implementation of step 2 can be seen in S331-S332.

It will be appreciated that the foregoing differences are one specific implementation of the differences. Desktop transport protocol is a specific implementation of a communication protocol.

3. And encoding the data to be shared based on the bandwidth occupied by the data to be shared to obtain a code stream.

In some implementations, the text region and the non-text region are encoded separately, i.e., the text region is encoded separately, in which case the specific manner of encoding the non-text region can be seen in S334-S335.

In some implementations, the text region and the non-text region are not distinguished, but rather a region of interest is identified and a code rate is obtained based on the region of interest, an example being equation (8).

It will be appreciated that the two implementations described above may be used in combination, namely: and respectively encoding the text region and the non-text region, taking the region of interest as an example in the process of calculating the code rate of the non-text region, replacing tb3 with tb2 and vr3, wherein vr3 is the ratio of the sum of the areas of the region of interest in the non-text region to the image resolution.

4. And sending the code stream to the client.

According to the encoding method provided by the embodiment, the bandwidth occupied by the data to be shared can be acquired more accurately by distinguishing the data to be shared from the target data, so that the code rate which is more matched with the actual condition of the network is acquired.

Fig. 6 is a schematic structural diagram of an encoding device for a virtual desktop image according to an embodiment of the present application, including:

and the first determining module is used for determining available bandwidth of a network for transmitting a code stream in response to receiving data to be shared, wherein the code stream carries the data to be shared.

The second determining module is configured to determine, based on a difference between the available bandwidth and a bandwidth occupied by target data, the bandwidth occupied by the data to be shared, where the target data includes desktop transmission protocol data, the desktop transmission protocol data is data set based on a desktop transmission protocol, and the desktop transmission protocol is used to transmit the data to be shared.

And the coding module is used for coding based on the bandwidth occupied by the data to be shared, obtaining the code stream and sending the code stream to the client.

The coding device of the embodiment can obtain the code rate which is more matched with the actual condition of the network, and the code stream coded by using the code rate can not only relieve the picture blocking caused by the reduction of the network bandwidth under the condition that the network bandwidth fluctuates, but also keep the continuity and the stability of pictures, and can realize the full utilization of the network bandwidth.

The details of the functions of the above modules may be referred to the above method embodiments, and are not described herein.

The embodiment of the application also discloses an electronic device, as shown in fig. 7, the electronic device 700 includes: a processor 701 and a memory 702. The memory 702 is used for storing a computer program, and the processor 701 is used for executing the computer program to implement the method for writing a registry disclosed in any of the above embodiments.

In particular, the processor 701 may be one or more central processing units (central processing units, CPU).

The memory 702 is used for storing various types of data, and a computer program for implementing an operating system and the encoding method described in the above embodiments.

The encoding method described in the above embodiment is implemented when the processor 701 runs a computer program.

It is to be appreciated that the electronic device 700 can also include a multimedia component 703, an I/O interface 704, and a communication component 705.

The multimedia component 703 can include a screen and an audio component. Wherein the screen may be, for example, a touch screen, the audio component being for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 702 or transmitted through the communication component 705. The audio assembly further comprises at least one speaker for outputting audio signals. The I/O interface 704 provides an interface between the processor 701 and other interface modules, which may be a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is for wired or wireless communication between the electronic device 700 and other devices. Examples of wireless communications include Wi-Fi, bluetooth, near field communications (Near Field Communication, NFC for short), 2G, 3G, or 4G, or a combination of one or more thereof, and thus the respective communications component 705 may include: wi-Fi part, bluetooth part, NFC part.

The embodiment of the application also discloses a readable storage medium for storing a computer program, wherein the computer program realizes the encoding method disclosed in the previous embodiment when being executed by a processor. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.

The computer readable storage medium may include: a U-disk, a removable hard disk, a Read-only memory (ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other.

Claims (12)

1. The coding method of the virtual desktop image is characterized by being applied to a server and comprising the following steps of:

in response to receiving the data to be shared, determining an available bandwidth of a network for transmitting the code stream; the data to be shared represents desktop image data to be displayed on a screen in a virtual desktop running on the server;

determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by target data, wherein the target data comprises desktop transmission protocol data, the desktop transmission protocol data is data set based on a desktop transmission protocol, and the desktop transmission protocol is used for transmitting the data to be shared;

encoding the data to be shared based on the bandwidth occupied by the data to be shared to obtain a code stream;

and sending the code stream to a client.

2. The method of claim 1, wherein the determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and a target data occupied bandwidth comprises:

and determining a first residual bandwidth as the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the desktop transmission protocol data, wherein the bandwidth occupied by the desktop transmission protocol data is determined based on the available bandwidth, a predetermined frame rate and the data volume of the desktop transmission protocol data.

3. The method of claim 2, wherein the target data further comprises:

retransmitted data;

the determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data further comprises:

and determining a second residual bandwidth as the bandwidth occupied by the data to be shared based on the difference between the first residual bandwidth and the retransmission occupied bandwidth.

4. A method according to any of claims 1-3, wherein the data to be shared comprises: data of a desktop image, the image including a text region and a non-text region;

the encoding the data to be shared based on the bandwidth occupied by the data to be shared includes:

determining a third residual bandwidth as the bandwidth occupied by the data to be shared based on the difference between the bandwidth occupied by the data to be shared and the bandwidth occupied by the individual encoding of the text region;

determining the code rate of the non-text region based on the third residual bandwidth;

and encoding the non-literal area based on the code rate of the non-literal area.

5. The method of claim 4, wherein determining the code rate of the non-literal region based on the third residual bandwidth comprises:

and determining the code rate of the non-text region based on the maximum value of a first value and a second value, wherein the second value is a preset minimum code rate, and the first value is determined based on the ratio of the area of the region of interest in the desktop image to the resolution of the desktop image and the third residual bandwidth.

6. A method according to any of claims 1-3, wherein said encoding said data to be shared based on bandwidth occupied by said data to be shared comprises:

determining a code rate based on the bandwidth occupied by the data to be shared;

and encoding the data to be shared based on the code rate.

7. A method according to any of claims 1-3, wherein said encoding said data to be shared based on bandwidth occupied by said data to be shared comprises:

determining a code rate based on the maximum value of a first value and a second value, wherein the first value is determined based on the bandwidth occupied by the data to be shared, and the second value is a preset minimum code rate;

and encoding the data to be shared based on the code rate.

8. The method of claim 7, wherein the data to be shared comprises data of a desktop image;

the first value is determined based on the bandwidth occupied by the data to be shared, and includes:

the first value is determined based on a ratio of an area of a region of interest in the desktop image to a resolution of the desktop image, and a bandwidth occupied by the data to be shared.

9. An encoding device for a virtual desktop image, comprising:

the first determining module is used for determining available bandwidth of a network for transmitting a code stream in response to receiving data to be shared, wherein the data to be shared represents desktop image data to be displayed on a screen in a virtual desktop running on a server side;

the second determining module is used for determining the bandwidth occupied by the data to be shared based on the difference between the available bandwidth and the bandwidth occupied by the target data, wherein the target data comprises desktop transmission protocol data, the desktop transmission protocol data is data set based on a desktop transmission protocol, and the desktop transmission protocol is used for transmitting the data to be shared;

and the encoding module is used for encoding the data to be shared based on the bandwidth occupied by the data to be shared, obtaining the code stream and sending the code stream to the client.

10. A data sharing system, comprising: an encoder for performing the method of encoding a virtual desktop image as claimed in any of claims 1-8.

11. An electronic device, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement the method of encoding a virtual desktop image as claimed in any one of claims 1-8.

12. A readable storage medium for storing a computer program, wherein the computer program when executed by a processor implements the method of encoding a virtual desktop image as claimed in any one of claims 1-8.