RU2785897C1 - Data redundancy application algorithm for the loudplay cloud gaming platform - Google Patents

Abstract

FIELD: computing technology.

SUBSTANCE: invention relates to the field of computing technology and network technologies. The technical result is achieved due to the forward error correction during the transmission of video stream data from the server to the client, performed by adding such additional service data to the transmitted information that allows errors to be corrected without an additional request, wherein when video stream data is being encoded, 2p verification characters are added on the server side to the block of m-characters; all m characters of the initial block are used in calculating each verification symbol, therefore, Fourier transforms are applied to accelerate the process of decoding the received data on the client; wherein a redundancy level changing algorithm depending on the duration of session, frame type, connection type, and the current state of the channel is applied; the streaming session is started with a video stream bitrate parameter equal to 10 Mbit/s; 100% redundancy is added during the first 30 seconds of the session, which is necessary to stabilise the data channel; and after 30 seconds, the redundancy level in the video stream is reduced to 20%, excluding the IDR-frame, the redundancy whereof is always maintained at 100%.

EFFECT: possibility of simultaneous increase in the smoothness of the image and up to a 30% reduction in the graphic image processing delay time.

15 cl, 3 dwg

Description

The claimed group of inventions relates to the field of computer technology and network technologies, in particular to the processing of tasks in a cloud service.

Currently, a large number of people are faced with the problem of low personal computer performance when running new computer games or resource-intensive graphic applications, such as design / CAD systems. The cost of a modern powerful computer is about $1000-1500 and is a luxury item in many emerging markets. Upgrading a computer is technically not always possible and requires special knowledge and free time.

In almost every area of consumer electronics, the battle for hardware performance is coming to an end. This has already happened in game consoles, it is happening in smartphones and in home computers. The development of data transmission networks leads to a change in the consumption model (from the purchase of local capacity to a subscription to cloud resources). For example, the availability of data transfer has led to the active development and reduction in the cost of cloud storage services. Already now, the large internal memory of the smartphone is not so important for consumers, since there are cloud storages where you can upload your videos and photos.

The main hardware problem of user devices today is the work with graphical computing. Previously, the transfer of this process to the cloud was hindered by restrictions in the data network. Today, these problems are solved in fixed broadband data networks and 5G mobile networks.

Server graphics cards are also regularly upgraded, GPU virtualization technologies are being actively developed, and new video stream transmission algorithms based on the use of data redundancy are being developed and implemented. Compared to standard protocols based on the re-request of lost data, the use of redundancy significantly improves the quality, smoothness and stability of the video stream on high-loss data networks, such as WIFI networks.

Currently, various methods and systems are known for implementing computer games for computers of various capacities. The closest analogue is the GeForce Now service (see https://gfii.ru/). The disadvantage of which is the low stability of the connection on bad networks, as well as restrictions on the number of supported games and applications.

The objective of the presented group of inventions is to eliminate at least some of the disadvantages inherent in the prior art with the achievement of a technical result in the form of providing the possibility of simultaneously increasing the smoothness of the image and reducing the delay time for processing a graphic image by up to 30%.

To achieve the above technical result, a group of inventions is claimed, including:

platform for virtualization and streaming of applications and computer games, including:

at least one server running a computer game or application, receiving control commands related to the game or application transmitted from a client controlled by the user, and in response executing the game program code to render a sequence of video images as a result of executing the game or application, when this is a continuous transmission of the video stream in real time to the client device,

at the same time, the application or game is launched on the server in a virtualization environment, which allows you to separate user applications and games from each other, eliminating their mutual influence, in addition, the virtualization environment allocates a separate virtual computer to each user, on which the user works with the application or plays the game, in the same way as if the game or application were run on a high-powered personal computer,

at least one client made for at least one user device, which is an application that receives a video stream transmitted from the server in real time, decodes it and displays it to the user,

while the client receives control commands from the input device and transmits them to the server,

at the same time, the client performs the functions of preliminary analysis of the state of the data transmission channel from the server to the client and preliminary analysis of the power of the user computer to issue recommendations to the user or automatically configure the parameters of the video stream and the parameters of the user device,

at the same time, after analyzing the power of the user device and the state of the data transmission channel from the client to the server, a user notification is generated about compliance or a warning about the mismatch between the power of the user device or the state of the data transmission channel from the client to the server with the minimum quality requirements,

that provides a game or application experience, similar to if the game or application would be launched on a high-performance personal computer and recommendations are displayed for setting up a user video stream, a user device to improve the quality of the game process or application experience, or automatically adjusting the video stream parameters and user settings devices to achieve the best quality of the game process or work with the application,

at the same time, when transmitting video stream data from the server to the client, forward error correction occurs by adding to the transmitted information such additional service data that allows errors to be corrected without an additional request. Practical application has shown that the addition of redundancy makes it possible to correct multiple bursts of errors.

On the server side, when encoding video stream data, 2p parity symbols are added to the block of m-symbols. When calculating each check symbol, all m symbols of the source block are used, so Fourier transforms are used to speed up the process of decoding the received data on the client.

To ensure high quality streaming on lossy networks, an algorithm for changing the redundancy level is used, depending on the session duration, frame type, connection type and current channel state: a streaming session is started with a video stream bitrate parameter of 10 Mbps. During the first 30 seconds of the session, 100% redundancy is added to stabilize the data link. After 30 seconds, the level of redundancy in the video stream is reduced to 20%, except for the IDR-frame, where the redundancy always remains at 100%. Further, depending on the type of client connection to the data network, the level of redundancy depends as follows:

1. If the Wifi signal level is less than 80%, then an additional 20% redundancy is added to the video stream;

2. If the signal type is Wifi with a frequency of 2.4 GTZ, then an additional 20% redundancy is added to the video stream;

Further, depending on the indications of the current losses on the channel (loss-rate) of data transmission, the redundancy value changes:

1. If loss-rate = 1…3%, 20% redundancy is added to the video stream;

2. If loss-rate = 4…5%, 40% redundancy is added to the video stream;

3. If loss-rate = 6…10%, 60% redundancy is added to the video stream;

4. If loss-rate = 11…20%, 100% redundancy is added to the video stream;

5. If loss-rate = 21…30%, 150% redundancy is added to the video stream;

6. If loss-rate = 31% and higher, 200% redundancy is added to the video stream;

The claimed group of inventions also includes a machine-implemented method for virtualization and streaming of applications and computer games, comprising the following steps:

- at least one user runs a client application on his user device, the client application displays an image that is produced on the virtualization server, the client application performs a preliminary analysis of the state of the data transmission channel from the client to the server, analysis of the power of the user computer to issue recommendations to the user or automatic configuration parameters of the video stream and parameters of the user device, when analyzing the power of the user device and the state of the data transmission channel from the client to the server, a user notification is generated about compliance or a warning about the inconsistency of the power of the user device or the state of the data transmission channel from the client to the server with the minimum quality requirements that provide game or application process, similar to if the game or application were run on a high-powered personal computer and output all recommendations on setting up the video stream of the user video stream, user device to improve the quality of the gaming process or work with the application, or the parameters of the video stream and the parameters of the user device are automatically configured to achieve the best quality of the gaming process or work with the application, then the user controls the computer game by entering control commands or application, while the commands from the client application to the server are transmitted via a data channel,

- at least one server running a computer game or application receives control commands related to the game or application transmitted from a client controlled by the user, and in response executes the game program code to render a sequence of video images as a result of executing the game or application, moreover, continuous transmission of the video stream in real time to the client device,

at the same time, the application or game is launched on the server in a virtualization environment that allows you to separate user applications and games from each other, excluding their mutual influence, in addition, the virtualization environment allocates a separate virtual computer for each user, on which the user works with the application or plays the game, similarly that if the game or application were run on a high-powered personal computer,

moreover, the reception, decoding and display of the video stream for the user is performed on the client application;

at the same time, when transmitting video stream data from the server to the client, forward error correction occurs by adding to the transmitted information such additional service data that allows you to correct errors without an additional request, and

on the server side, when encoding video stream data, 2p check symbols are added to a block of m-symbols, when calculating each check symbol, all m symbols of the source block are used, therefore, Fourier transforms are used to speed up the process of decoding the received data on the client,

at the same time, an algorithm for changing the redundancy level is applied, depending on the session duration, frame type, connection type and current channel state: the streaming session is started with a video stream bitrate parameter of 10 Mbps, 100% redundancy is added during the first 30 seconds of the session, necessary to stabilize the data transmission channel, after 30 seconds the level of redundancy in the video stream decreases to 20%, with the exception of the IDR-frame, where the redundancy always remains at 100%, then, depending on the type of client connection to the data transmission network, the level of redundancy depends in the following way:

- If the Wifi signal level is less than 80%, then an additional 20% redundancy is added to the video stream;

- If the signal type is Wifi with a frequency of 2.4 GHz, then an additional 20% redundancy is added to the video stream;

further, depending on the indications of the current losses on the channel (loss-rate) of data transmission, the redundancy value changes:

- If loss-rate = 1…3%, 20% redundancy is added to the video stream;

- If loss-rate = 4…5%, 40% redundancy is added to the video stream;

- If loss-rate = 6…10%, 60% redundancy is added to the video stream;

- If loss-rate = 11…20%, 100% redundancy is added to the video stream;

- If loss-rate = 21…30%, 150% redundancy is added to the video stream;

- If loss-rate = 31% or higher, 200% redundancy is added to the video stream.

In this case, the user device is a phone, tablet, PC, set-top box with various operating systems. In this case, the input device is a keyboard, mouse, game joysticks, manipulators, touch screen. In this case, the video game or application image is captured and encoded into the H.264 or H.265 transport video stream using a graphics card installed in the server, capture is performed in the memory of the graphics card, encoding is performed using a graphics card chip, the functionality of which is capture and packaging images to h.264 and h.265 transport video stream. At the same time, the data transmission channel from the server to the client uses 4G or 5G mobile data transmission technology. In this case, the request for a key frame (I-frame) in the h.246 and h.265 transport video stream by the client on the server is made only during the initial connection or when large losses occur on the data transmission channel (40-60%). At the same time, at least one server on which a computer game or application is launched, which receives control commands related to the game or application transmitted from the client controlled by the user, and in response executes the program code of the game to render a sequence of video images as a result of the execution of the game or application , while the application or game is launched on the server in a virtualization environment that allows you to separate user applications and games from each other, eliminating their mutual influence, in addition, the virtualization environment allocates a separate virtual computer to each user, on which the user works with the application or plays the game , similar to if the game or application were run on a high-powered personal computer, with at least one server used in a multi-access edge computing architecture, while in a multi-access edge computing (MEC) architecture, two or more servers can be combined using by the software-defined storage (sds) system. At the same time, the applications that are launched are resource-intensive applications. At the same time, the video stream parameters (bitrate, fps and screen resolution) are changed based on the forecast of the neural network, and the neural network is trained based on the collected data of user sessions (server performance, data transfer network performance, user device performance, game type) and a subjective assessment of the quality of the gaming process.

The declared platform for virtualization and streaming of applications and computer games meets the following requirements:

- registration of new users, authorization of existing users;

- checking the connection with several game servers and selecting the nearest game server;

- implementation of a demo mode (subscribers' access to a game demo session with a time limit without prepayment);

- the ability to select different server configurations and select the appropriate tariff. Cloud server rental.

- checking the quality of the user's connection throughout the entire interval of the game session and, in the case of the minimum allowable value, notifying the user about it;

- support for the function of adaptive change of video stream parameters, which changes the parameters of the video stream - bitrate (bitrate) and frame rate (FPS) of the broadcast, depending on the user's Internet connection capabilities;

- the ability to launch any applications and games using cloud technology, broadcast the corresponding video stream to the user's device;

- support for statistics and analysis of the time spent by users in various applications and games;

- the possibility of flexible modification of the user interface;

- providing protection against interception of commands from the user device;

- storage of user data (applications, files, games);

- ensuring round-the-clock monitoring of the technical condition of the platform components;

- the claimed technology provides a minimum delay, imperceptible to the end player;

- functioning of the system in 24x7 mode and availability of at least 99.97%.

In FIG. 1, under the following symbols, a description of the platform modules is presented:

pos. 1 - Server part of the platform;

pos. 2 - Virtualization system of the server part of the platform;

pos. 3 - Guest operating system/virtual machine;

pos. 4 - Disk space for storing guest operating system data;

pos. 5 - The module responsible for collecting and aggregating statistical data on the operation of the platform and the product, as well as monitoring the performance of the technical part of the platform and business indicators of the product;

pos. 6 - Billing and virtual infrastructure management module. Storage of client profiles and authentication data;

pos. 7 - Game showcase of the service. Contains a description of the service for users and the entrance to the personal account;

pos. 8 - Module responsible for accepting payments from users;

pos. 9 - Module that provides separation of infrastructure from the Internet;

pos. 10 - Game or application running on a guest virtual machine;

pos. 11 - Module for capturing, encoding and streaming images;

pos. 12 - Module for intercepting control signals transmitted from the client part of the platform;

pos. 13 - Client part of the platform;

pos. 14 - User device (PC, tablet, smartphone, set-top box);

pos. 15 - Software installed on the user device, providing authorization and authentication of the client;

pos. 16 - Software installed on the user device, which provides the reception of a video stream from the platform, as well as the transmission of control signals to the server part;

pos. 17 - Registration and authorization of users;

pos. 18 - Checking the parameters of the data transmission channel;

pos. 19 - Video stream transmission;

pos. 20 - Transmission of control signals.

In FIG. 2 shows the scheme of data recovery.

In FIG. 3 shows the data structure.

terminology used.

Streaming (English streaming) - image transmission through a data network. Virtualization is the provision of a set of computing resources or their logical association, abstracted from the hardware implementation, and at the same time providing logical isolation from each other of computing processes running on one physical resource.

An example of the use of virtualization is the ability to run multiple operating systems on one computer: while each of the instances of such guest operating systems works with its own set of logical resources (processor, RAM, storage devices), the provision of which from a common pool available at the hardware level is controlled by host operating system - hypervisor. Data transmission networks and storage networks can also be virtualized. Bitrate (from the English. bitrate) - the number of bits used to transmit / process data per unit of time. It is customary to use bitrate when measuring the effective transmission rate of a data stream over a channel, that is, the minimum size of a channel that can pass this stream without delay.

Claims (57)

1. Platform for virtualization and streaming of applications and computer games, including at least one server that runs a computer game or application that receives control commands related to the game or application transmitted from a client controlled by the user, and in response executes the program code of the game to render a sequence of video images resulting from the execution of a game or application, at the same time, there is a continuous transmission of the video stream in real time to the client device, at the same time, the application or game is launched on the server in a virtualization environment, which allows you to separate user applications and games from each other, eliminating their mutual influence, in addition, the virtualization environment allocates a separate virtual computer to each user, on which the user works with the application or plays the game, in the same way as if the game or application were run on a high-powered personal computer, at least one client made for at least one user device, which is an application that receives a video stream transmitted from the server in real time, decodes it and displays it to the user, while the client receives control commands from the input device and transmits them to the server, at the same time, the client performs the functions of preliminary analysis of the state of the data transmission channel from the server to the client and preliminary analysis of the power of the user computer to issue recommendations to the user or automatically configure the parameters of the video stream and the parameters of the user device, at the same time, after analyzing the power of the user device and the state of the data transmission channel from the client to the server, a user notification is generated about compliance or a warning about the inconsistency of the power of the user device or the state of the data transmission channel from the client to the server with the minimum quality requirements that ensure the gaming process or the process of working with application, just as if the game or application were run on a high-performance personal computer, and recommendations are displayed to configure the user's video stream, user device to improve the quality of the game process or application experience, or the video stream parameters and user device parameters are automatically adjusted to achieve the best quality of the gaming experience. process or work with the application, at the same time, when transmitting video stream data from the server to the client, forward error correction occurs by adding to the transmitted information such additional service data that allows you to correct errors without an additional request, and on the server side, when encoding video stream data, 2p check symbols are added to a block of m-symbols, when calculating each check symbol, all m symbols of the source block are used, therefore, Fourier transforms are used to speed up the process of decoding the received data on the client, at the same time, an algorithm for changing the redundancy level is applied, depending on the session duration, frame type, connection type and current channel state: the streaming session is started with the video stream bitrate parameter equal to 10 Mbps, 100% redundancy is added during the first 30 seconds of the session, necessary to stabilize the data transmission channel, after 30 s the redundancy level in the video stream decreases to 20%, with the exception of the IDR-frame, the redundancy in which always remains at 100%, then, depending on the type of client connection to the data network, the level of redundancy depends in the following way: if the Wi-Fi signal level is less than 80%, then an additional 20% redundancy is added to the video stream; if the signal type is Wi-Fi with a frequency of 2.4 GTZ, then an additional 20% redundancy is added to the video stream; further, depending on the indications of the current losses on the channel (loss-rate) of data transmission, the redundancy value changes: - with loss-rate = 1…3%, 20% redundancy is added to the video stream; - with loss-rate = 4…5%, 40% redundancy is added to the video stream; - with loss-rate = 6…10%, 60% redundancy is added to the video stream; - with loss-rate = 11…20%, 100% redundancy is added to the video stream; - with loss-rate = 21…30%, 150% redundancy is added to the video stream; - with loss-rate = 31% and higher, 200% redundancy is added to the video stream. 2. The platform for virtualization and streaming of applications and computer games according to claim 1, characterized in that the user device is a phone, tablet, PC, set-top box with various operating systems. 3. The platform for virtualization and streaming of applications and computer games according to claim 1, characterized in that the input device is a keyboard, mouse, game joysticks, manipulators, touch screen. 4. Platform for virtualization and streaming of applications and computer games according to claim 1, characterized in that - capturing and encoding an image of a video game or application into an H.264 or H.265 transport video stream is performed using a graphics card installed in the server, - capture is made from the video memory of the graphic card, encoding is performed using a graphic card chip, the functionality of which is to capture and pack the image into the h.264 and h.265 transport video stream. 5. Platform for virtualization and streaming of applications and computer games according to claim 1 or 4, characterized in that the data transmission channel from the server to the client uses 4G or 5G mobile data transmission technology. 6. The platform for virtualization and streaming of applications and computer games according to claim 1 or 4, characterized in that the request for a key frame (I-frame) in the h.246 and h.265 transport video stream by the client on the server is made only upon initial connection or when large losses on the data transmission channel (40-60%). 7. Platform for virtualization and streaming of applications and computer games according to claim 1 or 4, characterized in that it contains at least one server running a computer game or application, receiving control commands related to the game or application transmitted from a client controlled by the user, and in response executing the game program code to render a sequence of video images as a result of executing the game or application, when In this case, the application or game is launched on a server in a virtualization environment that allows you to separate user applications and games from each other, excluding their mutual influence, in addition, the virtualization environment allocates a separate virtual computer for each user, on which the user works with the application or plays the game, similarly that if the game or application were run on a high-powered personal computer, wherein at least one server is used in multi-access edge computing (MEC) architecture, however, in a multi-access edge computing (MEC) architecture, two or more servers can be combined using a software-defined storage (sds) system. 8. The platform for virtualization and streaming of applications and computer games according to claim 1, characterized in that the applications being launched are resource-intensive applications. 9. The platform for virtualization and streaming of applications and computer games according to claim 1, characterized in that the change in the parameters of the video stream (bitrate, fps and screen resolution) is performed based on the forecast of the neural network, and the training of the neural network is based on the collected data of user sessions (server performance, data network parameters, user device performance, game type) and a subjective assessment of the quality of the gameplay. 10. A machine-implemented method for virtualizing and streaming applications and computer games according to claim 1, comprising the steps: - at least one user runs a client application on his user device, the client application displays an image that is produced on the virtualization server, the client application conducts a preliminary analysis of the state of the data transmission channel from the client to the server, analyzes the power of the user computer to issue recommendations to the user or automatically configures the parameters of the video stream and the parameters of the user device, when analyzing the power of the user device and the state of the data transmission channel from the client to the server, a notification is generated to the user about compliance or a warning about the mismatch of the power of the user device or the state of the data transmission channel from the client to the server with the minimum quality requirements that ensure the game process or the process of working with the application, similar to if the game or application would be run on a high-powered personal computer, and recommendations are displayed on how to configure the video stream of the user's video stream, the user's device to improve the quality of the game process or work with the application, or the parameters of the video stream and the parameters of the user device are automatically adjusted to achieve the best quality of the game process or work with the application, further, the user, by entering control commands, controls the computer game or application, while the commands from the client application to the server are transmitted via a data transmission channel, - at least one server running a computer game or application receives control commands related to the game or application transmitted from a client controlled by the user, and in response executes the game program code to render a sequence of video images as a result of executing the game or application, moreover, continuous transmission of the video stream in real time to the client device, at the same time, the application or game is launched on the server in a virtualization environment, which allows you to separate user applications and games from each other, eliminating their mutual influence, in addition, the virtualization environment allocates a separate virtual computer for each user, on which the user works with the application or plays the game, in the same way as if the game or application were run on a high-powered personal computer, moreover, the reception, decoding and display of the video stream for the user is performed on the client application; at the same time, when transmitting video stream data from the server to the client, forward error correction occurs by adding to the transmitted information such additional service data that allows you to correct errors without an additional request, and on the server side, when encoding video stream data, 2p check symbols are added to a block of m-symbols, when calculating each check symbol, all m symbols of the source block are used, therefore, Fourier transforms are used to speed up the process of decoding the received data on the client, at the same time, an algorithm for changing the redundancy level is applied, depending on the session duration, frame type, connection type and current channel state: the streaming session is started with the video stream bitrate parameter equal to 10 Mbps, 100% redundancy is added during the first 30 seconds of the session, necessary to stabilize the data transmission channel, after 30 s the redundancy level in the video stream decreases to 20%, with the exception of the IDR-frame, the redundancy in which always remains at 100%, then, depending on the type of client connection to the data network, the level of redundancy depends in the following way: if the Wi-Fi signal level is less than 80%, then an additional 20% redundancy is added to the video stream; if the signal type is Wi-Fi with a frequency of 2.4 GHz, then an additional 20% redundancy is added to the video stream; further, depending on the indications of the current losses on the channel (loss-rate) of data transmission, the redundancy value changes: - with loss-rate = 1…3%, 20% redundancy is added to the video stream; - with loss-rate = 4…5%, 40% redundancy is added to the video stream; - with loss-rate = 6…10%, 60% redundancy is added to the video stream; - with loss-rate = 11…20%, 100% redundancy is added to the video stream; - with loss-rate = 21…30%, 150% redundancy is added to the video stream; - with loss-rate = 31% and higher, 200% redundancy is added to the video stream. 11. A computer-implemented method for virtualizing and streaming applications and computer games according to claim 9, wherein capturing and encoding an image of a video game or application into an H.264 or H.265 transport video stream using a graphics card installed in the server. 12. A machine-implemented method for virtualizing and streaming applications and computer games according to claim 9, wherein the user device is a phone, tablet, PC, set-top boxes with various operating systems. 13. A computer-implemented method for virtualizing and streaming applications and computer games according to claim 9, wherein the input device is a keyboard, mouse, game joysticks, manipulators, touch screen. 14. A machine-implemented method for virtualizing and streaming applications and computer games according to claim 9, characterized in that the applications being launched are resource-intensive applications. 15. A machine-implemented method for virtualizing and streaming applications and computer games according to claim 9, characterized in that the video stream parameters (bitrate, fps and screen resolution) are changed based on the neural network prediction, and the neural network is trained based on the collected user session data (server performance , data network parameters, user device performance, game type) and a subjective assessment of the quality of the gameplay.

Images