CN115941784B - Method and system for reliably downloading large files in edge weak network environment - Google Patents

Abstract

The application provides a method and a system for reliably downloading large files in an edge weak network environment, wherein the method comprises three key points of slicing, concurrent downloading and merging of a proxy end, and the slicing of the proxy end is to slice and number the large files requested by a client end; all fragments are downloaded through the thread pool in parallel, and retry is carried out for a certain number of times when the fragment downloading fails; and merging the downloaded fragments into one STREAMING HTTP Response according to the sequence numbers, and returning the fragments to the client. The application provides stable file transmission by integrating with the proxy on the basis of file blocking and multithreading downloading, and can effectively avoid the problem of transmission interruption caused by network fluctuation and server transmission limitation.

Description

A method and system for reliable downloading of large files in a marginal weak network environment

Technical Field

The present application relates to the technical field of data transmission in an edge weak network environment, and specifically to a method and system for reliable downloading of large files in an edge weak network environment.

Background technique

A thread can be understood as a download channel. One thread is a download channel for a file. Multi-threading means opening several download channels at the same time. When the server provides download services, downloaders share bandwidth. Under the same priority, the main server will evenly distribute the total download threads. The more threads there are, the faster the download speed will be. Currently, each download application itself supports block multi-threaded downloading.

In the application scenario of multi-threaded download, the server is usually connected to multiple users at the same time, and the bandwidth is shared among the users. If the priority of N users is the same, then the actual bandwidth of each user connected to the server is one-Nth of the server bandwidth. Therefore, if there are a large number of users, each user can only occupy a pitiful amount of bandwidth, and the download will be a long process. In the existing multi-threaded download technology, virtual machine images are stored in the object storage of the central cloud. When the edge cluster with poor network conditions downloads large image files, it is easy to have transmission interruptions with the object storage service end.

Summary of the invention

In order to solve the above technical problems, this application proposes a method and system for reliable downloading of large files in a weak edge network environment, which provides stable file transmission by integrating with a proxy based on file segmentation and multi-threaded downloading. The technical solution adopted by this application is as follows:

A method for reliably downloading large files in a weak edge network environment, the method comprising the following steps:

Step 1: The edge virtual machine server sends a request to the proxy according to the client operation;

Step 2: The proxy analyzes the request and interacts with the object storage service based on the analysis results.

Step 3: The proxy returns data to the edge virtual machine server based on the data interaction situation.

Furthermore, in step 2, the request analysis includes a usage analysis and an integrity analysis of the request.

Furthermore, in step 2, data interaction is performed with the object storage service end according to the analysis result, including:

When receiving a request, the proxy analyzes the purpose of the request to determine whether the request is for executing multi-segment download; if so, the proxy directly forwards the request for executing multi-segment download to the object storage service; if not, the proxy performs an integrity analysis to determine whether the request is a complete request, and if so, the proxy obtains file information and configuration information from the object storage service; if not, the request is discarded.

Furthermore, in step 2, data is interacted with the object storage service according to the analysis result, and the agent also splits the task according to the shard size of the file information and configuration information returned by the object storage service.

Furthermore, the task units are numbered from 0 to n or n+1, and the image fragments of the task units are downloaded concurrently by multiple threads; wherein n=file size/fragment size.

Furthermore, in step 3, the proxy returns data to the edge virtual machine server according to the data interaction situation, including: if the image fragment is downloaded successfully, it is merged according to the task unit number and returned to the client through the edge virtual machine server; if the image fragment fails to be downloaded, it is downloaded again until the number of download failures reaches a preset maximum number, the download task of the image fragment is interrupted, and the download failure message is returned to the client through the edge virtual machine server.

Furthermore, the merging according to the task unit numbers includes: according to the task unit numbers, aggregating the data after the task is split into a Streaming Http Response and sending it to the edge virtual machine server.

A system for reliable downloading of large files in an edge weak network environment is used to implement the above method, characterized in that the system includes an edge virtual machine server, an object storage service and a proxy.

Furthermore, the proxy end is used to receive a request from the edge virtual machine server end, analyze the request, and then interact with the object storage service end according to the analysis result;

The edge virtual machine server is used to send a request to the proxy terminal according to the client operation and receive the return data of the proxy terminal;

The object storage service end is used to provide file information, configuration information and mirror fragments of task units to the proxy end.

Furthermore, the system also includes a cache server for caching data exchanged between the edge virtual machine server and the proxy. The following technical effects can be obtained through the embodiments of the present application:

(1) It can ensure the reliability of large image downloads in edge weak network environments;

(2) It is easier to cache than client implementation. A 200-node edge computing cluster has at least 200 clients and only needs one proxy. To achieve reliable download of large files in the proxy, it only needs to cache them once at the proxy location.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

Figure 1 is a schematic diagram of the edge virtual machine startup process;

Figure 2 is a system structure block diagram for reliable downloading of large files in an edge weak network environment.

Detailed ways

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

The edge virtual machine startup process is shown in Figure 1. The download of the large image in the edge weak network environment is to avoid the problem of transmission interruption by adding download task block, multi-threaded download and download retry functions on the agent side, and ensure the consistency and validity of the data. The method includes the following steps:

Step 1: The edge virtual machine server sends a request to the proxy according to the client operation;

Step 2: The proxy analyzes the request and interacts with the object storage service based on the analysis results.

Step 3: The proxy returns data to the edge virtual machine server based on the data interaction situation.

In step 2, the request analysis includes a purpose analysis and a completeness analysis of the request;

In step 2, data is exchanged with the object storage service end based on the analysis results, including:

When receiving a request, the proxy analyzes the purpose of the request to determine whether the request is for executing multi-segment download; if so, the proxy directly forwards the request for executing multi-segment download to the object storage service; if not, the proxy performs an integrity analysis to determine whether the request is a complete request, and if so, the proxy obtains file information and configuration information from the object storage service; if not, the request is discarded.

In step 2, data is exchanged with the object storage service end according to the analysis result, and the agent end splits the task according to the shard size of the file information and configuration information returned by the object storage service end;

Number the task units from 0 to n or n+1, and concurrently download the image shards of the task units in multiple threads; where n = file size/shard size;

In step 3, the proxy returns data to the edge virtual machine server according to the data interaction situation, including: if the image fragment is downloaded successfully, it is merged according to the task unit number and returned to the client through the edge virtual machine server; if the image fragment fails to be downloaded, it is downloaded again until the number of download failures reaches a preset maximum number, the download task of the image fragment is interrupted, and the download failure message is returned to the client through the edge virtual machine server.

The merging according to the task unit number includes: according to the task unit number, aggregating the task split data into a Streaming Http Response and sending it to the edge virtual machine server. The proxy ensures the integrity and validity of the data through the sequence number when the task unit is split.

Figure 2 is a system structure diagram for reliable downloading of large files in an edge weak network environment. The system includes an edge virtual machine server, an object storage service, and a proxy.

The proxy end is used to receive requests from the edge virtual machine server end, analyze the requests and interact with the object storage service end according to the analysis results;

The edge virtual machine server is used to send a request to the proxy terminal according to the client operation and receive the return data of the proxy terminal;

The object storage service end is used to provide file information, configuration information and mirror fragments of task units to the proxy end.

The request analysis includes a purpose analysis and a completeness analysis of the request;

Data is exchanged with the object storage service based on the analysis results, including:

When receiving a request, the proxy analyzes the purpose of the request to determine whether the request is for executing multi-segment download; if so, the proxy directly forwards the request for executing multi-segment download to the object storage service; if not, the proxy performs an integrity analysis to determine whether the request is a complete request, and if so, the proxy obtains file information and configuration information from the object storage service; if not, the request is discarded.

The proxy splits the task according to the shard size of the file information and configuration information returned by the object storage service:

Number the task units from 0 to n or n+1, and concurrently download the image shards of the task units in multiple threads; where n = file size/shard size;

If the image fragments are downloaded successfully, they are merged according to the task unit numbers and returned to the client through the edge virtual machine server;

If the image segment fails to be downloaded, it is downloaded again until the number of download failures reaches a preset maximum number, the download task of the image segment is interrupted, and a download failure message is returned to the client through the edge virtual machine server.

The merging according to the task unit number includes: according to the task unit number, aggregating the task split data into a Streaming Http Response and sending it to the edge virtual machine server. The proxy ensures the integrity and validity of the data through the sequence number when the task unit is split.

The system also includes a cache server for caching data interacting between the edge virtual machine server and the proxy.

In summary, the method of this application has three key points: fragmentation, concurrent downloading and merging on the proxy side. The fragmentation on the proxy side is to divide the large file requested by the client into n or n+1 (n=file size/fragment size) fragments and number them; download all fragments concurrently through the thread pool, and retry a certain number of times when the fragment download fails; merge the downloaded fragments into a Streaming Http Response according to the sequence number, and return it to the client. The technical solution of this application has been tested in actual application. Tianyi Cloud Intelligent Edge Cloud Chongqing Beibei District 1 can successfully download various large image files using the improved proxy, and the download speed is the same as the download speed of the old proxy.

Although the specific implementation methods of the present application are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present application is defined by the appended claims. Those skilled in the art may make various changes or modifications to these implementation methods without departing from the principles and essence of the present application, but these changes and modifications all fall within the protection scope of the present application.

Claims (8)

1. The method for reliably downloading the large file in the edge weak network environment is characterized by comprising the following steps of:

step 1, an edge virtual machine server sends a request to a proxy end according to the operation of a client;

step 2, the agent end performs request analysis and performs data interaction with the object storage server end according to an analysis result;

step 3, the proxy end returns data to the edge virtual machine server end according to the data interaction condition;

In step 2, data interaction is performed with the object storage server according to the analysis result, and the method further comprises the following steps: the agent end performs task splitting according to the file information and the fragment size of the configuration information returned by the object storage server end;

Numbering task units from 0 to n or n+1, and concurrently downloading mirror image fragments of the task units in a multithreading manner; where n=file size/slice size.

2. The method of claim 1, wherein in step 2, the request analysis includes a usage analysis and an integrity analysis of the request.

3. The method according to claim 2, wherein in step 2, the data interaction with the object storage server according to the analysis result includes:

When a request is received, the proxy end performs purpose analysis of the request and judges whether the request is used for executing fragment downloading or not; if yes, the proxy end directly forwards the request for executing the fragment downloading to the object storage server end; if not, the proxy end carries out integrity analysis and judges whether the request is a complete request, if so, the proxy end acquires file information and configuration information from the object storage server end; if not, the request is discarded.

4. The method of claim 1, wherein in step 3, the agent returns data to the edge virtual machine server according to the data interaction condition, including: if the mirror image fragments are successfully downloaded, merging is carried out according to the serial numbers of the task units, and the mirror image fragments are returned to the client through the edge virtual machine server; if the image fragment downloading fails, re-downloading is performed until the downloading failure times reach the preset maximum times, the downloading task of the image fragment is interrupted, and the downloading failure message is returned to the client through the edge virtual machine server.

5. The method of claim 4, wherein the merging according to the task unit number comprises: and converging the data after task splitting into one StreamingHttpResponse according to the serial numbers of the task units and sending the data to the edge virtual machine server.

6. A system for reliably downloading large files in an edge weak network environment, which is used for realizing the method as claimed in any one of claims 1 to 5, and is characterized in that the system comprises an edge virtual machine service end, an object storage service end and a proxy end.

7. The system of claim 6, wherein the proxy is configured to receive the request from the edge virtual machine server, perform data interaction with the object storage server according to the analysis result after performing the request analysis;

The edge virtual machine server side is used for sending a request to the proxy side according to the operation of the client side and receiving the returned data of the proxy side;

The object storage server is used for providing file information, configuration information and mirror image fragments of the task units for the proxy.

8. The system of claim 7, further comprising a caching server for caching data interacted between the edge virtual machine server and the proxy.