CN110703980A - A file transmission method and device - Google Patents

Abstract

The invention discloses a file transmission method and device. The method includes: for any node server in the node server cluster, the node server can first generate corresponding meta information according to the received upload file, and then send the upload file and meta information to the cloud storage device, so that the upload file and the meta information can be sent to the cloud storage device. Cloud storage device for storage. In this way, on the one hand, multiple client devices in the client cluster can send uploaded files to the node server cluster at the same time, so as to achieve the purpose of fast file transfer and improve the speed of file transfer; on the other hand, each node server is receiving After the file is sent to the client device, the file can be synchronously stored in the same storage location in the cloud storage device, so that the user can directly manage the file through the cloud storage device, which is convenient for the user to centrally manage the file.

Description

A file transmission method and device

technical field

The present invention relates to the field of communication technologies, and in particular, to a file transmission method and device.

Background technique

With the rapid development of Internet technology, the amount of network information data is also increasing, and the need for enterprises to store massive data has become more important. The file storage system in the traditional sense has problems such as limited disk space and insufficient processing power, which cannot meet the user's large-capacity storage requirements.

An existing solution is to upload the files to be stored by the user to multiple node servers in a distributed manner by means of distributed storage, so as to meet the user's large-capacity storage needs. For example, if a user needs to upload a 1TB file, if a distributed method is adopted, 500GB of the file can be stored with node server A, and the other 500GB of file can be stored with node server B. However, since files are distributed in different node servers, when users manage the files, they need to manage some files stored in node server A and files stored in node server B at the same time, so it cannot meet the needs of users for centralized management. .

Another solution is to use a centralized cloud storage service to centrally upload files to be stored by users to a cloud storage device. However, although this method can meet the needs of users for centralized management of files, because the upload rate that the cloud storage device can provide is limited, when the data volume of the file is too large, it cannot meet the user's high concurrency. The need to upload files quickly and easily.

Based on this, there is an urgent need for a file transfer method, which is used to solve the problem that fast upload and centralized management cannot be taken into account in the process of file transfer in the prior art.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a file transmission method and device to solve the technical problem that the distributed storage of data in the prior art cannot meet the needs of users for centralized management.

An embodiment of the present invention provides a method for uploading files, and the method is applied to a distributed storage system, where the distributed storage system includes a cloud storage device, a node server cluster, and a client cluster; the node server cluster includes multiple nodes A server, the client cluster includes multiple client devices; the method includes:

For any node server in the node server cluster, the node server receives an upload file sent by any client device in the client cluster; the upload file is a set of upload files sent by the client cluster Each file in the uploaded file set corresponds to the same data subject;

generating, by the node server, meta information of the uploaded file, where the meta information includes a data subject to which the uploaded file belongs;

The node server sends the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and sends the meta information and the uploaded file to the cloud storage device. are stored in the target storage area; each uploaded file in the uploaded file set is stored in the target storage area.

In this way, for services with a large amount of data and transmission delay requirements, on the one hand, multiple client devices in the client cluster can send the uploaded file set of the same data subject to the node server cluster at the same time. Each node server provides available bandwidth to receive the uploaded file set sent by the client cluster, so as to achieve the purpose of fast file transfer and improve the file transfer rate; After each node server receives the upload file sent by the client device, it can synchronously store the upload file of the same data subject in the cloud storage device. In terms of the method of multiple node servers, in the embodiment of the present invention, the user can directly manage the files through the cloud storage device, thereby facilitating the centralized management of the files by the user. It can be seen from the above analysis that the same data subject can be provided with a data upload service that can not only meet the requirements of transmission delay, but also meet the needs of centralized storage.

In a possible implementation manner, the node server sends the uploaded file and the meta information to the cloud storage device, including:

sending, by the node server, a synchronization request to the cloud storage device, where the synchronization request includes the meta information;

After receiving the response message of the synchronization request returned by the cloud storage device, the node server sends the uploaded file to the cloud storage device; the response information is that the cloud storage device is responding to the meta information. The determination of sending after storing is sent after receiving the meta-information.

In this way, the node server can first send the meta information to the cloud storage device. Compared with the uploaded file, the meta information has a smaller amount of data, so it can be transferred to the cloud storage device relatively quickly, and when the bandwidth is sufficient Then transfer the uploaded file, thereby improving the efficiency of file transfer.

In a possible implementation manner, after the node server sends the uploaded file and the meta information to the cloud storage device, the method further includes:

The node server deletes the uploaded file and the meta information. In this way, the storage burden of the node server can be reduced, and the problem of duplication of file contents stored in the cloud storage device and the node server can be avoided.

An embodiment of the present invention provides a method for downloading files. The method is applied to a distributed storage system, where the distributed storage system includes a cloud storage device, a node server cluster, and a client cluster; the node server cluster includes multiple nodes A server, the client cluster includes multiple client devices; the method includes:

For any client device in the client cluster, the client device sends a download request to the cloud storage device, where the download request includes a target file identifier;

The client device receives a response message of the download request sent by the cloud storage device, and acquires the target file from the target node server according to the location information of the target node server included in the response message;

The location information of the target node server is determined according to the correspondence between the file identifier and the location information of the node server after the cloud storage device determines that no target file corresponding to the target file identifier is stored.

In this way, when the client device downloads the target file, if the file has not been synchronously stored in the cloud storage device, the client device can still download the target file from the corresponding node server, thereby preventing the file from being unsynchronized or during the file synchronization process. , the client device cannot download the target file.

In a possible implementation, the method further includes:

The client device receives a target file sent by the cloud storage device, where the target file is a file stored in the cloud storage device and corresponding to the target file identifier.

In this way, when the client device downloads the target file, if the file has been synchronously stored in the cloud storage device, the client device can directly obtain the target file.

An embodiment of the present invention provides a method for uploading files, and the method is applied to a distributed storage system, where the distributed storage system includes a cloud storage device, a node server cluster, and a client cluster; the node server cluster includes multiple nodes A server, the client cluster includes multiple client devices; the method includes:

The cloud storage device receives the upload file sent by any node server in the node server cluster, and the meta information of the upload file; the meta information includes the data subject to which the upload file belongs; the upload file is all the files in the upload file set sent by the client cluster, and each file in the upload file set corresponds to the same data subject;

The cloud storage device determines the target storage area of the uploaded file according to the meta information and the preset correspondence between the data entity and the storage area; the target storage area is used to store each upload in the set of uploaded files. document;

The cloud storage device stores the uploaded file and the meta information in the target storage area.

In this way, on the one hand, multiple client devices in the client cluster can send uploaded files to the node server cluster at the same time, so as to achieve the purpose of fast file transfer and improve the speed of file transfer; on the other hand, each node server is receiving After the files are sent to the client device, the files can be stored in the cloud storage device synchronously. Compared with the prior art, the files sent by multiple client devices are stored in multiple node servers in a distributed manner. In the embodiment of the invention, the user can directly manage the files in the cloud storage device, thereby facilitating the user to perform centralized management of the files.

In a possible implementation manner, the cloud storage device receives an uploaded file sent by any node server in the node server cluster, and meta information of the uploaded file, including:

receiving, by the cloud storage device, a synchronization request sent by the node server, where the synchronization request includes the meta information;

After determining that there is available transmission bandwidth, the cloud storage device returns a response message for the synchronization request to the node server, where the response message is used to instruct the node server to send the uploaded file to the cloud storage device ;

The cloud storage device receives the upload file sent by the node server.

In this way, the node server can first send the meta information to the cloud storage device. Compared with the uploaded file, the meta information has a smaller amount of data, so it can be transferred to the cloud storage device relatively quickly, and when the bandwidth is sufficient Then transfer the uploaded file, thereby improving the efficiency of file transfer.

In a possible implementation, the method further includes:

receiving, by the cloud storage device, a download request sent by any client device in the client cluster, where the download request includes a target file identifier;

If the cloud storage device determines that the target file corresponding to the target file identifier is not stored, the location information of the target node server is determined according to the corresponding relationship between the file identifier and the location information of the node server; the corresponding relationship is based on the metadata. information determined;

The cloud storage device sends a response message of the download request to the client, where the response message includes the location information of the target node server, and the response message is used to instruct the client device to The location information of the server obtains the target file from the target node server.

In this way, when the client device downloads the target file, if the file has not been synchronously stored in the cloud storage device, the client device can still download the target file from the corresponding node server, thereby preventing the file from being unsynchronized or during the file synchronization process. , the client device cannot download the target file.

In a possible implementation, the method further includes:

If it is determined that a target file corresponding to the target file identifier is stored, the cloud storage device sends the target file to the client.

In this way, when the client device downloads the target file, if the file has been synchronously stored in the cloud storage device, the client device can directly obtain the target file.

An embodiment of the present invention provides a server, and the server includes:

A transceiver unit, configured to receive an upload file sent by any client device in the client cluster; the upload file is a file in the upload file set sent by the client cluster, and each file in the upload file set correspond to the same data subject;

a processing unit, configured to generate meta information of the uploaded file, where the meta information includes the data subject to which the uploaded file belongs;

The transceiver unit is configured to send the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and sends the meta information and the meta information to the cloud storage device. The uploaded file is stored in the target storage area; each uploaded file in the upload file set is stored in the target storage area.

In a possible implementation manner, the transceiver unit is specifically used for:

Send a synchronization request to the cloud storage device, where the synchronization request includes the meta information; and after receiving a response message for the synchronization request returned by the cloud storage device, send the upload to the cloud storage device file; the response information is sent when the cloud storage device determines that there is available transmission bandwidth.

In a possible implementation manner, the processing unit is further configured to:

Delete the uploaded file and the meta information.

An embodiment of the present invention provides a client device, where the client device includes:

a sending unit, configured to send a download request to the cloud storage device, where the download request includes a target file identifier;

a receiving unit, configured to receive a response message of the download request sent by the cloud storage device, and obtain the target file from the target node server according to the location information of the target node server included in the response message;

The location information of the target node server is determined according to the correspondence between the file identifier and the location information of the node server after the cloud storage device determines that the target file corresponding to the target file identifier is not stored; the Correspondence is determined based on meta information.

In a possible implementation manner, the receiving unit is further configured to:

A target file sent by the cloud storage device is received, where the target file is a file stored in the cloud storage device and corresponding to the target file identifier.

An embodiment of the present invention provides a cloud storage device, and the cloud storage device includes:

A transceiver unit, configured to receive an upload file sent by any node server in the node server cluster, and meta information of the upload file; the meta information includes the data subject to which the upload file belongs; the upload file is all the files in the upload file set sent by the client cluster, and each file in the upload file set corresponds to the same data subject;

a processing unit, configured to determine the target storage area of the uploaded file according to the meta-information and the preset correspondence between the data entity and the storage area; the target storage area is used to store each uploaded file in the set of uploaded files file; and storing the uploaded file and the meta information to the target storage area.

In a possible implementation manner, the transceiver unit is specifically used for:

Receive a synchronization request sent by the node server, where the synchronization request includes the meta information; and after determining that there is available transmission bandwidth, return a response message of the synchronization request to the node server, where the response message is used for instructing the node server to send the upload file to the cloud storage device; and receiving the upload file sent by the node server.

In a possible implementation manner, the transceiver unit is further used for:

receiving a download request sent by any client device in the client cluster, where the download request includes a target file identifier;

The processing unit is also used to:

If it is determined that the target file corresponding to the target file identifier is not stored, the location information of the target node server is determined according to the corresponding relationship between the file identifier and the location information of the node server; the corresponding relationship is determined according to the meta information;

The transceiver unit is further configured to send a response message of the download request to the client, where the response message includes the location information of the target node server, and the response message is used to instruct the client device to The target file is obtained from the target node server according to the location information of the target node server.

In a possible implementation manner, the transceiver unit is further used for:

If it is determined that a target file corresponding to the target file identifier is stored, the target file is sent to the client.

An embodiment of the present application further provides an apparatus, which has the function of implementing the file transmission method described above. This function can be implemented by executing corresponding software in hardware. In a possible design, the device includes: a processor, a transceiver, and a memory; the memory is used to store computer-executed instructions, and the transceiver is used to realize the communication between the device and other devices The entity communicates, the processor is connected to the memory through the bus, and when the apparatus is running, the processor executes the computer-executable instructions stored in the memory to cause the apparatus to perform the file transfer method described above.

An embodiment of the present invention further provides a computer storage medium, where a software program is stored in the storage medium, and when the software program is read and executed by one or more processors, the files described in the above various possible implementation manners are implemented transfer method.

Embodiments of the present invention also provide a computer program product containing instructions, which, when executed on a computer, cause the computer to execute the file transfer method described in the various possible implementations above.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the accompanying drawings used in the description of the embodiments.

1 is a schematic diagram of a system architecture to which an embodiment of the present invention is applicable;

2 is a schematic flowchart corresponding to a file transmission method provided by an embodiment of the present invention;

3 is a schematic flowchart corresponding to a method for uploading a file by a client device according to an embodiment of the present invention;

4 is a schematic diagram of another system architecture provided by an embodiment of the present invention;

5 is a schematic flowchart corresponding to a method for synchronizing files by a node server according to an embodiment of the present invention;

6 is a schematic flowchart corresponding to another file transmission method provided by an embodiment of the present invention;

7 is a schematic diagram of the overall flow involved in Embodiment 2 of the present invention;

FIG. 8 is a schematic structural diagram of a server according to an embodiment of the present invention;

FIG. 9 is a schematic structural diagram of a client device according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a cloud storage device according to an embodiment of the present invention.

Detailed ways

The present application will be specifically described below with reference to the accompanying drawings, and the specific operation methods in the method embodiments can also be applied to the device embodiments.

FIG. 1 schematically shows a schematic diagram of a system architecture to which an embodiment of the present invention is applicable. As shown in FIG. 1 , a system architecture 100 to which the embodiment of the present invention is applicable includes a cloud storage device 101 , a node server cluster 102 and a client cluster 103 . The system 100 may be a distributed storage system; the node server cluster 102 may include multiple node servers, such as the node server 1021, the node server 1022 and the node server 1023 shown in FIG. 1 ; the client cluster 103 may also include multiple node servers client devices, such as client device 1031 , client device 1032 , and client device 1033 shown in FIG. 1 .

It should be noted that the client device in this application may be various types of devices, for example, a notebook computer, a smart phone, a tablet computer, a smart TV, etc., which are not specifically limited.

Example 1:

Based on the system architecture shown in FIG. 1 , FIG. 2 exemplarily shows a schematic flowchart corresponding to a file transfer method provided by an embodiment of the present invention, which specifically includes the following steps:

Step 201: The node server receives an upload file sent by any client device in the client cluster; the upload file is a file in the upload file set sent by the client cluster, and each file in the upload file set corresponds to the same data subject .

Step 202: The node server generates meta information of the uploaded file, where the meta information includes the data subject to which the uploaded file belongs.

Step 203, the node server sends the uploaded file and meta information to the cloud storage device.

Step 204, the cloud storage device receives the uploaded file sent by the node server and the meta information of the uploaded file.

Step 205, the cloud storage device determines the target storage area of the uploaded file according to the meta information and the preset correspondence between the data entity and the storage area.

Step 206, the cloud storage device stores the uploaded file and meta information in the target storage area.

It should be noted that the above steps 201 to 203 can be executed by any node server in the node server cluster 102 shown in FIG. 1 , for example, can be executed by the node server 1021 shown in FIG. 1 ; Step 206 may be performed by the cloud storage device 101 shown in FIG. 1 .

In this way, for services with a large amount of data and transmission delay requirements, on the one hand, multiple client devices in the client cluster can send the uploaded file set of the same data subject to the node server cluster at the same time. Each node server provides available bandwidth to receive the uploaded file set sent by the client cluster, so as to achieve the purpose of fast file transfer and improve the file transfer rate; After each node server receives the upload file sent by the client device, it can synchronously store the upload file of the same data subject in the cloud storage device. In terms of the method of multiple node servers, in the embodiment of the present invention, the user can directly manage the files through the cloud storage device, thereby facilitating the centralized management of the files by the user. It can be seen from the above analysis that the same data subject can be provided with a data upload service that can not only meet the requirements of transmission delay, but also meet the needs of centralized storage.

Specifically, before step 201 is executed, there are various communication relationships between the client and the node server. For example, the node server can receive the uploaded file sent by any client device in the client cluster. In an example, the node server The upload file sent by the client device 1031 shown in FIG. 1 can be received, the upload file sent by the client device 1032 can also be received, and the upload file sent by the client device 1033 can also be received.

In another example, the node server may also receive upload files sent by the client devices specified in the client device 1031 , the client device 1032 , and the client device 1033 shown in FIG. 1 . For example, the client device can send the upload file to the designated node server according to its own location information. For example, as shown in FIG. 3, an embodiment of the present invention provides a schematic flowchart of a method for a client device to determine a node server corresponding to an uploaded file. The method can be executed by the client device 1031 shown in FIG. 1, including: Follow the steps below:

Step 301: The client device sends an upload request to the DNS server, where the upload request includes location information of the client device.

The location information of the client device may refer to the IP address of the client device, or may also refer to the geographic location of the client device. As shown in Table 1, it is an example of the location information of the client device. It can be seen from the content shown in Table 1 that each client device has a corresponding geographic location. Taking the client device 1 shown in Table 1 as an example, the corresponding IP address of the client device 1 is: 222.64.232.189, the corresponding The geographic location is: Minhang District, Shanghai. For the location information of other client devices, please refer to the table for details, which will not be described in detail here.

Table 1: An example of location information for a client device

Changes in client devices IP address Geographical location Client Device 1 222.64.232.189 Minhang District, Shanghai Client Device 2 220.155.1.196 Haiding district, Beijing Client Device 3 112.10.39.255 Xiaoshan District, Hangzhou Client Device 4 221.226.140.255 Xuanwu District, Nanjing Client Device 5 218.82.20.255 Jing'an District, Shanghai

Step 302, the DNS server determines the target node server from the node server cluster according to the location information of the client device.

In this embodiment of the present invention, the DNS server may be used to provide a corresponding node server for any client device in the client cluster. As shown in FIG. 4 , which is a schematic diagram of another system architecture provided by an embodiment of the present invention, the system architecture 100 includes a cloud storage device 101 , a node server cluster 102 , a client cluster 103 , and a DNS server 104 . The DNS server 104 is connected to the client cluster 103 and the node server cluster 102, so that the DNS server 104 can, according to the location information of any client device in the client cluster 103 (eg The corresponding node server (eg, the node server 1021 ) is determined in 102 , so that the client device 1031 can communicate with the node server 1021 .

For example, taking the client device 1 shown in Table 1 as an example, the location information corresponding to the client device is Minhang District, Shanghai, then the DNS server can filter out the location information from the node server cluster, which is also Minhang District, Shanghai. the node server, and use this node server as the node server corresponding to the client device 1.

Step 303: The DNS server sends a response message of the upload request to the client device, where the response message includes the location information of the target node server.

Step 304, the client device receives the response message of the upload request sent by the DNS server.

Step 305, the client device sends the upload file to the target node server according to the location information of the target node server.

It should be noted that the method described in steps 301 to 305 is only an example, and in other possible examples, those skilled in the art can also determine the target node server according to experience and actual conditions. The load information of the node servers in the cluster is used to determine the target node server, which is not specifically limited.

In step 201, for the client cluster, all files uploaded by the client cluster can be regarded as a set of uploaded files, and the files uploaded by each client in the client cluster are the files in the set of uploaded files, and the upload Each file in the file collection corresponds to the same data subject.

The data subject may refer to the type corresponding to each file, such as company type, industry type, etc., which can be determined by those skilled in the art according to the actual situation. For example, if uploaded file 1 is a file of company A, the data entity corresponding to uploaded file 1 can be company A; uploaded file 2 is a file in domain B, then the data entity corresponding to uploaded file 2 can be domain B.

In step 202, the meta information is generated according to the uploaded file, and the meta information may include the data subject to which the uploaded file belongs, the uploaded file identifier, and the location information of the node server, which is not specifically limited.

In the specific implementation process, the node server can directly use the data subject to which the uploaded file belongs, the uploaded file identifier and the location information of the node server as meta information; The location information is encrypted, and the encrypted information is used as meta information, which is not specifically limited.

In step 203, the node server may directly send the uploaded file and the meta information to the cloud storage device; alternatively, the node server may first send the meta information to the cloud storage device, and then send the uploaded file to the cloud storage device; the node server may also send the meta information to the cloud storage device. It can be to send the uploaded file to the cloud storage device within a set time after receiving the uploaded file. Specifically, as shown in FIG. 5 , a schematic flowchart corresponding to a method for synchronizing files by a node server provided by an embodiment of the present invention includes the following steps:

Step 501: The node server sends a synchronization request to a cloud storage device, where the synchronization request includes meta information.

Step 502, the cloud storage device receives the synchronization request sent by the node server.

Step 503, after determining that there is available transmission bandwidth, the cloud storage device returns a response message of the synchronization request to the node server.

The cloud storage device can also determine whether there is available transmission bandwidth before returning a response message for the synchronization request to the node server, and return a response message to the node server when it is determined that there is an available transmission bandwidth, thereby improving the performance of the node server. The rate at which the node server sends uploaded files to the cloud storage device.

Step 504: After receiving the response message of the synchronization request, the node server sends the upload file to the cloud storage device.

In this way, the node server can first send the meta information to the cloud storage device. Compared with the uploaded file, the meta information has a smaller amount of data, so it can be transferred to the cloud storage device relatively quickly, and when the bandwidth is sufficient Then transfer the uploaded file, thereby improving the efficiency of file transfer.

After step 204 is executed, the node server may also delete the uploaded file and meta information, thereby reducing the storage burden of the node server and avoiding the problem of duplication of file contents stored in the cloud storage device and the node server.

Embodiment 2:

Based on the system architecture shown in FIG. 1 , FIG. 6 exemplarily shows a schematic flowchart corresponding to another file transfer method provided by an embodiment of the present invention, which specifically includes the following steps:

Step 601, the client device sends a download request to the cloud storage device, where the download request includes the target file identifier.

Step 602, the cloud storage device receives the download request sent by the client device.

Step 603: If the cloud storage device determines that the target file corresponding to the target file identifier is not stored, the cloud storage device determines the location information of the target node server according to the corresponding relationship between the file identifier and the location information of the node server.

Step 604, the cloud storage device sends a response message to the client for the download request, where the response message includes the location information of the target node server.

Step 605: The client device receives the response message of the download request sent by the cloud storage device, and obtains the target file from the target node server according to the location information of the target node server included in the response message.

It should be noted that the above steps 601 and 605 may be performed by any client device in the client cluster 103 shown in FIG. 1 , for example, may be performed by the client device 1031 shown in FIG. 1 ; the above steps Steps 602 to 604 may be performed by the cloud storage device 101 shown in FIG. 1 .

In this way, when the client device downloads the target file, if the file has not been synchronously stored in the cloud storage device, the client device can still download the target file from the corresponding node server, thereby preventing the file from being unsynchronized or during the file synchronization process. , the client device cannot download the target file.

Specifically, in step 603 and step 604, if the cloud storage device determines that the target file corresponding to the target file identifier is stored, it can directly send the target file to the client. In this way, when the client device downloads the target file, If the file has been synchronously stored in the cloud storage device, the client device can directly obtain the target file.

In other possible embodiments, the cloud storage device may not store any information related to the target file identifier. In other words, the cloud storage device has neither the target file corresponding to the target file identifier nor the target file identifier. The node server can return information that the file does not exist to the client device.

In the specific implementation process, the client can be set to directly connect to the cloud storage device when starting the download function; when the cloud storage device determines that the target file is not stored, the management server can also determine the location information of the target node server corresponding to the target file. .

In order to introduce the above-mentioned file transmission method more clearly, the following describes the process involved in the second embodiment of the present invention as a whole with reference to FIG. 7 . As shown in Figure 7, the following steps can be included:

Step 701: The client device sends a download request to the cloud storage device, where the download request includes the target file identifier.

Step 702: The cloud storage device receives the download request sent by the client device.

Step 703 , the cloud storage device determines whether a target file corresponding to the target file ID is stored according to the target file ID, and if so, executes step 704 ; if not, executes step 706 .

Step 704, the cloud storage device sends the target file to the client.

Step 705: The client device receives the target file sent by the cloud storage device.

Step 706: The cloud storage device determines whether there is a target node server corresponding to the target file ID according to the correspondence between the file ID and the location information of the node server. If so, go to Step 707; if not, go to Step 710.

Step 707, the cloud storage device determines the location information of the target node server.

Step 708: The cloud storage device sends a response message to the client for the download request, where the response message includes the location information of the target node server.

Step 709: The client device receives the response message of the download request sent by the cloud storage device, and obtains the target file from the target node server according to the location information of the target node server included in the response message.

Step 710, the cloud storage device returns information that the file does not exist to the client device.

It should be noted that the above step numbering is only an exemplary representation of an execution flow, and the present application does not specifically limit the sequence of each step. For example, in the above steps 703 and 706, the cloud storage device may first determine whether there is a target node server corresponding to the target file identifier according to the corresponding relationship between the file identifier and the location information of the node server, and then determine whether there is a target node server corresponding to the target file identifier according to the target file identifier. A target file corresponding to the target file identifier is stored, which is not specifically limited.

Embodiment three:

Based on the same inventive concept, FIG. 8 exemplarily shows a schematic structural diagram of a server provided by an embodiment of the present invention. As shown in FIG. 8 , the server 800 includes a transceiver unit 801 and a processing unit 802 .

The transceiver unit 801 is configured to receive an upload file sent by any client device in the client cluster; the upload file is a file in the upload file set sent by the client cluster, and each upload file set in the upload file set The files correspond to the same data subject;

a processing unit 802, configured to generate meta information of the uploaded file, where the meta information includes the data subject to which the uploaded file belongs;

The transceiver unit 801 is configured to send the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and sends the meta information to the cloud storage device. The uploaded file is stored in the target storage area; each uploaded file in the upload file set is stored in the target storage area.

In a possible implementation manner, the transceiver unit 801 is specifically used for:

Send a synchronization request to the cloud storage device, where the synchronization request includes the meta information; and after receiving a response message for the synchronization request returned by the cloud storage device, send the upload to the cloud storage device file; the response information is sent when the cloud storage device determines that there is available transmission bandwidth.

In a possible implementation manner, the processing unit 802 is further configured to:

Delete the uploaded file and the meta information.

Embodiment 4:

Based on the same inventive concept, FIG. 9 exemplarily shows a schematic structural diagram of a client device provided by an embodiment of the present invention. As shown in FIG. 9 , the client device 900 includes a sending unit 901 and a receiving unit 902 .

A sending unit 901, configured to send a download request to the cloud storage device, where the download request includes a target file identifier;

A receiving unit 902, configured to receive a response message of the download request sent by the cloud storage device, and obtain the target file from the target node server according to the location information of the target node server included in the response message ;

The location information of the target node server is determined according to the correspondence between the file identifier and the location information of the node server after the cloud storage device determines that the target file corresponding to the target file identifier is not stored; the Correspondence is determined based on meta information.

In a possible implementation manner, the receiving unit 902 is further configured to:

A target file sent by the cloud storage device is received, where the target file is a file stored in the cloud storage device and corresponding to the target file identifier.

Embodiment 5:

Based on the same inventive concept, FIG. 10 exemplarily shows a schematic structural diagram of a cloud storage device provided by an embodiment of the present invention. As shown in FIG. 10 , the cloud storage device 1000 includes a transceiver unit 1001 and a processing unit 1002 .

A transceiver unit 1001, configured to receive an upload file sent by any node server in the node server cluster, and meta information of the upload file; the meta information includes the data subject to which the upload file belongs; the upload file is The files in the uploaded file set sent by the client cluster, each file in the uploaded file set corresponds to the same data subject;

The processing unit 1002 is configured to determine the target storage area of the uploaded file according to the meta-information and the preset correspondence between the data entity and the storage area; the target storage area is used to store each item in the uploaded file set. uploading a file; and storing the uploaded file and the meta information to the target storage area.

In a possible implementation manner, the transceiver unit 1001 is specifically used for:

Receive a synchronization request sent by the node server, where the synchronization request includes the meta information; and after determining that there is available transmission bandwidth, return a response message of the synchronization request to the node server, where the response message is used for instructing the node server to send the upload file to the cloud storage device; and receiving the upload file sent by the node server.

In a possible implementation manner, the transceiver unit 1001 is further configured to:

receiving a download request sent by any client device in the client cluster, where the download request includes a target file identifier;

The processing unit 1002 is also used for:

If it is determined that the target file corresponding to the target file identifier is not stored, the location information of the target node server is determined according to the corresponding relationship between the file identifier and the location information of the node server; the corresponding relationship is determined according to the meta information;

The transceiver unit 1001 is further configured to send a response message of the download request to the client, where the response message includes the location information of the target node server, and the response message is used to instruct the client device according to the The location information of the target node server obtains the target file from the target node server.

In a possible implementation manner, the transceiver unit 1001 is further configured to:

If it is determined that a target file corresponding to the target file identifier is stored, the target file is sent to the client.

An embodiment of the present application further provides an apparatus, which has the function of implementing the file transmission method described above. This function can be implemented by executing corresponding software in hardware. In a possible design, the device includes: a processor, a transceiver, and a memory; the memory is used to store computer-executed instructions, and the transceiver is used to realize the communication between the device and other devices The entity communicates, the processor is connected to the memory through the bus, and when the apparatus is running, the processor executes the computer-executable instructions stored in the memory to cause the apparatus to perform the file transfer method described above.

An embodiment of the present invention further provides a computer storage medium, where a software program is stored in the storage medium, and when the software program is read and executed by one or more processors, the files described in the above various possible implementation manners are implemented transfer method.

Embodiments of the present invention also provide a computer program product containing instructions, which, when executed on a computer, cause the computer to execute the file transfer method described in the various possible implementations above.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Based on the content described above, the node server cluster included in the system to which the embodiment of the present invention is applicable includes multiple node servers, so that multiple client devices in the client cluster can upload the same data subject at the same time. The file set is sent to the node server cluster, and multiple node servers in the node server cluster can provide available bandwidth to receive the uploaded file set sent by the client cluster, so as to achieve the purpose of fast file transfer and improve the speed of file transfer; further , after receiving the file sent by the client device, any node server can synchronously store the file to the same storage location in the cloud storage device, so that the user can directly manage the file through the cloud storage device, which is convenient for the user to process the file. Centralized management. It can be seen from this that the method for providing data according to the embodiment of the present invention can not only provide an upload service that can meet the transmission delay requirement for large-capacity data, but also enable the large-capacity data to be centrally stored, thereby realizing the large-capacity data. The need for centralized management of data. The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Although preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of the present invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (20)

1. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:

aiming at any node server in the node server cluster, the node server receives an upload file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;

the node server generates meta information of the uploaded file, wherein the meta information comprises a data main body to which the uploaded file belongs;

the node server sends the uploaded file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information and stores the meta information and the uploaded file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.

2. The method of claim 1, wherein the node server sends the upload file and the meta information to the cloud storage device, and wherein the sending comprises:

the node server sends a synchronization request to the cloud storage device, wherein the synchronization request comprises the meta information;

after receiving a response message of the synchronization request returned by the cloud storage device, the node server sends the upload file to the cloud storage device; the response information is sent when the cloud storage device determines that available transmission bandwidth exists.

3. The method according to claim 1 or 2, wherein after the node server sends the upload file and the meta information to the cloud storage device, the method further comprises:

and the node server deletes the uploaded file and the meta information.

4. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:

for any client device in the client cluster, the client device sends a download request to the cloud storage device, wherein the download request comprises a target file identifier;

the client device receives a response message of the downloading request sent by the cloud storage device, and acquires the target file from a target node server according to the position information of the target node server included in the response message;

the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored; the correspondence is determined from the meta-information.

5. The method of claim 4, further comprising:

and the client device receives a target file sent by the cloud storage device, wherein the target file is a file stored in the cloud storage device and corresponding to the target file identifier.

6. The file transmission method is applied to a distributed storage system, wherein the distributed storage system comprises cloud storage equipment, a node server cluster and a client cluster; the node server cluster comprises a plurality of node servers, and the client cluster comprises a plurality of client devices; the method comprises the following steps:

the cloud storage equipment receives an uploaded file sent by any node server in the node server cluster and meta information of the uploaded file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;

the cloud storage equipment determines a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set;

and the cloud storage equipment stores the uploaded file and the meta information to the target storage area.

7. The method according to claim 6, wherein the receiving, by the cloud storage device, the uploaded file sent by any node server in the node server cluster and the meta information of the uploaded file comprises:

the cloud storage device receives a synchronization request sent by the node server, wherein the synchronization request comprises the meta information;

after determining that the available transmission bandwidth exists, the cloud storage device returns a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the upload file to the cloud storage device;

and the cloud storage equipment receives the uploaded file sent by the node server.

8. The method of claim 6, further comprising:

the cloud storage device receives a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;

if the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;

the cloud storage device sends a response message of the downloading request to the client, the response message comprises the position information of the target node server, and the response message is used for indicating the client device to acquire the target file from the target node server according to the position information of the target node server.

9. The method of claim 8, further comprising:

and if the cloud storage equipment determines that the target file corresponding to the target file identifier is stored, sending the target file to the client.

10. A server, characterized in that the server comprises:

the receiving and sending unit is used for receiving the uploading file sent by any client device in the client cluster; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;

the processing unit is used for generating meta information of the uploaded file, and the meta information comprises a data main body to which the uploaded file belongs;

the receiving and sending unit is configured to send the upload file and the meta information to the cloud storage device, so that the cloud storage device determines a target storage area according to the meta information, and stores the meta information and the upload file in the target storage area; and each uploading file in the uploading file set is stored in the target storage area.

11. The server according to claim 10, wherein the transceiver unit is specifically configured to:

sending a synchronization request to the cloud storage device, the synchronization request including the meta information; after receiving a response message of the synchronization request returned by the cloud storage device, sending the upload file to the cloud storage device; the response information is sent when the cloud storage device determines that available transmission bandwidth exists.

12. The server according to claim 10 or 11, wherein the processing unit is further configured to:

and deleting the uploaded file and the meta information.

13. A client device, the client device comprising:

a sending unit, configured to send a download request to the cloud storage device, where the download request includes a target file identifier;

a receiving unit, configured to receive a response message of the download request sent by the cloud storage device, and obtain the target file from a target node server according to location information of the target node server included in the response message;

the position information of the target node server is determined according to the corresponding relation between the file identifier and the position information of the node server after the cloud storage equipment determines that the target file corresponding to the target file identifier is not stored; the correspondence is determined from the meta-information.

14. The client device of claim 13, wherein the receiving unit is further configured to:

and receiving a target file sent by the cloud storage equipment, wherein the target file is a file stored in the cloud storage equipment and corresponding to the target file identifier.

15. A cloud storage device, the cloud storage device comprising:

the receiving and sending unit is used for receiving an uploading file sent by any node server in the node server cluster and meta information of the uploading file; the meta information comprises a data main body to which the uploading file belongs; the uploaded files are files in an uploaded file set sent by the client cluster, and each file in the uploaded file set corresponds to the same data main body;

the processing unit is used for determining a target storage area of the uploaded file according to the meta information and a preset corresponding relation between a data entity and the storage area; the target storage area is used for storing each uploading file in the uploading file set; and storing the uploaded file and the meta information to the target storage area.

16. The cloud storage device of claim 15, wherein the transceiver unit is specifically configured to:

receiving a synchronization request sent by the node server, wherein the synchronization request comprises the meta information; after determining that the available transmission bandwidth exists, returning a response message of the synchronization request to the node server, wherein the response message is used for indicating the node server to send the uploaded file to the cloud storage device; and receiving the uploading file sent by the node server.

17. The cloud storage device of claim 15, wherein the transceiving unit is further configured to:

receiving a downloading request sent by any client device in the client cluster, wherein the downloading request comprises a target file identifier;

the processing unit is further to:

if the target file corresponding to the target file identifier is determined not to be stored, determining the position information of the target node server according to the corresponding relation between the file identifier and the position information of the node server; the corresponding relation is determined according to the meta information;

the receiving and sending unit is further configured to send a response message of the download request to the client, where the response message includes location information of the target node server, and the response message is used to instruct the client device to acquire the target file from the target node server according to the location information of the target node server.

18. The cloud storage device of claim 17, wherein the transceiving unit is further configured to:

and if the target file corresponding to the target file identifier is determined to be stored, sending the target file to the client.

19. A computer-readable storage medium, characterized in that the storage medium stores instructions that, when executed on a computer, cause the computer to carry out performing the method of any one of claims 1 to 3, or the method of claim 4 or 5, or the method of any one of claims 6 to 9.

20. A computer device, comprising:

a memory for storing program instructions;

a processor for calling program instructions stored in said memory to execute the method of any one of the claims in accordance with the obtained program.

Images