CN107861686A - File memory method, service end and computer-readable recording medium - Google Patents

Abstract

The invention discloses a file storage method, which is applied to an Internet data center. The Internet data center includes a file processing system, a distributed file system, and a distributed storage system. The file processing system includes a server and a client. The method includes: The server receives the files uploaded by the sender through the client; caches the received files in a temporary folder, and records the storage location information of each file in the distributed storage system; merges the files in the temporary folder , to obtain the merged file, and store the merged file in the distributed file system; based on the merged file, update the corresponding storage location information in the distributed storage system. The invention also discloses a server and a computer-readable storage medium. The invention can increase the amount of data storage by merging files, and store files through a distributed file system, so that more files can be stored.

Description

File storage method, server and computer-readable storage medium

technical field

The present invention relates to the field of application technology, in particular to a file storage method, a server and a computer-readable storage medium.

Background technique

Traditional mass file storage is stored in a dedicated file server such as NAS (Network Attached Storage, Network Attached Storage). NAS is defined as a special dedicated data storage server, including storage devices (such as disk arrays, CD/DVD drives, etc.) , tape drive or removable storage media) and embedded system software that provides cross-platform file sharing. NAS usually occupies its own node on a LAN (Local Area Network, local area network), without the intervention of the application server, allowing users to access data on the network.

In the existing file storage architecture, multiple front-end servers share the back-end NAS device through a dedicated storage network, and the storage space on the back-end NAS device passes through CIFS (Commom Internet File System, common network file system), NFS (Network File System, network File system) protocol is shared with the front-end host, and concurrent reading and writing can be performed on the same directory or file at the same time. The file system is located in the back-end storage system, and the connection adopts standard Ethernet link and TCP (Transmission Control Protocol, Transmission Control Protocol)/IP (Internet Protocol, Internet Protocol) protocol, which can realize file storage sharing between multiple systems. However, with the passage of time and the development of business, the scale of data is getting larger and larger, and the storage capacity of NAS devices is limited. It is difficult for the traditional file storage mode to cope with the blowout development of data. In the case of a large file, the existing file storage method and the capacity of the stored data are difficult to meet the demand.

Contents of the invention

The main purpose of the present invention is to provide a file storage method, a server and a computer-readable storage medium, aiming to solve the technical problem that the existing file storage method is difficult to meet the storage requirements when the data capacity increases.

In order to achieve the above object, the present invention provides a file storage method, which is applied to an Internet data center. The Internet data center includes a file processing system, a distributed file system, and a distributed storage system. The file processing system includes a server and a client terminal, the file storage method includes:

The server of the file processing system receives the file uploaded by the sender through the client;

Cache the received files into a temporary folder, and record the storage location information of each file in the distributed storage system;

Merge each file in the temporary folder to obtain the merged file, and store the merged file in the distributed file system;

Based on the merged files, the corresponding storage location information in the distributed storage system is updated.

Optionally, the step of merging each file in the temporary folder to obtain the merged file includes:

The server scans each file in the temporary folder;

Acquiring the combined file, and determining among the scanned files a file whose capacity value after being combined with the combined file is smaller than a preset threshold value, and merging the determined file into the combined file.

Optionally, the method also includes:

When receiving the file query instruction, determine the index information corresponding to the file query instruction;

Find the merged file pointed to by the index information in the distributed file system;

Restore the merged file, so as to restore the file corresponding to the index information from the merged file.

Optionally, after the step of storing the merged file in the distributed file system, the method further includes:

The server generates file identification information and file hash information based on the files stored in the distributed file system;

feeding back the file identification information and file hash information to the sender through the client, so that the sender can transmit the file identification information and file hash information to the receiver;

When the client receives the file identification information sent by the recipient, extract the file corresponding to the file identification information in the distributed file system, and feed it back to the recipient, so that the recipient can pass the file The hash information checks the file, and when the check is successful, the file is retrieved.

Optionally, the number of the server includes multiple, the server and the client of the file processing system are connected through a gateway, and the way of uploading the file from the client to the server includes: the gateway sends the client to the server according to a preset strategy. The uploaded files are polled and uploaded to the server.

Optionally, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, the method further includes:

The server scans each file in the distributed file system to monitor the storage duration of each file;

When the storage duration of a file reaches a preset duration, delete the file in the distributed file system, and delete the storage location information of the file in the distributed storage system.

Optionally, the Internet data center also includes a distributed application program coordination service, and the server scans each file in the distributed file system to monitor the storage duration of each file. Before the step, the method further includes:

The server sends request information for deleting locks to the distributed application coordination service;

When the lock is acquired successfully, the step of scanning each file in the distributed file system to monitor the storage duration of each file is executed.

Optionally, the server is located in the primary Internet data center, and if there is a backup Internet data center in the system, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, The methods include:

The server synchronizes the stored files to the server of the file processing system where the standby Internet data center is located, so that the server of the file processing system where the standby Internet data center is located performs file storage operations.

In addition, in order to achieve the above object, the present invention also provides a server, which includes a memory, a processor, and a file storage program stored in the memory and operable on the processor, the file storage When the program is executed by the processor, the steps of the above-mentioned file storage method are implemented.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium, on which a file storage program is stored, and when the file storage program is executed by a processor, the above-mentioned file The steps of the storage method.

In the technical solution proposed by the present invention, the server of the file processing system first receives the file uploaded by the sender through the client, then caches the received file in a temporary folder, and records the storage location of each file in the distributed storage system information, and then merge each file in the temporary folder to obtain the merged file, store the merged file in the distributed file system, and finally update the corresponding file in the distributed storage system based on the merged file The location information is stored to facilitate subsequent reading of files according to the storage location information. In this solution, the received files are merged, and then the merged files are stored in the distributed file system. The file merger increases the amount of files that can be stored in the system. In addition, because the distributed file system has scalable Compared with the existing file storage method, this solution can store a larger amount of files and is more suitable for storing a large number of small files.

Description of drawings

Fig. 1 is a schematic diagram of the server structure of the hardware operating environment involved in the solution of the embodiment of the present invention;

Fig. 2 is a schematic flow chart of the first embodiment of the file storage method of the present invention;

Fig. 3 is a file storage architecture diagram of the present invention;

Fig. 4 is the schematic diagram that the document of the present invention merges;

FIG. 5 is a schematic flowchart of a second embodiment of the file storage method of the present invention;

Fig. 6 is an exemplary diagram of file transmission in the present invention;

FIG. 7 is a schematic flowchart of a third embodiment of the file storage method of the present invention;

Fig. 8 is an example diagram of file deletion in the present invention.

The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The solution of the embodiment of the present invention is mainly: the server of the file processing system first receives the file uploaded by the sender through the client, then caches the received file in a temporary folder, and records each file in the distributed storage system storage location information, and then merge each file in the temporary folder to obtain the merged file, store the merged file in the distributed file system, and finally update the distributed storage system based on the merged file The corresponding storage location information in the file is convenient for subsequent reading of the file according to the storage location information. In order to solve the problem that the existing file storage method is difficult to meet the storage demand.

It should be noted that the existing file storage method also has the following defects:

The file storage solution does not have a life cycle management function, and does not support functions such as temporary file expiration and deletion, which may easily lead to excessive data storage;

It is not suitable for the access of a large number of systems, and installation and deployment are relatively troublesome.

Based on the problems existing in the prior art, the present invention builds a FPS (File Process System, file processing system), the FPS can support the storage of massive data, and at the same time adopts the scheme of storing multiple copies of data across racks and computer rooms to ensure high service quality. availability. Its main application scenarios include:

(1) Provide an intermediate platform for file exchange between different systems, for example, system A provides reconciliation files to system B for reconciliation through the intermediate platform;

(2) Provide a data storage platform based on file life cycle management, which can support massive data storage, and files are stored for a period of time and need to be automatically deleted when they expire.

Technical terms of the present invention are introduced:

Hadoop: It is a distributed system infrastructure that enables users to build and use a distributed computing platform. Users can develop and run applications that process massive amounts of data on Hadoop.

HDFS: Distributed file system (Hadoop Distributed File System). HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost (low-cost) hardware; and it provides high throughput (high throughput) to access application data, suitable for those with large data sets (large data set )s application.

HBase: It is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. Belongs to the Hadoop ecosystem.

Zookeeper: It is a distributed, open source distributed application coordination service, an open source implementation of Google's Chubby, and an important component of Hadoop and Hbase. It is a software that provides consistent services for distributed applications. Its functions include: configuration maintenance, domain name service, distributed synchronization, group service, etc. Belongs to the Hadoop ecosystem.

TGW: The full name is Tencent GateWay. It is a system that realizes unified access of multiple networks, forwarding of external network requests, and supports automatic load balancing. TGW can be called a gateway.

NAS: Network Attached Storage, which is a device connected to the network and capable of data storage, so it is also called "network storage". It is a dedicated data storage server.

RMB: Message bus system, used for RPC message service between multiple systems.

As shown in FIG. 1 , FIG. 1 is a schematic diagram of the server-side structure of the hardware operating environment involved in the solution of the embodiment of the present invention.

The server in this embodiment of the present invention may be a PC (personal computer, personal computer), or a terminal device with a display function such as a tablet computer or a portable computer.

As shown in FIG. 1 , the server may include: a processor 1001 , such as a CPU, a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface (for example, for connecting a wired keyboard, a wired mouse, etc.), a wireless interface (for example, for Connect wireless keyboard, wireless mouse). The network interface 1004 may optionally include a standard wired interface (for connecting to a wired network) and a wireless interface (such as a WI-FI interface, a Bluetooth interface, an infrared interface, etc., for connecting to a wireless network). The memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Optionally, the server may also include a camera, an RF (Radio Frequency, radio frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like.

Those skilled in the art can understand that the server structure shown in FIG. 1 does not constitute a limitation on the server, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a file storage program. Among them, the operating system is a program that manages and controls the server and software resources, and supports the operation of the network communication module, user interface module, file storage program, and other programs or software; the network communication module is used to manage and control the network interface 1002; the user interface Modules are used to manage and control the user interface 1003 .

In the service end shown in Figure 1, the network interface 1004 is mainly used to connect the server end of the Internet data center, and carry out data communication with the server end of the Internet data center; the user interface 1003 is mainly used to connect the client (client) , performing data communication with the client; the server invokes the file storage program stored in the memory 1005 through the processor 1001, and performs the following steps:

Receive the file uploaded by the sender through the client;

Cache the received files into a temporary folder, and record the storage location information of each file in the distributed storage system;

Merge each file in the temporary folder to obtain the merged file, and store the merged file in the distributed file system;

Based on the merged files, the corresponding storage location information in the distributed storage system is updated.

Further, the server calls the file storage program stored in the memory 1005 through the processor 1001, so as to realize the steps of merging each file in the temporary folder and obtaining the merged file:

scan individual files in said temporary folder;

Acquiring the combined file, and determining among the scanned files a file whose capacity value after being combined with the combined file is smaller than a preset threshold value, and merging the determined file into the combined file.

Further, the server invokes the file storage program stored in the memory 1005 through the processor 1001 to implement the following steps:

When receiving the file query instruction, determine the index information corresponding to the file query instruction;

Find the merged file pointed to by the index information in the distributed file system;

Restore the merged file, so as to restore the file corresponding to the index information from the merged file.

Further, after the step of storing the merged file in the distributed file system, the server invokes the file storage program stored in the memory 1005 through the processor 1001 to implement the following steps:

Based on the files stored in the distributed file system, generate file identification information and file hash information;

feeding back the file identification information and file hash information to the sender through the client, so that the sender can transmit the file identification information and file hash information to the receiver;

When the client receives the file identification information sent by the recipient, extract the file corresponding to the file identification information in the distributed file system, and feed it back to the recipient, so that the recipient can pass the file The hash information checks the file, and when the check is successful, the file is retrieved.

Further, the number of the server includes multiple, the server and the client of the file processing system are connected through a gateway, and the way of uploading the file from the client to the server includes: the gateway uploads the client to the server according to a preset strategy. The file polling uploaded to the server.

Further, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, the server invokes the file storage program stored in the memory 1005 through the processor 1001 to implement the following steps:

Scan each file in the distributed file system to monitor the storage time of each file;

When the storage duration of a file reaches a preset duration, delete the file in the distributed file system, and delete the storage location information of the file in the distributed storage system.

Further, the Internet data center also includes a distributed application program coordination service, and the server scans each file in the distributed file system to monitor the storage duration of each file. Call the file storage program stored in the memory 1005 to realize the following steps:

Send the request information of deleting the lock to the distributed application coordination service;

When the lock is acquired successfully, the step of scanning each file in the distributed file system to monitor the storage duration of each file is executed.

Further, the server is located in the main Internet data center. If there is a backup Internet data center in the system, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, the The server calls the file storage program stored in the memory 1005 through the processor 1001 to achieve the following steps:

Synchronize the stored files to the server of the file processing system where the standby Internet data center is located, so that the server of the file processing system where the standby Internet data center is located performs file storage operations.

Based on the above hardware structure of the server, various embodiments of the file storage method of the present invention are proposed.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a first embodiment of a file storage method according to the present invention.

In this embodiment, the file storage method is applied to an Internet data center, and the Internet data center includes a file processing system, a distributed file system, and a distributed storage system, and the file processing system includes a server and a client, so The file storage methods described above include:

The server of the file processing system receives the files uploaded by the sender through the client; caches the received files in a temporary folder, and records the storage location information of each file in the distributed storage system; The files are merged to obtain the merged files, and the merged files are stored in the distributed file system; based on the merged files, the corresponding storage location information in the distributed storage system is updated.

In this embodiment, the file storage method is applied to the server corresponding to the file processing system FPS where the IDC (Internet Data Center, Internet Data Center) is located, and the server may be the server shown in FIG. 2 .

It should be noted that the embodiment of the present invention sets up FPS (File Process System, file processing system), which is a self-developed small file storage and processing system, specific life cycle management, cross-room disaster recovery and other functions.

In the implementation of the present invention, the structural diagram of the IDC can refer to Figure 3:

IDC includes file processing system FPS, distributed file system HDFS, and distributed storage system Hbase. Among them, FPS mainly includes two parts: FPS-Client (client) and FPS-Server (server). Business programs use the functions provided by FPS by integrating FPS-Client.

FPS-Client: FPS provides clients of java version, c language version and python version.

FPS-Server: The background server program of FPS, which provides the main business logic processing, including authority control, file storage and reading, file life cycle management and other functions.

It can be seen from FIG. 3 that the server and the client of the file processing system are connected through a gateway, and the gateway is represented by TGW. The TGW (Tencent Gataway) is used for load balancing when forwarding files.

It should be noted that the number of FPS servers and clients is not limited, and the number can be set according to the actual situation. When the number of servers includes multiple, the ways of uploading files from the client to the server include: : The gateway polls and uploads the files uploaded by the client to the server according to the preset strategy.

That is, all traffic that uploads files to FPS-Server through the gateway, that is, FPS-Client, will pass through TGW, and the file polling will be sent to each FPS-Server through TGW. After FPS-Server receives the file, it stores the content of the file in HDFS (Distributed File System), and records the storage location information of the file in HBASE (Distributed Storage System).

It should be noted that HDFS is a distributed file system that uses PCs as storage. Generally, files will have multiple backups, such as three backups. It is very convenient to dynamically increase machines linearly. Facing the exponential growth of Internet services, it can be easily It is convenient to achieve non-stop expansion, which can well meet business needs. At the same time, storing the file storage location information on HBase can provide hundreds of billions of file index storage. Like HDFS, HBase is a member of the Hadoop ecosystem, and it is also convenient to increase storage performance by adding nodes.

In the embodiment of the present invention, the interaction between the FPS-Client and the FPS-Server is performed through the HTTP (Hyper TextTransfer Protocol, hypertext transfer protocol) protocol.

The following are the specific steps for implementing file storage in this embodiment:

Step S10, the server of the file processing system receives the file uploaded by the sender through the client;

That is, the server receives the file uploaded by the client through the TGW.

Step S20, cache the received files into a temporary folder, and record the storage location information of each file in the distributed storage system;

In this embodiment, as shown in Figure 3, the FPS-Server receives the file sent by the sender through the FPS-Client. After the FPS-Server receives the file, it first caches the received file in a temporary folder. The folder is preferably a designated directory of HDFS. After the FPS-Server caches the files to the temporary folder, it first records the storage location information of each file in the distributed storage file.

Step S30, merge each file in the temporary folder to obtain the merged file, and store the merged file in the distributed file system;

Specifically, the step of "merging each file in the temporary folder to obtain the merged file" includes:

Step 1, the server scans the directory in the temporary folder to obtain a lock corresponding to the directory;

Step 2, when the lock is obtained, the server scans each file in the temporary folder;

Step 3: Obtain the combined file, and determine among the scanned files a file whose capacity value after being combined with the combined file is smaller than a preset threshold value, and merge the determined file into the combined file.

Further, after step 3, the method also includes:

Step 4, delete the merged files in the temporary folder.

That is, after the uploaded file is cached to the temporary folder, the FPS-Server scans the directory in the temporary folder to obtain the lock corresponding to the directory. If the lock can be obtained, the FPS-Server scans the directory in the temporary folder. The capacity value of each file, and preferably merge the files whose capacity value after merging with the combined file is less than a preset threshold. In this embodiment, the preset threshold is set according to the actual situation and is not limited here. The file merging method is as follows: obtain a preset combined file, scan each file in the temporary folder, determine among the scanned files the file whose capacity value after merging with the combined file is less than a preset threshold, and determine After the files are merged, the merged files can be stored in the distributed file system HDFS, and then the index information corresponding to the merged files in the distributed storage system can be updated according to the merged files .

In the embodiment of the present invention, each file in the temporary folder can be scanned regularly, or each file in the folder can be scanned in real time, and then the scanned small files can be merged, and the merged small files can be updated in HBase Store location information. Small files are merged by FPS-Server, so that the subsequent files stored in the distributed file system will not be too fragmented, which can reduce the space occupied by stored files and significantly save the memory space of cluster nodes.

For a better understanding of this embodiment, with reference to Fig. 4, at first, FPS-Server first scans the temporary directory of HDFS, then obtains the lock corresponding to the directory, and when the lock is obtained, scans each file in the directory, and then Obtain the index information of the small files in the distributed storage system, then obtain the combined files, merge the small files into the combined files, finally update the index information and delete the original small files in the respective distributed file systems.

Step S40, based on the merged file, update the corresponding storage location information in the distributed storage system.

After obtaining the merged file, FPS-Server updates the distributed storage system based on the merged file, so as to update the storage location information corresponding to the merged file in the distributed storage system, that is, FPS-Server updates the merged file The storage location information of the file in the distributed file system is updated to the distributed storage system, so that the corresponding information can be indexed according to the storage location information when the file is subsequently searched.

In addition, in the embodiment of the present invention, the method also includes:

Step A, when receiving the file query command, determine the index information corresponding to the file query command;

Step B, searching the merged file pointed to by the index information in the distributed file system;

Step C, restore the merged file, so as to restore the file corresponding to the index information from the merged file.

In the technical solution proposed in this embodiment, the server of the file processing system first receives the file uploaded by the sender through the client, then caches the received file in a temporary folder, and records the storage of each file in the distributed storage system. Location information, and then merge each file in the temporary folder to obtain the merged file, store the merged file in the distributed file system, and finally update the corresponding file in the distributed storage system based on the merged file The storage location information of the file is convenient for subsequent reading of the file according to the storage location information. In this solution, the received files are merged, and then the merged files are stored in the distributed file system. The file merger increases the amount of files that can be stored in the system. In addition, because the distributed file system has scalable Compared with the existing file storage method, this solution can store a larger amount of files and is more suitable for storing a large number of small files.

Further, referring to FIG. 5 , a second embodiment of the file storage method of the present invention is proposed based on the first embodiment.

The difference between the second embodiment of the file storage method and the first embodiment of the file storage method is that after the step S30, the method further includes:

Step S50, the server generates file identification information and file hash information based on the files stored in the distributed file system;

Step S60, feeding back the file identification information and file hash information to the sender through the client, so that the sender can transmit the file identification information and file hash information to the receiver;

Step S70, when the client receives the file identification information sent by the receiver, extract the file corresponding to the file identification information in the distributed file system, and feed it back to the receiver for the receiver The party checks the file through the file hash information, and obtains the file when the check is successful.

In this embodiment, after the FPS-Server stores the file in the distributed file system HDFS, according to the file stored in the HDFS, the file identification information (FileId) and the file hash information (FileHash) corresponding to the file are generated. After obtaining the File Id and File Hash, the FPS-Server feeds back the File Id and File Hash to the sender, so that the sender can transmit the File Id and File Hash to the receiver.

It should be noted that the sender and receiver interact through the RMB message service bus. After receiving the File Id and File Hash, the receiver uses the FPS-Client to send the File Id to the FPS-Server.

When the FPS-Server receives the File Id sent by the receiver through the FPS-Client, it extracts the file corresponding to the File Id in HDFS, and feeds it back to the receiver, so that the receiver passes the File Hash The file is verified and, if the verification is successful, the file is retrieved. That is to say, the receiver downloads from the FPS-Server through the File Id, and verifies the accuracy of the file through the File Hash. The download and the verification of the correctness of the file are both completed in the FPS-Client.

For a better understanding of this embodiment, with reference to Figure 6, the sender uploads the file to the FPS, and after the file is uploaded successfully, the FPS will return the File Id and File Hash of the file to the sender, and after the sender receives it, the File The Id and FileHash are sent to the receiver through the RMB message service bus. After receiving the message notification, the receiver uses the FPS-Client of FPS to send the File Id to the FPS-Server of FPS to download the file, and the FPS-Client checks the file through File Hash Verify the accuracy of the file. After the verification is successful, FPS-Client returns a message that the file has been successfully downloaded to the receiver.

In this embodiment, the file transfer between the sender and the receiver is realized through various systems in the Internet data center, and file verification is performed by file identification information and file hash information, which improves the accuracy of file transfer.

Further, referring to FIG. 7 , a third embodiment of the file storage method of the present invention is proposed based on the first embodiment.

The difference between the third embodiment of the file storage method and the first embodiment of the file storage method is that after the step S40, the method includes:

Step S80, the server scans each file in the distributed file system to monitor the storage duration of each file;

Step S90, when the storage duration of a file reaches a preset duration, delete the file in the distributed file system, and delete the storage location information of the file in the distributed storage system.

In this embodiment, the manner in which the server scans each file in the distributed file system is preferably regular scanning. Therefore, after the distributed file system HDFS stores files, the FPS-Server background will periodically perform the following operations: regularly scan expired files and delete files to save disk space. Specifically: the FPS-Server scans each file in the distributed file system HDFS to monitor the storage duration of each file, and when the storage duration of a file reaches a preset duration, the preset duration is set according to actual conditions, There is no limitation, for example, the preset period is 3 months. When the storage time of the file reaches the preset time, it means that the storage time of the file is longer. In order to realize the life cycle management of file storage, delete the file in the distributed file system and delete the distributed storage system Storage location information for the files described in .

In this embodiment, the Internet data center also includes a distributed application program coordination service, and before the step S80, the method also includes:

The server sends request information for deleting locks to the distributed application coordination service;

When the lock is acquired successfully, the step of scanning each file in the distributed file system to monitor the storage duration of each file is executed.

As shown in Figure 3, the application coordination service is represented by Zookeeper. Before deleting the file, the FPS-Server first sends a request message to Zookeeper to acquire the delete lock, and only executes step S80 if the lock can be successfully acquired.

For better understanding of the present invention, with reference to Fig. 8, FPS-Server regularly sends to Zookeeper the request information of acquiring deletion lock, if can acquire lock, promptly FPS-Server acquires lock success, at this moment, FPS-Server sends to distributed storage system Hbase requests expired data, that is, FPS-Server requests from Hbase the storage location information corresponding to the file whose storage duration reaches the preset duration, and deletes the requested storage location information after the request is received. Subsequently, FPS-Server requests Zookeeper to obtain a deletion lock, and after obtaining the lock, deletes the files whose storage duration reaches the preset duration in the distributed file system HDFS.

In this embodiment, by regularly deleting expired data, the file storage has a life cycle, and expired files can be deleted regularly, preventing excessive file size and improving the intelligence of file storage.

Further, a fourth embodiment of the file storage method of the present invention is proposed based on the first embodiment.

The difference between the fourth embodiment of the file storage method and the first to third embodiments of the file storage method is that the server is located in the primary Internet data center, and if there is a backup Internet data center in the system, the steps After S40, the method includes:

Step D, the server synchronizes the stored files to the server of the file processing system where the standby Internet data center is located, so that the server of the file processing system where the standby Internet data center is located performs file storage operations.

In this embodiment, the deployed Internet data center IDC includes multiple sets, for example, two sets of Internet data center IDCs, namely the master IDC and the secondary IDC, and the two IDCs communicate with each other through the network. The business system connects to the TGW to request file upload through the integrated FPS-Client, and the TGW routes the request to a certain FPS-Server in the main IDC according to the specified policy, and then officially starts uploading the file. FPS-Server temporarily stores files in HBASE, while the location information of files is stored in HBASE.

When the file in the primary IDC is successfully uploaded, the FPS-Server in the primary IDC will asynchronously upload the file to the FPS-Server in the standby IDC, so as to ensure the consistency of the files in the two clusters, and the synchronization of the primary and secondary FPS A logical backup is used.

In this embodiment, through file backup, in the case of failure of the primary IDC, the standby IDC can continue to provide services without affecting the storage and use of files, and the availability is higher.

In addition, an embodiment of the present invention also proposes a computer-readable storage medium, where a file storage program is stored on the computer-readable storage medium, and when the file storage program is executed by a processor, the following operations are implemented:

Receive the file uploaded by the sender through the client;

Cache the received files into a temporary folder, and record the storage location information of each file in the distributed storage system;

Merge each file in the temporary folder to obtain the merged file, and store the merged file in the distributed file system;

Based on the merged files, the corresponding storage location information in the distributed storage system is updated.

Further, when the file storage program is executed by the processor, it also realizes the operation of merging each file in the temporary folder to obtain the merged file:

scan individual files in said temporary folder;

Acquiring the combined file, and determining among the scanned files a file whose capacity value after being combined with the combined file is smaller than a preset threshold value, and merging the determined file into the combined file.

Further, when the file storage program is executed by the processor, the following operations are also implemented:

When receiving the file query instruction, determine the index information corresponding to the file query instruction;

Find the merged file pointed to by the index information in the distributed file system;

Restore the merged file, so as to restore the file corresponding to the index information from the merged file.

Further, after the step of storing the merged file in the distributed file system, when the file storage program is executed by the processor, the following operations are also implemented:

Based on the files stored in the distributed file system, generate file identification information and file hash information;

feeding back the file identification information and file hash information to the sender through the client, so that the sender can transmit the file identification information and file hash information to the receiver;

When the client receives the file identification information sent by the recipient, extract the file corresponding to the file identification information in the distributed file system, and feed it back to the recipient, so that the recipient can pass the file The hash information checks the file, and when the check is successful, the file is retrieved.

Further, the number of the server includes multiple, the server and the client of the file processing system are connected through a gateway, and the way of uploading the file from the client to the server includes: the gateway uploads the client to the server according to a preset strategy. The file polling uploaded to the server.

Further, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, when the file storage program is executed by the processor, the following operations are also implemented:

Scan each file in the distributed file system to monitor the storage time of each file;

When the storage duration of a file reaches a preset duration, delete the file in the distributed file system, and delete the storage location information of the file in the distributed storage system.

Further, the Internet data center also includes a distributed application program coordination service, and the server scans each file in the distributed file system to monitor the storage time of each file. Before the step, the file storage program is executed by the processor When executed, the following operations are also implemented:

Send the request information of deleting the lock to the distributed application coordination service;

When the lock is acquired successfully, the step of scanning each file in the distributed file system to monitor the storage duration of each file is executed.

Further, the server is located in the main Internet data center. If there is a backup Internet data center in the system, after the step of updating the corresponding storage location information in the distributed storage system based on the merged file, the When the above-mentioned file storage program is executed by the processor, the following operations are also realized:

Synchronize the stored files to the server of the file processing system where the standby Internet data center is located, so that the server of the file processing system where the standby Internet data center is located performs file storage operations.

It should be noted that, in this document, the terms "comprising", "comprising" or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements that are not expressly listed, or that are inherent to the process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

Claims (10)

1. a kind of file memory method, it is characterised in that applied to Internet data center, the Internet data center includes Document handling system, distributed file system, distributed memory system, the document handling system include service end and client End, the file memory method include：

The service end of document handling system receives the file of sender's upload by client；

By the file cache received into temporary folder, and record in distributed memory system the storage position of each file Confidence ceases；

Processing is merged to each file in temporary folder, the file after being merged, and the file after merging is deposited Store up in distributed file system；

Based on the file after merging, corresponding storage location information in distributed memory system is updated.

2. file memory method as claimed in claim 1, it is characterised in that each file in temporary folder enters Row merging treatment, include the step of file after being merged：

The service end scans each file in the temporary folder；

Composition file is obtained, and determines that the capability value after merging with the composition file is less than pre-set threshold value in the file of scanning File, by the Piece file mergence of determination into the composition file.

3. file memory method as claimed in claim 2, it is characterised in that methods described also includes：

When receiving file polling instruction, index information corresponding to file polling instruction is determined；

The merging file pointed by the index information is searched in distributed file system；

The file that merged is reduced, to restore file corresponding to the index information in file from having merged.

4. file memory method as claimed in claim 1, it is characterised in that the file storage by after merging to distribution After step in file system, methods described also includes：

The service end generates file identification information and file Hash information based on the file stored in distributed file system；

By the client feedback file identification information and file Hash information to described sender, so that described sender will File identification information and file Hash information are transmitted to recipient；

When receiving the file identification information that the recipient sends by the client, extracted in distributed file system File corresponding to the file identification information, and the recipient is fed back to, so that the recipient passes through file Hash information The file is examined, and the file is obtained when examining successfully.

5. file memory method as claimed in claim 1, it is characterised in that the number of the service end is described including multiple The service end of document handling system is connected with client by gateway, and the mode that file is uploaded to service end from client includes： The file poll that client uploads is uploaded in service end by gateway according to default strategy.

6. file memory method as claimed in claim 1, it is characterised in that the file based on after merging, renewal distribution In formula storage system the step of corresponding storage location information after, methods described also includes：

Each file in the service end scanning distributed file system, to monitor the storage duration of each file；

When documentary storage duration reaches preset duration, the file in the distributed file system is deleted, and delete Except the storage location information of file described in the distributed memory system.

7. file memory method as claimed in claim 6, it is characterised in that the Internet data center also includes distribution Application program coordination service, the service end scans each file in distributed file system, to monitor depositing for each file Before the step of storing up duration, methods described also includes：

The service end sends the solicited message for deleting lock to distributed application program coordination service；

Obtain lock successfully when, execution scans each file in distributed file system, during monitoring the storage of each file Long step.

8. the file memory method as described in claim any one of 1-7, it is characterised in that the service end is located at main internet In data center, in the case of standby Internet data center being in systems present, the file based on after merging, renewal distribution In formula storage system the step of corresponding storage location information after, methods described includes：

The service end by the service end of document handling system where the file synchronization of storage to standby Internet data center, with The service end of document handling system performs file storage operations where incense Internet data center.

9. a kind of service end, it is characterised in that the service end includes memory, processor and is stored on the memory simultaneously The file storage program that can be run on the processor, the file storage program are realized as weighed during the computing device Profit requires the step of file memory method any one of 1 to 8.

10. a kind of computer-readable recording medium, it is characterised in that be stored with file on the computer-readable recording medium and deposit Program is stored up, the file storage as any one of claim 1 to 8 is realized when the file storage program is executed by processor The step of method.