CN111046001A - A method, device, device and storage medium for creating files in batches - Google Patents
A method, device, device and storage medium for creating files in batches Download PDFInfo
- Publication number
- CN111046001A CN111046001A CN201911385174.3A CN201911385174A CN111046001A CN 111046001 A CN111046001 A CN 111046001A CN 201911385174 A CN201911385174 A CN 201911385174A CN 111046001 A CN111046001 A CN 111046001A
- Authority
- CN
- China
- Prior art keywords
- creation
- files
- target client
- file
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000013475 authorization Methods 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012790 confirmation Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Storage Device Security (AREA)
Abstract
本发明公开了一种批量创建文件的方法、装置、设备及存储介质，该方法包括：接收目的客户端发送的授权请求，并确定与授权请求对应的目录为指定目录；判断指定目录是否已被授权给除目的客户端之外的其他客户端，如果是，则等待直至完成与对指定目录具有权限的客户端对应全部文件的批量创建后，释放指定目录，将指定目录授权给目的客户端，如果否，则将指定目录授权给目的客户端；接收目的客户端发送的多个创建请求，基于接收的创建请求实现相应的多个文件的同时创建，存储创建得到的每个文件的元数据，并在完成目的客户端对应全部文件的批量创建后释放指定目录。本申请能够提高文件创建性能，提高分布式文件系统的性能。
The present invention discloses a method, device, equipment and storage medium for batch creation of files, the method comprising: receiving an authorization request sent by a destination client, and determining that the directory corresponding to the authorization request is a designated directory; judging whether the designated directory has been authorized to other clients except the destination client, if so, waiting until the batch creation of all files corresponding to the client having authority to the designated directory is completed, releasing the designated directory, and authorizing the designated directory to the destination client, if not, authorizing the designated directory to the destination client; receiving multiple creation requests sent by the destination client, realizing the simultaneous creation of multiple corresponding files based on the received creation requests, storing metadata of each created file, and releasing the designated directory after the batch creation of all files corresponding to the destination client is completed. The present application can improve the file creation performance and the performance of the distributed file system.
Description
Technical Field
The present invention relates to the field of distributed file system technology, and more particularly, to a method, an apparatus, a device, and a storage medium for creating files in batch.
Background
Distributed cluster storage is widely applied in various scenes of current big data (such as high performance, video monitoring, broadcast and television media resources and the like), mass data exist in a distributed file system, and the premise of big data calculation is to organize and store the mass data.
Disclosure of Invention
The invention aims to provide a method, a device, equipment and a storage medium for creating files in batch, which can improve the file creating performance and further improve the performance of a distributed file system.
In order to achieve the above purpose, the invention provides the following technical scheme:
a method of creating files in batches, comprising:
receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing the metadata of each file obtained by the creation, and releasing the specified directory after completing the batch creation of all the files corresponding to the target client.
Preferably, before receiving the authorization request sent by the destination client, the method further includes:
receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and applying for a file number in a preset range for the target client in advance;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to the preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
Preferably, after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further includes:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; wherein the metadata of the file includes the file name and the file number having an association.
Preferably, the method further comprises the following steps:
when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
Preferably, the method further comprises the following steps:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request of the target client record, which is retransmitted by the target client, after the fault is repaired; and the client records a creation request corresponding to each file to be created.
Preferably, after the target client is implemented to create all files in batch, the method further includes:
and returning confirmation information of successful batch creation of all the files corresponding to the target client to indicate the creation request of deleting the record of the target client.
Preferably, after the target client is implemented to create all files in batch, the method further includes:
and sending a marking instruction to the target client to indicate the target client to record a mark of successful creation of the batch files in a log of file creation.
An apparatus for creating files in batches, comprising:
a determination module to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
a determination module configured to: judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
a creation module to: receiving a plurality of creation requests sent by the target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing the specified directory after completing batch creation of all files corresponding to the target client.
An apparatus for creating files in batches, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method for batch creation of files as described in any one of the above when executing the computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method of bulk creating files as claimed in any one of the preceding claims.
The invention provides a method, a device, equipment and a storage medium for creating files in batches, wherein the method comprises the following steps: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files; judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory; receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing the metadata of each file obtained by the creation, and releasing the specified directory after completing the batch creation of all the files corresponding to the target client. According to the technical scheme, when the client needs to create the files in batch, the authority of the appointed directory can be obtained through application, if the appointed directory is not authorized to other clients currently, the authority of the appointed directory can be obtained, and then the files are created in batch under the appointed directory, otherwise, the authority of the appointed directory is obtained after the access authority of other clients to the appointed directory is removed, and then the files are created in batch under the appointed directory. Therefore, the files required to be created can be created in batches under the instruction of the client, the file creation performance is improved, and the performance of the distributed file system is further improved; in addition, by setting the exclusive right for the specified directory, the situation that the names of created files are the same and further file creation fails when multiple clients simultaneously achieve file creation under the specified directory can be avoided, effective implementation of batch creation of the files is guaranteed, the situation that when one client achieves file creation under the specified directory, other clients cannot access the information such as the valid files when accessing the specified directory can be avoided, and the effectiveness of the clients in obtaining the information is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for creating files in batch according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an apparatus for creating files in batch according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flowchart of a method for creating files in batch according to an embodiment of the present invention is shown, where the method may include:
s11: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client which needs to create the batch files.
The execution subject of the method for creating files in batches provided by the embodiment of the present invention may be a corresponding device, and the device may be disposed in a client, so that the execution subject of the method may be a server, and the following detailed description will be given by taking the execution subject of the method as a server.
When any client needs to create files in batch, the client needing to create the files in batch can be called as a target client, so that the file batch creation of any client is realized. When a target client needs to create files in batch, an authorization request can be sent to a server, after the server receives the authorization request sent by the target client, a directory corresponding to the authorization request, that is, a directory identifier carried in the authorization request can be determined, and the directory for which the target client needs to create files in batch, which can be called as an appointed directory. In addition, in this embodiment, creating a file in batch, that is, creating multiple files simultaneously, and a Server can provide an MDS (Meta Data Server) metadata service, the method for creating a file in batch implemented in this embodiment of the present application is implemented based on the MDS.
S12: judging whether the appointed directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the appointed directory is completed, releasing the appointed directory, authorizing the appointed directory to the target client, and if not, authorizing the appointed directory to the target client; wherein the specified directory allows access only to clients having rights to the specified directory.
It should be noted that, in this embodiment, a Cap (Capability, metadata operation authority) that monopolizes a directory is introduced, and when a certain client holds the Cap of a certain directory, other clients cannot operate the directory to perform operations such as reading and writing, file creation, and the like; specifically, when a certain directory is an appointed directory, the appointed directory can only be authorized to one client at the same time, the client authorized by the appointed directory is a client having access right to the appointed directory, namely, the client can realize operations such as information reading and writing, file creation and the like under the appointed directory, and other clients not authorized cannot access the appointed directory; because if a plurality of clients create files under a specified directory at the same time, the situation that the names of the files created by different clients are the same may occur, but the situation that the names of the files are the same is actually not allowed, the situation that the names of the created files are the same and further the file creation fails when the plurality of clients simultaneously achieve the file creation under the same directory can be avoided through the setting of the authority of the directory; moreover, if a certain client creates a file or reads and writes under a directory, other clients accessing the directory at the moment cannot access the latest data information, that is, cannot access valid data information, so that through the setting of the permission to the directory, the situation that when one client achieves file creation under a directory, other clients cannot access valid data information such as files and the like under the directory can be avoided. Therefore, in this embodiment, after determining the designated directory, the server may determine whether the designated directory is currently authorized to other clients except the destination client, that is, whether other clients except the destination client have the right to the designated directory, if so, the designated directory cannot be authorized to the destination client, so that the designated directory is released after the batch creation of all files that the client having the right to the designated directory needs to implement is completed, that is, after the right to the designated directory is released, the designated directory does not have the client having the right to the designated directory, and at this time, the designated directory may be authorized to the destination client; if not, the designated directory may be directly authorized to the destination client.
In addition, if the specified directory is authorized to other clients except the destination client, the destination client may wait until the specified directory has the authority of the specified directory after being released, or may implement the sequential creation of the files in a conventional manner, and may specifically perform setting according to actual needs, all of which are within the protection scope of the present invention.
S13: receiving a plurality of creation requests sent by a target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing a specified directory after batch creation of all files corresponding to the target client is completed.
After the target client side applies for obtaining the authority of the specified directory, the creation requests in a certain time (or certain items) can be packaged simultaneously, and the packaged result is sent to the server, so that the server analyzes the creation requests contained in the packaged result, the batch creation, namely simultaneous creation, of the corresponding files is further realized, the metadata of each file obtained by the creation is stored, and the specified directory is released after all files required to be created by the target client side are created and is available for other client sides to obtain. In addition, in this embodiment, the certain time may be total time required for instructing the server to implement batch creation of the files, and the certain entry may be total number of the files required for instructing the server to implement batch creation of the files, so as to implement batch creation of the files according to the requirement of the target client.
According to the technical scheme, when the client needs to create the files in batch, the authority of the appointed directory can be obtained through application, if the appointed directory is not authorized to other clients currently, the authority of the appointed directory can be obtained, and then the files are created in batch under the appointed directory, otherwise, the authority of the appointed directory is obtained after the access authority of other clients to the appointed directory is removed, and then the files are created in batch under the appointed directory. Therefore, the files required to be created can be created in batches under the instruction of the client, the file creation performance is improved, and the performance of the distributed file system is further improved; in addition, by setting the exclusive right for the specified directory, the situation that the names of created files are the same and further file creation fails when multiple clients simultaneously achieve file creation under the specified directory can be avoided, effective implementation of batch creation of the files is guaranteed, the situation that when one client achieves file creation under the specified directory, other clients cannot access the information such as the valid files when accessing the specified directory can be avoided, and the effectiveness of the clients in obtaining the information is guaranteed.
The method for creating files in batch provided by the embodiment of the invention can further include, before receiving an authorization request sent by a destination client:
receiving a connection request sent by a target client, responding to the connection request to establish connection with the target client, and pre-applying a file number in a preset range for the target client;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to a preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
It should be noted that, in this embodiment, in order to achieve simultaneous creation of multiple corresponding files, a file number of each file may be quickly obtained, and then the file creation rate is increased, so in this embodiment, after receiving a connection request sent by a destination client, a connection with the destination client may be established, and a file number (Inode, a file number that is included in a node) in a certain range (a preset range and may be set according to actual needs) is applied for the destination client from a metadata pool, and then allocation of the file number may be directly achieved when creating the file, and it is not necessary to obtain the file number from the metadata pool, and thus the file creation rate is greatly increased. After the server applies for a certain range of file numbers for the target client in advance, the server can inform the target client of all the file numbers distributed for the target client, so that the target client can know the file numbers of the files which are determined to be created based on the file numbers. In addition, if the target client needs to create files in batch and the file numbers allocated to the target client are used up, a certain range of file numbers can be allocated to the target client from the metadata pool, and so on, so as to ensure that the target client has sufficient file numbers to create the files.
The method for creating files in batch provided by the embodiment of the present invention may further include, after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; the metadata of the file includes a file name and a file number having an association.
The creation request sent by the destination client may carry a file name (i.e., a name of a file) set by the client for the corresponding file, so that the name of the file obtained by creation is set as the file name carried in the corresponding creation request, and the file name and the file number of each file are associated and stored as metadata, thereby facilitating searching for the file and the like. In addition, the information having the association may specifically be a CInode (metadata Inode information, which mainly includes file attributes) and a CDentry (metadata directory entry, which mainly includes file names), where the CInode includes a file number, and the CDentry includes a file name, so that the associated CInode and CDentry include an associated file name and file number, and further include the associated CInode and CDentry in the metadata for storage, for example, in a metadata pool.
The method for creating files in batch provided by the embodiment of the invention can further comprise the following steps:
when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
It should be noted that, the target client in this embodiment may package and send the creation requests corresponding to all files that the target client needs to create to the server, or may package and send the creation requests included in each batch to the server after dividing all files that the target client needs to create into multiple batches, which may be specifically determined according to actual situations; correspondingly, the application of the destination client for having the authority for the specified directory may include a time period in which the authority for the specified directory needs to be provided and the total number of all files that need to be created under the specified directory, while the incomplete destination client corresponds to the batch creation of all files, and specifically, the application of the destination client for having the authority for the specified directory may not have the time period in which the authority for the specified directory has the authority for the specified directory, and/or the number of the files created under the specified directory does not have the total number of all files that the destination client needs to create. In addition, the destination client can cache the creation request which needs to be sent and is not sent yet locally; if the target client side does not finish the batch creation of all files corresponding to the target client side, authorization requests sent by other client sides are received, whether the target client side has creation requests which are not sent to the server or not can be judged, if yes, the target client side can be informed to forcibly refresh the cached creation requests to the server, so that the server can realize corresponding file batch creation, file metadata storage and the like, if not, the server can finish the file batch creation, the file metadata storage and the like corresponding to the creation requests which are sent to the server by the target client side, and then the authority of the specified directory is released after the batch creation and the corresponding metadata storage of all files which need to be created by the target client side are realized, so that the use of the specified directory by other client sides can be facilitated.
The method for creating files in batch provided by the embodiment of the invention can further comprise the following steps:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request recorded by the target client and retransmitted by the target client after the fault is repaired; the client records a creation request corresponding to each file to be created.
The client can locally record the creation request to be sent before sending the creation request to the server, and further if the server fails during processing the batch creation request, the server needs to repair the failure, and after the server is repaired and can normally work, the client needs to resend the recorded creation request to the server, and the server realizes batch file creation again after receiving the creation request resent by the client, so that the effective realization of batch file creation by the client is ensured.
The method for creating files in batch provided by the embodiment of the invention can further include the following steps after the target client side is created in batch corresponding to all the files:
and returning confirmation information of successful batch creation of all the files corresponding to the target client to indicate the creation request of deleting the record of the target client.
After the server completes batch creation of all files corresponding to the target client, confirmation information can be returned to the target client, so that the target client can know the created information of all files to be created, and further delete the recorded creation request, thereby avoiding waste of storage space in the target client.
Moreover, after the target client is implemented to perform batch creation of all files, the method may further include:
and sending a marking instruction to the target client to indicate the target client to record the mark of successful creation of the batch file in the log of file creation.
By sending the marking instruction to the target client, the target client can be instructed to record the mark of successful creation of the batch file in the log after receiving the marking instruction, so as to be queried at a later stage.
In a specific implementation scenario, a method for creating files in batch provided in an embodiment of the present application may include:
(1) inode pre-allocation:
when the server is connected with the client, the server applies for Inode in a certain range from the metadata pool and informs the client;
(2) the Cap application:
when a client creates a file under a certain directory, applying for exclusive Cap of the directory to a server, and if no other client uses the directory, the server agrees to grant the exclusive Cap of the directory to the client; and if other clients are using the directory, the server refuses to authorize the exclusive Cap, and after all files of other clients are created, the server agrees to authorize the exclusive Cap of the directory to the client.
(3) Sending in batches:
after applying for exclusive Cap of the directory, the client packs the creation request within a certain time (or a certain request entry) and sends the request to the server, and the client records the sent request (for fault processing); if a certain time or an item period is not reached, other clients apply for the exclusive Cap of the directory to the server, and the server informs the client of forcibly refreshing the creation request in the cache and then releases the exclusive Cap;
(4) the server processes the batch requests:
after receiving the batch creation request of the client, the server analyzes the batch creation request, creates a corresponding file, associates the CInode and the CDentry of the file, drops all created metadata into a metadata pool, and then responds (can comprise confirmation information and a marking instruction) to the client to complete the request;
(5) and a finishing stage:
after receiving the server response, the client clears the sent request recorded in the client, and marks that the batch creation service is completed;
(6) and (3) fault repair:
if the server fails during processing of the batch creation request, the client needs to resend the batch creation request to the server for processing again in the server recovery phase.
The method for creating the files in batch in the distributed file system can greatly improve the file creating performance and the metadata creating performance, particularly the creating performance of small file scenes, and further improve the creating performance of the distributed file system.
An embodiment of the present invention further provides a device for creating files in batch, and as shown in fig. 2, the device specifically includes:
a determining module 11, configured to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create the batch files;
a judging module 12, configured to: judging whether the appointed directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the appointed directory is completed, releasing the appointed directory, authorizing the appointed directory to the target client, and if not, authorizing the appointed directory to the target client; wherein, the appointed directory only allows the client with the authority to access the appointed directory;
a creation module 13 for: receiving a plurality of creation requests sent by a target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing a specified directory after batch creation of all files corresponding to the target client is completed.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a connection module for: before receiving an authorization request sent by a target client, receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and pre-applying a file number in a preset range for the target client;
an assignment module to: after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request, file numbers belonging to a preset range are respectively distributed to each file obtained by the creation; wherein the metadata of the file includes a file number of the file.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a setup module to: after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request, respectively setting corresponding file names for each file obtained by creation, and establishing association between the file name of each file obtained by creation and a file number; the metadata of the file includes a file name and a file number having an association.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
an indication module to: when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a reconstruction module to: if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request recorded by the target client and retransmitted by the target client after the fault is repaired; the client records a creation request corresponding to each file to be created.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a return module to: and after the batch creation of all files corresponding to the target client is realized, returning confirmation information of successful batch creation of all files corresponding to the target client to indicate the creation request of deleting the record of the target client.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a sending module configured to: after the target client side is established in batch corresponding to all the files, a marking instruction is sent to the target client side so as to indicate the target client side to record a mark of successful establishment of the batch files in a log of file establishment.
An embodiment of the present invention further provides a device for creating files in batch, where the device may include:
a memory for storing a computer program;
a processor for implementing the steps of the method for batch creation of files as described in any one of the above when executing a computer program.
The embodiment of the invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for creating the files in batches as described above can be implemented.
It should be noted that for the description of the relevant parts in the apparatus, the device, and the storage medium for creating files in batches provided in the embodiment of the present invention, reference is made to the detailed description of the corresponding parts in the method for creating files in batches provided in the embodiment of the present invention, and details are not repeated here. In addition, parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for creating files in batches, comprising:
receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing the metadata of each file obtained by the creation, and releasing the specified directory after completing the batch creation of all the files corresponding to the target client.
2. The method of claim 1, wherein before receiving the authorization request sent by the destination client, the method further comprises:
receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and applying for a file number in a preset range for the target client in advance;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to the preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
3. The method of claim 2, wherein after enabling the simultaneous creation of the respective plurality of files based on the received creation request, further comprising:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; wherein the metadata of the file includes the file name and the file number having an association.
4. The method of claim 1, further comprising:
when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
5. The method of claim 1, further comprising:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request of the target client record, which is retransmitted by the target client, after the fault is repaired; and the client records a creation request corresponding to each file to be created.
6. The method according to claim 5, wherein after the batch creation of all files corresponding to the destination client is realized, the method further comprises:
and returning confirmation information of successful batch creation of all the files corresponding to the target client to indicate the creation request of deleting the record of the target client.
7. The method according to claim 6, wherein after the batch creation of all files corresponding to the destination client is realized, the method further comprises:
and sending a marking instruction to the target client to indicate the target client to record a mark of successful creation of the batch files in a log of file creation.
8. An apparatus for creating files in batches, comprising:
a determination module to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
a determination module configured to: judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
a creation module to: receiving a plurality of creation requests sent by the target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing the specified directory after completing batch creation of all files corresponding to the target client.
9. An apparatus for creating files in batches, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method of bulk creating files according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method for bulk creation of files according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911385174.3A CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
| PCT/CN2020/110731 WO2021128892A1 (en) | 2019-12-28 | 2020-08-24 | Method and apparatus for creating files in batches, device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911385174.3A CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111046001A true CN111046001A (en) | 2020-04-21 |
| CN111046001B CN111046001B (en) | 2023-03-14 |
Family
ID=70241006
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911385174.3A Active CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111046001B (en) |
| WO (1) | WO2021128892A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112597120A (en) * | 2020-11-30 | 2021-04-02 | 新华三大数据技术有限公司 | Catalog management method and device |
| WO2021128892A1 (en) * | 2019-12-28 | 2021-07-01 | 浪潮电子信息产业股份有限公司 | Method and apparatus for creating files in batches, device, and storage medium |
| CN113849467A (en) * | 2021-09-26 | 2021-12-28 | 金蝶软件（中国）有限公司 | Data processing method, device and equipment and readable storage medium |
| CN115580610A (en) * | 2022-09-22 | 2023-01-06 | 广州文远知行科技有限公司 | Method, device and equipment for uploading batch files and readable storage medium |
| CN115720223A (en) * | 2022-10-17 | 2023-02-28 | 北京思特奇信息技术股份有限公司 | A method and system for file current limiting transmission control based on Linux system |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760853B (en) * | 2021-08-16 | 2024-02-20 | 联想凌拓科技有限公司 | Directory processing method, server and storage medium |
| CN116611988B (en) * | 2023-05-23 | 2024-04-26 | 释空(上海)展示制作有限公司 | Image processing method, electronic device, and storage medium |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100049921A1 (en) * | 2008-08-25 | 2010-02-25 | International Business Machines Corporation | Distributed Shared Caching for Clustered File Systems |
| CN103179185A (en) * | 2012-12-25 | 2013-06-26 | 中国科学院计算技术研究所 | Method and system for creating files in client cache of distributed file system |
| US20130218934A1 (en) * | 2012-02-17 | 2013-08-22 | Hitachi, Ltd. | Method for directory entries split and merge in distributed file system |
| CN103597503A (en) * | 2011-06-27 | 2014-02-19 | 阿玛得斯两合公司 | Method and system for a pre-shopping reservation system with increased search efficiency |
| CN104021137A (en) * | 2014-04-21 | 2014-09-03 | 中国科学院计算技术研究所 | Method and system for opening and closing file locally through client side based on catalogue authorization |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| US9307025B1 (en) * | 2011-03-29 | 2016-04-05 | Riverbed Technology, Inc. | Optimized file creation in WAN-optimized storage |
| CN105512279A (en) * | 2015-12-04 | 2016-04-20 | 华为技术有限公司 | Metadata access method, related equipment and system |
| CN105630810A (en) * | 2014-10-30 | 2016-06-01 | 曙光信息产业股份有限公司 | Method for uploading mass small files in distributed storage system |
| CN106462544A (en) * | 2014-03-31 | 2017-02-22 | 亚马逊科技公司 | Session management in distributed storage systems |
| CN106485156A (en) * | 2016-09-22 | 2017-03-08 | 中广核工程有限公司 | A kind of apparatus and method for files in batch mandate |
| CN109408474A (en) * | 2018-11-09 | 2019-03-01 | 郑州云海信息技术有限公司 | A kind of file creating method and client |
| CN110138881A (en) * | 2019-06-05 | 2019-08-16 | 安徽三实信息技术服务有限公司 | A kind of distributed memory system and its storage method |
| CN110597773A (en) * | 2019-08-28 | 2019-12-20 | 重庆爱奇艺智能科技有限公司 | Method and apparatus for sharing files between computer device and virtual reality device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102024016B (en) * | 2010-11-04 | 2013-03-13 | 曙光信息产业股份有限公司 | Rapid data restoration method for distributed file system (DFS) |
| US10157183B2 (en) * | 2015-07-28 | 2018-12-18 | International Business Machines Corporation | Real time and scalable space-time data recovery |
| CN111046001B (en) * | 2019-12-28 | 2023-03-14 | 浪潮电子信息产业股份有限公司 | Method, device and equipment for creating files in batch and storage medium |
-
2019
- 2019-12-28 CN CN201911385174.3A patent/CN111046001B/en active Active
-
2020
- 2020-08-24 WO PCT/CN2020/110731 patent/WO2021128892A1/en active Application Filing
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100049921A1 (en) * | 2008-08-25 | 2010-02-25 | International Business Machines Corporation | Distributed Shared Caching for Clustered File Systems |
| US9307025B1 (en) * | 2011-03-29 | 2016-04-05 | Riverbed Technology, Inc. | Optimized file creation in WAN-optimized storage |
| CN103597503A (en) * | 2011-06-27 | 2014-02-19 | 阿玛得斯两合公司 | Method and system for a pre-shopping reservation system with increased search efficiency |
| US20130218934A1 (en) * | 2012-02-17 | 2013-08-22 | Hitachi, Ltd. | Method for directory entries split and merge in distributed file system |
| CN103179185A (en) * | 2012-12-25 | 2013-06-26 | 中国科学院计算技术研究所 | Method and system for creating files in client cache of distributed file system |
| CN106462544A (en) * | 2014-03-31 | 2017-02-22 | 亚马逊科技公司 | Session management in distributed storage systems |
| CN104021137A (en) * | 2014-04-21 | 2014-09-03 | 中国科学院计算技术研究所 | Method and system for opening and closing file locally through client side based on catalogue authorization |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| CN105630810A (en) * | 2014-10-30 | 2016-06-01 | 曙光信息产业股份有限公司 | Method for uploading mass small files in distributed storage system |
| CN105512279A (en) * | 2015-12-04 | 2016-04-20 | 华为技术有限公司 | Metadata access method, related equipment and system |
| CN106485156A (en) * | 2016-09-22 | 2017-03-08 | 中广核工程有限公司 | A kind of apparatus and method for files in batch mandate |
| CN109408474A (en) * | 2018-11-09 | 2019-03-01 | 郑州云海信息技术有限公司 | A kind of file creating method and client |
| CN110138881A (en) * | 2019-06-05 | 2019-08-16 | 安徽三实信息技术服务有限公司 | A kind of distributed memory system and its storage method |
| CN110597773A (en) * | 2019-08-28 | 2019-12-20 | 重庆爱奇艺智能科技有限公司 | Method and apparatus for sharing files between computer device and virtual reality device |
Non-Patent Citations (4)
| Title |
|---|
| MENG LIU ET AL.: "Privacy-Preserving Detection of Statically Mutually Exclusive Roles Constraints Violation in Interoperable Role-Based Access Control", 《2017 IEEE TRUSTCOM/BIGDATASE/ICESS》 * |
| 刘芬: "分布式文件系统客户端元数据缓存一致性的实现及优化", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 杨洪章等: "分布式文件系统中海量小文件异步创建技术", 《网络新媒体技术》 * |
| 樊金生等: "基于角色的访问控制扩展模型及其实现", 《计算机工程与设计》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021128892A1 (en) * | 2019-12-28 | 2021-07-01 | 浪潮电子信息产业股份有限公司 | Method and apparatus for creating files in batches, device, and storage medium |
| CN112597120A (en) * | 2020-11-30 | 2021-04-02 | 新华三大数据技术有限公司 | Catalog management method and device |
| CN112597120B (en) * | 2020-11-30 | 2022-05-27 | 新华三大数据技术有限公司 | Catalog management method and device |
| CN113849467A (en) * | 2021-09-26 | 2021-12-28 | 金蝶软件（中国）有限公司 | Data processing method, device and equipment and readable storage medium |
| CN115580610A (en) * | 2022-09-22 | 2023-01-06 | 广州文远知行科技有限公司 | Method, device and equipment for uploading batch files and readable storage medium |
| CN115580610B (en) * | 2022-09-22 | 2024-03-19 | 广州文远知行科技有限公司 | Batch file uploading method, device, equipment and readable storage medium |
| CN115720223A (en) * | 2022-10-17 | 2023-02-28 | 北京思特奇信息技术股份有限公司 | A method and system for file current limiting transmission control based on Linux system |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021128892A1 (en) | 2021-07-01 |
| CN111046001B (en) | 2023-03-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046001A (en) | A method, device, device and storage medium for creating files in batches | |
| US10776285B2 (en) | Data write method, apparatus, and system | |
| US9563636B2 (en) | Allowing writes to complete without obtaining a write lock to a file | |
| US4961224A (en) | Controlling access to network resources | |
| KR20020090520A (en) | Parallel logging method of transaction processing system | |
| CN105653406A (en) | Method for managing distributed storage system and distributed storage system | |
| CN105701156A (en) | Distributed file system management method and device | |
| JP2008046860A (en) | File management system and file management method | |
| CN107566470B (en) | Method and device for managing virtual machine in cloud data system | |
| CN117591034B (en) | Storage disk arbitration method, device, equipment and storage medium | |
| CN111309799A (en) | Method, device and system for realizing data merging and storage medium | |
| CN109144403B (en) | Method and equipment for switching cloud disk modes | |
| US7395283B1 (en) | Method and apparatus for making independent data copies in a data processing system | |
| CN111176559B (en) | Data writing method and device and dual-activity system | |
| CN107704596A (en) | A kind of method, apparatus and equipment for reading file | |
| US8677363B2 (en) | Method for managing, tracking and distributing job programs for processing to a plurality of execution computers | |
| CN109032536A (en) | Date storage method, device, system and equipment based on distributed cluster system | |
| CN105790975A (en) | Service processing operation execution method and device | |
| CN112162949B (en) | File management method, system, electronic device and storage medium | |
| CN111274176B (en) | Information processing method, electronic equipment, system and storage medium | |
| CN110688057B (en) | Distributed storage method and device | |
| CN109828720B (en) | Data storage method, device, server and storage medium | |
| CN108345431B (en) | Method and device for reading data | |
| JP3212787B2 (en) | Transfer data management method and data transfer method between host and terminal | |
| CN113568881A (en) | Method, system, equipment and medium for realizing read-write sharing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |