CN113051244A - Data access method and device, and data acquisition method and device - Google Patents
Data access method and device, and data acquisition method and device Download PDFInfo
- Publication number
- CN113051244A CN113051244A CN202110303163.7A CN202110303163A CN113051244A CN 113051244 A CN113051244 A CN 113051244A CN 202110303163 A CN202110303163 A CN 202110303163A CN 113051244 A CN113051244 A CN 113051244A
- Authority
- CN
- China
- Prior art keywords
- data
- end node
- node
- address
- identifier
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000004590 computer program Methods 0.000 claims description 13
- 230000005012 migration Effects 0.000 claims description 13
- 238000013508 migration Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 4
- 238000012217 deletion Methods 0.000 claims description 2
- 230000037430 deletion Effects 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 16
- 238000004891 communication Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 239000002699 waste material Substances 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/148—File search processing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a data access method and device, relates to the technical field of cloud computing, and further relates to the technical field of cloud storage. The specific implementation scheme is as follows: sending a front-end data reading request to a front-end node of the distributed file system based on the data reading request of the client; receiving data sent by a front end node, an address of a rear end node and a data access identifier; sending a back-end data read request to a back-end node of the distributed file system based on the address and the data access identifier of the back-end node; receiving data corresponding to the data access identification returned by the back-end node; and combining the data returned by the front-end node and the data returned by the back-end node, and sending the combined data to the client. The embodiment improves the flexibility of data transmission of the distributed file system.
Description
Technical Field
The present application relates to the field of computer technologies, to the field of cloud computing technologies, and further to the field of cloud storage technologies, and in particular, to a data access method and apparatus, a data acquisition method and apparatus, an electronic device, a computer-readable medium, and a computer program product.
Background
When a low-speed medium and a high-speed medium are adopted by a current mainstream distributed file system to store cold and hot data respectively, the high-speed medium is generally regarded as a front-end node, and the low-speed medium is regarded as a back-end node.
In a distributed file system, for a read request of a client, data corresponding to the read request is generally searched in a front-end node, if all corresponding data of the read request are found, the data are directly returned, otherwise, the data are read from a back-end node to the front-end node, the front-end node returns the client and updates the data, and the data access mode not only brings extra data transmission overhead, but also the updating mechanism of the front-end node is not flexible enough.
Disclosure of Invention
A data access method and apparatus, a data acquisition method and apparatus, an electronic device, a computer-readable medium, and a computer program product are provided.
According to a first aspect, there is provided a data access method, the method comprising: sending a front-end data reading request to a front-end node of the distributed file system based on the data reading request of the client; receiving data sent by a front end node, an address of a rear end node and a data access identifier; sending a back-end data read request to a back-end node of the distributed file system based on the address and the data access identifier of the back-end node; receiving data corresponding to the data access identification returned by the back-end node; and combining the data returned by the front-end node and the data returned by the back-end node, and sending the combined data to the client.
According to a second aspect, there is provided a data acquisition method, the method comprising: receiving and analyzing a front-end data reading request to obtain a local data identifier and a data access identifier; acquiring the address of the back-end node corresponding to the data access identifier; acquiring data corresponding to the local data identifier; sending data, a data access identifier and an address of a back-end node.
According to a third aspect, there is provided a data access apparatus comprising: a front-end sending unit configured to send a front-end data read request to a front-end node of the distributed file system based on the data read request of the client; the front-end receiving unit is configured to receive the data sent by the front-end node, the address of the back-end node and the data access identification; a back-end sending unit configured to send a back-end data read request to a back-end node of the distributed file system based on the address and the data access identifier of the back-end node; the back-end receiving unit is configured to receive data corresponding to the data access identification returned by the back-end node; a combined data unit configured to combine data returned by the front-end node and data returned by the back-end node; and the data sending unit is configured to send the combined data to the client.
According to a fourth aspect, there is provided a data acquisition apparatus comprising: the request receiving unit is configured to receive and analyze a front-end data reading request to obtain a local data identifier and a data access identifier; an address acquisition unit configured to acquire an address of a backend node corresponding to the data access identifier; a data acquisition unit configured to acquire an address of a backend node corresponding to the data access identifier; and the data sending unit is configured to send the data, the data access identification and the address of the back-end node.
According to a fifth aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method as described in any one of the implementations of the first aspect or the second aspect.
According to a sixth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform a method as described in any one of the implementations of the first or second aspect.
According to a seventh aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a method as described in any of the implementations of the first or second aspect.
According to the data access method and device provided by the embodiment of the application, firstly, a front-end data read request is sent to a front-end node of a distributed file system based on a data read request of a client; secondly, receiving data sent by a front end node, an address of a rear end node and a data access identifier; thirdly, based on the address and the data access identification of the back-end node, sending a back-end data read request to the back-end node of the distributed file system; receiving data corresponding to the data access identification returned by the back-end node from the second time; and finally, combining the data returned by the front-end node and the data returned by the back-end node, and sending the combined data to the client. Therefore, when the client reads the data of the distributed file system, only the front end node needs to provide the address and the data access identification of the rear end node, and the data of the rear end node does not need to be transmitted to the front end node, so that the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
According to the data acquisition method and device provided by the embodiment of the application, firstly, after a front-end data reading request is received and analyzed, a local data identifier and a data access identifier are obtained; secondly, acquiring the address of the back-end node corresponding to the data access identifier; thirdly, acquiring data corresponding to the local data identifier; and finally, sending the data, the data access identification and the address of the back-end node. Therefore, the front end node of the distributed file system obtains and sends the address of the back end node based on the front end data reading request, so that an access party directly reads the data of the back end node based on the address of the back end node without transmitting the data of the back end node to the front end node, the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a flow diagram according to one embodiment of a data access method of the present application;
FIG. 2 is a schematic structural diagram of a client connected to a distributed file system according to the present application;
FIG. 3 is a flow diagram according to one embodiment of a data acquisition method of the present application;
FIG. 4 is a schematic block diagram of an embodiment of a data access device according to the present application;
FIG. 5 is a schematic block diagram of an embodiment of a data acquisition device according to the present application;
fig. 6 is a block diagram of an electronic device for implementing the data access method or the data acquisition method according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In order to understand the technical solutions mentioned in the embodiments of the present application, several technical terms referred to in the present application are briefly described below.
The distributed file system is formed by a plurality of nodes in a computer cluster in a physical structure, and the distributed file system refers to a file system in which physical storage resources managed by the file system are not necessarily directly connected to local nodes, but are connected with the nodes through a computer network. The distributed file system provides a logical tree file system structure for resources distributed at any position on the network, so that users can access shared files distributed on the network more conveniently.
The hot data is defined according to the frequency of the data being used, such as the data which is frequently inquired and used within half a year by the user.
Cold data, defined in terms of how often data is used, may be, for example, data that is not used often by a user for two years, and may be stored in a lower performance, less expensive file system.
FIG. 1 shows a flow 100 according to one embodiment of a data access method of the present application, the data access method comprising the steps of:
In this embodiment, the execution main body on which the data access method operates may be a protocol server in a distributed File system, for example, an nfs (network File system) server or an smb (server Message block) server that manages a transmission protocol in the distributed File system, and the execution main body may perform information interaction with a client, and when the client has a requirement for reading data from the distributed File system, may further receive a data reading request sent by the client, and convert the received data reading request into a front-end data reading request for accessing each node, specifically referring to fig. 2, a schematic structural diagram that the distributed File system is connected with the client.
In this embodiment, the data read request is a data read request sent by a client to a data distributed file system, and in order to obtain data corresponding to the data read request in the distributed file system, the execution main body needs to convert the data read request of the client into a front-end data read request that can access a front-end node, where a sorting order of data or identifiers in the data read request may be consistent with that in the front-end data read request.
The distributed file system is a system which performs its own functions and collaborates with each other and provides storage services to the outside uniformly, and can mainly meet the following requirements (but not limited to): basic read-write function, performance, scalability, reliability, availability. In the distributed file system, data is stored in a data server cluster, and the data servers in the cluster are responsible for each other and cooperate with each other to provide services of the whole distributed file system. In this embodiment, the distributed file system may be a distributed file system used in cloud storage, the cloud storage is an important component in implementing a cloud computing system architecture, and the cloud storage integrates a large number of storage devices of different types in a network through a cluster application, a grid technology, a distributed file system, and the like, and provides a data storage function and a service access function to the outside. Briefly, cloud storage is the management and use of virtualized storage resources.
In this embodiment, the front-end node and the back-end node are both one or more data servers in a data server cluster, and the front-end node and the back-end node may be defined by a user according to a storage requirement and a storage medium of the data server, for example, hot data is stored in a high-speed medium, which is the front-end node, cold data is stored in a low-speed medium, which is the back-end node; the front-end node and the back-end node can also be obtained by processing the sequencing based on the client request, for example, a data server accessed by the NFS server or the SMB server for the first time is the front-end node, and data servers in the data server cluster except the data server accessed by the NFS server or the SMB server for the first time are the back-end nodes.
The front-end node and the back-end node are relatively speaking, in the access flow of other clients, the front-end node of the embodiment may be used as the back-end node of the access flow, and the back-end node of the embodiment may be used as the front-end node of the access flow.
In this embodiment, the front-end data read request is a request sent by the execution main body to the front-end node to read data of the front-end node, and based on different contents of the analyzed front-end data read request, the contents fed back to the execution main body by the front-end node may be different.
In this embodiment, based on the content of the front-end data read request being different, the content fed back to the execution main body by the front-end node is different. The front-end node analyzes the front-end data read request in receiving the front-end data read request, and identifies a local data identifier in the front-end data read request and a data access identifier which is used for distinguishing the local data identifier and can access other data servers in the distributed file system.
For example, when parsing the front-end data read request includes: and when the local data is identified, the front-end node feeds back data corresponding to the local data identification to the execution main body.
The front-end data read request includes: and when the local data identifier and the access data identifier exist, the front-end node feeds back data corresponding to the local data identifier to the execution main body, and inquires the address of the rear-end node corresponding to the access data identifier.
In this embodiment, the address of the back-end node is recorded in the front-end node, and the back-end node can be determined and accessed through the address of the back-end node. The data access identifier is an identifier recorded in the front-end node and corresponding to a data server in the distributed file system, and data of the back-end node corresponding to the data access identifier can be accessed through the data access identifier.
In this embodiment, the back-end data read request is a request sent by the execution main body to the back-end node to read data of the back-end node, and based on different contents of the analyzed back-end data read request, the contents fed back to the execution main body by the back-end node may be different.
The execution body on which the data access method operates encapsulates the data access identification in the back-end data read request and sends the back-end data read request to the data server indicated by the address of the back-end node.
And step 104, receiving data corresponding to the data access identification returned by the back-end node.
In this embodiment, after receiving the back-end data read request, the back-end node feeds back data corresponding to the data access identifier to the execution main body, and the execution main body of the data access method may receive the data corresponding to the data access identifier returned by the back-end node.
In this embodiment, the front-end data read request may include a local data identifier and a data access identifier, where the local data identifier is an identifier of data stored in the front-end node, and the data access identifier is an identifier of data stored in the back-end node. The local data identification and the data access identification have a sequence in the front-end data reading request, the sequence between the data of the front-end node corresponding to the local data identification and the data of the rear-end node corresponding to the data access identification can be obtained based on the sequence of the local data identification and the data access identification in the front-end data reading request, and the data returned by the front-end node and the data returned by the rear-end node are based on the sequence.
The front-end data reading request is obtained by converting a data reading request of a client, and the data reading request of the client also comprises a local data identifier and a data access identifier. Optionally, the data of the front-end node corresponding to the local data identifier and the data of the back-end node corresponding to the data access identifier may be obtained based on the sequence of the local data identifier and the data access identifier in the data reading request of the client, and the data returned by the front-end node and the data returned by the back-end node may be combined based on the sequence.
And step 106, sending the combined data to the client.
In this embodiment, after obtaining the data reading request of the client, the execution main body sequentially sends a front-end data reading request to the front-end node and a back-end data reading request to the back-end node, combines data returned by the front-end node and the back-end node according to a data sequence required in the data reading request of the client, and finally sends the combined data to the client.
Optionally, the execution body may write data in the data write request to the front-end node after obtaining the data write request of the client, and further, after the hot data in the front-end node becomes cold data, the scheduling module in the execution body may brush down the cold data in the front-end node to the back-end node. Optionally, the execution main body may further obtain an address of the backend node first, access the data in the data write request to the backend according to the address of the backend node, and write the data in the data write request to the backend node.
In a distributed file system, a conventional data access method is as follows: aiming at a data reading request of a client, firstly, a protocol server searches data to be read corresponding to the data reading request in a front-end node, and if all the data to be read corresponding to the data reading request are in the front-end node, the data of the front-end node is read; and if all the data to be read corresponding to the data reading request are not in the front-end node, controlling the front-end node to request the back-end node for missing data, storing the missing data in the front-end node, combining the missing data and the data of the front-end node, returning the combined data to the protocol server, and asynchronously updating the data read from the back-end node to the front-end node.
Compared with the traditional data access method, the data access method provided by the application is more efficient and flexible under the following scenes:
on one hand, in the flow of the data access method, data existing in the back end node is directly requested by the execution main body, and a path from the back end to the front end and then to the protocol server under the traditional scene is cancelled, so that the additional cost of network bandwidth is reduced, and the access delay of a read request can be effectively reduced.
On the other hand, the data access method saves the link of direct interaction of the front end node and the back end node, also avoids the problem that all read data of the back end node are directly updated to the front end node, and can provide a reliable basis for the front end node to better utilize the local storage space when the storage space of the front end node is limited.
The data access method provided by the embodiment of the application comprises the steps that firstly, a front-end data read request is sent to a front-end node of a distributed file system based on a data read request of a client; secondly, receiving data sent by a front end node, an address of a rear end node and a data access identifier; thirdly, based on the address and the data access identification of the back-end node, sending a back-end data read request to the back-end node of the distributed file system; receiving data corresponding to the data access identification returned by the back-end node from the second time; and finally, combining the data returned by the front-end node and the data returned by the back-end node, and sending the combined data to the client. Therefore, when the client reads the data of the distributed file system, only the front end node needs to provide the address and the data access identification of the rear end node, and the data of the rear end node does not need to be transmitted to the front end node, so that the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
In this embodiment, when the front-end node and the back-end node store data in a hierarchical manner (for example, the back-end node is the next-level storage device of the front-end node), if data migration needs to be performed between the front-end node and the back-end node (for example, data in the back-end node needs to be migrated to the front-end node), a flow of a conventional data access method may be initiated once for the data of the front-end node, and the data of the back-end node is migrated to the front-end node by the conventional data access method.
The advantages of using the traditional data access method for data migration are as follows: the traditional reading interface is directly used, extra functions are not needed to be provided, but the traditional data access method is adopted to transfer the data of the back-end node to the front-end node, and meanwhile, extra reading IO is also introduced. Since the storage location of the data of the distributed file system is transparent to the client, migrating data by the conventional data access method only reads the data of the whole data block to ensure that all data of the back-end node is synchronized to the front-end node, for example, assuming that a certain data block has 64M data, 63M data is in the front-end node, and only 1M data is in the back-end node, the data migrated to the back-end node by the conventional data access method introduces reading of the 63M data in the front-end node outside the migration, thereby introducing an additional read IO operation, which affects the performance of the distributed file system.
In some optional implementations of this embodiment, when the front-end node and the back-end node store data hierarchically, the data access method provided by this implementation may further include: querying the data state of the front-end node and the data state of the back-end node; data in the location of the back-end node is migrated to the front-end node in response to the data of the front-end node being empty and the back-end node having the data.
In this embodiment, the front-end node may record a data directory of data blocks of a related node (a node in communication with the front-end node, such as a back-end node) in the distributed file system, where the data directory includes, for example, data blocks corresponding to the related nodes, sizes of the data blocks, and the like. Further, the data directory may also record the data status (status includes empty and has a value) in each data segment of the data block of the relevant node, for example, the data in the 2 nd data segment of the data block of a certain node is empty.
The execution main body can acquire the data state of the front end node and the data state of the back end node through communication with the front end node, wherein the data state of the front end node refers to the state of data stored locally in the front end node; the data state of the back-end node refers to the state of the data stored in the back-end node.
The execution main body compares the state of the data stored by the front end node with the state of the data stored by the back end node, so that whether the data stored in the front end node and the data stored in the back end node are consistent or not can be determined, a certain section of data in a data block corresponding to the front end node and the back end node is empty, a certain section of data in the data block corresponding to the back end node has a value, and the section of data in the back end node needs to be migrated to the front end node.
In this optional implementation, the same location refers to a data location in the front-end node corresponding to the back-end node.
For a distributed file system in units of data blocks as storage, the same location may be a data segment of a data block in the front-end node, and the data segment of the data block of the front-end node corresponds to a data segment of a database of the back-end node. Further, each data block in the front-end node is assumed to be divided into 64 segments, and the same location may be the data segment represented by the same index. And the same index refers to any value between 0 and 63. For example, the data segment with the front end node being 2 and the data segment with the back end node being 2 in the same data block represent the same range of data.
In order to avoid the influence on the normal IO performance of the distributed file system, optionally, the priority of the step of migrating data in the position of the back-end node to the front-end node may be set to be smaller than the priority of the step of receiving the data read request of the client, priority control is defined on the data migration tasks of the front-end node and the back-end node, and a lower priority is set for the data migration step of the background by default, so as to ensure the access delay of the external IO request.
In the optional implementation mode, by inquiring the data states of the front-end node and the back-end node, when the data of the front-end node at the same position is empty and the back-end node has the data, the data in the position of the back-end node is migrated to the front-end node, so that the data migration between the front-end node and the back-end node can be quickly and conveniently realized when the IO performance of the distributed file system is not influenced and the front-end node and the back-end node are stored in a grading manner, and the flexibility of the data migration between multiple nodes is improved.
When the front-end node and the back-end node store hierarchically, the front-end node stores hot data and the back-end node stores cold data, the cold data needs to be brushed down to the back-end node after the hot data of the front-end node becomes cold, and the cold data in the front-end node is deleted to free up space of the front-end node for more hot data to use. When the data of the back-end node is pulled back to the front-end node (that is, the data of the front-end node in the same position is empty and the back-end node has the data), optionally, the front-end node may be used as a data cache to directly copy the data in the same position of the back-end node to the front-end node and maintain the data in the same position of the back-end node unchanged. The optional mode is suitable for the scene that the storage space of the rear end node is far larger than that of the front end node and the rear end node, the cost is lower, or the data migration frequency of the front end node and the rear end node is higher.
When the front-end node and the back-end node store hierarchically and data of the back-end node needs to be pulled back to the front-end node, in some optional implementation manners of this embodiment, after data in the same position of the back-end node is migrated to the front-end node, the data in the position of the back-end node is deleted.
After data is migrated from the back-end node to the front-end node, the data exists in the front-end node and the back-end node at the same time, based on the principle that data storage of the distributed file system is based on the front-end node, the data in the back-end node is redundant in practice, and after the data is migrated, the IO performance of the distributed file system cannot be influenced by adding a data deleting step of the back-end node.
In this optional implementation, deleting the data in the location of the backend node can reduce the waste of the storage space of the backend node.
FIG. 3 shows a flow 300 according to an embodiment of a data acquisition method of the present application, the data access method comprising the steps of:
In this embodiment, the data obtaining method is applied to the distributed file system, and the execution subject on which the data obtaining method operates may be any one data server in a data server cluster of the distributed file system, where the any one data server is also a front-end node. The front-end node can be defined by a user according to storage requirements and a storage medium of a data server, for example, hot data is stored in a high-speed medium, namely the front-end node; the front-end nodes can also be obtained by processing sequencing based on client requests, for example, a data server accessed by an NFS server or an SMB server for the first time in the file distribution system is the front-end node.
In this embodiment, the front-end data read request is a request for reading data in the execution main body, which is sent to the execution main body by a protocol server in the distributed file system, and the contents obtained by the execution main body are different based on different contents of the analyzed front-end data read request.
For example, when parsing the front-end data read request includes: and when the local data identifier is obtained, the execution main body obtains the data corresponding to the local data identifier, wherein the local data identifier refers to the identifier of the data stored in the execution main body.
In this embodiment, the data access identifier is recorded in the execution body, and the identifier corresponding to the data server in the distributed file system may access the data of the backend node corresponding to the data access identifier through the data access identifier.
In this embodiment, the front-end data read request includes: accessing the data identification, the executing agent needs to query the address of the backend node corresponding to the accessing data identification.
In the distributed file system, the back-end node is a node opposite to the execution subject, when the execution subject is a high-speed medium, and a low-speed medium arranged in the distributed file system is the back-end node; and when the data server accessed by the NFS server or the SMB server for the first time is a front-end node, the data servers except the data server accessed by the NFS server or the SMB server for the first time in the distributed file system are back-end nodes.
Each data server in the data server cluster of the distributed file system stores a directory of another data server, and the directory records information such as addresses and data states of the other data servers. The executing body can obtain the address of the back-end node connected with the executing body by inquiring the directory of other data servers and sends the address of the back-end node to the NFS server or the SMB server.
In a distributed file system, the NFS server or SMB server, after obtaining the address of the backend node, can access the backend node through the address of the backend node.
In this embodiment, the front-end data read request includes: and the local data identifier is an identifier indicating data in the execution main body, and the execution main body can obtain the data corresponding to the local data identifier through the local data identifier.
In this embodiment, after obtaining the data corresponding to the local data identifier, the data access identifier, and the address of the backend node, the execution main body may send the data corresponding to the local data identifier, the data access identifier, and the address of the backend node to the NFS server or the SMB server.
Further, after obtaining the address of the backend node, the NFS server or the SMB server can access the backend node through the address of the backend node, and read the data in the backend node through the data access identifier.
According to the data acquisition method provided by the embodiment of the application, firstly, after a front-end data reading request is received and analyzed, a local data identifier and a data access identifier are obtained; secondly, acquiring the address of the back-end node corresponding to the data access identifier; thirdly, acquiring data corresponding to the local data identifier; and finally, sending the data, the data access identification and the address of the back-end node. Therefore, the front end node of the distributed file system obtains and sends the address of the back end node based on the front end data reading request, so that an access party directly reads the data of the back end node based on the address of the back end node without transmitting the data of the back end node to the front end node, the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
In some optional implementation manners of this embodiment, the data obtaining method further includes: receiving a data state query request; sending the local data state and the data state of the back-end node based on the data state query request; and when the local data at the same position is empty and the back-end node has the data, receiving and storing the data in the back-end node position.
In this optional implementation manner, the data state query request may be a request sent by a scheduling module in a protocol server of the distributed file system, where the data state query request includes: a local data state query request and a back-end node data state query request; based on the data state query request, the data state of each node in the locally stored distributed file system may be obtained, for example, the 2 nd bit of the data block of a certain node is empty.
In this embodiment, the execution principal may record a data directory of data blocks of a related node (a node in communication with the execution principal, such as a backend node) in the distributed file system, where the data directory includes, for example, data blocks corresponding to the related nodes, sizes of the data blocks, and the like. Further, the data directory may also record the data status (status includes empty and has a value) in each data segment of the data block of the relevant node, for example, the data in the 2 nd data segment of the data block of a certain node is empty. The execution agent, through the data directory, can determine the data state of the backend node.
The execution main body compares the local data storage state with the storage state of the back-end node, whether the execution main body is consistent with the data stored in the back-end node or not can be determined, when a certain section of data in a data block corresponding to the back-end node in the execution main body is empty, the back-end node has a value corresponding to the certain section of data in the data block, and the section of data in the back-end node needs to be migrated to the execution main body.
In this optional implementation, the same location refers to a data location corresponding to the backend node in the slave execution body.
For a distributed file system with data blocks as storage units, the same location may be a data segment of a data block in the execution body, and the data segment of the data block of the execution body corresponds to a data segment of a database of the backend node. Further, each data block in the execution body is assumed to be divided into 64 segments, and the same position may be the data segment represented by the same index. And the same index refers to any value between 0 and 63. For example, the data segment with body 2 and the data segment with backend node 2 represent the same range of data in the same data block.
In the optional implementation mode, by inquiring the data states of the front-end node and the back-end node, when the data of the front-end node at the same position is empty and the back-end node has the data, the data in the position of the back-end node is migrated to the front-end node, so that the data migration between the front-end node and the back-end node can be quickly and conveniently realized when the IO performance of the distributed file system is not influenced and the front-end node and the back-end node are stored in a grading manner, and the flexibility of the data migration between multiple nodes is improved.
With further reference to fig. 4, as an implementation of the method shown in the above figures, the present application provides an embodiment of a data access apparatus, which corresponds to the embodiment of the method shown in fig. 1, and which is particularly applicable to various electronic devices.
As shown in fig. 4, the data access apparatus 400 provided in the present embodiment includes: front-end transmitting section 401, front-end receiving section 402, back-end transmitting section 403, back-end receiving section 404, combined data section 405, and data transmitting section 406. The front-end sending unit 401 is configured to send a front-end data read request to a front-end node of the distributed file system based on a data read request of a client. The front-end receiving unit 402 is configured to receive the data sent by the front-end node, the address of the back-end node, and the data access identifier. The backend sending unit 403 is configured to send a backend data read request to the backend node of the distributed file system based on the address and the data access identifier of the backend node. The backend receiving unit 404 is configured to receive the data corresponding to the data access identifier returned by the backend node. The above-mentioned combined data unit 405 is configured to combine data returned by the front-end node and data returned by the back-end node. The data sending unit 406 is configured to send the combined data to the client.
In the present embodiment, in the data access apparatus 400: the detailed processing of the front-end sending unit 401, the front-end receiving unit 402, the back-end sending unit 403, the back-end receiving unit 404, the combined data unit 405, and the data sending unit 406 and the technical effects thereof may refer to the related descriptions of step 101, step 102, step 103, step 104, step 105, and step 106 in the corresponding embodiment of fig. 1, which are not described herein again.
In some optional implementations of this embodiment, the apparatus 400 further includes: query unit (not shown), migration unit (not shown). Wherein the query unit may be configured to query the data state of the front-end node and the data state of the back-end node. The migration unit may be configured to migrate data in the location of the back-end node to the front-end node in response to the data of the front-end node in the same location being empty and the back-end node having the data.
In some optional implementations of this embodiment, the apparatus 400 further includes: a deletion unit (not shown in the figure). Wherein the deleting unit may be configured to delete data in a location of the backend node.
In the data access device provided in the embodiment of the present application, first, the front end sending unit 401 sends a front end data read request to a front end node of a distributed file system based on a data read request of a client; secondly, the front end receiving unit 402 receives the data sent by the front end node, the address of the back end node and the data access identifier; thirdly, the back end sending unit 403 sends a back end data read request to the back end node of the distributed file system based on the address and the data access identifier of the back end node; from then, the backend receiving unit 404 receives the data corresponding to the data access identifier returned by the backend node; finally, the combined data unit 405 combines the data returned by the front-end node and the data returned by the back-end node, and the data sending unit 406 sends the combined data to the client. Therefore, when the client reads the data of the distributed file system, only the front end node needs to provide the address and the data access identification of the rear end node, and the data of the rear end node does not need to be transmitted to the front end node, so that the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
As shown in fig. 5, the data acquisition apparatus 500 provided in the present embodiment includes: a request receiving unit 501, an address acquiring unit 502, a data acquiring unit 503, and a data transmitting unit 504. The request receiving unit 501 is configured to receive and analyze a front-end data read request, so as to obtain a local data identifier and a data access identifier. The address obtaining unit 502 is configured to obtain an address of the backend node corresponding to the data access identifier. The data obtaining unit 503 is configured to obtain an address of the backend node corresponding to the data access identifier. The data transmission unit 504 is configured to transmit data, a data access identifier, and an address of a backend node.
In the present embodiment, in the data acquisition apparatus 500: the specific processing of the request receiving unit 501, the address obtaining unit 502, the data obtaining unit 503, and the data sending unit 504 and the technical effects thereof can refer to the related descriptions of step 301, step 302, step 303, and step 304 in the corresponding embodiment of fig. 3, which are not described herein again.
In some optional implementations of this embodiment, the apparatus 500 further includes: a query unit (not shown), a status sending unit (not shown), and a data receiving unit (not shown). Wherein, the query unit may be configured to receive a data status query request. The state sending unit may be configured to send the local data state and the data state of the backend node based on the data state query request. The data receiving unit may be configured to receive and store the data in the backend node location when the local data in the same location is empty and the backend node has the data.
In the data acquisition apparatus provided in the embodiment of the present application, first, after the request receiving unit 501 receives and analyzes a front-end data read request, a local data identifier and a data access identifier are obtained; secondly, the address obtaining unit 502 obtains the address of the back-end node corresponding to the data access identifier; again, the data obtaining unit 503 obtains data corresponding to the local data identifier; finally, the data sending unit 504 sends the data, the data access identifier, and the address of the backend node. Therefore, the front end node of the distributed file system obtains and sends the address of the back end node based on the front end data reading request, so that an access party directly reads the data of the back end node based on the address of the back end node without transmitting the data of the back end node to the front end node, the additional transmission overhead and the waste of storage space of the data are reduced, and the flexibility of data transmission of the distributed file system is improved.
There is also provided, in accordance with an embodiment of the present application, an electronic device, a readable storage medium, and a computer program product.
FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 executes the respective methods and processes described above, such as the data access method or the data acquisition method. For example, in some embodiments, the data access method or data acquisition method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into the RAM603 and executed by the computing unit 601, one or more steps of the data access method or the data acquisition method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data access method or the data acquisition method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present application can be achieved, and the present invention is not limited herein.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (13)
1. A method of data access, the method comprising:
sending a front-end data reading request to a front-end node of the distributed file system based on the data reading request of the client;
receiving data sent by the front-end node, an address of a back-end node and a data access identifier;
sending a back-end data read request to a back-end node of the distributed file system based on the address of the back-end node and the data access identifier;
receiving data corresponding to the data access identification returned by the back-end node;
and combining the data returned by the front-end node and the data returned by the back-end node, and sending the combined data to the client.
2. The method of claim 1, further comprising:
querying the data state of the front-end node and the data state of the back-end node;
migrating data in the location of the back-end node to the front-end node in response to the data of the front-end node being empty and the back-end node having data in the same location.
3. The method of claim 2, further comprising:
deleting data in the location of the backend node.
4. A method of data acquisition, the method comprising:
receiving and analyzing a front-end data reading request to obtain a local data identifier and a data access identifier;
acquiring the address of the back-end node corresponding to the data access identifier;
acquiring data corresponding to the local data identifier;
and sending the data, the data access identification and the address of the back-end node.
5. The method of claim 4, further comprising:
receiving a data state query request;
sending local data state and data state of a back-end node based on the data state query request;
when local data in the same location is empty and the backend node has data, the data in the location of the backend node is received and stored.
6. A data access apparatus, the apparatus comprising:
a front-end sending unit configured to send a front-end data read request to a front-end node of the distributed file system based on the data read request of the client;
the front-end receiving unit is configured to receive the data sent by the front-end node, the address of the back-end node and the data access identification;
a back-end sending unit configured to send a back-end data read request to a back-end node of the distributed file system based on the address of the back-end node and the data access identifier;
the back-end receiving unit is configured to receive data corresponding to the data access identification returned by the back-end node;
a combined data unit configured to combine data returned by the front-end node and data returned by the back-end node;
a data sending unit configured to send the combined data to the client.
7. The apparatus of claim 6, the apparatus further comprising:
a query unit configured to query the data state of the front-end node and the data state of the back-end node;
a migration unit configured to migrate data in the location of the back-end node to the front-end node in response to data of the front-end node being empty and the back-end node having data in the same location.
8. The apparatus of claim 7, further comprising:
a deletion unit configured to delete data in the location of the backend node.
9. A data acquisition apparatus, the apparatus comprising:
the request receiving unit is configured to receive and analyze a front-end data reading request to obtain a local data identifier and a data access identifier;
an address acquisition unit configured to acquire an address of a backend node corresponding to the data access identifier;
a data acquisition unit configured to acquire an address of a backend node corresponding to the data access identifier;
a data sending unit configured to send the data, the data access identifier, and the address of the backend node.
10. The apparatus of claim 9, the apparatus further comprising:
a query unit configured to receive a data status query request;
a state sending unit configured to send local data state and data state of a backend node based on the data state query request;
a data receiving unit configured to receive and store data in the location of the backend node when local data in the same location is empty and the backend node has the data.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-5.
13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110303163.7A CN113051244B (en) | 2021-03-22 | 2021-03-22 | Data access method and device, and data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110303163.7A CN113051244B (en) | 2021-03-22 | 2021-03-22 | Data access method and device, and data acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113051244A true CN113051244A (en) | 2021-06-29 |
| CN113051244B CN113051244B (en) | 2023-07-21 |
Family
ID=76514523
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110303163.7A Active CN113051244B (en) | 2021-03-22 | 2021-03-22 | Data access method and device, and data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113051244B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114442962A (en) * | 2022-02-15 | 2022-05-06 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and storage medium |
| CN117270782A (en) * | 2023-10-11 | 2023-12-22 | 北京火山引擎科技有限公司 | Data reading and writing method and device and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060136574A1 (en) * | 2004-12-16 | 2006-06-22 | International Business Machines Corporation | System and method for request priority transfer across nodes in a multi-tier data processing system network |
| US20130121075A1 (en) * | 2011-11-11 | 2013-05-16 | Neil David Hutchison | Systems and methods for operating multi-bank nonvolatile memory |
| CN107864215A (en) * | 2017-11-21 | 2018-03-30 | 中国科学院上海微系统与信息技术研究所 | Peer-to-peer network file system, access control/management method/system and terminal |
| US20180107408A1 (en) * | 2016-10-14 | 2018-04-19 | Netapp, Inc. | Read and Write Load Sharing in a Storage Array Via Partitioned Ownership of Data Blocks |
| CN111680015A (en) * | 2020-05-29 | 2020-09-18 | 北京百度网讯科技有限公司 | File resource processing method, device, equipment and medium |
| CN111966471A (en) * | 2020-10-21 | 2020-11-20 | 北京首都在线科技股份有限公司 | Access method, device, electronic equipment and computer storage medium |
| CN112084204A (en) * | 2020-09-10 | 2020-12-15 | 北京百度网讯科技有限公司 | Browsing data processing method, device, terminal and storage medium |
-
2021
- 2021-03-22 CN CN202110303163.7A patent/CN113051244B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060136574A1 (en) * | 2004-12-16 | 2006-06-22 | International Business Machines Corporation | System and method for request priority transfer across nodes in a multi-tier data processing system network |
| US20130121075A1 (en) * | 2011-11-11 | 2013-05-16 | Neil David Hutchison | Systems and methods for operating multi-bank nonvolatile memory |
| US20180107408A1 (en) * | 2016-10-14 | 2018-04-19 | Netapp, Inc. | Read and Write Load Sharing in a Storage Array Via Partitioned Ownership of Data Blocks |
| CN107864215A (en) * | 2017-11-21 | 2018-03-30 | 中国科学院上海微系统与信息技术研究所 | Peer-to-peer network file system, access control/management method/system and terminal |
| CN111680015A (en) * | 2020-05-29 | 2020-09-18 | 北京百度网讯科技有限公司 | File resource processing method, device, equipment and medium |
| CN112084204A (en) * | 2020-09-10 | 2020-12-15 | 北京百度网讯科技有限公司 | Browsing data processing method, device, terminal and storage medium |
| CN111966471A (en) * | 2020-10-21 | 2020-11-20 | 北京首都在线科技股份有限公司 | Access method, device, electronic equipment and computer storage medium |
Non-Patent Citations (2)
| Title |
|---|
| TENG MA 等: "AsymNVM: An Efficient Framework for Implementing Persistent Data Structures on Asymmetric NVM Architecture", 《ASPLOS \'20: PROCEEDINGS OF THE TWENTY-FIFTH INTERNATIONAL CONFERENCE ON ARCHITECTURAL SUPPORT FOR PROGRAMMING LANGUAGES AND OPERATING SYSTEMS》, pages 757 * |
| 陈亮;郭成城;: "嵌入式集群系统的自动管理监测系统设计", 计算机工程与应用, no. 13 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114442962A (en) * | 2022-02-15 | 2022-05-06 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and storage medium |
| CN114442962B (en) * | 2022-02-15 | 2024-07-02 | 北京百度网讯科技有限公司 | Data reading method, device, equipment and storage medium |
| CN117270782A (en) * | 2023-10-11 | 2023-12-22 | 北京火山引擎科技有限公司 | Data reading and writing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113051244B (en) | 2023-07-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109254733B (en) | Method, device and system for storing data | |
| US11301154B2 (en) | Distributed storage method and device | |
| CN111309732B (en) | Data processing method, device, medium and computing equipment | |
| US8510499B1 (en) | Solid state drive caching using memory structures to determine a storage space replacement candidate | |
| CN114116613A (en) | Metadata query method, equipment and storage medium based on distributed file system | |
| CN113961510B (en) | File processing method, device, equipment and storage medium | |
| CN114756509B (en) | File system operation method, system, device and storage medium | |
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| EP4120060A1 (en) | Method and apparatus of storing data,and method and apparatus of reading data | |
| CN113806300B (en) | Data storage method, system, device, equipment and storage medium | |
| KR20210040864A (en) | File directory traversal method, apparatus, device, and medium | |
| US20220342888A1 (en) | Object tagging | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device | |
| CN115168440A (en) | Data read-write method, distributed storage system, device, equipment and storage medium | |
| CN110737397B (en) | Method, apparatus and computer program product for managing a storage system | |
| CN112783887A (en) | Data processing method and device based on data warehouse | |
| CN113742131B (en) | Method, electronic device and computer program product for storage management | |
| US20250086067A1 (en) | Method and apparatus for fast generating a full snapshot, electronic device and storage medium | |
| CN117667879A (en) | Data access method and device, storage medium and electronic equipment | |
| CN114490540B (en) | Data storage method, medium, device and computing equipment | |
| CN115905121A (en) | File processing method, device, equipment and storage medium | |
| JP7458610B2 (en) | Database system and query execution method | |
| US10067678B1 (en) | Probabilistic eviction of partial aggregation results from constrained results storage | |
| CN103502953B (en) | The method and apparatus improving the concurrency performance of distributed objects storage system | |
| CN113821519A (en) | Data processing method and field-driven design architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |