CN113535087A - Data processing method, server and storage system in data migration process - Google Patents
Data processing method, server and storage system in data migration process Download PDFInfo
- Publication number
- CN113535087A CN113535087A CN202110791646.6A CN202110791646A CN113535087A CN 113535087 A CN113535087 A CN 113535087A CN 202110791646 A CN202110791646 A CN 202110791646A CN 113535087 A CN113535087 A CN 113535087A
- Authority
- CN
- China
- Prior art keywords
- server
- data
- migration
- target
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a server and a storage system in a data migration process, and relates to the field of the Internet. A data processing method in a data migration process comprises the following steps: receiving a data processing request of a user in a data migration process; obtaining a target data pool corresponding to the data processing request; determining a server to be processed from a source server and a target server according to the migration state of the target data pool; the source server is a server to which the target data pool belongs before migration, and the target server is a server to which the target data pool belongs after migration; and carrying out corresponding data processing on the server to be processed. The invention can normally process the data processing request in the process of data migration.
Description
Technical Field
The invention relates to the field of internet, in particular to a data processing method, a server and a storage system in a data migration process.
Background
In the related art, when data migration is performed, in order to avoid generating dirty data, for example, new data to be migrated is generated during the data migration, or original data is modified during the migration process, a service needs to be suspended, and the service is reopened after the data migration is completed.
Disclosure of Invention
The invention mainly aims to provide a data processing method, a server and a storage system in a data migration process, and aims to solve the technical problem that service needs to be suspended in the data migration process.
In order to achieve the above object, the present invention provides a data processing method in a data migration process, the method comprising the following steps:
receiving a data processing request of a user in a data migration process;
obtaining a target data pool corresponding to the data processing request;
determining a server to be processed from a source server and a target server according to the migration state of the target data pool; the source server is a server to which the target data pool belongs before migration, and the target server is a server to which the target data pool belongs after migration;
and carrying out corresponding data processing on the server to be processed.
In one embodiment, the data processing request is a write request;
the step of determining a server to be processed from a source server and a target server according to the migration state of the target data pool includes:
and if the target data pool is in a migration state, taking the target server as a server to be processed.
In one embodiment, the data processing request is a read request;
the step of determining a server to be processed from a source server and a target server according to the migration state of the target data pool comprises the following steps:
and if the target data pool is in a migration state, taking the source server as a server to be processed.
In an embodiment, the step of obtaining the target data pool corresponding to the data processing request includes:
analyzing the reading request to obtain a target data file corresponding to the reading request;
taking the data pool to which the target data file belongs as the target data pool;
the step of taking the source server as a server to be processed if the target data pool is in a migration state includes:
if the target data pool is in a migration state, determining the migration state of the target data file;
and if the migration state is a state in the migration or a state before the migration, taking the source server as a server to be processed.
In an embodiment, if the target data pool is in the migration state, after determining the migration state of the target data file, the method further includes:
and if the migration state is the stable state, taking the target server as a server to be processed.
In an embodiment, when the number of servers in the storage system increases after the data migration, and before the step of receiving a data processing request of a user during the data migration, the method further includes:
determining a new server from the storage system;
according to a server storage space utilization table, screening a first storage server with the highest space utilization rate from all the servers and all the newly added servers, and screening a second storage server with the highest disk space from all the newly added servers; the server storage space utilization table stores storage space utilization information of all servers and all newly added servers and disk space information of all the newly added servers;
screening out a data pool to be allocated with the largest occupied space from all data pools of the first storage server;
allocating the second storage server as a target server of the data pool to be allocated, and generating migration allocation information;
updating the server storage space utilization table based on the migration allocation information, returning to execute the step of screening out the first storage server with the highest space utilization rate from all the servers and all the newly-added servers according to the server storage space utilization table, circulating to the server storage space utilization table, wherein the mean square deviation of the storage space utilization rates of all the storage servers and all the newly-added servers does not exceed a preset threshold value, and allocating the server to which the data migration belongs to after the data migration to all the data pools in the storage system.
In an embodiment, after data migration, when the number of servers in the storage system is reduced, the server reserved after data migration is information of a third server, and the deleted server is a fourth server;
before the step of receiving a data processing request of a user in the data migration process, the method further includes:
screening a fifth server with the lowest space utilization rate from all the third servers according to the server storage space utilization rate table;
screening out a data pool to be distributed with the largest occupied space from all data pools of all the fourth servers;
allocating the fifth server as a target server of the data pool to be allocated, and generating migration allocation information;
updating the server storage space utilization table based on the migration allocation information, returning to execute the step of screening out a fifth server with the lowest space utilization rate from all the third servers according to the server storage space utilization table, and circulating that the mean square error of the storage space utilization rates of all the third servers in the server storage space utilization table does not exceed a preset threshold value so as to allocate the server to which the data migration belongs to all the data pools in the storage system.
In a second aspect, the present invention further provides a data processing apparatus in a data migration process, including:
the request receiving module is used for receiving a data processing request of a user in the data migration process; the data processing request comprises a read request or a write request;
the request analysis module is used for analyzing the data processing request to obtain a target data pool corresponding to the data processing request;
the server determining module is used for determining a server to be processed from a source server and a target server according to the migration state of the target data pool; the source server is a server to which the target data pool belongs before migration, and the target server is a server to which the target data pool belongs after migration; and the data processing module is used for carrying out corresponding data processing on the server to be processed.
In a third aspect, the present invention further provides a metadata server, including a memory, a processor, and a data processing program stored on the memory and executable on the processor in a data migration process, where the data processing program in the data migration process is configured to implement the steps of the foregoing method.
In a fourth aspect, the present invention further provides a distributed storage system, including:
a metadata server; and
at least one server, all of which are connected to the metadata server.
In a fifth aspect, the present invention also proposes a computer program product comprising executable program code, wherein said program code, when executed by a processor, performs the data processing method in the data migration procedure as described above.
The data processing method in the data migration process provided by the embodiment of the invention is characterized in that when a target data pool corresponding to a data processing request is in a migration state, a server to be processed is determined from a source server before data migration and a target server after data migration, and data processing is carried out on the server to be processed. Therefore, in the process of data migration, the read request and the write request can be processed normally, dirty data is not generated, and the data migration is ensured to be performed normally without interruption.
Drawings
FIG. 1 is a diagram illustrating a metadata server according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a data processing method during data migration according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a data processing method during data migration according to a second embodiment of the present invention;
FIG. 4 is a flowchart illustrating a data processing method during data migration according to a third embodiment of the present invention;
FIG. 5 is a flowchart illustrating a data processing method during data migration according to a fourth embodiment of the present invention;
FIG. 6 is a functional block diagram of a data processing apparatus according to a first embodiment of the present invention during a data migration process;
fig. 7 is a schematic diagram of detailed functional modules of the first embodiment of the data processing apparatus in the data migration process according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the related art, when data migration is performed, in order to avoid generating dirty data, for example, new data to be migrated is generated during the data migration, or original data is modified during the migration process, a service needs to be suspended, and the service is reopened after the data migration is completed.
To solve this problem, various embodiments of the data processing method in the data migration process of the present invention are proposed. When the target data pool corresponding to the data processing request is in a migration state, the method determines the server to be processed from the source server before data migration and the target server after data migration, and processes the data. Therefore, in the process of data migration, the read request and the write request can be processed normally, dirty data is not generated, and the migration process is guaranteed to be performed normally and not interrupted.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a recommendation device of a metadata server in a hardware operating environment according to an embodiment of the present invention.
Typically, the metadata server comprises at least one processor 301, a memory 302, and a data handler in a data migration process stored on the memory and executable on the processor.
The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 301 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable logic Array). The processor 301 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen.
In some embodiments, the metadata server may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected by a bus or signal lines. Various peripheral devices may be connected to communication interface 303 via a bus, signal line, or circuit board. The communication interface 303 is used for receiving a data processing request sent by a user. In some embodiments, processor 301, memory 302, and communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302 and the communication interface 303 may be implemented on a single chip or circuit board, which is not limited in this embodiment.
Those skilled in the art will appreciate that the architecture shown in FIG. 1 does not constitute a limitation of a metadata server, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
Further, the invention also provides a storage system, which further comprises the metadata server and at least one storage server. Wherein all storage servers are connected with the metadata server.
An embodiment of the present invention provides a data processing method in a data migration process, and referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of the data processing method in the data migration process according to the present invention.
In this embodiment, the data processing method in the data migration process may be applied to a metadata server in a distributed storage system, where the distributed storage system further includes at least one storage server. The following is a further description of the embodiment as applied to a metadata server in a distributed storage system. Those skilled in the art can easily think of the situation when the method is used in a traditional centralized storage system or a cloud storage system or other storage systems on the basis of the disclosure of the present application, and the details of this embodiment are not described herein.
The metadata server has only one server S0At the same time, server S0Having the entire data pool set Cp. When there are two servers, server S0、S1Having the entire set of data pools C in commonpWhich isIn, S0Owning data pool set Cp0，S1Owning data pool set Cp1，Cp0∈Cp，Cp1∈Cp，Cp0∩Cp1＝φ，Cp0∪Cp1＝Cp. Multiple servers and so on.
In this embodiment, the data processing method in the data migration process includes the following steps:
step S101, in the data migration process, receiving a data processing request of a user.
And the data migration process indicates that at least one part of all the data needing migration is not completed. In this step, the data processing request for transacting the service by the user is received during the data migration process, and in this process, the data migration task is not suspended, and the data processing request initiated by the user is not suspended. Wherein the data processing request comprises a read request or a write request.
And step S102, obtaining a target data pool corresponding to the data processing request.
The data processing request carries information of the data file it points to, which data file has a unique ID. The unique ID may be generated from the storage server network card physical MAC address + file name. And the data pool pointed by the data file in the data processing request is the target data pool.
In this step, the target data pool may be obtained by parsing the data processing request, or the target data pool may be directly extracted from the target data pool.
Step S103, determining a server to be processed from a source server and a target server according to the migration state of the target data pool; the source server is a server to which the target data pool belongs before migration, and the target server is a server to which the target data pool belongs after migration.
In this embodiment, the metadata server is provided with a first status information table therein. The first state information table stores migration state information of all data pools and storage server information to which any data pool belongs, and the storage server information comprises source server information to which the data pools belong before migration and target server information to which the data pools belong after migration.
For example, the first state information table is a mapping relationship table between the data pool and the server shown in table 1.
TABLE 1
The first state information table stores migration state information of all data pools and storage server information to which the data pools belong.
The migration state information includes three types, namely a stable state, a state to be migrated and a state in migration:
the stable state is that the data has been migrated or has been completely copied to the target server on the migration path. At this time, the storage server mapped by the data pool in the first state information table is the current server, that is, the migrated target server. E.g., in the storage server column of the first state information table, for the data pool PiThe storage server information thereof is' Sk”，SkThe target server to which the migration is completed belongs.
The to-be-migrated state is that the data pool has been allocated with a corresponding target server, but has not yet started to be migrated. At this time, the storage server mapped by the data pool in the first state information table includes a source server to which the data pool belongs before migration and a target server to which the data pool belongs after migration. E.g., in the storage server column of the first state information table, for the data pool PiThe storage server information thereof is' Si|Sk", thereby reflecting its migration path, SiFor a data pool PiSource server, S, to which data belong before migrationkThe target server to which the migration belongs.
The in-migration state, i.e., the data in the data pool, is being migrated, but there are some outstanding migrations. At this time, the storage server mapped by the data pool in the first state information table includes a source server to which the data pool belongs before migration and a target server to which the data pool belongs after migration.
The three states can be represented by using different assignments in the data pool and server mapping relationship table. For example, PState0 represents a stable state, PState1 represents a to-be-migrated state, and PState2 represents a migration-in-process state.
In this step, the metadata server may read the assignment of the migration state information of the target data pool in the first state information table, thereby determining which of the stable state, the to-be-migrated state, and the migration state the target data pool is in.
After the migration state of the target data pool is determined, the server to be processed can be determined from the source server and the target server. The server to be processed can perform corresponding operations, so that corresponding data processing can be executed in response to the data processing request in the data migration process.
And step S104, carrying out corresponding data processing on the server to be processed.
After the servers to be processed, which need to perform data processing, are determined, corresponding data processing is performed on the corresponding servers to be processed, for example, data files are written according to a write request, or data files are read according to a read request.
In this embodiment, after the migration state of the target data pool is determined, a server to be processed is determined from the source server before data migration and the target server after data migration, and data processing is performed on the server. Therefore, in the process of data migration, the data processing request can be processed normally, dirty data is not generated, and the migration process is guaranteed to be performed normally and not interrupted.
It is easy to understand that if the target data pool is in a to-be-migrated state, the data file in the target data pool has not yet started to be migrated, so that the source server of the target data pool can be used as a to-be-processed server for reading, the generated data can be migrated in the subsequent migration, and thus dirty data is not generated, and data processing can also be normally performed.
If the target data pool is in a stable state, all data files of the target data pool are migrated, and therefore the target server of the target data pool can be used as a server to be processed for reading and writing. At this time, since the state of the target data pool is stable, dirty data is not generated, and data processing can be normally performed.
In a specific embodiment, the data processing request is a write request. At this time, in this embodiment, the method includes:
step S101', in the data migration process, a write request of a user is received.
And step S102', obtaining a target data pool corresponding to the write request.
Because the data processing request is a write request, the write request is to newly add a data file in the storage system. The metadata server can analyze the data processing request to obtain a target data pool to which the data file to be written in the write request belongs.
Step S103', if the target data pool is in a migration state, the target server is used as a server to be processed.
And step S104, carrying out corresponding data processing on the server to be processed.
Specifically, if the data processing request is a write request, the target server after the target data pool is migrated is used as a server to be processed to perform data processing, that is, the newly added data file is written into the target server to which the data pool is to be migrated.
And determining that the to-be-processed server of the write request is the target server after the target data pool is migrated, namely, directly writing the newly-added data file into the target server, thereby preventing the source server from generating dirty data in the migration process. The specific location where the written new data file is written may be the storage space allocated to the target data pool. Therefore, in the process of data migration, the read request and the write request can be processed normally, dirty data is not generated, and the migration process is guaranteed to be performed normally and not interrupted.
In a specific embodiment, the data processing request is a read request. At this time, in this embodiment, the method includes:
step S101', in the data migration process, a reading request of a user is received.
And step S102', obtaining a target data pool corresponding to the write request.
Because the data processing request is a read request, the read request is to read the stored data file, and the metadata server can analyze the data processing request to obtain a target data pool to which the data file to be read by the read request belongs.
Step S103', if the target data pool is in a migration state, the source server is used as a server to be processed.
And step S104, carrying out corresponding data processing on the server to be processed.
Specifically, if the data processing request is a read request, the source server before the migration of the target data pool is used as a server to be processed to perform data processing, that is, the target file is read from the source server.
And determining that the pending server of the read request is the source server before the migration of the target data pool, namely reading the required data file from the source server, thereby preventing the reading failure.
In this embodiment, after the migration state of the target data pool is determined, if the data processing request is a write request, the target server after the migration of the target data pool is used as a to-be-processed server to perform data processing, that is, a newly-added data file is written into the target server to which the data pool is to be migrated, and if the data processing request is a read request, the source server before the migration of the target data pool is used as the to-be-processed server to perform data processing, that is, the target file is read from the source server. Therefore, in the process of data migration, the read request and the write request can be processed normally, dirty data is not generated, and the migration process is guaranteed to be performed normally and not interrupted.
In addition, based on the first embodiment of the data processing method in the data migration process of the present invention, a second embodiment of the data processing method in the data migration process of the present invention is proposed. Referring to fig. 3, fig. 3 is a flowchart illustrating a data processing method in a data migration process according to a second embodiment of the present invention. It is easy to understand that, due to different space occupation in the data pools, there are more target data files in the data pools with higher space occupation, and it is also necessary to further determine the migration status of the target data files to avoid data reading failure.
The data processing method in the data migration process in the embodiment includes the following steps:
step S201, in the data migration process, a read request of a user is received.
Step S202, analyzing the reading request to obtain a target data file corresponding to the reading request.
And step S203, taking the data pool to which the target data file belongs as a target data pool.
Step S204, if the target data pool is in a migration state, determining the migration state of the target data file.
The target data file is the data file to be read by the read request. In this embodiment, the metadata server may read the migration status information of any data file through the second status information table.
As shown in table 2, the second state information table is a mapping relationship table between the data file and the data pool built in the metadata server.
TABLE 2
The second state information table stores migration state information of all data files and data pool information to which any data file belongs. The migration state information includes three states, namely a stable state, a state to be migrated and a state in migration.
And the stable state is that the data file is completely migrated or is completely copied to the target server on the migration path. At this time, the server information mapped by the data pool to which the data file belongs in the second state information table may be obtained by reading the first state information table in real time.
The to-be-migrated state is a state in which the data pool to which the data file belongs enters the to-be-migrated state or a state in migration, but the data file has not yet started to be migrated.
The in-migration state is that the data file is being migrated.
It will be readily appreciated that the three states can be represented by using different assignments in the second state information table. Such as: FState0 represents a steady state, FState1 represents a state to be migrated, and FState2 represents a state in migration.
In this step, the metadata server may determine the migration status information of the target data file by reading the second status information table, so as to determine which of the stable status, the to-be-migrated status, and the migration status the target data file is specifically in.
Step S205, if the target data file is in the migration state or the pre-migration state, the source server is taken as the server to be processed.
Specifically, if the metadata server determines that the target data file is in the migration state by reading the second state information table, the metadata server may determine the server information mapped by the data pool to which the target data file belongs by reading the first state information table.
And if the target data pool is in a migration state, the server information mapped by the target data pool comprises a source server to which the target data pool belongs before data migration and a target server to which the target data pool belongs after data migration. At this time, in order to avoid reading failure, the source server to which the target data pool data belongs before being migrated may be used as a server to be processed, so that the target data file may be successfully read.
And when the target data pool belongs to the state before the migration, the server information mapped by the target data pool also comprises a source server to which the target data pool belongs before the migration and a target server to which the target data pool belongs after the migration. At this time, in order to avoid reading failure, the source server to which the target data pool data belongs before being migrated may be used as a server to be processed, so that the target data file may be successfully read.
It is easy to understand that, if the target data file is in a stable state, at this time, the target data file has already been migrated, and at this time, the target server to which the data of the target data pool is migrated may be used as a server to be processed, so that the target data file may be successfully read.
And S206, carrying out corresponding data processing on the server to be processed.
After the server to be processed, which needs to execute data processing, is determined, the target data file is read from the corresponding server to be processed.
In this embodiment, the migration state information of the target data file in the data pool is further determined for the data pool in the migration state, so that successful data processing can be further ensured in the data migration process, and the reliability of data processing is ensured.
In some embodiments, a first state information table is deployed in the metadata server, migration state information of all data pools and storage server information to which any data pool belongs are stored in the first state information table, and the storage server information includes source server information to which the data pool belongs before migration and target server information to which the data pool belongs after migration. After the data migration is completed, any data file and any data pool are in a stable state, and at the moment, the metadata server can update the storage server information in the first state information table according to the target server to which the data pool belongs. After updating, the information of the storage server mapped by any data pool in the first state information table only has the information of the target server to which the data pool belongs after migration, namely the information of the current server to which the data pool belongs.
In addition, an embodiment of the present invention provides a third embodiment of a data processing method in a data migration method, and referring to fig. 4, fig. 4 is a flowchart illustrating the third embodiment of the data processing method in the data migration method according to the present invention.
As will be readily appreciated, in the distributed storage system, migration tasks are generated in the distributed storage system due to the increase in the number of storage servers, that is, the change in the state of the distributed storage system.
In this embodiment, before step S101, the following steps are further included:
step A10, determining the new server from the storage system.
In this embodiment, the first state change information of the storage system may be received. The first state change information is used for indicating that the number of the servers in the storage system is increased, the first state change information comprises newly added server information, and the newly added server information comprises disk space information of the newly added server.
So that a newly added server of the storage system can be determined based on the first state information.
After receiving the first state change information of the storage system, the metadata server can obtain information such as the number of the newly added servers, addresses, the size of the disk space of each newly added server, the data pool of each newly added server, the size of the disk space of each data pool and the like. The disk space information of the newly added server can be recorded by newly creating a temporary table.
E.g., the number of data storage servers n becomes m, where m>n>0. The number k of the newly added data storage servers is m-n, and the set of the newly added data storage servers is CsNewly added data storage server Sn0、Sn1、……Snk-1∈Cs。
Step A20, according to the utilization table of the storage space of the server, screening out the first storage server with the highest space utilization rate from all the storage servers and all the newly added servers, and screening out the second storage server with the largest disk space from all the newly added servers. The server storage space utilization table stores storage space utilization information of all servers and all newly added servers, and disk space information of all the newly added servers.
The metadata server is internally provided with a server storage space utilization table, the storage space utilization rate of each storage server is rho, and rho is U/D, wherein U is the disk space of the storage server occupied by the data file, and D is the total disk space of the storage server.
And meanwhile, the metadata server can update the server storage space utilization table according to the first state change information, so that the server storage space utilization table stores the storage space utilization information of all the storage servers and all the newly added servers. At this time, the metadata server adds the storage space utilization rate information of the newly added servers and the total disk space information of each newly added server to the server storage space utilization table according to the received first state change information, so that the use condition of each storage server can be grasped.
In this step, the metadata server reads the rho value in the utilization table of the storage space of the server, and can screen rho from all the storage servers and all the newly added serversmaxThe first storage server Sm. Metadata server from all newly added servers set CsSecond storage server S with largest disk space is screened outni。
Wherein, ifmaxWhen a plurality of storage servers belong to the storage server group, the storage server which reaches the space utilization rate firstly is determined as the first storage server.
And step A30, screening out a data pool to be allocated, which occupies the largest space, from all data pools of the first storage server.
And A40, allocating the second storage server as a target server to which the data pool with the largest space occupation in the data pool to be allocated belongs after migration, and generating migration allocation information.
The metadata server can also be provided with a space occupation statistical table of the data pool, and reads the space occupation statistical table of the data pool, so that the first storage server S can be screened outmData pool P with maximum space occupied by inner magnetic diskm。PmAllocating the migrated target object of the data pool to be data migrated to a second storage server Sni. The metadata server may generate migration allocation information according to the allocation result and store it. The migration allocation information may be saved as a temporary file.
And step A50, updating the utilization rate table of the storage space of the server based on the migration allocation information, returning to execute step A20, and circulating that the mean square error of the utilization rates of the storage space of all the storage servers and all the newly added servers in the utilization rate table of the storage space of the server does not exceed a preset threshold value so as to allocate the server to which the data is migrated to all the data pools in the storage system.
In this step, due to PmIs distributed to SniResult in SmThe storage space utilization of (a) may change after migration. At this time, the server-stored space utilization table may be updated to reflect the change to the server-stored space utilization table. And due to PmIs distributed to Sni，SniThe disk space of the new server may also change, and the metadata server may update the disk space information of the new server according to the migration allocation information, such as the temporary table.
The metadata server again performs step A20, and may filter out the first storage server Sm2And a second storage server S with the largest disk space after allocationni2. Then the S is mixedm2Data pool P with maximum occupied space of medium diskm2As a data pool to be migrated, the migrated target object is allocated as the second storage server Sni2. As will be readily appreciated, during the screening process, SmOr belong to a newly added server set Cs. And circulating until the mean square error of the storage space utilization of all the storage servers in the storage system does not exceed a preset threshold, such as 10 percent of the preset threshold. And executing the steps once each time, the metadata server can obtain one piece of migration distribution information, and after the circulation is finished, a plurality of pieces of migration distribution information can be obtained and the distribution results of the corresponding data pools are stored. Thereby distributing the server to which the data is migrated for all the data pools in the storage system
At this time, the storage server information to which the corresponding data pool belongs in the first state information table may be updated according to the plurality of pieces of migration allocation information, so that the storage server information includes source server information to which the data pool belongs before migration and target server information to which the data pool belongs after migration.
The metadata server adds the migration paths formed after the distribution as the storage server information to which the data pool belongs to the first state information table based on all the migration distribution information, so that the disk utilization rate of all the servers in the storage system can be relatively even after the data migration, and the server resources can be fully utilized. After the first state information table is updated, all migration allocation information may be deleted.
E.g., in the storage server column of the first state information table, for the data pool PmCan store the server information by SmIs updated to Sm|Sni", reflecting its migration path.
In addition, on the basis of the third embodiment of the data processing method in data migration of the present invention, a fourth embodiment of the data processing method in data migration of the present invention is proposed. Referring to fig. 5, fig. 5 is a flowchart illustrating a data processing method in data migration according to a fourth embodiment of the present invention.
As will be readily appreciated, in a distributed storage system, since the number of storage servers is reduced, i.e., a part of the storage servers remains in the distributed storage system, but another part of the storage servers leaves the distributed storage system, resulting in a migration task in the distributed storage system, it is necessary to migrate the reduced data in the part of the storage servers to the reserved storage servers.
In this embodiment, before step S101, the data migration method further includes the following steps:
and step B10, screening out a fifth storage server with the lowest space utilization rate from all the third storage servers according to the server storage space utilization rate table.
In this step, the metadata server may receive second state change information of the storage system; the second state change information is used for indicating that the number of the servers in the storage system is reduced, and the second state change information comprises information of a third storage server, information of a reduced fourth storage server and a mapping table of a data pool and a server in a fourth storage, wherein the information of the third storage server is reserved after the number of the servers in the storage system is reduced.
After receiving the state change information of the storage system, the metadata server can obtain the information such as the number and the addresses of the reduced storage servers, the data pools of each reduced storage server, the size of the disk space of each data pool and the like.
E.g., the number of data storage servers n becomes m, where n>m>0. Reducing the number k of data storage servers to n-m, and the set of all fourth storage servers is CdFourth storage Server Sd0、Sd1、……Sdk-1∈Cd。
The mapping table of data pools to servers in the fourth storage is used to indicate to which fourth storage server any data pool of all the fourth storage servers belongs.
The metadata server screens out the space utilization rate rho from all third storage servers reserved after capacity reduction according to the rho value in the server storage space utilization tableminThe fifth storage server.
And step B20, screening out a data pool to be allocated with the largest occupied space from all data pools of all fourth servers.
And step B30, allocating the fifth storage server as a target server of the data pool to be allocated, and generating migration allocation information.
The metadata server can also have a space occupation statistical table of the data pool, and the metadata server reads the space occupation statistical table of the data pool, so that the reduced data storage server set C can be obtaineddScreening out data pool P with largest occupied spacedm。PdmAnd distributing the migrated target object of the data pool to be data migrated as a fifth storage server. The metadata server may generate migration allocation information according to the allocation result and store it. The migration allocation information may be a temporary file.
And B40, updating the server storage space utilization rate table and the mapping table of the data pools and the servers in the fourth storage based on the migration allocation information, returning to execute the step B10, and circulating that the mean square error of the storage space utilization rates of all the third storage servers in the server storage space utilization rate table does not exceed a preset threshold value so as to allocate the servers to which the data are migrated to all the data pools in the storage system.
In this step, due to PdmAnd the allocation to the fifth storage server results in that the storage space utilization of the fifth storage server changes after the migration. At this time, the server-stored space utilization table may be updated to reflect the change to the server-stored space utilization table. And due to PdmTo a fifth storage server, a fourth set of storage servers CdThe metadata server may update the mapping table of the data pool and the server in the fourth storage according to the migration allocation information. Such as the above PdmDeleted from the table.
The metadata server again performs step B10 and may filter out a new fifth storage server. Allocating the new fifth storage server as the data pool P with the largest space occupation in all the data pools of all the fourth storage servers after the initial allocationdm2And the target server to which the migration belongs generates migration allocation information.
And circulating until the mean square error of the storage space utilization of all the storage servers in the storage system does not exceed a preset threshold, such as 10 percent of the preset threshold. And executing the steps once each time, the metadata server can obtain one piece of migration distribution information, and after the circulation is finished, a plurality of pieces of migration distribution information can be obtained and the distribution results of the corresponding data pools are stored. And distributing the servers to which the data are migrated to all the data pools in the storage system.
In this embodiment, the storage server information to which the corresponding data pool belongs in the first state information table may also be updated according to the plurality of migration allocation information, so that the storage server information includes the source server to which the data pool belongs before the data migration and the target server to which the data pool belongs after the data migration.
And updating all migration allocation information by the metadata server, and adding the migration paths formed after allocation as the storage server information to which the data pool belongs to the first state information table, so that the disk utilization rate of all servers in the storage system is relatively even after data migration, and the server resources are fully utilized.
Referring to fig. 6, fig. 6 is a block diagram of a data processing apparatus according to a first embodiment of the present invention during a data migration process.
As shown in the figure, the data processing apparatus 100 in the data migration process in this embodiment includes:
a request receiving module 110, configured to receive a data processing request of a user during a data migration process;
a data pool obtaining module 120, configured to obtain a target data pool corresponding to the data processing request;
a server determining module 130, configured to determine a server to be processed from the source server and the target server according to the migration state of the target data pool; the source server is a server to which the target data pool belongs before migration, and the target server is a server to which the target data pool belongs after migration; and
and the data processing module 140 is configured to perform corresponding data processing on the server to be processed.
In this embodiment, after determining the migration state of the target data pool, the server determining module 130 in the data processing apparatus in the data migration process determines a server to be processed from the source server before the data migration and the target server after the data migration, and performs data processing through the data processing module 140, so that in the data migration process, the read request and the write request can be normally processed, dirty data is not generated, and the migration process is ensured to be normally performed and not to be interrupted.
In one embodiment, if the data processing request is a write request, the server determining module 130 is configured to determine the target server as a to-be-processed server if the target data pool is in a migration state.
In one embodiment, if the data processing request is a read request, the server determining module 130 is configured to determine that the source server is a to-be-processed server if the target data pool is in a migration state.
As an implementation manner, the data pool obtaining module 120 is further configured to parse the read request to obtain a target data file corresponding to the read request; and taking the data pool to which the target data file belongs as the target data pool.
The server determination module 130 is configured to determine a migration status of the target data file; and if the migration state is a state in the migration or a state before the migration, taking the source server as a server to be processed.
Referring to fig. 7, as an embodiment, the data processing apparatus in the data migration process further includes:
a new server determining module 400, configured to determine a new server from the storage system.
The screening module 600 is configured to screen a first storage server with the highest space utilization rate from all the servers and all the newly added servers according to a server storage space utilization rate table, and screen a second storage server with the largest disk space from all the newly added servers; the server storage space utilization table stores storage space utilization information of all servers and all newly added servers and disk space information of all the newly added servers;
a data pool screening module, configured to screen a to-be-allocated data pool with the largest occupied space from all data pools of the first storage server
The migration allocation module 700 is configured to allocate the second storage server as a target server of the data pool to be allocated, and generate migration allocation information;
and a circulation execution module 800, configured to update the server storage space utilization table and the disk space information of the newly added server based on the migration allocation information, and return to execute the step of screening out a first storage server with the highest space utilization rate from all the servers and all the newly added servers according to the server storage space utilization table, and screening out a second storage server with the highest disk space from all the newly added servers, where a mean square error of the storage space utilization rates of all the storage servers and all the newly added servers in the server storage space utilization table is not greater than a preset threshold value, so as to allocate the server to which the data migration belongs after the data migration for all the data pools in the storage system.
As an embodiment, in a data processing apparatus in a data migration process:
the screening module 500 is further configured to screen a fifth storage server with a lowest space utilization rate from all the third storage servers according to the server storage space utilization rate table; screening out a data pool to be distributed with the largest occupied space from all data pools of all the fourth servers;
the migration allocation module 600 is further configured to allocate the fifth storage server as a target server of the server to be allocated, and generate migration allocation information;
the loop execution module 700 is further configured to update the server storage space utilization table based on the migration allocation information, and return to execute the step of screening out a fifth server with a lowest space utilization rate from all the third servers according to the server storage space utilization table, and loop until a mean square error of storage space utilization rates of all the third servers in the server storage space utilization table does not exceed a preset threshold value, so as to allocate the server to which the data migration belongs to after the data migration to all the data pools in the storage system.
Other embodiments or specific implementation manners of the data processing apparatus in the data migration process of the present invention may refer to the above method embodiments, and are not described herein again.
Further, the present invention also provides a computer-readable storage medium, in which a data processing program in the data migration process is stored, and the data processing program in the data migration process is configured to implement the steps of the data processing method in the data migration process. Therefore, a detailed description thereof will be omitted. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application. The program instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Further, the present invention also provides a computer program product comprising executable program code, wherein the program code, when executed by a processor, performs the data processing method in the data migration process as in the above embodiments.
It should be noted that the above-described embodiments of the apparatus are merely schematic, where units illustrated as separate components may or may not be physically separate, and components illustrated as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110791646.6A CN113535087B (en) | 2021-07-13 | 2021-07-13 | Data processing method, server and storage system in data migration process |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110791646.6A CN113535087B (en) | 2021-07-13 | 2021-07-13 | Data processing method, server and storage system in data migration process |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113535087A true CN113535087A (en) | 2021-10-22 |
| CN113535087B CN113535087B (en) | 2023-10-17 |
Family
ID=78127752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110791646.6A Active CN113535087B (en) | 2021-07-13 | 2021-07-13 | Data processing method, server and storage system in data migration process |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535087B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115454932A (en) * | 2022-09-16 | 2022-12-09 | 北京火山引擎科技有限公司 | A data processing method and device |
| CN115543225A (en) * | 2022-11-30 | 2022-12-30 | 苏州浪潮智能科技有限公司 | Data processing method, system, device, computer storage medium and electronic equipment |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101557320A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Disaster tolerance realizing method and communication equipment thereof |
| US20100094948A1 (en) * | 2008-10-10 | 2010-04-15 | International Business Machines Corporation | Workload migration using on demand remote paging |
| CN102541467A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Scheduling method for read-write requests of object data server |
| US20150006833A1 (en) * | 2013-06-26 | 2015-01-01 | International Business Machines Corporation | Optimizing storage utilization by modifying a logical volume offset |
| CN104731905A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Volume reducing method for simplified storage pool |
| CN108268501A (en) * | 2016-12-30 | 2018-07-10 | 中国移动通信集团北京有限公司 | Method for processing business and device in a kind of online data transition process |
| US20180260241A1 (en) * | 2016-08-24 | 2018-09-13 | Ashish Khurange | Method and system of migrating applications to a cloud-computing environment |
| WO2018177107A1 (en) * | 2017-03-29 | 2018-10-04 | 腾讯科技（深圳）有限公司 | Data migration method, migration server, and storage medium |
| CN108924202A (en) * | 2018-06-25 | 2018-11-30 | 郑州云海信息技术有限公司 | A kind of the data disaster tolerance method and relevant apparatus of distributed type assemblies |
| US10146467B1 (en) * | 2012-08-14 | 2018-12-04 | EMC IP Holding Company LLC | Method and system for archival load balancing |
| CN110196752A (en) * | 2018-06-07 | 2019-09-03 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN111013131A (en) * | 2019-10-31 | 2020-04-17 | 咪咕互动娱乐有限公司 | Delayed data acquisition method, electronic device and storage medium |
-
2021
- 2021-07-13 CN CN202110791646.6A patent/CN113535087B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100094948A1 (en) * | 2008-10-10 | 2010-04-15 | International Business Machines Corporation | Workload migration using on demand remote paging |
| CN101557320A (en) * | 2009-05-25 | 2009-10-14 | 杭州华三通信技术有限公司 | Disaster tolerance realizing method and communication equipment thereof |
| CN102541467A (en) * | 2011-12-06 | 2012-07-04 | 华中科技大学 | Scheduling method for read-write requests of object data server |
| US10146467B1 (en) * | 2012-08-14 | 2018-12-04 | EMC IP Holding Company LLC | Method and system for archival load balancing |
| US20150006833A1 (en) * | 2013-06-26 | 2015-01-01 | International Business Machines Corporation | Optimizing storage utilization by modifying a logical volume offset |
| CN104731905A (en) * | 2015-03-24 | 2015-06-24 | 浪潮集团有限公司 | Volume reducing method for simplified storage pool |
| US20180260241A1 (en) * | 2016-08-24 | 2018-09-13 | Ashish Khurange | Method and system of migrating applications to a cloud-computing environment |
| CN108268501A (en) * | 2016-12-30 | 2018-07-10 | 中国移动通信集团北京有限公司 | Method for processing business and device in a kind of online data transition process |
| WO2018177107A1 (en) * | 2017-03-29 | 2018-10-04 | 腾讯科技（深圳）有限公司 | Data migration method, migration server, and storage medium |
| CN110196752A (en) * | 2018-06-07 | 2019-09-03 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN108924202A (en) * | 2018-06-25 | 2018-11-30 | 郑州云海信息技术有限公司 | A kind of the data disaster tolerance method and relevant apparatus of distributed type assemblies |
| CN111013131A (en) * | 2019-10-31 | 2020-04-17 | 咪咕互动娱乐有限公司 | Delayed data acquisition method, electronic device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 刘典型;: "多虚拟机下基于内存缓存的动态块迁移算法", 计算机应用与软件, no. 03, pages 17 - 21 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115454932A (en) * | 2022-09-16 | 2022-12-09 | 北京火山引擎科技有限公司 | A data processing method and device |
| CN115543225A (en) * | 2022-11-30 | 2022-12-30 | 苏州浪潮智能科技有限公司 | Data processing method, system, device, computer storage medium and electronic equipment |
| CN115543225B (en) * | 2022-11-30 | 2023-03-24 | 苏州浪潮智能科技有限公司 | Data processing method, system, device, computer storage medium and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113535087B (en) | 2023-10-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10545781B2 (en) | Dynamically deployed virtual machine | |
| CN107943421B (en) | Partition division method and device based on distributed storage system | |
| JP5932043B2 (en) | Volatile memory representation of non-volatile storage set | |
| EP4052126B1 (en) | Management of multiple physical function non-volatile memory devices | |
| US20200081874A1 (en) | Data placement and sharding | |
| CN114244717B (en) | Method, device, computer equipment and medium for configuring virtual network card resources | |
| CN104111897A (en) | Data processing method, data processing device and computer system | |
| CN113674131A (en) | Hardware accelerator equipment management method and device, electronic equipment and storage medium | |
| JP2023036774A (en) | Access control method of shared memory, access control device of shared memory, electronic apparatus, and autonomous vehicle | |
| CN104461735A (en) | Method and device for distributing CPU resources in virtual scene | |
| CN105302536A (en) | Configuration method and apparatus for related parameters of MapReduce application | |
| CN113535087A (en) | Data processing method, server and storage system in data migration process | |
| US11403026B2 (en) | Method, device and computer program product for managing storage system | |
| CN116149846A (en) | Application performance optimization method and device, electronic equipment and storage medium | |
| US11977785B2 (en) | Non-volatile memory device-assisted live migration of virtual machine data | |
| WO2020253490A1 (en) | Resource allocation method, apparatus and device | |
| CN109634524B (en) | A data partition configuration method, device and device for a data processing daemon | |
| US11954534B2 (en) | Scheduling in a container orchestration system utilizing hardware topology hints | |
| CN110447019B (en) | Memory allocation manager and method for managing memory allocation performed thereby | |
| CN109725981B (en) | A virtual machine upgrade method and related equipment | |
| CN115150268A (en) | Network configuration method, device, and electronic device for Kubernetes cluster | |
| CN114461318A (en) | A method, device and electronic device for cloud desktop management based on cloud terminal | |
| CN112395076A (en) | Network data processing method, equipment and storage medium | |
| CN119051995B (en) | Method and device for allocating authority among multiple members | |
| US20160378510A1 (en) | Configuration Method, Data Exchange Method and Server System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |