CN104239493A - Cross-cluster data migration method and system - Google Patents

Abstract

An embodiment of the invention provides a cross-cluster migration method and system. According to the cross-cluster migration method and system, persistence of data inside a distributed database of a source cluster before migration can be achieved due to data operation interruption through all child nodes of the source cluster and persistence of memory data of the distributed database of the source cluster; the data transmission amount can be reduced due to compression of data tables in the distributed database of the source cluster, the compressed data tables in the distributed database of the source cluster are migrated to a target cluster, and the migration efficiency is improved; then occupied storage space and total file blocks of the data tables in the distributed database of the source cluster before migration are matched with occupied space and total file blocks of the data tables of the target cluster, after migration and accordingly the migration integrity can be verified according to a matching result.

Description

Cross-cluster data migration method and system

technical field

The embodiments of the present invention relate to the technical field of databases, and in particular to a cross-cluster data migration method and system.

Background technique

With the development of Internet applications, the number of users has increased rapidly, and the number of data storage has increased exponentially. The traditional single-database storage technology cannot meet the access requirements of massive data. HDFS (Hadoop Distributed File System, distributed file system) and distributed database applications And born.

HBase (Hadoop Database, distributed database) is a scalable, column-oriented distributed database that uses HDFS as a file storage system to store data in the form of data tables, and can support billions of data on the basis of a common hardware environment. Large data tables with row-level rows and millions of columns, and support random storage and read operations on data of this scale. Because of its high reliability, high scalability, support for random access, and support for MapReduce (mapping simplification) parallel computing, it has been widely used. Among them, Hadoop is a distributed system infrastructure developed by the "Apache" foundation. Users can develop distributed programs without knowing the underlying details of the distribution, and make full use of the power of the cluster to achieve high-speed computing and massive data storage. Pick.

In the actual application process, data migration is inevitably involved, especially when an online HBase cluster needs to be offline, or when the computer room is maintained and relocated, it will face the urgent task of massive data migration, that is, to migrate the data tables of the old cluster to The new cluster continues to provide massive data access services for access business parties.

Existing data migration technologies usually use Hadoop's data copy component for distributed copying, so as to achieve the purpose of migrating data tables in one cluster to a new cluster. When the data copy is complete, start the new cluster-related service process.

The disadvantages of the above data migration technology are: the integrity of the migrated data cannot be guaranteed; the migration time is strictly dependent on the size of the migrated data, making it difficult to control the migration time. It is difficult to ensure that the migration work is completed within a short migration window, that is, the migration efficiency is low.

Contents of the invention

Embodiments of the present invention provide a cross-cluster data migration method and system to ensure the integrity and efficiency of cross-cluster data migration.

In the first aspect, the embodiment of the present invention provides a cross-cluster data migration method, including:

The master control node of the source cluster invokes the stop command to control each child node of the source cluster to stop data operations;

The master control node of the source cluster uses the clear buffer component of the distributed database of the source cluster to persist the data in the distributed database memory into the distributed file system HDFS;

The master control node of the source cluster compresses the data tables contained in the distributed database of the source cluster using the set compression algorithm;

The master control node of the source cluster counts the first storage space size and the first total file block number of HDFS occupied by the data tables in the distributed database of the source cluster;

The master control node of the source cluster migrates the data tables in the distributed database in the source cluster to the distributed database of the target cluster based on the pre-acquired mapping relationship between the IP addresses of the nodes contained in the target cluster and the host name;

If the data migration completion message returned by the web management interface of the mapping and simplification process of the source cluster is obtained, the master control node of the target cluster counts the size of the corresponding second storage space of HDFS occupied by the data table in the distributed database of the target cluster and a second total number of file blocks, and matching the size of the second storage space and the second total number of file blocks with the size of the first storage space and the first total number of file blocks;

If the matching is successful, the master control node of the target cluster uses the decompression algorithm corresponding to the set compression algorithm to decompress the data table migrated to the target cluster;

The master control node of the target cluster starts the target cluster based on the start policy.

In the second aspect, the embodiment of the present invention also provides a cross-cluster data migration system, including a source cluster and a target cluster, the source cluster includes a master node and at least one child node, and the target cluster includes a master node and at least a child node;

The master control node of the source cluster includes:

The stop module is used to call the stop command to control each child node of the source cluster to stop data operations;

Persistence module, for utilizing the empty buffer component of the distributed database of source cluster, the data in the memory of described distributed database is persisted in the distributed file system HDFS;

The compression module is used to compress the data tables contained in the distributed database of the source cluster using a set compression algorithm;

Statistical module, for the first storage space size and the first total file block number of HDFS occupied by the data table in the distributed database of the source cluster;

The migration module is used to migrate the data tables in the distributed database in the source cluster to the distributed database of the target cluster based on the pre-acquired mapping relationship between the IP addresses of the nodes contained in the target cluster and the host names;

The master control node of the target cluster includes:

The statistical module is used to obtain the data migration completion message returned by the webpage management interface of the mapping and simplification process of the source cluster, then count the second storage space size and the size of the corresponding HDFS occupied by the data table in the distributed database of the target cluster a second total number of file blocks, and matching the size of the second storage space and the second total number of file blocks with the size of the first storage space and the first total number of file blocks;

The decompression module is used to decompress the data table migrated to the target cluster by using the decompression algorithm corresponding to the set compression algorithm if the matching is successful;

The startup module is configured to start the target cluster based on the startup policy.

The cross-cluster data migration method and system provided by the embodiments of the present invention can realize the distribution of the source cluster before the migration by stopping the data operation of each child node of the source cluster and persisting the data in the memory of the distributed database of the source cluster. Data persistence in the distributed database; by compressing the data tables in the distributed database of the source cluster, the amount of data transmission can be reduced, and the compressed data tables in the distributed database of the source cluster can be migrated to the target cluster. Improve the migration efficiency; then by comparing the storage space size and total number of file blocks occupied by the data tables in the distributed database of the source cluster before migration with the storage space size and total file blocks occupied by the data tables of the target cluster after migration The number of blocks is matched, and the integrity of the migration can be verified according to the matching result.

Description of drawings

In order to illustrate the present invention more clearly, the accompanying drawings that need to be used in the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art , on the premise of not paying creative labor, other drawings can also be obtained based on these drawings.

FIG. 1 is a flowchart of a cross-cluster data migration method provided by Embodiment 1 of the present invention;

FIG. 2 is a flowchart of a cross-cluster data migration method provided by Embodiment 3 of the present invention;

FIG. 3a is a schematic structural diagram of a master control node of a source cluster in a cross-cluster data migration system provided by Embodiment 4 of the present invention;

FIG. 3 b is a schematic structural diagram of a master control node of a target cluster in a cross-cluster data migration system provided by Embodiment 4 of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Example. It can be understood that the specific embodiments described here are only used to explain the present invention, rather than limit the present invention. Based on the embodiments of the present invention, all those skilled in the art can obtain without creative work. Other embodiments all belong to the protection scope of the present invention. In addition, it should be noted that, for the convenience of description, only parts related to the present invention are shown in the drawings but not all content.

Embodiment one

Please refer to FIG. 1 , which is a flowchart of a cross-cluster data migration method provided by Embodiment 1 of the present invention. The method of the embodiment of the present invention is applicable to a cross-cluster data migration system, the system includes a source cluster and a target cluster, the source cluster includes a master control node and at least one child node, and the target cluster includes a master control node and at least one child node . Among them, the master control node and at least one child node of the source cluster form HDFS, and the data tables to be migrated are stored in the source cluster; the master control node and at least one child node of the target cluster can also form HDFS, which is used to migrate and store data tables in the source cluster data sheet.

The method includes:

Step 110, the master control node of the source cluster invokes a stop command to control each child node of the source cluster to stop data operations;

This step is specifically to make the data in each node before migration persistent by stopping the data operation of each child node of the source cluster. Specifically, the business party corresponding to each sub-node may be notified to stop data writing or reading operations, and then a stop command may be invoked to cause each sub-node of the source cluster to stop the data operation. Of course, you can also directly invoke the stop command to stop data operations on each child node of the source cluster.

Step 120, the master control node of the source cluster uses the clear buffer component of the distributed database of the source cluster to persist the data in the memory of the distributed database to HDFS;

This step is specifically to persist the data in the distributed database of the source cluster.

Wherein the clear buffer component is used to persist the data temporarily stored in the memory of the distributed database to the disk of HDFS.

Step 130, the master control node of the source cluster compresses the data tables contained in the distributed database of the source cluster using a set compression algorithm;

This step is specifically to compress the data table to be migrated in the source cluster. Specifically, the uncompressed data tables may be compressed by checking the compression status of each data table, specifically, LZO (Lempel-Ziv-Oberhumer) compression algorithm, SNAPPY compression algorithm or other compression algorithms may be used.

Among them, the LZO compression algorithm is a compression algorithm with high compression ratio and extremely fast decompression speed. It is lossless compression, that is, the compressed data can be accurately restored. The SNAPPY compression algorithm is a development package for compression and decompression, designed to provide high compression speed and reasonable compression ratio.

Distributed databases use HDFS as a file storage system to store data in the form of data tables, and can support large data tables with billions of rows and millions of columns on the basis of common hardware environments. Compression can effectively reduce the data transfer rate and help improve data migration efficiency.

Step 140, the master control node of the source cluster counts the size of the first storage space and the first total number of file blocks in HDFS occupied by the data tables in the distributed database of the source cluster;

In this step, the data table contained in the distributed database of the source cluster can be stored in the root directory of the distributed database as the data table to be migrated. Since the distributed database of the source cluster uses HDFS as the file storage system, based on HDFS forms a disk storage space, which is the sum of the disk storage spaces of each child node of the source cluster, and the storage space occupied by the data table in the HDFS disk storage space is the first storage space. space.

Since the amount of data stored in the data table to be migrated is very large, in the actual storage process, distributed storage is used for the data table to be migrated, that is, the data table to be migrated is divided into blocks to form multiple files Blocks, which store different file blocks in the disks of different child nodes of the source cluster. The first total number of file blocks refers to the sum of the number of file blocks corresponding to the data table to be migrated.

Step 150, the master control node of the source cluster migrates the data tables in the distributed database in the source cluster to the distributed database of the target cluster based on the pre-acquired mapping relationship between the IP addresses of the nodes contained in the target cluster and the host names middle;

This step is specifically to migrate the data tables in the distributed database in the source cluster to the distributed database in the target cluster.

It should be noted that in step 110, only the data access service provided by the distributed database of the source cluster is stopped, while the HDFS service of the source cluster is still running normally.

It should also be noted that, according to the mapping relationship between the IP addresses and host names of the nodes contained in the target cluster, the master control node of the source cluster can find the HDFS of the target cluster, so based on the mapping relationship, the distributed The data tables in the database are migrated to the HDFS of the target cluster. Since the distributed database of the target cluster uses HDFS as the file storage system, the data tables in the distributed database of the source cluster can be migrated to the distributed database of the target cluster.

Step 160, if the data migration completion message returned by the webpage management interface of the MapReduce (mapping simplification) process of the source cluster is obtained, the master control node of the target cluster counts the corresponding HDFS occupied by the data table in the distributed database of the target cluster The size of the second storage space and the second total number of file blocks, and matching the size of the second storage space and the second total number of file blocks with the size of the first storage space and the first total number of file blocks;

This step is specifically after monitoring that the migration of the data table is completed, first counting the second storage space size and the second total file block number of the corresponding HDFS occupied by the data table of the distributed database of the target cluster, and then counting the first The size of the second storage space is compared with the size of the first storage space, and the second total number of file blocks is compared with the first total number of file blocks.

In this step, the size of the second storage space and the second total number of file blocks are similar to the size of the first storage space and the first total number of file blocks, and will not be repeated here.

Among them, MapReduce is a general programming model for implementing distributed parallel computing tasks, and is used to process parallel operations on large-scale data. The specific migration status can be monitored through the web management interface of the MapReduce process, such as real-time migration speed, migration percentage, estimated remaining time, and description information of the migrated data.

Step 170, if the matching is successful, the master control node of the target cluster uses the decompression algorithm corresponding to the set compression algorithm to decompress the data table migrated to the target cluster;

The successful matching in this step means that the size of the first HDFS storage space occupied by the data tables in the distributed database of the source cluster before migration is the same as the second storage space of HDFS of the target cluster occupied after migration, and the migration The total number of file blocks before is consistent with the total number of file blocks after migration, that is, if the matching is successful, the data tables in the distributed database of the source cluster are completely migrated to the distributed database of the target cluster.

This step is specifically to decompress and migrate the data table to the data table in the target cluster after the data table is completely migrated.

Step 180, the master control node of the target cluster starts the target cluster based on the start policy.

This step is specifically to start the target cluster, so that each node of the target cluster can work normally.

In the technical solution of this embodiment, the data persistence in the distributed database of the source cluster before migration can be realized by stopping the data operation of each child node of the source cluster and persisting the data in the memory of the distributed database of the source cluster ; By compressing the data tables in the distributed database of the source cluster, the amount of data transfer can be reduced, and the compressed data tables in the distributed database of the source cluster are migrated to the target cluster, which improves the migration efficiency; and then by Match the storage space and total number of file blocks occupied by the data tables in the distributed database of the source cluster before migration with the storage space and total number of file blocks occupied by the data tables of the target cluster after migration. The matching results verify the integrity of the migration.

Embodiment two

In this embodiment, on the basis of the foregoing embodiments, before the master control node of the source cluster counts the size of the first storage space of the HDFS occupied by the data tables in the distributed database of the source cluster and the first total number of file blocks, it also includes :

The master control node of the source cluster uses the complete file merging component of the distributed database of the source cluster to clear the data tables in the disk storage space of the distributed database of the source cluster that meet the preset clearing strategy.

In this step, after the data table to be migrated is compressed, the invalid data table in the disk storage space of the distributed database of the source cluster is cleared, so as to further reduce the amount of data to be migrated and improve migration efficiency.

Wherein, the preset clearing strategy can be implemented in multiple ways, for example including at least one of the following:

Using the data table with the deletion mark in the disk storage space of the distributed database of the source cluster as the data table to be cleared;

Using the data table that has reached the time-to-live in the disk storage space of the distributed database of the source cluster as the data table to be cleared;

It should be noted that the survival time of the data table can be preset according to the requirement, and expired data tables can be eliminated according to the survival time of the data table. Taking the e-commerce platform as an example, usually, the survival time of the corresponding data table will be set according to the duration of the promotional activity, such as 30 days, 7 days, or a certain period of time in a specific day, such as the day of the store celebration from 10:00 a.m. From 10:00 p.m. to 10:00 p.m., at the end of the store celebration, the survival time of the data table for this store celebration is the expired data table. By clearing the expired data table, it is beneficial to save storage space and improve migration efficiency.

A data table whose maximum number of versions is greater than a threshold value in the disk storage space of the distributed database of the source cluster is used as a data table to be cleared.

It should be noted that the maximum number of versions of the data table can be set in advance according to needs, usually set to 3. For data tables that are updated frequently, it can be set to 1, so that invalid data tables can be quickly eliminated, which is conducive to saving storage space and improving migration efficiency.

In the technical solution of this embodiment, after persisting the data in the distributed database of the source cluster before migration, the data table to be migrated is compressed, which can reduce the amount of data to be transmitted, and by clearing the distribution of the source cluster The invalid data tables in the disk storage space of the distributed database can further reduce the amount of migrated data, and migrate the compressed and cleared data tables in the distributed database of the source cluster to the target cluster, which improves the migration efficiency; through Match the storage space and total number of file blocks occupied by the data table of the distributed database of the source cluster before migration with the storage space and total number of file blocks occupied by the data table of the target cluster after migration. The results verify the integrity of the migration.

In the above solution, the empty buffer component and the complete file merge component can be triggered by calling the distributed database command line interface of the source cluster.

Wherein, the command line interface is an interactive interface between the operating system and the user. In the Linux operating system, the command line interface is called the shell, and its main function is to provide services for users, such as receiving input data from the keyboard or displaying execution results on the screen.

In the above solution, before the master control node of the target cluster adopts the decompression algorithm corresponding to the set compression algorithm to decompress the data table migrated to the target cluster, it is also preferred to include:

The master control node of the target cluster invokes the consistency detection component in the target cluster to detect the consistency of the data tables contained in the distributed database of the target cluster;

If they are consistent, trigger the master control node of the target cluster to use the decompression algorithm corresponding to the set compression algorithm to decompress the data table migrated to the target cluster.

It should be noted that checking the consistency of the data table refers to checking whether the description information of the data table is consistent with the attribute information of the data table actually existing in the HDFS of the target cluster. If consistent, trigger the master control node of the target cluster to use the decompression algorithm corresponding to the set compression algorithm to decompress the data table migrated to the target cluster; if inconsistent, use the consistency detection component to repair. This step is a supplementary verification after data table migration integrity verification, which can improve the consistency of data tables migrated to the target cluster.

Embodiment three

Please refer to FIG. 2 , which is a flowchart of a cross-cluster data migration method provided by Embodiment 3 of the present invention. On the basis of the foregoing embodiments, this embodiment provides an optimal solution for the master control node of the target cluster to start the target cluster based on the start policy. This preferred method includes:

Step 210, the master control node of the target cluster invokes a startup command to start the target cluster;

Step 220, if there is no error log information or warning log information in the log file associated with the distributed database of the target cluster, the master control node of the target cluster calls the overall health check component in the distributed database of the target cluster to check the target cluster overall health of

This step is specifically to check the log files associated with the distributed database in the nodes contained in the target cluster. If there is error log information or warning log information, call the relevant components of the distributed database of the target cluster to solve the problem according to the prompt; if not If there is error log information or warning log information, use the health check component of the distributed database of the target cluster to check the overall health of the target cluster.

Wherein, checking the overall health of the target cluster includes checking whether the data table of the target cluster is in a normal state.

Step 230, the master control node of the target cluster invokes the command line interface of the distributed database to set the state of the data table to an enabled state.

Specifically, this step is to maintain the state of the data table in the target cluster in the normal state of “enabled” according to the detection result of the overall health of the target cluster in step 220 .

In the technical solution of this embodiment, after starting the target cluster, check the log files associated with the distributed database in the nodes included in the target cluster, if there is error log information or warning log information, then call the distribution of the target cluster according to the prompt Relevant components of the distributed database to solve the problem; if there is no error log information or warning log information, use the health check component of the distributed database of the target cluster to check the overall health of the target cluster; and based on the overall health of the target cluster As a result of the detection, the state of the data table in the target cluster is maintained in a normal state of "enabled", so that the data table in the target cluster can provide normal access services.

In the above solution, if the data table contained in the distributed database of the target cluster is not enabled on the distributed database management page of the target cluster, the command line interface of the distributed database invoked by the master control node of the target cluster will be After the state of the data table is set to the enabled state, it also preferably includes:

Send the mapping relationship between the IP addresses of the nodes contained in the target cluster and the host name, and the connection information in the distributed database of the target cluster to the business party, and notify the business party to verify the data service of the target cluster.

Embodiment four

Please refer to Figure 3a and Figure 3b. Embodiment 4 of the present invention provides a cross-cluster data migration system, the system includes: a source cluster and a target cluster, the source cluster includes a master control node and at least one child node, and the target cluster includes a master control node and at least one child node node.

The master control node of the source cluster includes: a stop module 310 , a persistence module 320 , a compression module 330 , a statistics module 340 and a migration module 350 .

The master control node of the target cluster includes: a statistical module 360 , a decompression module 370 and a startup module 380 .

Among them, the stop module 310 is used to call the stop command to control each sub-node of the source cluster to stop data operation; Persistence in HDFS; Compression module 330 is used to compress the data tables contained in the distributed database of the source cluster using a set compression algorithm; Statistical module 340 is used to count the data tables contained in the distributed database of the source cluster The size of the first storage space and the first total number of file blocks of the occupied HDFS; the migration module 350 is used to transfer the distributed database in the source cluster to The data table of is migrated to the distributed database of the target cluster;

Wherein, the statistical module 360 is used to obtain the data migration completion message returned by the webpage management interface of the mapping and simplification process of the source cluster, then count the second storage space of the corresponding HDFS occupied by the data table in the distributed database of the target cluster size and the second total number of file blocks, and match the size of the second storage space and the second total number of file blocks with the size of the first storage space and the first total number of file blocks; the decompression module 370 is used if If the matching is successful, the data table migrated to the target cluster is decompressed using the decompression algorithm corresponding to the set compression algorithm; the startup module 380 is used to start the target cluster based on the startup policy.

In the technical solution of this embodiment, the data persistence in the distributed database of the source cluster before migration can be realized by stopping the data operation of each child node of the source cluster and persisting the data in the memory of the distributed database of the source cluster ; By compressing the data tables in the distributed database of the source cluster, the amount of data transfer can be reduced, and the compressed data tables in the distributed database of the source cluster are migrated to the target cluster, which improves the migration efficiency; and then by Match the storage space and total number of file blocks occupied by the data tables in the distributed database of the source cluster before migration with the storage space and total number of file blocks occupied by the data tables of the target cluster after migration. The matching results verify the integrity of the migration.

In the above solution, the master control node of the source cluster preferably also includes: a clearing module, used for counting the size of the first storage space and the first total file blocks of HDFS occupied by the data tables in the distributed database of the source cluster Before counting, use the complete file merging component of the distributed database of the source cluster to clear the data tables in the disk storage space of the distributed database of the source cluster that meet the preset clearing policy.

In the above solution, the preset clearing strategy includes at least one of the following:

Using the data table with the deletion mark in the disk storage space of the distributed database of the source cluster as the data table to be cleared;

Using the data table that has reached the time-to-live in the disk storage space of the distributed database of the source cluster as the data table to be cleared;

A data table whose maximum number of versions is greater than a threshold value in the disk storage space of the distributed database of the source cluster is used as a data table to be cleared.

In the above solution, the persistence module 310 is specifically configured to trigger the empty buffer component by calling the distributed database command line interface of the source cluster, and persist the data in the distributed database memory to the distributed file system HDFS The clearing module is specifically used to call the distributed database command line of the source cluster before the first storage space size of the HDFS occupied by the data tables in the distributed database of the statistical source cluster and the first total file block number The interface triggers the complete file merging component to clear the data tables conforming to the preset clearing policy in the disk storage space of the distributed database of the source cluster.

In the above solution, the master control node of the target cluster preferably further includes: a consistency detection module, which is used to decompress the data table migrated to the target cluster before using the decompression algorithm corresponding to the set compression algorithm , call the consistency detection component in the target cluster to detect the consistency of the data tables contained in the distributed database of the target cluster; if consistent, trigger the master control node of the target cluster to adopt the decompression corresponding to the set compression algorithm The algorithm decompresses the data tables migrated to the target cluster.

In the above solution, the startup module 390 preferably includes: a startup unit, an overall health degree detection unit, and a data table status setting unit.

Among them, the startup unit is used to call the startup command to start the target cluster; the overall health detection unit is used to call the distributed database of the target cluster if there is no error log information or warning log information in the log file associated with the distributed database of the target cluster The overall health check component in the system checks the overall health of the target cluster; the data table state setting unit is used to call the command line interface of the distributed database to set the state of the data table to an enabled state.

In the above solution, the startup module 390 may also include: a data service verification unit, configured to pass through the distributed database management page of the target cluster, if the data table contained in the distributed database of the target cluster is not in an enabled state, After calling the command line interface of the distributed database to set the state of the data table to the enabled state, the mapping relationship between the IP addresses of the nodes contained in the target cluster and the host name, and the connection in the distributed database of the target cluster The information is sent to the business party, and the business party is notified to verify the data service of the target cluster.

The master control node of the source cluster and the master control node of the target cluster in the cross-cluster data migration system provided by the embodiment of the present invention can execute the cross-cluster data migration method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial Effect.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; preferred implementations in the examples are not to limit them, and for those skilled in the art, this The invention is capable of various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (10)

1. across a company-data moving method, it is characterized in that, comprising:

Each child node that the main controlled node of source cluster calls control source cluster of ceasing and desisting order stops data manipulation;

The main controlled node of source cluster utilizes the clearing buffers area assembly of the distributed data base of source cluster, by the data persistence in described distributed data base internal memory in distributed file system HDFS;

The main controlled node of source cluster controls the tables of data comprised the distributed data base of source cluster, adopts the compression algorithm of setting to compress;

First storage size of the HDFS shared by tables of data in the distributed data base of the main controlled node Statistic Source cluster of source cluster and the first general act block number;

The IP address of node that the main controlled node of source cluster comprises based on the target cluster obtained in advance and the mapping relations of Hostname, migrate in the distributed data base of described target cluster by the tables of data in distributed data base in the cluster of source;

If the Data Migration that the management of webpage interface getting the mapping abbreviation process of source cluster returns completes message, then the main controlled node of target cluster adds up the second storage size and the second general act block number of the HDFS of the correspondence that the tables of data in the distributed data base of target cluster is occupied, and described second storage size and the second general act block number is mated with described first storage size and described first general act block number;

If the match is successful, then the main controlled node of target cluster adopts the decompression algorithm corresponding with the compression algorithm of described setting to carry out decompress(ion) to the tables of data migrated in target cluster;

The main controlled node of target cluster, based on startup strategy, starts described target cluster.

2. method according to claim 1, is characterized in that, before first storage size of the HDFS shared by the tables of data in the distributed data base of the main controlled node Statistic Source cluster of source cluster and the first general act block number, also comprises:

The main controlled node of source cluster utilizes the complete file merge module of the distributed data base of source cluster, removes in the disk storage space of the distributed data base of described source cluster the tables of data meeting and preset and remove strategy.

3. method according to claim 2, is characterized in that, described default removing strategy comprises following at least one item:

Using the tables of data with deletion mark in the disk storage space of the distributed data base of described source cluster as tables of data to be cleaned;

Using the tables of data reaching life span in the disk storage space of the distributed data base of described source cluster as tables of data to be cleaned;

Maximum version number in the disk storage space of the distributed data base of described source cluster is greater than the tables of data of threshold value as tables of data to be cleaned.

4. method according to claim 2, is characterized in that, triggers clearing buffers area assembly and complete file merge module by the distributed data base command line interface calling source cluster.

5. method according to claim 1, is characterized in that, adopts before the decompression algorithm corresponding with the compression algorithm of described setting carry out decompress(ion) to the tables of data migrated in target cluster, also comprise at the main controlled node of target cluster:

Consistency detection assembly in the main controlled node invocation target cluster of target cluster, the consistance of the tables of data that the distributed data base detecting target cluster comprises;

If consistent, then the main controlled node of trigger target cluster adopts the decompression algorithm corresponding with the compression algorithm of described setting to carry out decompress(ion) to the tables of data migrated in target cluster.

6. according to the arbitrary described method of claim 1-5, it is characterized in that, the main controlled node of target cluster, based on startup strategy, starts described target cluster, comprising:

The main controlled node of target cluster calls startup command and starts target cluster;

If there is not error-logging information or warning log information in the journal file of the distributed data base association of target cluster, holistic health degree inspection assembly in the distributed data base of the then main controlled node invocation target cluster of target cluster, checks the holistic health degree of target cluster;

The state of described tables of data is set to enabled state by the command line interface that the main controlled node of target cluster calls distributed data base.

7. method according to claim 6, it is characterized in that, at the distributed database management page by target cluster, if the tables of data that the distributed data base of target cluster comprises is not in enabled state, the command line interface that then main controlled node of target cluster calls distributed data base also comprises after the state of described tables of data is set to enabled state:

The IP address of the node that target cluster is comprised and the mapping relations of Hostname, and the link information in the distributed data base of target cluster is sent to business side, and notify that the data, services of described business side to target cluster is verified.

8. across a company-data migratory system, comprise source cluster and target cluster, described source cluster comprises main controlled node and at least one child node, and described target cluster comprises main controlled node and at least one child node, it is characterized in that:

The main controlled node of described source cluster comprises:

Stopping modular, stops data manipulation for each child node calling control source cluster of ceasing and desisting order;

Persistence module, for utilizing the clearing buffers area assembly of the distributed data base of source cluster, by the data persistence in described distributed data base internal memory in distributed file system HDFS;

Compression module, for the tables of data comprised the distributed data base of source cluster, adopts the compression algorithm of setting to compress;

Statistical module, for the first storage size and the first general act block number of the HDFS shared by the tables of data in the distributed data base of Statistic Source cluster;

Transferring module, for the IP address of node that comprises based on the target cluster obtained in advance and the mapping relations of Hostname, migrates in the distributed data base of described target cluster by the tables of data in distributed data base in the cluster of source;

The main controlled node of described target cluster comprises:

Statistical module, if the Data Migration returned for the management of webpage interface of the mapping abbreviation process getting source cluster completes message, then add up the second storage size and the second general act block number of the HDFS of the correspondence that the tables of data in the distributed data base of target cluster is occupied, and described second storage size and the second general act block number are mated with described first storage size and described first general act block number;

Decompression module, if for the match is successful, then adopts the decompression algorithm corresponding with the compression algorithm of described setting to carry out decompress(ion) to the tables of data migrated in target cluster;

Start module, for based on startup strategy, start described target cluster.

9. system according to claim 8, is characterized in that, the main controlled node of described source cluster also comprises:

Remove module, before first storage size of the HDFS shared by the tables of data in the distributed data base of Statistic Source cluster and the first general act block number, utilize the complete file merge module of the distributed data base of source cluster, remove in the disk storage space of the distributed data base of described source cluster the tables of data meeting and preset and remove strategy.

10. system according to claim 9, is characterized in that, described default removing strategy comprises following at least one item:

Using the tables of data with deletion mark in the disk storage space of the distributed data base of described source cluster as tables of data to be cleaned;

Using the tables of data reaching life span in the disk storage space of the distributed data base of described source cluster as tables of data to be cleaned;

Maximum version number in the disk storage space of the distributed data base of described source cluster is greater than the tables of data of threshold value as tables of data to be cleaned.