CN105045929A - MPP architecture based distributed relational database - Google Patents

Abstract

The invention provides a distributed relational database based on MPP architecture, which relates to the field of database, big data, and distributed computing. The database includes four modules, which are respectively a global transaction manager responsible for global transaction processing and a load balancing system. Responsible for the load balancing management of the cluster, the cluster coordination manager for coordinating the work between the data nodes, the data nodes of the relational database based on PowerDB deployment, the present invention aims at the PowerDB relational database, and establishes a cluster in a distributed environment, by adopting MPP architecture technology, Shared between nodes? Nothing, to ensure that when a single point of failure occurs, the cluster will not be affected. The cluster can be expanded horizontally to achieve PB-level data storage and support massive parallel data writing.

Description

A Distributed Relational Database Based on MPP Architecture

technical field

The invention relates to the field of databases, big data, and distributed computing, provides a massive data storage solution based on a relational database for big data processing, and supports two application scenarios of OLTP and OLAP.

Background technique

Information technology has developed rapidly in recent years, and the amount of information in enterprises and society has grown geometrically, which poses a huge challenge to the storage and processing of big data. How to build big data clusters and realize the storage of massive data is an important issue in the field of big data. the primary problem faced.

Based on the hadoop open source big data architecture, it is a widely used solution at present. The distributed file system provided by hadoop can solve the problem of PB-level data storage. At the same time, through the products under the hadoop ecosystem, columnar storage and data warehouse can be realized. Level data management partially solves the problem of massive data access, but the storage solution provided by Hadoop has the following deficiencies. First, it does not support SQL services. At present, a large number of business systems are developed based on SQL and cannot be smoothly migrated to the hadoop platform. Second, the single-process parallel writing capability is insufficient. Neither HDFS nor HBase has a single process capable of parallel writing, and cannot meet the needs of massive information collection.

Contents of the invention

PowerDB is a database system developed based on a single server. The technical problem to be solved in the present invention is to establish a cluster in a distributed environment for the PowerDB relational database. By adopting the MPP architecture technology, SharedNothing between each node ensures a single point of failure , does not affect the cluster work, the cluster can be expanded horizontally, realizes PB-level data storage, and supports massive parallel data writing.

In order to meet the needs of mass data storage based on SQL, the present invention provides a distributed relational database based on MPP architecture. The database is a distributed architecture and includes four modules, namely global transaction management module (Power-GTM), load Balance system (Power-Proxy), data coordination manager (Power-COORD), data node (Power-DataNode); among them:

The global transaction manager is responsible for global transaction processing. The load balancing system is responsible for the load balancing management of the cluster. The cluster coordination manager is used to coordinate the work among the data nodes. The data nodes are relational databases deployed based on PowerDB.

As a further improvement of the present invention, a cluster generally has only one global transaction management module.

As a further improvement of the present invention, a single global transaction management module can configure a standby (StandBy) node.

As a further improvement of the present invention, there may be multiple coordination managers in a cluster.

As a further improvement of the present invention, the annoying physical structure and logical structure of the data nodes remain completely consistent with PowerDB, that is, a single relational database system is maintained under one server.

Optionally, to improve cluster reliability, GTM provides active/standby mode. Through GTM-Standby configuration, multiple GTM nodes can be established in the cluster. Only one node works at a time, and data is synchronized from GTM-host through stream replication technology To GTM-Standby, when a single point of failure occurs on GTM-host, GTM-Standby will automatically become GTM-host and undertake global transaction management.

Optionally, for data nodes, its reliability is also achieved through multi-node redundancy design, that is, each data node is designed with one or more Standby nodes, and the data is synchronized to the standby node through stream replication technology during the writing process , when a single point of failure occurs, the system automatically switches to ensure that the cluster works 7*24 hours.

Optionally, use the roundrobin algorithm for table building operations, so that the distributed cluster can respond to data write operations most efficiently. Considering the cluster's own overhead, for a cluster with N data nodes, compared with a single server application, you can get about 0.7*N High performance to meet the needs of high-concurrency writing services for massive data.

Optionally, an automatic installation and deployment tool based on windows and Linux developed in java.

Optionally, the distributed cluster management tools mainly include remote login and management, query analysis manager, cluster monitoring tool, etc.

Description of drawings

Fig. 1 is the architecture diagram of the autonomous controllable distributed relational database of the present invention;

Fig. 2 is a simple and efficient distributed relational database cluster of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings. It should be understood that the embodiments described here are only used to explain the present invention, but not to limit the present invention.

PowerDB is a relational database system. It is developed based on the open source database postgreSQL. PostgreSQL is an open source database system that supports concurrent operations and is well compatible with ORACLE. By adopting the MVCC (multi-version concurrency control) mechanism, the ability to write data is improved. Through Divide the data table space into blocks to support stand-alone concurrency. PowerDB relational database maintains the postgreSQL kernel unchanged, develops external tools to meet the needs of business system development and DBA work, and supports SQL2008 standards.

(1) Design of autonomous and controllable distributed relational database architecture.

The system supports the construction of mainstream Linux and windows environments. It is recommended to use the X86 hardware environment to save cluster construction costs.

Considering performance and management issues, it is recommended to install it under Linux, such as UBUNTU, RedHat, centOS, etc.

GTM has high requirements on server reliability. It is recommended to use a commercial server. Since GTM does not save any data, it does not require hard disks and memory requirements. Generally, 16GB can meet the needs.

Each data node generally deploys three functions of Power-proxy, Power-COORD, and Power-DataNode at the same time, so as to maximize the use of server resources. High memory and hard disk resources need to be configured, typically Xeon E5 CPU, 64GB memory, and 6TB hard disk.

(2) Architecture planning of autonomous and controllable distributed relational database.

Taking 1 GTM node and 5 data nodes as an example, make the following plan:

For each component of the database cluster, the corresponding machine name and port number should be assigned, the following is the planning table.

(3) Distributed database management mechanism

Global transactions are managed through GTM, and the database table is horizontally divided into multiple data blocks, which are stored in the corresponding data nodes (Power-DataNode). Insert, query, modify and other services. (1) Data insertion process. GTM calculates which data node the data should be placed in according to the distributed algorithm. The algorithm includes the hash algorithm (generate a hash function according to the field range to determine the data storage node), the roundrobin algorithm (random distribution to each data node), etc. (2) Single table data query. GTM distributes the query command to each data node, and each node uploads the query result to GTM, and GTM organizes the query data set and sends it to the user. (3) Multi-table association query. Organized by the Power-COORD coordination manager, generally each data node is deployed with a coordination manager. The coordination manager is responsible for managing the associated data queried from other nodes, and performing associated calculations with the data of this node to form a single node query result. (4) Index. The index adopts a two-tier architecture, that is, each data node maintains its own index, and GTM also maintains a simple index, and sends commands to each data node for the index response process.

(4) Distributed database initialization

Deploy the application to the relevant directory of each node, such as the /usr/local/PowerDB directory.

Install and set up SSH to realize cluster key-free login.

Run initgtm, initdb and other tools to initialize Power-GTM, Power-Proxy, Power-COORD, and Power-DataNode.

You can also use the installation and deployment tool to implement the above process.

(5) Startup of the cluster.

The first step is to start the GTM service to make the global transaction manager work normally.

The second step is to start the Power-Proxy of each data node separately. Realize the association between nodes and GTM.

The third step is to start the Power-DataNode of each data node respectively.

The fourth step is to start the Power-COORD of each data node separately.

In the fifth step, a corresponding table of other nodes is established on each data node.

The above process can be realized through the cluster installation and deployment tool.

(6) Database testing and application

Create a distributed database through the SQL query manager. The establishment process is consistent with relational database access. Create database users and grant different permissions.

Create a node group (group) and use it for table creation operations.

Create database tables. Typical commands are:

Createtablet1(idint, ageint) distributebyroundrobintogroupgp1;

The above command creates a table called t1, which uses the random distribution algorithm, and uses the node group of gp1.

The reading and writing test of the database table can basically reach a writing speed of 100,000 records/second on a single machine, and the writing speed is expected to reach a speed of 0.7*N*100,000 records/second in a distributed environment.

Establish a data access environment through JDBC, ODBC, OLEDB, and realize application system development.

(7) Establishment of highly reliable distributed database system.

The master/slave node mode can be configured to implement a high-reliability cluster solution, so that when a single node failure occurs in the cluster, it will not cause data loss or cluster shutdown, thereby improving cluster reliability.

For the master/slave setting of the GTM node, you need to configure the standby to be on in the configuration file, and configure the machine name or IP address and port number for hot standby.

The master/slave setting of the data node is also implemented through the configuration file, and the basic method is the same.

The above process can be realized by installing the deployment tool.

(8) Independent and controllable distributed database maintenance.

When the capacity of the distributed database does not meet the requirements, it can be expanded horizontally, that is, by adding data nodes to achieve this function.

New nodes need to be configured with three modules: Power-Proxy, Power-COORD, and Power-DataNode.

Configure the standby node as required.

Start the service functions of the three modules.

Add the new node to the cluster group, so that the new node will take effect, and the newly written data will be partially saved to the new node.

When a single point of failure occurs in the system, the faulty node needs to be uninstalled, after offline repair, join the cluster, and resume work after data synchronization.

The above process can be completed by installing the deployment tool.

(9) Close the database cluster.

The first step is to send a message to the user to confirm that the cluster will be shut down, and you can wait for the user's work to complete, or delay for a certain period of time.

The second step is to close the data node service.

The third step is to close the coordination node service.

The fourth step is to close the load balancing node service.

The fifth step is to close the global transaction node service.

For those of ordinary skill in the technical field of the present invention, without departing from the concept and spirit of the present invention, some simple deduction or replacement should be considered as belonging to the protection scope of the present invention.

Claims (12)

1. the distributed relation database based on MPP framework, it is characterized in that: this database comprises four modules, global transaction management module (Power-GTM), SiteServer LBS (Power-Proxy), data harmonization manager (Power-COORD), back end (Power-DataNode) respectively, wherein:

Global transaction manager is responsible for global transaction process, and SiteServer LBS is responsible for the load balancing management of cluster, and cluster-coordinator manager is for coordinating the work between each back end, and back end is the relational database disposed based on PowerDB.

2. a kind of distributed relation database based on MPP framework according to claim 1, is characterized in that: a cluster generally only has a global transaction management module.

3. a kind of distributed relation database based on MPP framework according to claim 2, is characterized in that: single global transaction management module can configure secondary node.

4. a kind of distributed relation database based on MPP framework according to claim 1, is characterized in that: have multiple coordination manager in a cluster.

5. a kind of distributed relation database based on MPP framework according to claim 1, is characterized in that: maintain single relational database system under a station server.

6. a kind of distributed relation database based on MPP framework according to claim 1, is characterized in that: global transaction management module provides active/standby pattern.

7. a kind of distributed relation database based on MPP framework according to claim 6, is characterized in that: set up multiple global transaction management node in the cluster.

8. a kind of distributed relation database based on MPP framework according to claim 7, is characterized in that: the same time only has a node job.

9. a kind of distributed relation database based on MPP framework according to claim 1, it is characterized in that: each back end designs one or more secondary nodes, data are in ablation process, by stream reproduction technology, be synchronized on secondary node, when there is Single Point of Faliure, system switches automatically.

10. a kind of distributed relation database based on MPP framework according to claim 1, is characterized in that: adopt roundrobin algorithm to carry out building table handling, makes distributed type assemblies can the most efficient response data write operation.

11. a kind of distributed relation databases based on MPP framework according to claim 1, is characterized in that: with java exploitation based on the Auto-mounting deployment tool under windows, Linux.

12. a kind of distributed relation databases based on MPP framework according to claim 1, is characterized in that: distributed type assemblies management tool mainly comprises Telnet and management, query analysis manager, cluster monitoring instrument etc.