CN103167035A - Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) - Google Patents
Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) Download PDFInfo
- Publication number
- CN103167035A CN103167035A CN2013100826089A CN201310082608A CN103167035A CN 103167035 A CN103167035 A CN 103167035A CN 2013100826089 A CN2013100826089 A CN 2013100826089A CN 201310082608 A CN201310082608 A CN 201310082608A CN 103167035 A CN103167035 A CN 103167035A
- Authority
- CN
- China
- Prior art keywords
- namenode
- hdfs
- file system
- node
- glusterfs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of distributed file systems, in particular to a method for solving NameNode single-point failure in a hadoop distributed file system (HDFS). The method comprises the following steps of: S1, starting GlusterFS service, and mounting a GlusterFS file system on a node A; S2, starting HDFS NameNode service on the node A, and storing metadata of the HDFS NameNode in the GlusterFS file system; and S3, when the node A fails, mounting the GlusterFS file system on the other node B, and starting the HDFS NameNode service. By adopting the provided technical scheme, the NameNode position can smoothly slides during HDFS NameNode single-point failure, so that the risk and loss which are brought by the NameNode single-point failure to HDFS cluster are avoided.
Description
Technical field
The present invention relates to the distributed file system technology field, particularly the solution of NameNode single point failure in a kind of HDFS cluster.
Background technology
Hadoop Distributed File System is called for short HDFS, is the distributed file system of increasing income.HDFS inside is divided into two class roles to cluster: the NameNode(namenode) and the DataNode(back end), a HDFS cluster generally comprises a NameNode and a plurality of DataNode, NameNode is management document NameSpace and the master server of regulating the client-access file, DataNode controlled and store by NameNode real data from server.As the key node of HDFS cluster, the single point failure of NameNode becomes the greateset risk that this distributed system faces.For this reason, Hadoop community and each Main Branches have all been studied a lot of solutions and have been solved this problem.
Gluster File System is called for short GlusterFS, is another distributed file system of increasing income, and is mainly used in group system, and it configures simple and has good extensibility.
If GlusterFS can be applied in HDFS, solve the problem of NameNode single point failure, just can greatly improve the ability of HDFS reply node failure.
Summary of the invention
(1) technical problem to be solved
The object of the present invention is to provide the solution of NameNode single point failure in a kind of HDFS, thereby realize the smooth migration of NameNode position when the NameNode single point failure.
(2) technical scheme
In order to solve the problems of the technologies described above, the present invention proposes the solution of NameNode single point failure in a kind of HDFS, said method comprising the steps of:
S1, start the GlusterFS service, and on node A carry GlusterFS file system;
S2, start HDFS NameNode service on described node A, and with the meta-data preservation of HDFSNameNode in described GlusterFS file system;
S3, when described node A lost efficacy, the described GlusterFS file system of carry on another Node B, and start HDFS NameNode service.
Optionally, described method is applied in the OPV-Suite cluster management system.
Optionally, in step S1, described in carry GlusterFS file system on node A by operation mount order realization under Windows or (SuSE) Linux OS.
Optionally, in step S3, described inefficacy comprises outage or suspension.
(3) beneficial effect
The technical scheme that adopts the present invention to propose is preserved the metadata of oneself with the GlusterFS file system of carry due to HDFS NameNode, so just guaranteed the Information Security of NameNode metadata.When a certain Node lost efficacy, because the metadata of NameNode is not lost, so it can work on other Node, whole process has guaranteed the smooth migration of NameNode position, thereby has effectively avoided the risk and the loss that bring to the HDFS cluster because of the NameNode single point failure.
Description of drawings
Fig. 1 is the basic flow sheet of the solution of NameNode single point failure in the HDFS that proposes of the present invention.
Embodiment
Below in conjunction with drawings and Examples, the specific embodiment of the present invention is described in further detail.
The present invention proposes the solution of NameNode single point failure in a kind of HDFS, as shown in Figure 1, said method comprising the steps of:
S1, start the GlusterFS service, and on node A carry GlusterFS file system;
S2, start HDFS NameNode service on described node A, and with the meta-data preservation of HDFSNameNode in described GlusterFS file system;
S3, when described node A lost efficacy, the described GlusterFS file system of carry on another Node B, and start HDFS NameNode service.
The below utilizes Chinese cypress OPV-Suite cluster management system that implementation of the present invention is illustrated.
At first to utilize the OPV-Suite service control method to guarantee that GlusterFS service meeting starts prior to the HDFS service.At first the OPV-Suite cluster management system can start a GlusterFS service, and makes a Node come carry GlusterFS file system, then starts HDFS NameNode service on same Node.Carry GlusterFS file system can realize by operation mount order under Windows or (SuSE) Linux OS on a Node.HDFS NameNode preserves the metadata of oneself with the GlusterFS file system of carry, so just guaranteed the Information Security of NameNode metadata.If the Node at a certain moment NameNode place outage or suspension, OPV-Suite service control method can be at once on another Node carry GlusterFS file system and start another NameNode service.Because the metadata of NameNode is not lost, so it can work on other Node.Whole process has guaranteed the smooth migration of HDFS NameNode position, thereby effectively avoids the single point failure of HDFS NameNode.
The above is only the preferred embodiment of the present invention; should be pointed out that for the person of ordinary skill of the art, under the prerequisite that does not break away from the technology of the present invention principle; can also make some improvement and replacement, these improvement and replacement also should be considered as protection scope of the present invention.
Claims (4)
1. the solution of NameNode single point failure in a HDFS, is characterized in that, said method comprising the steps of:
S1, start the GlusterFS service, and on node A carry GlusterFS file system;
S2, start HDFS NameNode service on described node A, and with the meta-data preservation of HDFSNameNode in described GlusterFS file system;
S3, when described node A lost efficacy, the described GlusterFS file system of carry on another Node B, and start HDFS NameNode service.
2. the solution of NameNode single point failure in HDFS according to claim 1, is characterized in that, described method is applied in the OPV-Suite cluster management system.
3. the solution of NameNode single point failure in HDFS according to claim 1, it is characterized in that, in step S1, described in carry GlusterFS file system on node A by operation mount order realization under Windows or (SuSE) Linux OS.
4. the solution of NameNode single point failure in HDFS according to claim 1, is characterized in that, in step S3, described inefficacy comprises outage or suspension.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100826089A CN103167035A (en) | 2013-03-15 | 2013-03-15 | Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013100826089A CN103167035A (en) | 2013-03-15 | 2013-03-15 | Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103167035A true CN103167035A (en) | 2013-06-19 |
Family
ID=48589771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013100826089A Pending CN103167035A (en) | 2013-03-15 | 2013-03-15 | Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103167035A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103327105A (en) * | 2013-06-26 | 2013-09-25 | 北京汉柏科技有限公司 | Automatic slave node service recovering method of hadoop system |
| CN103634372A (en) * | 2013-11-04 | 2014-03-12 | 天津汉柏信息技术有限公司 | Method for storing big data |
| CN104536988A (en) * | 2014-12-10 | 2015-04-22 | 杭州斯凯网络科技有限公司 | MonetDB distributed computing storage method |
| CN105677703A (en) * | 2015-12-25 | 2016-06-15 | 曙光云计算技术有限公司 | NAS file system, and access method and apparatus thereof |
| CN110989929A (en) * | 2019-11-22 | 2020-04-10 | 浪潮电子信息产业股份有限公司 | MON service migration method, device, equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101997823A (en) * | 2009-08-17 | 2011-03-30 | 联想(北京)有限公司 | Distributed file system and data access method thereof |
| US20110246460A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating datasets for analysis |
| CN102420854A (en) * | 2011-11-14 | 2012-04-18 | 西安电子科技大学 | Distributed file system for cloud storage |
| US20120131275A1 (en) * | 2010-11-18 | 2012-05-24 | Promise Technology, Inc | Network-attached storage system |
| CN102737130A (en) * | 2012-06-21 | 2012-10-17 | 广州从兴电子开发有限公司 | Method and system for processing metadata of hadoop distributed file system (HDFS) |
-
2013
- 2013-03-15 CN CN2013100826089A patent/CN103167035A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101997823A (en) * | 2009-08-17 | 2011-03-30 | 联想(北京)有限公司 | Distributed file system and data access method thereof |
| US20110246460A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating datasets for analysis |
| US20120131275A1 (en) * | 2010-11-18 | 2012-05-24 | Promise Technology, Inc | Network-attached storage system |
| CN102420854A (en) * | 2011-11-14 | 2012-04-18 | 西安电子科技大学 | Distributed file system for cloud storage |
| CN102737130A (en) * | 2012-06-21 | 2012-10-17 | 广州从兴电子开发有限公司 | Method and system for processing metadata of hadoop distributed file system (HDFS) |
Non-Patent Citations (2)
| Title |
|---|
| S. A. WEIL, S. A. BRANDT, E. L. MILLER, D. D. E. LONG: "Ceph: A Scalable, High-Performance Distributed", 《DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING》, 20 April 2007 (2007-04-20), pages 1 - 27 * |
| YOUWEI WANG, JIANG ZHOU, CAN MA, WEIPING WANG, DAN MENG: "Clover: A distributed file system of expandable metadata service derived from HDFS", 《2012 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING》, 31 December 2012 (2012-12-31), pages 126 - 128 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103327105A (en) * | 2013-06-26 | 2013-09-25 | 北京汉柏科技有限公司 | Automatic slave node service recovering method of hadoop system |
| CN103327105B (en) * | 2013-06-26 | 2016-09-07 | 北京汉柏科技有限公司 | Slave node service automatic recovery method in hadoop system |
| CN103634372A (en) * | 2013-11-04 | 2014-03-12 | 天津汉柏信息技术有限公司 | Method for storing big data |
| CN104536988A (en) * | 2014-12-10 | 2015-04-22 | 杭州斯凯网络科技有限公司 | MonetDB distributed computing storage method |
| CN104536988B (en) * | 2014-12-10 | 2017-10-24 | 杭州斯凯网络科技有限公司 | MonetDB Distributed Calculation storage methods |
| CN105677703A (en) * | 2015-12-25 | 2016-06-15 | 曙光云计算技术有限公司 | NAS file system, and access method and apparatus thereof |
| CN110989929A (en) * | 2019-11-22 | 2020-04-10 | 浪潮电子信息产业股份有限公司 | MON service migration method, device, equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3477482A3 (en) | Intelligent snapshot tiering | |
| US10565071B2 (en) | Smart data replication recoverer | |
| CN103167035A (en) | Method for solving NameNode single-point failure in hadoop distributed file system (HDFS) | |
| WO2013192198A3 (en) | Enhanced data management virtualization system | |
| EP4254220A3 (en) | Methods and systems for digital reward processing | |
| NZ705517A (en) | Physical security system having multiple server nodes | |
| PH12015502654A1 (en) | Sharing a virtual hard disk across multiple virtual machines | |
| CN102411637A (en) | Metadata management method of distributed file system | |
| CN102761528A (en) | System and method for data management | |
| WO2014028273A3 (en) | System and method for secure synchronization of data across multiple computing devices | |
| IN2013CH01467A (en) | ||
| WO2014022755A3 (en) | Centralized information management policies and enforcement | |
| CA2922665C (en) | Distributed file system using consensus nodes | |
| WO2014082043A3 (en) | Streaming restore of a database from a backup system | |
| MX2014001725A (en) | Zero-click photo upload. | |
| WO2012057581A3 (en) | Cloud computing system and data synchronization method therefor | |
| WO2013066898A3 (en) | Cooperative storage management | |
| WO2014147085A3 (en) | Method and system for byzantine fault tolerant data replication | |
| IN2014CH01236A (en) | ||
| CN104104739A (en) | Data synchronization method for metadata service cluster of cloud storage system | |
| CN104021200A (en) | Data synchronizing method and device of database | |
| WO2016100545A8 (en) | Computer servers for datacenter management | |
| CN104410674A (en) | A WEB session synchronization method of a single sign on system | |
| CN105681420A (en) | Method for realizing backup of switch operating system based on Docker | |
| WO2015017065A4 (en) | Coerced encryption on connected devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20130619 |