CN104102710A - Massive data query method - Google Patents

Abstract

The invention discloses a massive data query method, which is characterized in that it includes: establishing an index mapping between an HBase non-rowkey query field and a rowkey; when querying, according to the index mapping relationship, the corresponding query field is queried in SolrCloud rowkey; use the rowkey to search in HBase, and display the query results in pages.

Description

A Massive Data Query Method

technical field

The invention relates to the field of big data, in particular to a massive data query method based on SolrCloud and HBase.

Background technique

Big data (Big data) is usually used to describe the large amount of unstructured and semi-structured data created by a company, which takes too much time and money to download to a relational database for analysis. Big data analysis is often associated with cloud computing, because real-time analysis of large data sets requires frameworks like MapReduce and HBase to distribute work to tens, hundreds, or even thousands of computers. Compared with traditional data warehouse applications, big data analysis has the characteristics of large data volume and complex query and analysis. Big data requires special techniques to efficiently handle large volumes of data that tolerate elapsed time. Technologies applicable to big data, including massively parallel processing (MPP) databases, data mining grids, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

Solr is an independent enterprise-level search application server that provides an API interface similar to Web-service. Users can submit XML files in a certain format to the search engine server through http requests to generate indexes; they can also submit search requests through Http Get operations and get returned results in XML or json format. SolrCloud is a distributed search solution based on Solr and Zookeeper after Solr4.0. SolrCloud is a deployment method of Solr based on Zookeeper.

HBase is a distributed, column-oriented open source database, which is derived from the Google paper "Bigtable: A Distributed Storage System for Structured Data" written by Fay Chang. HBase–Hadoop Database is a high-reliability, high-performance, column-oriented, and scalable distributed storage system. Using HBase technology, a large-scale structured storage cluster can be built on a cheap PC Server. While HBase provides support for highly concurrent read and write operations, it also has some significant defects: because HBase only sorts rowkeys (row key values), HBase cannot quickly search and retrieve fields other than rowkeys. At the same time, HBase cannot realize query-based paging display and page-by-page query. Therefore, designing a massive data query method based on SolrCloud and HBase can effectively solve these problems.

Contents of the invention

In order to solve the above technical problems, the present invention provides a massive data query method and device, which realize flexible multi-condition query of massive data, fuzzy query and pagination of query results.

A massive data query method, comprising:

Establish HBase non-rowkey rowkey query field and rowkey index mapping;

When querying, according to the index mapping relationship, the rowkey corresponding to the query field is queried in SolrCloud;

Use the rowkey to search in HBase, and display the query results in pages.

Preferably, when the data in HBase changes, the index mapping in SolrCloud is regularly updated.

Preferably, the index mapping is stored in a distributed manner,

When the master server receives the update of the index mapping, it sends the updated index mapping to other replica servers of the same shard;

When the replica server receives the update of the index mapping, it sends the updated index mapping to the master server to which it belongs.

Preferably, a Mapreduce model is used to speed up the establishment of the index mapping.

A massive data query device, comprising:

The mapping module establishes an index mapping with rowkey for HBase non-rowkey query fields;

The query module, according to the index mapping relationship, first queries the HBase rowkey corresponding to the query field in SolrCloud, and then uses the rowkey to query the required data in HBase;

The display module displays the query results in pages to the user.

Preferably, the update module regularly updates the index mapping in SolrCloud when the data in HBase changes.

Preferably, the synchronization module sends the updated index mapping to other replica servers of the same fragment when the device acts as the master server.

Preferably, the synchronization module, when the device serves as a replica server, after the update module updates the index mapping, the synchronization module sends the updated index mapping to the master server to which it belongs.

The technical solution of this application uses SolrCloud to store and maintain the index mapping from non-rowkey fields that need to be queried to rowkey in HBase, find the corresponding rowkey according to the query conditions, and then use the rowkey to search for data in HBase, thereby realizing flexible Multi-condition query of massive data, fuzzy query and pagination of query results; at the same time, SolrCloud is deployed in a distributed manner, which can realize centralized information storage, automatic fault tolerance, near real-time search and automatic load balancing.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is the Solr+HBase query schematic diagram of the embodiment of the present invention;

Fig. 2 is the HBase rowkey index mapping schematic diagram of the embodiment of the present invention;

Fig. 3 is the SolrCloud cluster distribution schematic diagram of the embodiment of the present invention;

Fig. 4 is a flow chart of a massive data query method according to an embodiment of the present invention;

FIG. 5 is a structural diagram of a massive data query device according to an embodiment of the present invention.

Detailed ways

The present invention adopts a method based on SolrCloud+HBase, which can establish an index mapping with a rowkey for a specified non-rowkey field in HBase. When querying, first find the rowkey corresponding to the field to be queried, and then search in HBase, avoiding direct query by HBase When querying a problem with a single condition. When the present invention displays query results, it can be displayed in pages; thereby providing convenient and easy-to-implement multi-condition query and query result paging, and at the same time providing full-text index and fuzzy query capabilities that traditional HBase storage does not possess. For requests for quantity statistics, the results can be obtained directly through Solr's index mapping, and there is no need to query HBase.

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Each record in HBase is indexed in an orderly manner according to the rowkey. The indexing method is the same as that described in Figure 2. It is a multi-level index and adopts a positioning similar to a 3-layer B+ tree. The first is to find the location of the root region from zookeeper, so as to load the -ROOT-region. -ROOT-region is the first region of the .META. table, which stores the location information of all other regions of the .META. table. The .META. table is resident in the memory of all RegionServers, where the region location information of all data tables is stored. When querying in HBase through rowkey, it searches for -ROOT-, .META. Then locates the region where the data is located, and then retrieves valid data from the region. Since the indexes at each step are already established and ordered, it is very efficient to use this rowkey-based query in HBase. But for non-rowkey queries, the efficiency drops significantly.

In order to solve this problem, the present invention uses SolrCloud to pre-establish a group of index mappings of its corresponding rowkey to the non-rowkey query field. The query process is shown in Figure 1. First, the HBase rowkey corresponding to the query condition is found in SolrCloud, and then Use the rowkey to query data in HBase, and finally, return the query result to the client. This method can greatly improve query efficiency.

SolrCloud is deployed in a distributed manner, which can realize centralized information storage, automatic fault tolerance, near real-time search and automatic load balancing. As shown in Figure 3, this is a SolrCloud cluster with 6 nodes (servers). The index mapping is distributed in two Shards (shards). Each Shard contains three Solr nodes, one Leader (master) node, and two Shards. A Replica (replica) node. Each shard has 3 copies at the same time. When 2 nodes are down at the same time, the system can still work normally. All status information of the cluster is maintained uniformly by the Zookeeper cluster. For these 6 nodes, any node can accept the update request of the index mapping, thus realizing load balancing. For example, when the node Server4 receives an update request about the index mapping of Shard1, Server4 will forward the information to the Leader node to which the index mapping should belong, that is, Server1. After the update of the Server1 node is completed, the version number and index mapping are sent to other Replicas nodes belonging to the same shard, namely Server2 and Server3, to complete the synchronization.

The massive data query method provided by the present invention, as shown in Figure 4, includes:

Step 401, establishing an index mapping with a rowkey for an HBase non-rowkey query field.

When HBase data is created, SolrCloud is used to create an index mapping between non-rowkey fields and rowkeys according to the set query conditions. The query condition is set for HBase non-rowkey fields.

At the stage of establishing Solr index mapping, the Mapreduce model can be used to accelerate the establishment of index mapping.

Step 402, when querying, according to the index mapping relationship, the corresponding rowkey is queried in SolrCloud.

When a query is required, according to the index mapping relationship, first query the HBase rowkey corresponding to the query condition in SolrCloud, and then use the rowkey to query the required data in HBase.

Preferably, the index mapping is stored in a distributed manner,

When the master server receives the update of the index mapping, it sends the updated index mapping to other replica servers of the same shard;

When the replica server receives the update of the index mapping, it sends the updated index mapping to the master server to which it belongs.

Step 403, displaying the query results to the user in pages.

After the data is obtained in HBase according to the rowkey, when displaying to the user, it will be displayed to the user according to the set paging method.

In the original method of HBase, the query results do not support paging display, and users can only view all the query results. The improvement of the present invention is that the query results are displayed in pages, for example, 20 items are displayed on each page, and the user can know the displayed items at a glance.

Preferably, the method may further include: regularly updating the index mapping in SolrCloud when the data in HBase changes.

The present invention also provides a corresponding massive data query device, as shown in Figure 5, including:

The mapping module establishes an index mapping with rowkey for HBase non-rowkey query fields;

The query module, according to the index mapping relationship, first queries the HBase rowkey corresponding to the query condition in SolrCloud, and then uses the rowkey to query the required data in HBase;

The display module displays the query results in pages to the user.

Preferably, the device further includes an update module, which regularly updates the index mapping in SolrCloud when the data in the HBase changes.

The massive data query device of the present invention can be used as a server node, as shown in FIG. 3 , can be set in each server to form a cluster.

Preferably, the device further includes a synchronization module. When the device serves as a copy server, after the update module updates the index mapping, the synchronization module sends the updated index mapping to the master server to which it belongs.

Preferably, the synchronization module sends the updated index mapping to other replica servers of the same fragment when the device acts as the master server.

Application example

1. Definition and configuration of Solr schema (architecture) files

Modify the schema.xml file and add fields that need to be indexed. At the same time, modify the original uniqueKey and set the rowkey in the HBase table to Solr's uniqueKey.

2. Establishment of index mapping

Create a Solr index for the data in HBase through HBase API full table scan (Scan) or through MapReduce.

3. Realization of query and paging

When querying, one or a group of rowkeys corresponding to the query conditions are found in Solr. After obtaining these rowkeys, the grouped rowkeys are used to query in HBase, so that the actual results can be queried and paging search can be realized.

Those skilled in the art can understand that all or part of the steps in the above method can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, and the like. Optionally, all or part of the steps in the foregoing embodiments may also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the foregoing embodiments may be implemented in the form of hardware, or may be implemented in the form of software function modules. This application is not limited to any specific form of combination of hardware and software.

The above descriptions are only preferred examples of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (8)

1. a mass data inquiry method, is characterized in that, comprising:

Set up the index-mapping of the non-line unit value of HBase rowkey inquiry field and rowkey;

When inquiry, according to described index-mapping relation, in SolrCloud, inquire rowkey corresponding to inquiry field;

Use described rowkey to search in HBase, and by Query Result Pagination Display.

2. the method for claim 1, is characterized in that,

When data in HBase change, the index-mapping in regular renewal SolrCloud.

3. method as claimed in claim 2, is characterized in that,

Described index-mapping is distributed storage,

In the time of the renewal of master server reception hint mapping, the index-mapping of renewal is sent on other replica servers of same burst;

In the time of the renewal of replica server reception hint mapping, on the master server under the index-mapping of renewal is sent to.

4. the method for claim 1, is characterized in that,

Use Mapreduce model to accelerate the foundation of index-mapping.

5. a mass data inquiry unit, is characterized in that, comprising:

Mapping block, the index-mapping to the foundation of the non-rowkey inquiry of HBase field with rowkey;

Enquiry module according to index-mapping relation, first inquires the corresponding HBase rowkey of this inquiry field in SolrCloud, re-uses this rowkey and in HBase, inquires about required data;

Display module, by Query Result to user's Pagination Display.

6. device as claimed in claim 5, is characterized in that, also comprises:

Update module, in the time of data change in HBase, the index-mapping in regular renewal SolrCloud.

7. device as claimed in claim 5, is characterized in that, also comprises:

Synchronization module,, sends to the index-mapping of renewal on other replica servers of same burst during as master server at this device.

8. device as claimed in claim 7, is characterized in that,

Synchronization module, at this device during as replica server, after update module is upgraded index-mapping, on the master server under synchronization module sends to the index-mapping of renewal.