CN105512216A - Data storage and reading method, device and system - Google Patents

Abstract

The invention discloses a data storage and reading method, device and system. The data reading method includes: obtaining the data value value corresponding to the first-level keyword key in the database, and obtaining the second-level key; dividing the second-level key into N shares according to the obtained parameter N, wherein each second-level key The key corresponds to the input of a calculation unit, and N is greater than or equal to 1 and less than or equal to the number of second-level keys; the value corresponding to the corresponding second-level key is read concurrently through the calculation unit. By means of the technical solution of the present invention, hasoop? In the data access scope of MR, the database set in the secondary mode is also included in the scope of big data processing, making Hadoop? MR can directly read data in large batches from the database, improving the operating efficiency of hadoopMR.

Description

Data storage and reading method, device and system

technical field

The invention relates to the technical field of databases, in particular to a data storage and reading method, device and system.

Background technique

Hadoop is currently the most used and mature open source big data storage and computing platform in the industry. It includes many components, among which MapReduce (ie MR) is the programming model on the Hadoop platform. It is suitable for distributed computing under large amount of data. Redis is an open source index/value (key/value) memory database that supports storage in multiple data formats.

HadoopMR provides many data source reading implementations, such as files, hbase databases, etc., but it does not have good support for reading large batches of data directly from redis. Specifically, HadoopMR is a powerful tool for distributed computing of big data, and the input is generally It is a hdfs file. Redis is a key/value memory database, which generally reads a single value based on a single key. Therefore, there is currently no way to directly read data from redis to hadoopMR in large batches.

Contents of the invention

In view of the above problems, the present invention is proposed to provide a data storage and reading method, device and system that overcome the above problems or at least partially solve the above problems.

The invention provides a data storage method, comprising:

The keyword key in the database is divided into two levels according to the size of the granularity, wherein the data value corresponding to the first-level key with a larger granularity is the second-level key with a smaller granularity;

Save the data to be stored as the value of the second-level key.

The present invention also provides a kind of data reading method, is used for Hadoop programming model MR to read data from database, comprises:

Obtain the data value value corresponding to the first-level keyword key in the database, and obtain the second-level key;

Divide the second-level key into N shares according to the obtained parameter N, where each second-level key corresponds to the input of a computing unit, and N is greater than or equal to 1 and less than or equal to the number of second-level keys;

The value corresponding to the corresponding second-level key is read concurrently through the computing unit.

The present invention also provides a data storage device, comprising:

The grading module is used to divide the keyword key in the database into two levels according to the granularity, wherein the data value value corresponding to the first-level key with a larger granularity is the second-level key with a smaller granularity;

The storage module is used to save the data to be stored as the value of the second-level key.

The present invention also provides a kind of data reading device, is arranged on Hadoop programming model MR, comprises:

The obtaining module is used to obtain the data value value corresponding to the first-level keyword key in the database, and obtain the second-level key;

The segmentation module is used to divide the second-level key into N shares according to the obtained parameter N, wherein each second-level key corresponds to the input of a computing unit, and N is greater than or equal to 1 and less than or equal to the number of second-level keys ;

The reading module is used to concurrently read the value corresponding to the corresponding second-level key through the computing unit.

The present invention also provides a data storage and reading system, comprising the above-mentioned data storage device and the above-mentioned data reading device.

The beneficial effects of the present invention are as follows:

By setting the database to the secondary mode, the data access scope of hadoopMR is expanded, and the database set to the secondary mode is also included in the scope of big data processing, so that hadoopMR can directly read data from the database in large quantities, which improves the performance of hadoopMR. operating efficiency.

The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable , the specific embodiments of the present invention are enumerated below.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:

Fig. 1 is the flowchart of the data storage method of the embodiment of the present invention;

Fig. 2 is the schematic diagram of the Redis database of the embodiment of the present invention;

Fig. 3 is the flowchart of the data reading method of the embodiment of the present invention;

Fig. 4 is the schematic diagram that HadoopMR of the embodiment of the present invention reads data from redis database;

5 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;

6 is a schematic structural diagram of a data reading device according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a data storage and reading system according to an embodiment of the present invention.

detailed description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

In order to solve the problem that hadoopMR cannot directly read data from the Redis database in large quantities in the prior art, the present invention provides a data storage and reading method, device and system. The present invention is further described below in conjunction with the accompanying drawings and embodiments Detailed description. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Method embodiment one

According to an embodiment of the present invention, a data storage method is provided. FIG. 1 is a flow chart of the data storage method according to an embodiment of the present invention. As shown in FIG. 1 , the data storage method according to an embodiment of the present invention includes the following processing:

Step 101, divide the keyword key in the database into two levels according to the granularity, wherein the data value value corresponding to the first-level key with a larger granularity is the second-level key with a smaller granularity; preferably, the granularity Including: the size of the time range. That is to say, the key of the first-level storage is the time range of hours, such as the data of a certain hour, and the value is the list of the keys of the second-level storage; the key of the second-level storage is the time range of minutes, such as within 1 minute The data, value is the reported data.

Step 102, save the data to be stored as the value of the second-level key.

Preferably, in the embodiment of the present invention, the database is a Redis database. That is to say, the technical solution of the embodiment of the present invention adopts a two-level key/value association in Redis to adapt to batch read operations. Fig. 2 is the schematic diagram of the Redis database of the embodiment of the present invention, as shown in Fig. 2, the value under the key stored in the first level is the list of keys stored in the second level, such as key1, key2, ..., keyn. The value of the key stored in the second level is the reported data. For example, the value of key1 is dataset1, and the value of key2 is dataset2.

In the technical solution of the embodiment of the present invention, the data in Redis is stored in two levels, so that hadoopMR can directly read the stored data in Redis in batches, so as to perform distributed calculation on the stored data in Redis. The fact that hadoopMR can directly read the stored data in redis in batches is described in detail in the second method embodiment.

Method embodiment two

According to an embodiment of the present invention, a kind of data reading method is provided, is used for Hadoop programming model MR to read data from database, and Fig. 3 is the flowchart of the data reading method of the embodiment of the present invention, as shown in Fig. 3, The data reading method according to the embodiment of the present invention includes the following processing:

Step 301, obtain the data value value corresponding to the first-level keyword key in the database, and obtain the second-level key;

Step 302, dividing the second-level key into N shares according to the obtained parameter N, wherein each second-level key corresponds to an input of a computing unit, and the N is greater than or equal to 1 and less than or equal to the number of second-level keys number;

In step 302, preferably, the acquiring the parameter N specifically includes: determining the parameter N according to the number of computing units performing concurrent computing.

Step 303, concurrently read the value corresponding to the corresponding second-level key through the calculation unit.

The following is a detailed description of the Hadoop programming model MR reading data from the redis database. Fig. 4 is a schematic diagram of hadoopMR reading data from a redis database according to an embodiment of the present invention, as shown in Fig. 4 , specifically including the following processing:

Step 1: In hadoopMR, first assemble the first-level key according to the time to be read (for example, 2015111016 means November 10, 2015 from 15:00 to 16:00), read the first-level storage in redis, and obtain the corresponding secondary storage All keys.

Step 2: Divide all secondary keys into N shares according to the parameter N passed in by the external program, and each share corresponds to the input of a computing unit mapper, where N is at most equal to the number of all secondary keys, and N indicates how many mappers are specified Programs to perform concurrent computations.

Step 3: Each mapper of hadoopMR reads the secondary key/value assigned to it; all mappers can be read together and stored in redis within the hour from 15:00 to 16:00 on November 10, 2015 all data for .

To sum up, with the help of the technical solution of the embodiment of the present invention, by setting the database in the secondary mode, the data access range of hadoopMR is expanded, and the database set in the secondary mode is also included in the scope of big data processing, so that HadoopMR can directly read data in large batches from the database, improving the operating efficiency of hadoopMR. Directly reading the memory database has higher performance than reading files, and takes less time to read data.

Device embodiment

According to an embodiment of the present invention, a data storage device is provided. FIG. 5 is a schematic structural diagram of a data storage device according to an embodiment of the present invention. As shown in FIG. 5 , the data storage device according to an embodiment of the present invention includes: a classification module 50 , and the storage module 52, each module of the embodiment of the present invention will be described in detail below.

The classification module 50 is used to divide the keyword key in the database into two levels according to the granularity, wherein the data value value corresponding to the first-level key with a larger granularity is a second-level key with a smaller granularity; preferably, the The size of granularity mentioned above includes: the size of the time range. The database is a Redis database.

The storage module 52 is configured to store the data to be stored as the value of the second-level key.

The specific processing of each module in the embodiment of the present invention has been described in detail in the first method embodiment, and will not be repeated here.

Device embodiment two

According to an embodiment of the present invention, a data reading device is provided, which is set in the Hadoop programming model MR. FIG. 6 is a schematic structural diagram of a data reading device according to an embodiment of the present invention. As shown in FIG. 6, according to an embodiment of the present invention The data reading device includes: an acquisition module 60, a segmentation module 62, and a reading module 64, each module of the embodiment of the present invention will be described in detail below.

Obtaining module 60, is used for obtaining the data value value corresponding to first level keyword key in the database, obtains second level key;

The segmentation module 62 is configured to divide the second-level key into N shares according to the obtained parameter N, wherein each second-level key corresponds to an input of a computing unit, and the N is greater than or equal to 1 and less than or equal to the second The number of level keys; the segmentation module 62 to which it belongs is specifically configured to: determine the parameter N according to the number of calculation units performing concurrent calculations.

The reading module 64 is configured to concurrently read the value corresponding to the corresponding second-level key through the calculation unit.

The specific processing of each module in the embodiment of the present invention has been described in detail in the first method embodiment, and will not be repeated here.

System embodiment

According to an embodiment of the present invention, a data storage and reading system is provided. FIG. 7 is a schematic structural diagram of a data storage and reading system according to an embodiment of the present invention. As shown in FIG. 7 , the data storage and reading system according to an embodiment of the present invention The system includes: the data storage device 70 of the first embodiment of the device above, and the data reading device 72 of the second embodiment of the device above. The above devices have been described in detail in the corresponding device embodiments, and will not be repeated here.

To sum up, with the help of the technical solution of the embodiment of the present invention, by setting the database in the secondary mode, the data access range of hadoopMR is expanded, and the database set in the secondary mode is also included in the scope of big data processing, so that HadoopMR can directly read data in large batches from the database, improving the operating efficiency of hadoopMR. Directly reading the memory database has higher performance than reading files, and takes less time to read data.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and equivalent technologies thereof, the present invention also intends to include these modifications and variations.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other device. Various generic systems can also be used with the teachings based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not specific to any particular programming language. It should be understood that various programming languages can be used to implement the content of the present invention described herein, and the above description of specific languages is for disclosing the best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, in order to streamline this disclosure and to facilitate an understanding of one or more of the various inventive aspects, various features of the invention are sometimes grouped together in a single embodiment, figure, or its description. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art can understand that the modules in the client in the embodiment can be adaptively changed and installed in one or more clients different from the embodiment. The modules in the embodiments can be combined into one module, and furthermore can be divided into a plurality of sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method or method so disclosed may be used in any combination, except that at least some of such features and/or processes or units are mutually exclusive. All processes or units of the client are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will understand that although some embodiments described herein include some features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention. and form different embodiments. For example, in the following claims, any one of the claimed embodiments may be used in any combination.

The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to realize some or all of the functions of some or all of the components in the client loaded with sorted URLs according to embodiments of the present invention . The present invention can also be implemented as an apparatus or an apparatus program (for example, a computer program and a computer program product) for performing a part or all of the methods described herein. Such a program for realizing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

Claims (11)

1. a date storage method, is characterized in that, comprising:

Keyword key in database is divided into two-stage according to granule size, and wherein, the data value value that the first order key that granularity is larger is corresponding is the second level key that granularity is less;

The value of the data needing to store as second level key is preserved.

2. the method for claim 1, is characterized in that, described granule size comprises: the size of time range.

3. the method for claim 1, is characterized in that, described database is Redis database.

4. a method for reading data, reads data for Hadoop programming model MR from database, it is characterized in that, comprising:

Obtain the data value value that in database, first order keyword key is corresponding, obtain second level key;

Described second level key is divided into N part by the Parameter N according to obtaining, wherein, the input of every part of corresponding computing unit of second level key, described N is more than or equal to 1 and is less than or equal to the number of second level key;

By the value corresponding to the corresponding second level key of the concurrent reading of computing unit.

5. method as claimed in claim 4, it is characterized in that, the Parameter N of described acquisition specifically comprises:

According to the number of the computing unit that execution concurrence calculates, determine Parameter N.

6. a data storage device, is characterized in that, comprising:

Diversity module, for the keyword key in database is divided into two-stage according to granule size, wherein, the data value value that the first order key that granularity is larger is corresponding is the second level key that granularity is less;

Memory module, for preserving the value of the data needing to store as second level key.

7. device as claimed in claim 6, it is characterized in that, described granule size comprises: the size of time range.

8. device as claimed in claim 6, it is characterized in that, described database is Redis database.

9. a data fetch device, is arranged at Hadoop programming model MR, it is characterized in that,

Acquisition module, for obtaining the data value value that in database, first order keyword key is corresponding, obtains second level key;

Cutting module, for described second level key being divided into N part according to the Parameter N obtained, wherein, the input of every part of corresponding computing unit of second level key, described N is more than or equal to 1 and is less than or equal to the number of second level key;

Read module, for passing through the value corresponding to the corresponding second level key of the concurrent reading of computing unit.

10. device as claimed in claim 9, is characterized in that, affiliated cutting module specifically for:

According to the number of the computing unit that execution concurrence calculates, determine Parameter N.

11. 1 kinds of data store reading system, it is characterized in that, comprise the data storage device according to any one of claim 6-8 and the data fetch device according to any one of claim 9-10.