CN104050291A - Parallel processing method and system for account balance data - Google Patents

Abstract

The invention discloses a method for parallel processing of account balance data. The method includes: one or more Map nodes executing the first task read different slice data of account balance detailed data, and generate the data in the read slice data. The first output parameter and the second output parameter of each balance record; wherein, the first output parameter includes at least an account ID, and the second output parameter is set as account status information, and the account status information includes at least: balance value, transaction date, and transaction number of the day; one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task, and according to the first output parameter and the second output parameter of the balance record The two output parameters respectively generate the daily average balance value records of each account; among them, the balance records with the same first output parameter are read by the same Reduce node. The invention can quickly acquire the statistical results of the daily average balance of the account under a large amount of data. The invention also discloses a parallel processing system for account balance data.

Description

A parallel processing method and system for account balance data

technical field

The invention relates to the technical field of big data processing, in particular to a method and system for parallel processing of account balance data under a large amount of data.

Background technique

Data is an indispensable information that almost all business activities such as production, operation, and strategy rely on. Data is like the eyes of business operators. Data can reflect business problems, just like the helmsman relies on navigation. As human society enters the information age, data has become a strategic resource as important as water and oil. Enterprises are currently facing massive growth in data volume. For example, a recent IDC report predicts that by 2020, the amount of data worldwide will expand 50-fold. At present, the size of big data is still a changing indicator, and the size of a single data set ranges from tens of terabytes to several petabytes. In addition, data can be generated from a variety of unexpected sources.

Traditional business data has evolved over time in a standard format that can be recognized by standard business intelligence software. Compared with traditional business data, big data has a multi-layered structure, which means that big data will present various forms and types. Due to the irregular and ambiguous nature of big data, it is difficult or even impossible to analyze it using traditional application software.

Today, the challenge for enterprises is to extract value from various forms of complex data.

Contents of the invention

The technical problem to be solved by the present invention is to provide a parallel processing method and system for account balance data, which can quickly obtain statistical results of account daily average balance under a large amount of data.

In order to solve the above technical problems, the present invention provides a method for parallel processing of account balance data, the method comprising:

One or more Map nodes performing the first task read different fragmented data of the account balance detailed data, and generate the first output parameter and the second output parameter of each balance record in the read fragmented data; wherein, the The first output parameter includes at least an account ID, and the second output parameter is set as account status information, and the account status information includes at least: balance value, transaction date and transaction number of the day;

One or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task, and generate the daily balance records of each account according to the first output parameter and the second output parameter of the balance record respectively. Average balance value records; where the balance records with the same first output parameter are read by the same Reduce node.

Further, the method also includes the following features:

The generating of the daily average balance value records of each account according to the first output parameter and the second output parameter of the balance record includes:

According to the account ID in the first output parameter, traverse each balance record of the same account, determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, and calculate each day The balance value of the account is averaged within the start and end time range of the query to obtain the daily average balance value of the account, and a record of the daily average balance value of the account is generated.

Further, the method also includes the following features:

After the Reduce node performing the first task respectively generates the daily average balance value records of each account according to the first output parameter and the second output parameter of the balance record, it also includes:

One or more Map nodes performing the second task read the daily average balance value records of different accounts, and generate the first output parameter and the second output parameter of the read daily average balance value records; wherein, the daily average balance value The first output parameter of the value record is set to the interval where the daily average balance value is located, and the second output parameter of the daily average balance value record is set to 1;

One or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task, and according to the first output parameter and the second output parameter of the daily average balance value record Counting the number of accounts in each daily average balance value interval, including: according to the daily average balance value interval in the first output parameter, traversing each daily average balance value record in the same daily average balance value interval, and recording each daily average balance value The second output parameter of the value record is accumulated to obtain the number of accounts in the daily average balance value interval; wherein, the daily average balance value records with the same first output parameter are read by the same Reduce node.

Further, the method also includes the following features:

According to the second output parameter of the balance record, determine the balance value of the account for each day within the query start and end time range, including:

From the day of the query start time to the day of the query end time, judge whether there is a balance record for each day. If it exists, use the balance value of the balance record with the largest transaction number on that day as the final balance value of the day. If it does not exist, trace back to earlier than For the latest date with a balance record on that day, the balance value of the balance record with the largest transaction sequence number on the day of the latest date is taken as the final balance value of the day.

Further, the method also includes the following features:

Before one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task, it also includes:

Calculate the hash value of the first parameter of each balance record, and establish a mapping relationship between the hash value of the first parameter and the Reduce node that executes the first task; wherein, the mapping relationship is used for the execution of the first task. The Reduce node of the task reads the corresponding balance record according to the mapping relationship.

Further, the method also includes the following features:

Before one or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task, it also includes:

Calculate the hash value of the first parameter of each daily average balance value record, and establish a mapping relationship between the hash value of the first parameter and the Reduce node executing the second task; wherein, the mapping relationship is used for the execution of the second task The Reduce node of the second task reads the corresponding daily average balance value record according to the mapping relationship.

Further, the method also includes the following features:

Before one or more Map nodes performing the first task read different fragmented data of the account balance detailed data, it also includes:

Determine the reading range of account balance detailed data according to the start and end time of the query, including: determining the full balance detailed data and the incremental balance detailed data up to the day when the query ends as the reading range of account balance detailed data;

Fragmenting the account balance detail data belonging to the reading range, and establishing a mapping relationship between each fragment and the Map node performing the first task; wherein, the mapping relationship is used for the Map node performing the first task The corresponding fragment data is read according to the mapping relationship.

In order to solve the above technical problems, the present invention also provides a parallel processing system for account balance data, the system includes:

The Map processing module includes one or more Map nodes that execute the first task; each Map node that executes the first task is used to read different fragment data of the account balance detailed data, and generates each item in the read fragment data. The first output parameter and the second output parameter of the balance record; wherein, the first output parameter includes at least an account ID, and the second output parameter is set as account status information, and the account status information includes at least: balance value, Transaction date and transaction serial number of the day;

The Reduce processing module includes one or more Reduce nodes that perform the first task; each Reduce node that performs the first task is used to read the different balance records that the Map node that performs the first task has processed, according to the balance record The first output parameter and the second output parameter respectively generate the daily average balance value records of each account; wherein, the balance records with the same first output parameter are read by the same Reduce node.

Further, the system also includes the following features:

The Reduce node performing the first task is used to generate the daily average balance value record of each account according to the first output parameter and the second output parameter of the balance record, including: according to the account ID in the first output parameter , traverse the balance records of the same account, determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, and calculate the balance value of each day within the query start and end time range Take the average to obtain the daily average balance value of the account, and generate the daily average balance value record of the account.

Further, the system also includes the following features:

The Map processing module also includes one or more Map nodes that execute the second task, and the Reduce processing module also includes one or more Reduce nodes that execute the second task;

Each Map node performing the second task is used to read the daily average balance value records of different accounts, and generate the first output parameter and the second output parameter of the read daily average balance value records; wherein, the daily average balance value The first output parameter of the record is set to the interval where the daily average balance value is located, and the second output parameter of the record of the daily average balance value is set to 1;

Each Reduce node executing the second task is used to read the different daily average balance value records processed by the Map node executing the second task, and make statistics according to the first output parameter and the second output parameter of the daily average balance value record The number of accounts in each daily average balance value interval, including: according to the daily average balance value interval in the first output parameter, traverse each daily average balance value record in the same daily average balance value interval, and record each daily average balance value The second output parameter of the record is accumulated to obtain the number of accounts in the daily average balance value interval; wherein, the daily average balance value records with the same first output parameter are read by the same Reduce node.

Further, the system also includes the following features:

The Reduce node executing the first task is used to determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, including: from the day of the query start time to the query end time On the current day, judge whether there is a balance record for each day. If it exists, take the balance value of the balance record with the largest transaction number on the current day as the final balance value of the current day. The balance value of the balance record with the largest transaction sequence number on the most recent date mentioned above is taken as the final balance value of the day.

Further, the system also includes the following features:

The Reduce processing module also includes the first task routing module:

The first task routing module is used to calculate the first parameter of each balance record before one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task Hash value, establishes the mapping relationship between the hash value of the first parameter and the Reduce node performing the first task; wherein, the mapping relationship is used for the Reduce node performing the first task according to the mapping relationship Read the corresponding balance record.

Further, the system also includes the following features:

The Reduce processing module also includes a second task routing module:

The second task routing module is used to calculate each daily average balance value record before one or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task The hash value of the first parameter of the value record, establishes the mapping relationship between the hash value of the first parameter and the Reduce node performing the second task; wherein, the mapping relationship is used for the Reduce node performing the second task according to The mapping relationship reads the corresponding daily average balance value records.

Further, the system also includes the following features: the Map processing module also includes a fragmentation module:

The sharding module is used to determine the reading range of the account balance detail data according to the query start and end time before one or more Map nodes performing the first task read different shard data of the account balance detail data, including: The full balance detailed data and the incremental balance detailed data up to the day when the query is terminated are determined as the reading range of the account balance detailed data; the account balance detailed data belonging to the reading range is fragmented, and each fragment is established and executed A mapping relationship of the Map node of the first task; wherein, the mapping relationship is used for the Map node executing the first task to read corresponding fragment data according to the mapping relationship.

Compared with the prior art, the present invention provides a method and system for parallel processing of account balance data. Based on MapReduce, the large-scale account balance detailed data is divided into several parts and handed over to Map nodes for parallel processing. In the Map stage, the data is processed according to the account. After the processing is completed, it is grouped according to the account id and routed to multiple Reduce nodes for parallel processing, so as to quickly obtain the statistical results of the average daily balance of the account under a large amount of data, with high processing efficiency and strong scalability.

Description of drawings

FIG. 1 is a flow chart of obtaining the daily average balance value of each user in a parallel processing method of account balance data according to an embodiment of the present invention.

Fig. 2 is a flow chart of counting the number of accounts in each daily average balance value interval in a parallel processing method of account balance data according to an embodiment of the present invention.

FIG. 3 is a schematic structural diagram of a parallel processing system for account balance data according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a processing architecture of account balance data based on MapReduce in an application example of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined arbitrarily with each other.

As shown in Figure 1, an embodiment of the present invention provides a method for parallel processing of account balance data, the method comprising:

S10, one or more Map nodes performing the first task read different fragmented data of the account balance detailed data, and generate the first output parameter and the second output parameter of each balance record in the read fragmented data; wherein , the first output parameter includes at least an account ID, the second output parameter is set as account status information, and the account status information includes at least: balance value, transaction date and transaction number of the day;

S20, one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task, and generate each account according to the first output parameter and the second output parameter of the balance record The daily average balance value record; wherein, the balance records with the same first output parameter are read by the same Reduce node;

The method may also include the following features:

Preferably, before one or more Map nodes performing the first task read different fragmented data of account balance detailed data, it also includes:

Determine the reading range of account balance detailed data according to the start and end time of the query, including: determining the full balance detailed data and the incremental balance detailed data up to the day when the query ends as the reading range of account balance detailed data;

Fragmenting the account balance detail data belonging to the reading range, and establishing a mapping relationship between each fragment and the Map node performing the first task; wherein, the mapping relationship is used for the Map node performing the first task The corresponding fragment data is read according to the mapping relationship.

Preferably, the generating of the daily average balance value records of each account according to the first output parameter and the second output parameter of the balance record includes:

According to the account ID in the first output parameter, traverse each balance record of the same account, determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, and calculate each day The balance value of the account is averaged within the start and end time range of the query to obtain the daily average balance value of the account, and a record of the daily average balance value of the account is generated.

Preferably, according to the second output parameter of the balance record, the balance value of the account for each day within the query start and end time range is determined, including:

From the day of the query start time to the day of the query end time, judge whether there is a balance record for each day. If it exists, use the balance value of the balance record with the largest transaction number on that day as the final balance value of the day. If it does not exist, trace back to earlier than For the latest date with a balance record on that day, the balance value of the balance record with the largest transaction sequence number on the day of the latest date is taken as the final balance value of the day.

Preferably, before one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task, further comprising:

Calculate the hash value of the first parameter of each balance record, and establish a mapping relationship between the hash value of the first parameter and the Reduce node that executes the first task; wherein, the mapping relationship is used for the execution of the first task. The Reduce node of the task reads the corresponding balance record according to the mapping relationship.

Preferably, the hash value of the first parameter of each balance record is the modulus of the first parameter to the total number of Reduce nodes executing the first task;

Preferably, as shown in Figure 2, after step S20, it also includes:

S30, determining the interval in which the daily average balance of each account is located, and counting the number of accounts in each interval;

Preferably, determine the interval in which the daily average balance of each account is located, and count the number of accounts in each interval, including:

S301, one or more Map nodes executing the second task read the daily average balance value records of different accounts, and generate the first output parameter and the second output parameter of the read daily average balance value records; wherein, the daily average balance value records The first output parameter of the average balance value record is set to the interval where the daily average balance value is located, and the second output parameter of the daily average balance value record is set to 1;

S302. One or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task, and according to the first output parameter and the second output parameter of the daily average balance value record, The output parameter counts the number of accounts in each average daily balance value interval, including: according to the daily average balance value interval in the first output parameter, traverse each daily average balance value record in the same daily average balance value interval, and record each daily average balance value interval The second output parameter of the average balance value record is accumulated to obtain the number of accounts in the daily average balance value interval; wherein, the daily average balance value records with the same first output parameter are read by the same Reduce node.

Preferably, before one or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task, it also includes:

Calculate the hash value of the first parameter of each daily average balance value record, and establish a mapping relationship between the hash value of the first parameter and the Reduce node executing the second task; wherein, the mapping relationship is used for the execution of the second task The Reduce node of the second task reads the corresponding daily average balance value record according to the mapping relationship.

Preferably, the hash value of the first parameter of each daily average balance value record takes the modulus of the first parameter to the total number of Reduce nodes executing the second task;

Wherein, the Map nodes executing the second task and the Map nodes executing the first task are the same batch of nodes or different batches of nodes, that is, the Map nodes can execute the second task only after executing the first task. Similarly, the Reduce nodes executing the second task and the Reduce nodes executing the first task are the same batch of nodes or different batches of nodes, that is, the Reduce nodes can execute the second task only after executing the first task.

As shown in Figure 3, an embodiment of the present invention provides a parallel processing system for account balance data, the system includes:

The Map processing module includes one or more Map nodes that execute the first task; each Map node that executes the first task is used to read different fragment data of the account balance detailed data, and generates each item in the read fragment data. The first output parameter and the second output parameter of the balance record; wherein, the first output parameter includes at least an account ID, and the second output parameter is set as account status information, and the account status information includes at least: balance value, Transaction date and transaction serial number of the day;

The Reduce processing module includes one or more Reduce nodes that perform the first task; each Reduce node that performs the first task is used to read the different balance records that the Map node that performs the first task has processed, according to the balance record The first output parameter and the second output parameter respectively generate the daily average balance value records of each account; wherein, the balance records with the same first output parameter are read by the same Reduce node.

The system may also include the following features:

Preferably, the Reduce node executing the first task is used to generate the daily average balance value records of each account according to the first output parameter and the second output parameter of the balance record, including: according to the first output parameter Account ID of the same account, traverse each balance record of the same account, determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, and calculate the balance value of each day at the start and end of the query Take the average within the time range to obtain the daily average balance value of the account, and generate the daily average balance value record of the account.

Preferably, the Map processing module further includes one or more Map nodes executing the second task, and the Reduce processing module further includes one or more Reduce nodes executing the second task;

Each Map node performing the second task is used to read the daily average balance value records of different accounts, and generate the first output parameter and the second output parameter of the read daily average balance value records; wherein, the daily average balance value The first output parameter of the record is set to the interval where the daily average balance value is located, and the second output parameter of the record of the daily average balance value is set to 1;

Each Reduce node executing the second task is used to read the different daily average balance value records processed by the Map node executing the second task, and make statistics according to the first output parameter and the second output parameter of the daily average balance value record The number of accounts in each daily average balance value interval, including: according to the daily average balance value interval in the first output parameter, traverse each daily average balance value record in the same daily average balance value interval, and record each daily average balance value The second output parameter of the record is accumulated to obtain the number of accounts in the daily average balance value interval; wherein, the daily average balance value records with the same first output parameter are read by the same Reduce node.

Preferably, the Reduce node executing the first task is used to determine the balance value of the account for each day within the query start and end time range according to the second output parameter of the balance record, including: from the current day of the query start time to On the day of the query termination time, determine whether there is a balance record for each day. If it exists, use the balance value of the balance record with the largest transaction number on that day as the final balance value of the day. If it does not exist, trace back to the latest date with a balance record that is earlier than the current day , taking the balance value of the balance record with the largest transaction sequence number on the most recent date as the final balance value of the day.

Preferably, the Reduce processing module also includes a first task routing module:

The first task routing module is used to calculate the first parameter of each balance record before one or more Reduce nodes executing the first task read the different balance records processed by the Map node executing the first task Hash value, establishes the mapping relationship between the hash value of the first parameter and the Reduce node performing the first task; wherein, the mapping relationship is used for the Reduce node performing the first task according to the mapping relationship Read the corresponding balance record.

Preferably, the Reduce processing module also includes a second task routing module:

The second task routing module is used to calculate each daily average balance value record before one or more Reduce nodes executing the second task read the different daily average balance value records processed by the Map node executing the second task The hash value of the first parameter of the value record, establishes the mapping relationship between the hash value of the first parameter and the Reduce node performing the second task; wherein, the mapping relationship is used for the Reduce node performing the second task according to The mapping relationship reads the corresponding daily average balance value records.

Preferably, the Map processing module also includes a fragmentation module:

The sharding module is used to determine the reading range of the account balance detail data according to the query start and end time before one or more Map nodes performing the first task read different shard data of the account balance detail data, including: The full balance detailed data and the incremental balance detailed data up to the day when the query is terminated are determined as the reading range of the account balance detailed data; the account balance detailed data belonging to the reading range is fragmented, and each fragment is established and executed A mapping relationship of the Map node of the first task; wherein, the mapping relationship is used for the Map node executing the first task to read corresponding fragment data according to the mapping relationship.

Application example

An application example is given below: statistics of the average daily balance of each account from January 1 to January 8 this year, and the distribution range of the average daily balance of each account. (Suppose there are two accounts: id001 and id002 respectively)

The original detailed data of the two accounts are shown in Table 1, including part of the full data of last year (2013) and incremental data from January 1 to January 8 of this year (2014).

date Account ID balance (yuan) serial number 20140101 002 20 1 20140102 002 10 1 20140102 002 50 2 20140103 002 30 1 20140103 002 15 2

20140106 002 40 1 20140106 002 50 2 20140106 002 60 3 20140108 002 30 1 20131230 002 90 1 20140106 001 60 1 20140108 001 30 1 20131225 001 90 1

Table 1

As shown in Figure 4, two tasks are started based on MapReduce to process and calculate account information in parallel.

The first task divides the input large-scale data file into several fragments and sends them to the Map node for parallel processing. In the Map stage, the data is output in the form of <account id, account information>. This step classifies and pre-classifies the massive data Processing, directing the detailed data of the same account to the same Reduce node for combined processing, for example, grouping and routing to multiple Reduce nodes for parallel processing according to the account id hash Hash value (modulo the number of Reduce nodes); The output detailed data of each account is processed, including:

a) Read the incremental files from the full amount of files last year to the query termination time;

b) The Map stage classifies all the data and outputs <account id, account detail data information>

c) In the Reduce stage, the detailed data in the same account id is selected for organization and processing, and the balance value of each account is determined for each day in the statistical cycle from the query start time to the end time, and then calculated for each account in the statistical cycle average daily balance.

All data of account id001 and account id002 are fetched and processed by two Reduce nodes respectively. For each account, the Reduce node judges whether there is a balance record for each day from the day of the query start time to the day of the query end time. If there is, the balance value of the balance record with the largest transaction number on the day is taken as the final balance value of the day. If it does not exist, trace back to the latest date that is earlier than the current day and has a balance record, and use the balance value of the balance record with the largest transaction sequence number on the day of the latest date as the final balance value of the current day.

For example, on the day when the query starts (January 1 this year), and there is a balance record in account 002, the balance value "20 yuan" in the balance record of account 002 on January 1, 2014 with transaction number 1 is taken as the balance value of account 002. The balance value on January 1, 2014; if the 001 account has no balance record on January 1, 2014, the balance record closest to this year in the full amount of data in 2013 (on December 25, 2013, the transaction number is 1) The balance value "90 yuan" is used as the balance value of the 001 account on January 1, 2014.

For the 002 account, the balance value of the 002 account determined by the first Reduce node within the query start and end time range for each day, as shown in Table 2;

query date Balance value (yuan) 20140101 20 20140102 50 20140103 15 20140104 15 20140105 40 20140106 60 20140107 60 20140108 30

Table 2

For account 001, the balance value of account 002 determined by the first Reduce node within the query start and end time range for each day, as shown in Table 3;

date balance value 20140101 90 20140102 90

20140103 90 20140104 90 20140105 90 20140106 60 20140107 60 20140108 30

table 3

The daily average balance of account 002 as of January 8, 2014 is calculated as follows: (20+50+15+15+40+60+60+30)/8=36.25 (yuan);

The average daily balance of account 001 as of January 8, 2014 is calculated as follows: (90+90+90+90+90+60+60+30)/8=75;

The second task calculates the distribution of the daily average balance of each account based on the calculation results of the first task.

According to the daily average balance output by the first task, the daily average balance is judged in the Map stage of the second task. For example, set intervals [0,15), [15,50), [50,100]. Then the interval of the average daily balance of account 002 is [15,50), and the interval of the average daily balance of account 001 is [50,100]. For account 002, the interval [15,50] is used as the first output parameter (key), and 1 is used as the second output parameter (value); for account 001, the interval [50,100] is used as the first output parameter (key), and 1 is used as the second output parameter (value). The second output parameter (value). At the Reduce end of the second task, the value in each interval is accumulated and output, and the result is the account distribution number in each interval.

The method and system for parallel processing of account balance data provided by the above embodiments, based on MapReduce, divides large-scale account balance detailed data into several parts and hands them to Map nodes for parallel processing. In the Map stage, the data is classified according to accounts. After the processing is completed According to the account id, it is grouped and routed to multiple Reduce nodes for parallel processing, so as to quickly obtain the statistical results of the average daily balance of the account under a large amount of data, with high processing efficiency and strong scalability.

Those skilled in the art can understand that all or part of the steps in the above method can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, and the like. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Correspondingly, each module/unit in the above embodiments can be implemented in the form of hardware, or can be implemented in the form of software function modules. The form is realized. The present invention is not limited to any specific combination of hardware and software.

It should be noted that the present invention can also have other various embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and deformations according to the present invention, but these Corresponding changes and deformations should belong to the scope of protection of the appended claims of the present invention.

Claims (14)

1. a method for parallel processing for account balance data, the method comprises:

The Map node of one or more execution first tasks reads the different fragment datas of account balance detailed data, generates the first output parameter and second output parameter of each remaining sum record in the fragment data reading; Wherein, described the first output parameter at least comprises account ID, and described the second output parameter is set as account status information, and described account status information at least comprises: remaining sum value, trade date and daylight trading sequence number;

The Reduce node of one or more execution first tasks reads the complete different remaining sum records of Map node processing of described execution first task, and the first output parameter recording according to described remaining sum and the second output parameter generate respectively the average daily remaining sum value record of each account; Wherein, the remaining sum record that the first output parameter is identical is read by same Reduce node.

2. the method for claim 1, is characterized in that:

Described the first output parameter recording according to described remaining sum and the second output parameter generate respectively the average daily remaining sum value record of each account, comprising:

According to the account ID in described the first output parameter, travel through each remaining sum record of same account, determine the remaining sum value of every day within inquiry beginning and ending time account according to the second output parameter of described remaining sum record, the remaining sum value of every day is averaged to the average daily remaining sum value that obtains the account within the described inquiry beginning and ending time, generates the average daily remaining sum value record of the account.

3. the method for claim 1, is characterized in that, generates respectively after the average daily remaining sum value record of each account according to the first output parameter and second output parameter of described remaining sum record at the Reduce of described execution first task node, also comprises:

The Map node of one or more execution the second tasks reads the average daily remaining sum value record of different accounts, generates the first output parameter and second output parameter of the average daily remaining sum value record reading; Wherein, the first output parameter of described average daily remaining sum value record is set as the interval at described average daily remaining sum value place, and the second output parameter of described average daily remaining sum value record is set as 1;

The Reduce node of one or more execution the second tasks reads the complete average daily remaining sum value record of difference of Map node processing of described execution the second task, according to the account number in the first output parameter of described average daily remaining sum value record and the second output parameter statistics each average daily remaining sum value interval, comprise: according to the average daily remaining sum value interval in described the first output parameter, travel through each the average daily remaining sum value record in same average daily remaining sum value interval, the second output parameter of each average daily remaining sum value record is added up, obtain the account number in this average daily remaining sum value interval; Wherein, the average daily remaining sum value record that the first output parameter is identical is read by same Reduce node.

4. method as claimed in claim 2, is characterized in that:

The remaining sum value of determining every day within inquiry beginning and ending time account according to the second output parameter of described remaining sum record, comprising:

From the same day of inquiry initial time to the same day of inquiring about the termination time, judge and whether have remaining sum record every day, as existed, resulting balance value using the remaining sum value of the remaining sum record of day trade transaction sequence number maximum as the same day, if do not existed, reviewed early than the same day and there is the nearest date that remaining sum records, the resulting balance value using the remaining sum value of the remaining sum record of the transaction sequence number maximum of described nearest that day on date as the same day.

5. the method for claim 1, is characterized in that:

Before the Reduce of one or more execution first tasks node reads the complete different remaining sums records of the Map node processing of described execution first task, also comprise:

Calculate the cryptographic hash of the first parameter of each remaining sum record, set up the mapping relations of the cryptographic hash of the first parameter and the Reduce node of described execution first task; Wherein, described mapping relations read corresponding remaining sum record for the Reduce node for described execution first task according to described mapping relations.

6. method as claimed in claim 3, is characterized in that:

Before the Reduce node of one or more execution the second tasks reads the complete average daily remaining sum value record of difference of the Map node processing of described execution the second task, also comprise:

Calculate the cryptographic hash of the first parameter of each average daily remaining sum value record, set up the cryptographic hash of the first parameter and the mapping relations of the Reduce node of second task of execution; Wherein, described mapping relations read corresponding average daily remaining sum value record for the Reduce node for described execution the second task according to described mapping relations.

7. the method for claim 1, is characterized in that:

Read the different fragment datas of account balance detailed data at the Map of one or more execution first tasks node before, also comprise:

Determine and comprise the read range of account balance detailed data according to the inquiry beginning and ending time: by full dose balance detail data be defined as the read range of account balance detailed data by the end of the increment balance detail data on inquiry same day termination time;

By the account balance detailed data burst belonging in this read range, set up the mapping relations of the Map node of each burst and execution first task; Wherein, described mapping relations read corresponding fragment data for the Map node for described execution first task according to described mapping relations.

8. a parallel processing system (PPS) for account balance data, this system comprises:

Map processing module, comprises the Map node of one or more execution first tasks; Each Map node of carrying out first task, for reading the different fragment datas of account balance detailed data, generates the first output parameter and second output parameter of each remaining sum record in the fragment data reading; Wherein, described the first output parameter at least comprises account ID, and described the second output parameter is set as account status information, and described account status information at least comprises: remaining sum value, trade date and daylight trading sequence number;

Reduce processing module, comprises the Reduce node of one or more execution first tasks; Each Reduce node of carrying out first task is for reading the complete different remaining sum records of Map node processing of described execution first task, and the first output parameter recording according to described remaining sum and the second output parameter generate respectively the average daily remaining sum value record of each account; Wherein, the remaining sum record that the first output parameter is identical is read by same Reduce node.

9. system as claimed in claim 8, is characterized in that:

The Reduce node of described execution first task is for generating respectively the average daily remaining sum value record of each account according to the first output parameter of described remaining sum record and the second output parameter, comprise: according to the account ID in described the first output parameter, travel through each remaining sum record of same account, determine the remaining sum value of every day within inquiry beginning and ending time account according to the second output parameter of described remaining sum record, the remaining sum value of every day is averaged to the average daily remaining sum value that obtains the account within the described inquiry beginning and ending time, generate the average daily remaining sum value record of the account.

10. system as claimed in claim 8, is characterized in that, described Map processing module also comprises the Map node of one or more execution the second tasks, and described Reduce processing module also comprises the Reduce node of one or more execution the second tasks;

Each Map node of carrying out the second task, for reading the average daily remaining sum value record of different accounts, generates the first output parameter and second output parameter of the average daily remaining sum value record reading; Wherein, the first output parameter of described average daily remaining sum value record is set as the interval at described average daily remaining sum value place, and the second output parameter of described average daily remaining sum value record is set as 1;

Each Reduce node of carrying out the second task is for reading the complete average daily remaining sum value record of difference of Map node processing of described execution the second task, according to the account number in the first output parameter of described average daily remaining sum value record and the second output parameter statistics each average daily remaining sum value interval, comprise: according to the average daily remaining sum value interval in described the first output parameter, travel through each the average daily remaining sum value record in same average daily remaining sum value interval, the second output parameter of each average daily remaining sum value record is added up, obtain the account number in this average daily remaining sum value interval; Wherein, the average daily remaining sum value record that the first output parameter is identical is read by same Reduce node.

11. systems as claimed in claim 9, is characterized in that:

The Reduce node of described execution first task is for determining the remaining sum value of every day within inquiry beginning and ending time account according to the second output parameter of described remaining sum record, comprise: from the same day of inquiry initial time to the same day of inquiring about the termination time, judge and whether have remaining sum record every day, as existed, resulting balance value using the remaining sum value of the remaining sum record of day trade transaction sequence number maximum as the same day, if do not existed, reviewed early than the same day and there is the nearest date that remaining sum records, resulting balance value using the remaining sum value of the remaining sum record of the transaction sequence number maximum of described nearest that day on date as the same day.

12. systems as claimed in claim 8, is characterized in that, described Reduce processing module also comprises first task routing module:

Described first task routing module, for read the complete different remaining sums records of the Map node processing of described execution first task at the Reduce of one or more execution first tasks node before, calculate the cryptographic hash of the first parameter of each remaining sum record, set up the mapping relations of the cryptographic hash of the first parameter and the Reduce node of described execution first task; Wherein, described mapping relations read corresponding remaining sum record for the Reduce node for described execution first task according to described mapping relations.

13. systems as claimed in claim 10, is characterized in that, described Reduce processing module also comprises the second task routing module:

Described the second task routing module, for read the complete average daily remaining sum value record of difference of the Map node processing of described execution the second task at the Reduce node of one or more execution the second tasks before, calculate the cryptographic hash of the first parameter of each average daily remaining sum value record, set up the cryptographic hash of the first parameter and the mapping relations of the Reduce node of second task of execution; Wherein, described mapping relations read corresponding average daily remaining sum value record for the Reduce node for described execution the second task according to described mapping relations.

14. systems as claimed in claim 8, is characterized in that, described Map processing module also comprises burst module:

Described burst module, for before the Map of one or more execution first tasks node reads the different fragment datas of account balance detailed data, determine and comprise the read range of account balance detailed data according to the inquiry beginning and ending time: by full dose balance detail data be defined as the read range of account balance detailed data by the end of the increment balance detail data on inquiry same day termination time; By the account balance detailed data burst belonging in this read range, set up the mapping relations of the Map node of each burst and execution first task; Wherein, described mapping relations read corresponding fragment data for the Map node for described execution first task according to described mapping relations.