CN104391748A - Mapreduce calculation process optimization method - Google Patents

Abstract

The invention discloses a method for optimizing the mapreduce calculation process. First, the original data file is divided into several files, and one of the unprocessed files is selected as the input of the sub-job to determine whether there is a file that needs to be merged. If there is no file, Then submit the task; start the Map task with the same processing process, execute the Map operation, sort, merge and partition the Map output, receive the Map output result, execute the Reduce operation, and save the output result; if there are files that need to be merged, submit the task, Start the Map task with multiple processing, send different input data to the corresponding Map, perform Map operations, multi-output sorting, merging, and partitioning, and finally check whether there are still data files in the original data file set that have not been processed. If not, End the program, otherwise, re-partition the good data file and execute this process again. The invention disperses the output time, reduces the instantaneous network transmission flow, reduces the occupancy rate of the local disk, and improves the MapReduce calculation process.

Description

A mapreduce calculation process optimization method

technical field

The invention relates to the technical field of computer software and parallel computing, and is specifically described as an optimization method for improving the MapReduce computing process by reducing the amount of intermediate data stored on a local disk during program operation and reducing disk load.

Background technique

With the rapid development of computer technology and Internet technology, the network penetration rate and the scale of Internet users are also increasing year by year. The double stimulation of the continuous increase of user scale and the rapid growth of data processing has brought new challenges to Internet applications. Massive data requires a huge amount of storage resources as the basis, and the increasing dependence of network applications on data makes the demand for the ability to calculate and process massive data more and more intense. The cost of maintaining data storage for these applications and data calculation Processing costs are getting higher and higher. Hadoop has developed vigorously in just a few years after its appearance, which proves its huge technical capabilities and application value, but it should be noted that Hadoop is still very young and incomplete in many aspects. It is necessary and meaningful to further improve the performance of MapReduce by carrying out research on the perfection and optimization of Hadoop.

MapReduce is a software architecture proposed by Google for parallel computing of large-scale data sets (greater than 1TB). The concepts "Map" and "Reduce", as well as their main ideas, are borrowed from functional programming languages, along with features borrowed from vector programming languages. [1] The current software implementation is to specify a Map (mapping) function to map a set of key-value pairs into a new set of key-value pairs, and specify a concurrent Reduce (simplification) function to ensure that all mapped keys Each of the value pairs shares the same set of keys.

Contents of the invention

The technical problem to be solved by the present invention is: in order to improve the MapReduce task processing ability, aiming at the high memory usage rate in the MapReduce task processing process, the concentrated consumption of network resources and network congestion, and the current situation of resource shortage caused by high disk load, a MapReduce calculation process is provided Optimization.

The technical scheme adopted in the present invention is:

A method for optimizing the mapreduce calculation process. First, the original data file is divided into several files, and one of the unprocessed files is selected as the input of the sub-job to determine whether there is a file that needs to be merged. If not, the task is submitted; Start the Map task with the same processing process, execute the Map operation, sort, merge, and partition the Map output, receive the Map output result, execute the Reduce operation, and save the output result; if there is a file that needs to be merged, submit the task, start with multiple Process the Map task, send different input data to the corresponding Map, perform Map operations, multi-output sorting, merging, and partitioning, and finally check whether there are data files in the original data file set that have not been processed. If not, end the program, otherwise , re-partitioned data files to perform this process again. the

The architecture involved in the method includes: an input data processing module, a data result merging module, a calculation process and a merging process combination module, wherein:

The input data processing module is responsible for splitting the original data into several parts. This splitting is not the system's fragmentation processing of the input data. What is needed is manual division, and the data size is much larger than the fragmented data block size. The data starts a job processing, and the sub-jobs are executed in sequence; at any time during the program running, only one sub-job is running on the system, and only a part of the original data is operated, so the intermediate data generated by the sub-jobs can be achieved in this way. Relatively few, shorten the running time of a single job, and delete the intermediate data in time;

Since the output of each sub-job is only for the calculation of part of the data, not the calculation results of the entire original data, there are a lot of repetitions among the results of each part, and the data result merging module is responsible for merging the repeated results to reduce redundant intermediate data;

The calculation process and merging process combination module is responsible for combining the sub-job calculation process and the merging process together, and merging while calculating. Except for the first sub-job, the next sub-job receives the original input data and the previous sub-job at the same time. According to the calculation results, according to the different application types, the map function of the combined data is adjusted accordingly, so that after the Map process is completed, the intermediate data is a set of key-value pairs with the same format, which will not affect the subsequent Reduce tasks.

The realization process of described method is as follows:

1) Build a Hadoop cluster, and allocate 2G and 20G input data files to the unoptimized single-job MapReduce program and the optimized multi-job MapReduce program respectively;

2) Split the input data of 2G and 20G into several data files and then iteratively run the sub-jobs, start a job processing for each job, execute each sub-job in sequence, and use multiple input operations to combine the sub-job calculation process and The merging process is combined and processed together, and merged while calculating. Except for the first sub-job, the next sub-job receives the original input data and the calculation result of the previous sub-job at the same time. According to the different application types, the map of the merged data The function is adjusted accordingly.

Beneficial effects of the present invention: Since the input data is dispersed into multiple jobs, the intermediate data output by the Map of each job is relatively small, and the execution time of each job is short, so that the output data of the Map stage under the job can be timely Deleted from the local disk, avoiding its long-term occupation of the disk. In fact, in the optimized MapReduce program, the actual intermediate data generated is not less than that of the unoptimized program, but the output time is dispersed, thereby dispersing the disk I/O The operation reduces the instantaneous network transmission traffic, and the subsequent stage of the Reduce operation is performed on the intermediate data in time, which reduces the occupancy rate of the local disk, thereby improving the MapReduce calculation process.

Description of drawings

Fig. 1 is a flow chart of the optimized MapReduce calculation process of the present invention.

Detailed ways

Below according to accompanying drawing of description, in conjunction with specific embodiment, the present invention is further described:

A mapreduce calculation process optimization method. First, the original data file needs to be divided into several files, and one of the unprocessed files is selected as the input of the sub-job to determine whether there are files that need to be merged. If not, submit the task. ;Start the Map task with the same processing process, execute the Map operation, sort, merge and partition the Map output, receive the Map output result, execute the Reduce operation, and save the output result; if there is a file that needs to be merged, submit the task, start the multi This is a Map task for processing, sending different input data to the corresponding Map, performing Map operations, multi-output sorting, merging, and partitioning, and finally checking whether there are still data files in the original data file set that have not been processed. If not, end the program. Otherwise, re-partition the good data file and execute this process again. the

The architecture involved in the method includes: an input data processing module, a data result merging module, a calculation process and a merging process combination module, wherein:

The input data processing module is responsible for splitting the original data into several parts. This splitting is not the system's fragmentation processing of the input data. What is needed is manual division, and the data size is much larger than the fragmented data block size. The data starts a job processing, and the sub-jobs are executed in sequence; at any time during the program running, only one sub-job is running on the system, and only a part of the original data is operated, so the intermediate data generated by the sub-jobs can be achieved in this way. Relatively few, shorten the running time of a single job, and delete the intermediate data in time;

Since the output of each sub-job is only for the calculation of part of the data, not the calculation results of the entire original data, there are a lot of repetitions among the results of each part, and the data result merging module is responsible for merging the repeated results to reduce redundant intermediate data;

The calculation process and merging process combination module is responsible for combining the sub-job calculation process and the merging process together, and merging while calculating. Except for the first sub-job, the next sub-job receives the original input data and the previous sub-job at the same time. According to the calculation results, according to the different application types, the map function of the combined data is adjusted accordingly, so that after the Map process is completed, the intermediate data is a set of key-value pairs with the same format, which will not affect the subsequent Reduce tasks.

The realization process of described method is as follows:

1) Build a Hadoop cluster, and allocate 2G and 20G input data files to the unoptimized single-job MapReduce program and the optimized multi-job MapReduce program respectively;

2) Split the input data of 2G and 20G into several data files and then iteratively run the sub-jobs, start a job processing for each job, execute each sub-job in sequence, and use multiple input operations to combine the sub-job calculation process and The merging process is combined and processed together, and merged while calculating. Except for the first sub-job, the next sub-job receives the original input data and the calculation result of the previous sub-job at the same time. According to the different application types, the map of the merged data The function is adjusted accordingly.

After the calculation process is finished, we found that in the optimized MapReduce program, the input data is distributed among multiple jobs, the intermediate data output by the Map of each job is relatively small, and the execution time of each job is short. As a result, the output data of the Map stage under this job can be deleted from the local disk in time, avoiding its long-term occupation of the disk. In the optimized MapReduce program, the total amount of intermediate data actually generated is not less than that of the unoptimized program, but we disperse its output time, thereby dispersing disk I/O operations, reducing instantaneous network transmission traffic, and timely processing The intermediate data is subjected to the Reduce operation in the latter stage, which reduces the occupancy rate of the local disk. It can be seen that the optimized MapReduce program has a better test effect.

Claims (3)

1. a mapreduce computation process optimization method, it is characterized in that: first raw data file is divided into some parts of files, from not processed file set, select a input as subjob, judge whether the file needing to merge, if do not exist, then submit task to; Startup has the Map task of same process, performs Map operation, receives Map Output rusults, perform Reduce operation, preserve Output rusults Map after exporting sequence, merging, subregion; The file merged is needed if exist, then submit task to, start the Map task with multiple process, different input data are sent to corresponding Map, perform Map operation, multi output sequence, merging, subregion, finally check raw data file concentrates whether data file is not processed in addition, if nothing, terminates program, otherwise, again ready-portioned data file is performed this process again.

2. a kind of mapreduce computation process optimization method according to claim 1, it is characterized in that, the architecture that described method relates to comprises: input data processing module, data result merge module, computation process and merging process binding modules, wherein:

Input data processing module, raw data is responsible for split into some parts, this splits is not the burst process of system to input data, it is desirable that manually divide, size of data is also far longer than fragment data block size, for each number is according to startup one-stop operation process, each subjob performs in order successively; In program operation, any instant system all only has a subjob in operation, and only some raw data is operated;

Data result merges module, is responsible for the result of repetition to merge, and reduces redundancy intermediate data;

Computation process and merging process binding modules, be responsible for subjob computation process and merging process to combine process, calculating limit, limit merges, except first subjob, next subjob receives the result of calculation of original input data and a upper subjob simultaneously, according to the difference of application type, the map function being combined data does corresponding adjustment.

3. a kind of mapreduce computation process optimization method according to claim 1 and 2, it is characterized in that, the performing step of described method is as follows:

1) hadoop cluster is built, respectively to the input data file of many operations MapReduce programme distribution 2G, the 20G after the single job MapReduce program do not optimized and optimization;

2) operation of subjob iteration is carried out after the input Data Division of 2G, 20G being become some parts of data files, to the process of job initiation one-stop operation each time, each subjob performs in order successively, multi input is utilized to operate the process that subjob computation process and merging process to be combined, calculating limit, limit merges, except first subjob, next subjob receives the result of calculation of original input data and a upper subjob simultaneously, according to the difference of application type, the map function of the data be combined adjusts accordingly.