CN111326216B - A fast partitioning method for big data gene sequencing files - Google Patents

Abstract

The present invention relates to the field of high-performance computing, in particular to a method for quickly splitting large data gene sequencing files, so that in the multi-node gene analysis process, there is no need to actually split the sequencing files, no sub-files are generated, and a flexible partitioning scheme is provided according to subsequent analysis programs, so that the load of each node is more balanced, hard disk reading and writing is reduced, and partitioning efficiency is improved.

Description

A fast partitioning method for big data gene sequencing files

technical field

The invention relates to the field of high-performance computing, in particular to a method for quickly splitting large data gene sequencing files.

Background technique

With the rapid development of the general health field, genetic analysis technology has played an increasingly important role. The gene sequencer produces a large number of sequencing files, and the most commonly used sequencing file format is the fastq format. Each sequencing file ranges from a few gigabytes to tens of gigabytes to hundreds of gigabytes. How to quickly process these big data has increasingly become the bottleneck of genetic analysis.

Because the sequencing file is very large, it takes a lot of time to analyze and process with a single node, so multiple nodes are required to perform parallel computing to reduce the time for gene analysis. This requires dividing the sequencing file, each node only processes a part of the sequencing file, and finally merges the processing results, so as to obtain the complete result of genetic analysis in a short period of time.

When multiple nodes are used to process sequencing files, the common segmentation method is to divide the sequencing files equally according to the number of nodes, then generate multiple sub-files, write them to the hard disk, and each node reads the corresponding sub-files for processing. Although this method is simple and convenient, it will increase the read and write burden of the hard disk.

Also, common segmentation methods may affect the results of subsequent procedures. In sequencing analysis, sequence comparison software such as bwa and bowtie are usually used for analysis and comparison. For example, the bwa program reads files by blocks, and processes one block of the fastq file each time during its operation. Since the common segmentation method does not take this into consideration, the result of bwa is affected, and it is easy to cause inconsistency in the comparison results.

Contents of the invention

The present invention provides a method for quickly dividing large data gene sequencing files, including:

Step 101, setting the size of the file block;

Step 102, the fastq file is analyzed and counted according to the file block size set in step 101, and is divided into a plurality of file blocks; the position information of each file block and the total number of file blocks are saved in the information file;

Step 103, calculate the number of file blocks to be processed by each node according to the number of nodes and the number of cores of each node, and determine the start position and end position of the file part to be processed by each node in the fastq file;

Step 104, according to the start position and end position of the file part to be processed by each node determined in step 103 in the fastq file, generate a read instruction, and provide it to a subsequent program through a pipeline.

Preferably, step 102 in the above method further includes: if the end position of the file block is in the middle of a certain sequence, extending the file block to the end of the sequence.

Preferably, the position information of the file block in the above method includes the start position and the end position of the file block.

Preferably, the value range of the file block size in the above method may be between 1M and 100M.

Preferably, the number of file blocks to be processed by each node in step 103 of the above method is calculated according to the following formula:

Among them, B _i is the number of file blocks processed by the i-th node;

c _i is the number of cores of the i-th node;

B _t is the total number of file blocks;

n is the total number of nodes;

j is an integer ranging from 1 to n;

c _j is the number of cores of the jth node.

According to another aspect of the present invention, there is provided a method for rapidly dividing large data gene sequencing files, including:

Step 201, analyze and count the fastq files according to the sequence, and obtain the position information of each sequence and the total number of sequences;

Step 202, according to the number of nodes and the number of cores of each node, calculate the number of sequences that each node needs to process, and determine the start position and end position of the file part to be processed by each node in the fastq file;

Step 203, according to the start position and end position of the file part to be processed by each node obtained in step 202 in the fastq file, generate a read instruction, and provide it to a subsequent program through a pipeline.

Preferably, the position information of the sequence in the above method includes the start position and end position of the sequence.

Preferably, the number of sequences that each node needs to process in step 202 of the above method is calculated according to the following formula:

Among them, S _i is the number of sequences processed by the i-th node;

c _i is the number of cores of the i-th node;

S _t is the total sequence number;

n is the total number of nodes;

j is an integer ranging from 1 to n;

c _j is the number of cores of the jth node.

A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, any one of the above-mentioned methods is implemented.

A computer device includes a memory and a processor, and a computer program capable of running on the processor is stored in the memory, and is characterized in that any one of the above methods is implemented when the processor executes the program.

Aiming at the deficiencies of the prior art, the present invention adopts a strategy of lazily dividing fastq files and does not generate sub-files, thereby avoiding the reading, writing and storage of sub-files. And added a variety of division methods for subsequent analysis software. The method of the invention reduces the times of reading and writing of the hard disk, improves the speed of file division, and eliminates comparison errors.

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

Fig. 1 is a schematic flowchart of a method for dividing by blocks according to an embodiment of the present invention.

Fig. 2 is a schematic flowchart of a method for dividing by sequence according to an embodiment of the present invention.

Detailed ways

Before explaining this method in detail, briefly introduce the format of the fastq file. The fastq file is a text file, with four lines per sequence, the first line is the name information of the sequence, the second line is the base sequence, the third line is the description information, and the fourth line is the quality score information of the sequence. Each sequence is not exactly the same length. Sequencing files are divided into single-end sequencing files and paired-end sequencing files. A single-end sequencing file contains only one file, and a paired-end sequencing file contains a pair of files. Each sequence in this pair of files corresponds to each other.

According to an embodiment of the present invention, a method for dividing by blocks is introduced with reference to FIG. 1 , and the method includes the following steps.

Step 101, setting the size of the file block, preferably, its value range can be between 1M and 100M.

The inventor found that the gene sequencing analysis tool bwa reads the sequencing files in blocks during the comparison process. The inventors found that when multiple nodes are used to run the bwa tool to compare the fastq files in parallel, dividing the fastq files by blocks is beneficial to load balancing. The size of the file block can take different values according to different analysis software and the processing capability of the nodes. Preferably, the value range can be between 1M and 100M. In an embodiment of the present invention, when the size of the file block is 10M, better processing speed and load balancing effect can be obtained.

Step 102, analyze and count the fastq file according to the block size set in the previous step, and divide it into multiple file blocks; and save the analysis result into the information file.

When analyzing a fastq file by block size, take a block size of 10M as an example, offset 10M bytes backward from the start position of the file, and if this byte data happens to be at the end of a sequence, set the start position of the first file block to 0 and the end position to 10M. If this byte data is in the middle of a sequence, sets the end of the first file block to the end of the sequence. After finding the first file block position of the fastq file, store its start position and end position in the information file. It can be seen that the size of the first file block is greater than or equal to 10M and contains complete sequence data. Then, offset the end position of the first file block by one byte as the start position of the second file block, continue to offset backward by 10M bytes, if the current byte data is at the end of a sequence, set the current byte position as the end position of the second block, and if the current byte data is at the beginning or middle of a sequence, set the end position of the second block as the end position of the sequence. By analogy, analyze until the end of the fastq file, find the start position and end position of all file blocks and store them in the information file. According to an embodiment of the present invention, obviously, only the starting position can also be stored in the information file. It can be seen that the size of the last file block may be less than 10M, therefore, except the last file block, the size of each other file block will be slightly different, floating around 10M. Moreover, each file block contains multiple complete sequences, and a sequence only exists in one file block. According to an embodiment of the present invention, during the analysis process, the number of file blocks is also accumulated and stored in the information file.

For the paired-end sequencing files, if the two files have the same size, perform block-by-block statistical analysis according to one of the files, and if the two files are inconsistent in size, then the two files are divided according to another method of dividing by sequence in the present invention.

Step 103, according to the number of nodes and the number of cores of each node, calculate the number of file blocks that each node needs to process, and according to the statistical information obtained in step 102, determine the start position and end position of the file part to be processed by each node in the fastq file.

Specifically, in the multi-node gene analysis process, the number of cores and computing power of each computing node are different. Therefore, when dividing the sequencing files, these situations must be taken into consideration, and the processing range for each node should be determined to make the load more balanced.

According to an embodiment of the present invention, the number of file blocks processed by each node is calculated according to Formula 1.

Among them, B _i is the number of blocks processed by the i-th node;

c _i is the number of cores of the i-th node;

B _t is the total number of file blocks;

n is the total number of nodes;

j is an integer ranging from 1 to n;

c _j is the number of cores of the jth node.

After the number of file blocks to be processed by each node is calculated, the start position and end position of the file part to be processed by each node in the original sequencing file can be determined through the information file.

Step 104, generate a read command according to the starting position and ending position of the file part to be processed by each node obtained in step 103 in the original sequencing file, and provide it to the subsequent program through a pipeline.

The pipeline command is an existing technology, and a brief introduction is made here by taking the Linux system as an example. Linux pipes use the vertical bar "|" to connect multiple commands, which is called a pipe character. The specific syntax format of the Linux pipeline is as follows:

command1|command2

In the present invention, command1 is an instruction for reading the fastq file range, and command2 is an instruction for block analysis tools such as bwa.

According to another aspect of the present invention, the inventor also found that bowtie processes fastq files according to sequence, so when the subsequent processing program is a tool for sequence analysis such as bowtie, fastq files need to be divided according to sequence.

In the following, according to an embodiment of the present invention, a method for dividing by sequence is introduced with reference to FIG. 2 , and the method includes the following steps.

Step 201, analyze and count the fastq files according to the sequence. While analyzing the fastq file, save the analysis results to the info file.

Specifically, during the analysis process, the start and end positions of each sequence are analyzed and recorded, and the number of sequences is counted and saved to an information file.

For paired-end sequencing files, if the two files have the same size, perform statistical analysis by sequence according to one of the files, and if the two files are inconsistent in size, perform statistical analysis on the two files according to the sequence respectively.

Step 202, according to the number of nodes and the number of cores of each node, calculate the number of sequences that each node needs to process, and according to the statistical information obtained in step 201, determine the start position and end position of the file part to be processed by each node in the fastq file.

According to an embodiment of the present invention, in the multi-node gene analysis process, the number of cores and computing power of each computing node are different, so when dividing the sequencing files, these situations should be considered, and the processing range should be determined for each node to make the load more balanced.

The number of sequences processed by each node is calculated according to formula 2.

Among them, S _i is the number of sequences processed by the i-th node;

c _i is the number of cores of the i-th node;

S _t is the total sequence number;

n is the total number of nodes;

j is an integer ranging from 1 to n;

c _j is the number of cores of the jth node.

After the number of sequences to be processed by each node is calculated, the start position and end position of the file part to be processed by each node in the original sequencing file can be determined through the information file.

Step 20304, generate a read command according to the starting position and ending position of the file part to be processed by each node in the original sequencing file obtained in step 202, and provide it to the subsequent program through the pipeline.

The pipeline command format is as follows:

command1|command2

Among them, command1 is the command to read the range of the fastq file, and command2 is the command to analyze tools such as bowtie by sequence.

The present invention proposes a method for quickly dividing large data gene sequencing files, so that in the multi-node gene analysis process, there is no need to perform actual segmentation on the sequencing files, no sub-files are generated, and a flexible division scheme is provided according to the follow-up analysis program, thereby making the load of each node more balanced, reducing hard disk reading and writing, and improving division efficiency.

It should be noted that not all the steps described in the foregoing embodiments are necessary, and those skilled in the art may make appropriate trade-offs, replacements, modifications, etc. according to actual needs.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limit them. Although the present invention has been described in detail above with reference to the embodiments, those skilled in the art should understand that modifications or equivalent replacements to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all should be covered by the claims of the present invention.

Claims (6)

1. A rapid partitioning method for big data gene sequencing files, comprising:

step 101, setting the size of a file block;

102, analyzing and counting fastq files according to the file block sizes set in the step 101, and dividing the fastq files into a plurality of file blocks; storing the position information of each file block and the total number of the file blocks into an information file, wherein the position information of the file blocks comprises a starting position and an ending position of the file blocks;

step 103, calculating the number of file blocks to be processed by each node according to the number of nodes and the number of cores of each node, and determining the starting position and the ending position of the file part to be processed by each node in the fastq file;

step 104, generating a reading instruction according to the starting position and the ending position of the file part to be processed by each node in the fastq file, which are determined in step 103, providing the reading instruction to a subsequent program in a pipeline mode,

the number of file blocks to be processed by each node in step 103 is calculated according to the following formula:

wherein B is _i The number of file blocks processed for the ith node;

c _i the number of cores for the ith node;

B _t the total file block number;

n is the total node number;

j is an integer ranging from 1 to n;

c _j the number of cores for the j-th node.

2. The rapid partitioning method for big data gene sequencing file of claim 1, said step 102 further comprising: if the ending position of a file block is in the middle of a sequence, the file block is extended to the end of the sequence.

3. The rapid partitioning method for big data gene sequencing file as claimed in claim 1, wherein the file block size has a value ranging from 1M to 100M.

4. A rapid partitioning method for big data gene sequencing files, comprising:

step 201, analyzing and counting fastq files according to sequences, obtaining position information of each sequence and total number of the sequences, and storing the position information of each sequence into an information file, wherein the position information of each sequence comprises a starting position and an ending position of each sequence;

step 202, calculating the number of sequences to be processed of each node according to the number of nodes and the number of cores of each node, and determining the starting position and the ending position of a file part to be processed of each node in a fastq file;

step 203, generating a read instruction according to the start position and the end position of the file part to be processed by each node in the fastq file obtained in step 202, providing the read instruction to a subsequent program in a pipeline manner,

the number of sequences that each node needs to process in step 202 is calculated according to the following formula:

wherein S is _i The number of sequences processed for the ith node;

c _i the number of cores for the ith node;

S _t is the total number of sequences;

n is the total node number;

j is an integer ranging from 1 to n;

c _j the number of cores for the j-th node.

5. A computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the method according to any of claims 1-4.

6. A computer device comprising a memory and a processor, on which memory a computer program is stored which can be run on the processor, characterized in that the processor implements the method according to any of claims 1-4 when executing the program.