CN114420210B - Rapid trimming method and system for biological sequencing sequence - Google Patents

Abstract

The invention provides a method and a system for quickly trimming a biological sequencing sequence, which belong to the technical field of biological information, and the scheme comprises the following steps: obtaining a biological sequencing sequence to be trimmed; performing a read operation, a trim operation, and a write operation on the biological sequencing sequence; the read operation, the trim operation and the write operation are decoupled based on a producer-consumer model, and asynchronous execution is realized; and the formatting process of the biological sequencing sequence is transferred from the read operation to the trim operation.

Description

A kind of biological sequencing sequence rapid trimming method and system

technical field

The invention belongs to the technical field of biological information, and in particular relates to a method and system for rapid trimming of biological sequencing sequences.

Background technique

The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

In next-generation sequencing, the nucleic acid sequence to be sequenced is connected with the adapter sequence (Adapter) so that it can be recognized by the sequencer. However, when the length of the nucleic acid sequence is shorter than the read length of the sequencing platform, the sequenced gene sequence fragment ( Called Read) will contain both the nucleic acid sequence to be sequenced and all or part of the linker sequence. In addition, in NGS (Next Generation Sequencing: Next Generation Sequencing) sequencing, the confidence of the sequencing results will become lower at the end of the cycle (tail cycle), and some low-quality sequencing sequences are obtained. Sequencing sequences contaminated by sequencing adapters or low-quality sequencing processes often lead to unsatisfactory downstream analysis (such as gene alignment work, etc.) missing link.

With the advancement of modern sequencers, the ever-increasing throughput and sequence lengths pose new challenges for pruning work. The performance of some current processing tools (such as speed and accuracy) has been difficult to meet the sequencing data of this scale, making The preprocessing step has become the bottleneck of data analysis, and an ultrafast and accurate sequencing data adapter quality trimming tool for NGS data preprocessing is still in urgent need.

At present, many tools are used for the trimming of sequencing adapters and low-quality data in next-generation sequencing data, such as: Trimmomatic, Fastp, Ktrim, etc. These tools use different Trim algorithms to realize Adapter in NGS sequencing data -Trim and Quality-Trim.

The inventor found that Ktrim has better performance on NGS short-read sequencing data than several other tools. However, with the development and popularization of solid-state disk technology and the advancement of disk array technology, Ktrim's current processing rate The difference between the read and write peaks of the hard disk is large, which cannot meet the current performance requirements for biological data preprocessing, and after testing, Ktrim's thread scalability is not good, and it is difficult to continue to improve the processing speed of programs using more than four threads.

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention provides a method and system for rapid pruning of biological sequencing sequences. The solution greatly improves the rate of I/O by using a lightweight I/O framework; at the same time, on this basis The vectorization method is used to optimize the Adapter search process, which effectively improves the processing efficiency of rapid pruning of biological sequencing sequences.

According to a first aspect of the embodiments of the present invention, a rapid trimming method for biological sequencing sequences is provided, including:

Obtain the biological sequencing sequence to be trimmed;

A read operation, a trim operation, and a write operation are performed on the biological sequencing sequence; wherein, the read operation, the trimming operation, and the writing operation are decoupled based on a producer-consumer model to achieve asynchronous execution; and the biological sequencing sequence The formatting process is shifted from the read operation to the trim operation.

Further, the read operation, the pruning operation and the writing operation are respectively implemented by independent threads, wherein one read thread and one write thread are provided, and one or more pruning threads are provided.

Further, the read operation is used to read the biological sequencing sequence in a block manner through a read thread, and store the read block objects in the first data queue.

Further, the trimming operation is used to obtain data from the first data queue through a trimming thread, format the biological sequencing sequence, and remove low-quality base sequences and linker sequences in the biological sequencing sequence; The latter sequence is stored in the second data queue.

Further, obtaining the linker sequence in the pruning thread includes: using each base in the biological sequencing sequence as a character; and using several bit operations to obtain the position of the linker sequence in the preset length sequence data based on the vector register. .

Further, the writing operation is used to obtain the processed biological sequencing sequence from the second data queue through a writing thread, and store it.

According to a second aspect of the embodiments of the present invention, a rapid trimming system for biological sequencing sequences is provided, including:

a data acquisition unit, which is used to acquire the biological sequencing sequence to be trimmed;

A data processing unit, which is used to perform read operation, trim operation and write operation on the biological sequencing sequence; wherein, the read operation, trim operation and write operation are decoupled based on the producer-consumer model to realize asynchronous execution ; and the formatting process of the biological sequencing sequence is transferred from the read operation to the trimming operation.

Compared with the prior art, the beneficial effects of the present invention are:

(1) The present invention provides a method and system for rapid pruning of biological sequencing sequences. The solution greatly improves the rate of I/O by using a lightweight I/O framework; The vectorization method optimizes the Adapter search process and effectively improves the processing efficiency of rapid pruning of biological sequencing sequences.

(2) The present invention solves the problem of low IO efficiency of Ktrim, and improves the input and output rate of file data to reach or approach the performance peak value of disk read and write. The scheme realizes decoupling, asynchrony and speed balance among reading threads, working threads and writing threads by adopting the producer-consumer model; at the same time, the task of data formatting is transferred from the reading thread to the working thread, and the data is written back. Prepared tasks are shifted from the writer thread to the worker thread to maximize the speed of the reader and writer threads.

(3) The scheme of the present invention uses the data pool DataPool to reuse chunk objects, reduces the creation and destruction of objects, and reduces the overhead of creating objects; at the same time, when transferring data between reading threads, working threads and writing threads, use data as much as possible. pointer, without data copying, which effectively reduces the overhead of memory copying;

(4) In the solution of the present invention, the reading thread, the working thread (ie, the trimming thread) and the writing thread in the whole workflow are created only once, and are not destroyed until all their tasks are completed, which effectively reduces the overhead of thread creation.

Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will become apparent from the description which follows, or may be learned by practice of the invention.

Description of drawings

The accompanying drawings forming a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention.

1 is a schematic diagram of the flow of chunk objects between a read thread and a worker thread (that is, a Trim thread) described in an embodiment of the present invention;

FIG. 2 is a schematic diagram of the relationship between a worker thread (that is, a Trim thread) and a writing thread described in an embodiment of the present invention;

3 is a schematic structural diagram of the lightweight IO framework described in the embodiment of the present invention;

FIG. 4 is a schematic diagram of the overall framework described in the embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

Embodiments of the invention and features of the embodiments may be combined with each other without conflict.

Terminology Explanation:

Adapter: Adapter sequence, refers to the additional sequence fragments measured due to sequencing technology or sequencing platform during the biological sequence sequencing process.

Trim: An important link in the quality control of sequencing data, mainly to remove low-quality base sequences and adapter sequences.

Example 1:

The purpose of this embodiment is to provide a rapid trimming method for biological sequencing sequences.

First of all, it should be noted that the present invention is an improvement based on the Ktrim tool. Compared with several other existing tools (such as Trimmomatic and Fastp), Ktrim has better performance on NGS short-read sequencing data. Tested on a computing server device equipped with an Intel Xeon CPU and a standard 64-bit CentOS system, using single-ended test data (10 million pieces of sequencing data in FASTQ format, about 2.1GB in size), on four threads the tool can To achieve its best performance, the processing time is 7.26 seconds, and the processing rate is about 300MB/sec; using paired-end test data (two single-end sequencing data files, a total of 20 million pieces of data in FASTQ format, about 4.2GB in size), The best performance is achieved on four threads, the processing time is 11.29 seconds, and the best performance processing rate is about 370MB/sec. The author of the Ktrim tool mentions in his paper (Sun K.Ktrim:an extra-fast and accurate adapter-and quality-trimmer for sequencing data) that Ktrim maintains a high accuracy while processing speed is the fastest among several other tools (Trim galore, Trimmomatic, SeqPurgea) ~2-18 times.

Although Ktrim has better performance than other tools, with the development and popularization of solid-state disk technology and the advancement of disk array technology, the current processing rate of Ktrim is far from the peak read and write value of hard disks, which cannot meet the current demand for The performance requirements of biological data preprocessing, and after testing, the thread scalability of Ktrim is not good, and it is difficult to continue to improve the processing speed of programs using more than four threads. Therefore, Ktrim still has a lot of room for improvement in computing performance. .

Through analysis and testing, we found that the performance bottleneck of Ktrim is mainly in the parsing of FASTQ files. The inefficient IO method limits the processing performance of the program, which is a typical IO-intensive application. At the same time, look for Adapter in Ktrim The process is also relatively time-consuming. In view of the above situation, we propose a fast trimming method for biological sequencing sequences (hereinafter referred to as the RabbitTrim tool), which uses the lightweight FASTQ data IO framework on the basis of Ktrim (as shown in the figure). 3 shows the lightweight IO framework structure diagram), which greatly improves the IO rate. On this basis, the vectorization method is used to optimize the Adapter (connector sequence) search process, so that the processing efficiency of the program is improved. It is a significant improvement, which can be close to the peak hard disk read and write (about 2GB/sec) of our test platform.

Specifically, the performance bottleneck of the Ktrim tool is mainly in the IO of data files, including:

(1) Read thread pressure is high. A read thread is used to read the input data file in Ktrim. The read thread not only needs to read the string data, but also needs to parse the data according to the FASTQ format. The work of data formatting greatly increases the burden of the read thread, resulting in the production of FASTQ format. Data efficiency is low.

(2) The number of data prefetching is small. Ktrim supports multi-threaded mode, but in multi-threaded mode, it only prefetches one more piece of data (one BATCH data), and there is still a problem with the input data supply when the calculation part is executed faster.

(3) There are dependencies among the various parts of the workflow. Partial parallelism is achieved between the read operation and the Trim operation (that is, the trim operation), but when the execution rates of the two operations do not match, it will cause one side to wait for the other side, and there is a string between the Trim operation and the write operation. In the line method, the next block is processed only after a block of data is processed and output is completed. There is a large dependency between read operations, trim operations, and write operations.

(4) The process of finding the connector is slow. The Ktrim tool uses the char * strstr(const char * haystack , const char * needle ) function in the <string.h> library to find the "seed" of the connector, which is relatively slow .

In order to solve the above problems, the present invention provides a rapid trimming method for biological sequencing sequences, including:

Obtain the biological sequencing sequence to be trimmed;

A read operation, a trim operation, and a write operation are performed on the biological sequencing sequence; wherein, the read operation, the trimming operation, and the writing operation are decoupled based on a producer-consumer model to achieve asynchronous execution; and the biological sequencing sequence The formatting process is shifted from the read operation to the trim operation.

Further, the read operation, the pruning operation and the writing operation are respectively implemented by independent threads, wherein one read thread and one write thread are provided, and one or more pruning threads are provided.

Further, the read operation is used to read the biological sequencing sequence in a block manner through a read thread, and store the read block objects in the first data queue.

Further, the creation of the block objects introduces the idea of a data pool, and only a preset number of block objects are created for repeated use.

Further, the trimming operation is used to obtain data from the first data queue through a trimming thread, format the biological sequencing sequence, and remove low-quality base sequences and linker sequences in the biological sequencing sequence; The latter sequence is stored in the second data queue.

Further, obtaining the linker sequence in the pruning thread includes: using each base in the biological sequencing sequence as a character; and using several bit operations to obtain the position of the linker sequence in the preset length sequence data based on the vector register. .

Further, the writing operation is used to obtain the processed biological sequencing sequence from the second data queue through a writing thread, and store it.

Further, the threads corresponding to the read operation, the pruning operation and the write operation are created only once, and are destroyed after the processing task is completed.

Further, the formatting specifically includes performing data parsing according to the FASTQ format.

Specifically, in order to facilitate understanding, the scheme of the present invention is described in detail below in conjunction with the accompanying drawings:

The solution of the present invention mainly improves the problems existing in the Ktrim tool, including the following two aspects: the improvement of the input and output process and the improvement of the Adapter query process, specifically:

(1) Improvement of the input and output process

For the input and output process, the RabbitTrim tool proposed by the present invention uses a lightweight IO framework to complete data input, parsing and output, wherein the framework is based on the producer-consumer model, which combines read operations, Trim operations, The three write operations are decoupled to achieve asynchronous execution. Specifically, the processing process is as follows:

A producer-consumer model is established between the read operation and the Trim operation. The read thread acts as the producer and the Trim thread acts as the consumer. In order to ensure the order and correctness of the read data, the producer is set to one, and the consumer can Set to one or more. As mentioned above, the read thread in Ktrim is not only responsible for reading data but also for formatting data, which puts a lot of pressure on the read thread. Because the speed of constructing a data object when formatting data is slower than reading a string of a specified size from a file, the present invention transfers the formatting work to the consumer (that is, the Trim thread), effectively improving the productivity of the producer. (i.e. read thread) efficiency; at the same time, the present invention can balance the speed of consumers and consumers by increasing the number of Trim threads.

The solution of the present invention reads data in the form of chunks, the reading thread reads the data into the chunk object, and then stores the chunk object in the queue for use by the Trim thread, in order to reduce the creation of repeated chunk objects and the application of memory for them At the same time, the idea of data pool is adopted, and only chunk objects of fixed data are created for repeated use. Figure 1 shows the flow of chunk objects between the read thread and the Trim thread.

Secondly, a producer-consumer model is also established between the Trim thread and the writer thread. The Trim thread is the producer and the writer thread is the consumer. In order to ensure the correctness of the data written back, only one writer thread is used in the program. , because the speed of writing to disk is slow, the process of converting FASTQ object data back to string data in the program is placed in the Trim thread, and the relationship is shown in Figure 2.

To sum up, for the input and output process, the solution of the present invention has the following improvements compared to the existing Ktrim tool:

(1) Effectively solve the problem of low IO efficiency of Ktrim, and improve the input and output rate of file data to reach or approach the peak performance of disk read and write:

(2) Use the producer-consumer model to achieve decoupling, asynchrony, and speed balance among read threads, worker threads, and write threads;

(3) Transfer the task of data formatting from the reading thread to the working thread, and transfer the task of writing back data from the writing thread to the working thread to ensure the speed of the reading thread and the writing thread as much as possible;

(4) Use the data pool DataPool to reuse chunk objects, reduce the creation and destruction of objects, and reduce the overhead of creating objects;

(5) When transferring data between read threads, worker threads (ie Trim threads) and write threads, use data pointers as much as possible instead of data copying, which effectively reduces the overhead of memory copying;

(6) In the entire workflow, the read thread, each worker thread and the write thread are created only once, and are not destroyed until all their tasks are completed, reducing the overhead of thread creation.

(2) Improvement of Adapter query process

The core part of the Ktrim tool is to find the position of the possible Adapter, that is, the "seed" of the Adapter, which is a nucleotide sequence with a length of 3. The process of finding the "seed" can be abstracted as a sequence of tens to Find a substring of length 3 in hundreds of strings; the existing Ktrim uses the char * strstr(const char * haystack , const char * needle ) function in the <string.h> library to optimize the IO module Later, this simple method is inefficient and becomes a hot spot when the program is running.

，符号

表示取上整。其具体过程如下述伪代码所示：The solution of the present invention is based on the vectorization components in modern computers, and uses the vectorization method to find the positions of all "seed", specifically: each base in the sequenced sequence can be regarded as a character, we use AVX512 vector The register can process 64 bytes of data at one time, that is, 64 bases. Through the AVX512 vector register and instruction set, using multiple bit operations (three XOR operations, two left shift operations, and two OR operations) we can To get the position of "seed" in the sequence data of length 64, we divide the sequence data of length L into N groups, each of which contains 64 bases. The location of the "seed". in

,symbol

Indicates rounding up. The specific process is shown in the following pseudo code:

Algorithm 1 vectorized-find-seed

Input: seq sequencing sequence, index1 seed 1, index2 seed 2, all positions where seed1 seed 1 occurs, all positions where seed2 seed 2 occurs, seedtable position lookup table

Output: seed1 and seed2

1: function VECTORIZED-FIND-SEED(seq, index1, index2, seed1, seed2, seedtable)

2: len ← length(seq) ▷ sequence length

3: num ← ceil(len − 2/62) ▷The number of iterations needed

4: i ← 0

5: v11 ← init(index1[0]) ▷Initialize the avx512 register with a certain base of the "seed"

6: v12 ← init(index1[1])

7: v13 ← init(index1[2])

8: v21 ← init(index2[0])

9: v22 ← init(index2[1])

10: v23 ← init(index2[2])

11: whilei< num do

12: v1 ← load(seq + 62 ∗ i)▷Load the sequence data required for the i-th iteration into the avx512 register v1

13: res11 ← XOR(v1, v11) ▷ XOR operation

14: res21 ← XOR(v1, v21)

15: v1 ← leftShift(v1, 8) ▷v1 register data is shifted left by 8bit

16: res12 ← XOR(v1, v12)

17: res22 ← XOR(v1, v22)

18: res12 ← OR(res11, res12) ▷ OR the result of two XORs

19: res22 ← OR(res21, res22)

20: v1 ← leftShift(v1, 8)

21: res13 ← XOR(v1, v13)

22: res23 ← XOR(v1, v23)

23: res13 ← OR(res12, res13)

24: res23 ← OR(res22, res23)

25: res1 ← mask(res13) ▷Use mask to convert the result of OR operation to mask64

26: res2 ← mask(res23)

27: j ← 0

28: while j < 8 do

29: t1 ← res1 AND 255 ▷The value represented by the last 8 bits of mask64

30: t2 ← res2 AND 255

31: res1 ← rightShift(res1, 8) ▷mask64 right shift 8bit

32: res2 ← rightShift(res2, 8)

33: putsseedtable[i][j][t1] into seed1▷Use the table lookup method to query the "seed" position in the array seed-table

34: putsseedtable[i][j][t2] into seed2

35: j++

36: end while

37: i++

38: end while

39: end function

Among them, the division according to 62 instead of 64 is because we are looking for a base sequence of length 3, when the last two (63rd, 64th) bases of a 64-base sequence are the same as the first two of "seed". The bases are completely matched, but since the next base data cannot be read in, the final result is considered not to be the starting position of "seed", but in fact, if the next base happens to be the third position of "seed" If the bases match, this position is the position of the "seed". Therefore, when we divide, we need to consider the last two digits of the previous 64-base sequence, and the first two of the next 64-base sequence should be them. Those two bases have been read redundantly, which is quite For the first time, 64 bases were actually read. From the second time, only 62 new bases were read, so it was divided according to the size of 62 bases. L-2 was the first time. Read 64 bases, round up because L-2 is not exactly a multiple of 62. If the last remaining bases are less than 62, a new round is needed to complete.

Further, the use of the AVX512 vector register given above is for better explanation, and it can be understood that vector registers with other storage capacities may also be used in the specific implementation process.

Further, in order to prove the effectiveness of the solution of the present invention (as shown in Fig. 4 is a schematic diagram of its overall frame structure), the following experimental tests were carried out:

Among them, Low-Quality-Trim in Figure 4 represents low-quality base trimming work. According to the quality scores in the FASTQ format data, a fixed-size sliding window is used to scan from the 3' end to the 5' end of the sequencing data. When the average quality score of the bases in the sliding window is less than the user-specified threshold, the sliding window will be Move one base to the left and continue to judge until the average quality score of the bases in the sliding window meets the threshold, the sliding window stops sliding, and then the base sequence at the position of the sliding window and its back (near the 3' end) is clipped.

Adapter-Trim in Figure 4 represents the adapter in the trimming sequence, that is, the adapter used in the sequencing, and the adapter is essentially a nucleotide sequence. First, according to the first three bases of the adapter sequence specified by the user, use it as the "seed", find all the positions where the "seed" appears in the sequencing sequence, and use these positions as candidate positions. Then traverse the candidate positions in the sequence from left to right, compare the sequence at the candidate position with the adapter sequence specified by the user, calculate the Hamming edit distance, and use the first candidate position whose distance meets the user-specified threshold as the clipping position. The base sequence at the candidate position and its back (near the 3' end) is trimmed. This process uses a vectorization method, which speeds up the search for candidate positions and improves the processing efficiency of worker threads. The vectorization process uses the above algorithm 1.

In order to evaluate the correctness and performance of the RabbitTrim program, we conducted a series of experiments and analyses on different simulated data and real data. The specific experiments are as follows:

(1) Correctness test

In this embodiment, simulation data is used for correctness verification. We used simulation data generation software to generate single-end data and paired-end data at different sequencing error rates. After experimental testing, our RabbitTrim is consistent with the original Ktrim in correctness.

(2) Performance test

In this embodiment, a plurality of simulated data and real data are used to conduct performance testing experiments, and the running time and thread scalability of RabbitTrim, Ktirm, and Trimmomatic are recorded. Let's take a real data as an example. The data comes from GEO (GSE81178) in the NCBI (National Center for Biotechnology Information) database, using GSM2144218 ~ GSM2144224 and combining the file name suffix with read1.fastq into pooled.read1.fq, the file The suffix name read2.fastq is merged into pooled.read2.fq, and the size of each file is 19GB.

Table 1 Single-ended data runtime (seconds):

Number of threads Tool name 1 2 4 6 8 RabbitTrim 42.68 21.4 11.14 9.52 9.58 Ktrim 51.86 38.76 26.58 26.86 26.69 Trimmomatic 167.23 86.52 87.51 84.25 86.64

Table 2 Single-ended data speedup ratio:

Number of threads Tool name 1 2 4 6 8 RabbitTrim 1 1.99 3.83 4.49 4.46 Ktrim 1 1.34 1.95 1.93 1.94 Trimmomatic 1 1.93 1.91 1.98 1.93

Table 3 Double-ended data runtime (seconds):

Number of threads Tool name 1 2 4 6 8 RabbitTrim 62.88 32.77 22.69 22.73 22.51 Ktrim 96.04 68.56 69.05 66.81 69.32 Trimmomatic 364.38 149.72 124.76 124.6 123.19

Table 4 Double-ended data speedup ratio:

Number of threads Tool name 1 2 4 6 8 RabbitTrim 1 1.92 2.77 2.77 2.79 Ktrim 1 1.4 1.39 1.44 1.39 Trimmomatic 1 2.43 2.92 2.92 2.96

From the experimental results in Table 1 to Table 4, we can see that the performance of RabbitTrim has been greatly improved compared with Ktrim, the single-thread execution speed is improved, the thread scalability is better, and the optimal processing speed is about 3 times that of Ktrim. , about 2GB/sec (Ktrim is about 730MB/sec on the same platform), reaching the peak disk read/write performance (IOBound) of the test platform.

The solution of the present invention solves the problem of low IO efficiency by optimizing the input and output modules and the vectorization module, and improves the speed of finding an Adapter. After experimental tests, RabbitTrim is consistent with Ktrim in correctness, but the performance has been greatly improved, and the data processing speed can be close to the peak performance of disk read and write (~2GB/sec). On the same experimental platform, the peak performance of RabbitTrim for processing data is about 3 times that of the original Ktrim, which greatly improves the speed of next-generation sequencing data preprocessing, which is of great significance for speeding up downstream analysis tasks. Information workers provide a faster quality-adapter-trim tool.

Embodiment 2:

The purpose of this embodiment is to provide a rapid trimming system for biological sequencing sequences.

A rapid trimming system for biological sequencing sequences, including:

a data acquisition unit, which is used to acquire the biological sequencing sequence to be trimmed;

A data processing unit, which is used to perform read operation, trim operation and write operation on the biological sequencing sequence; wherein, the read operation, trim operation and write operation are decoupled based on the producer-consumer model to realize asynchronous execution ; and the formatting process of the biological sequencing sequence is transferred from the read operation to the trimming operation.

In further embodiments, there is also provided:

An electronic device includes a memory, a processor, and computer instructions stored on the memory and executed on the processor, and when the computer instructions are executed by the processor, the method described in the first embodiment is completed. For brevity, details are not repeated here.

It should be understood that, in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate array FPGA or other programmable logic devices , discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory may include read-only memory and random access memory and provide instructions and data to the processor, and a portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the method described in the first embodiment is completed.

The method in the first embodiment can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here.

Those of ordinary skill in the art can realize that the unit, that is, the algorithm step of each example described in conjunction with this embodiment, can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The method and system for rapid trimming of biological sequencing sequences provided by the above embodiments can be implemented and have broad application prospects.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. a kind of biological sequencing sequence fast trimming method, is characterized in that, comprises: Obtain the biological sequencing sequence to be trimmed; A read operation, a trim operation, and a write operation are performed on the biological sequencing sequence; wherein, the read operation, the trimming operation, and the writing operation are decoupled based on a producer-consumer model to realize asynchronous execution; and the biological sequencing sequence The formatting process is transferred from the read operation to the pruning operation; the reading operation, the pruning operation and the writing operation are implemented by independent threads, and the producer-consumer model is used to realize the relationship between the reading thread, the pruning thread and the writing thread. Decoupling, asynchronous and speed balance; The read operation is used to read the biological sequencing sequence in a block mode through a read thread, and store the read block objects in the first data queue; The trimming operation is used to obtain data from the first data queue through a trimming thread, format the biological sequencing sequence, and remove low-quality base sequences and linker sequences in the biological sequencing sequence; Stored in the second data queue; the formatting process specifically includes performing data analysis according to the FASTQ format; The writing operation is used for acquiring the processed biological sequencing sequence from the second data queue through a writing thread and storing it. 2. A kind of fast trimming method of biological sequencing sequence as claimed in claim 1, is characterized in that, described read operation, trimming operation and write operation adopt independent thread to realize respectively, wherein, read thread and write thread are all provided with A, pruning thread is set with one or more. 3 . The method for fast pruning of biological sequencing sequences according to claim 1 , wherein the creation of the block objects introduces the idea of a data pool, and only a preset number of block objects are created for repeated use. 4 . 4. The method for fast trimming of biological sequencing sequences according to claim 1, wherein obtaining an adapter sequence in the trimming thread comprises: treating each base in the biological sequencing sequence as a character; The vector register uses several bit operations to obtain the position of the linker sequence in the preset length sequence data. 5 . The method for fast pruning of biological sequencing sequences according to claim 1 , wherein the threads corresponding to the read operation, the pruning operation and the write operation are created only once, and are destroyed after the processing task is completed. 6 . 6. A system for rapid trimming of biological sequencing sequences, comprising: a data acquisition unit, which is used to acquire the biological sequencing sequence to be trimmed; a data processing unit, which is used for reading, trimming, and writing the biological sequencing sequence; Wherein, based on the producer-consumer model, the read operation, the trimming operation and the writing operation are decoupled to realize asynchronous execution; and the formatting process of the biological sequencing sequence is transferred from the read operation to the trimming operation; the Read operations, pruning operations, and write operations are implemented by independent threads, and the producer-consumer model is used to achieve decoupling, asynchrony, and speed balance among read threads, pruning threads, and write threads; The read operation is used to read the biological sequencing sequence in a block manner through a read thread, and store the read block objects in the first data queue; The trimming operation is used to obtain data from the first data queue through a trimming thread, format the biological sequencing sequence, and remove low-quality base sequences and linker sequences in the biological sequencing sequence; Stored in the second data queue; the formatting process specifically includes performing data analysis according to the FASTQ format; The writing operation is used to obtain the processed biological sequencing sequence from the second data queue through a writing thread and store it.

Images