CN118537853A - Improved YOLOv-based leukocyte classification method - Google Patents

Abstract

The present invention belongs to the technical field of medical microscopic image classification, and in particular, proposes a five-classification method of white blood cells based on improved YOLOv7, and realizes the recognition of blood cell images through deep learning technology. The present invention uses the YOLOv7 network as the main structure, replaces the four parallel convolutional layers of the first and fourth ELAN-1 modules in the Backbone part with GhostV2 modules, reduces the amount of parameters and generates more feature maps, combines the SimAM attention mechanism and MHSA to improve the ELAN-2 module in the Head part, generates more feature maps without increasing the amount of parameters, makes the algorithm pay more attention to the granularity characteristics of the white blood cell surface, and uses different improvement methods for the ELAN modules at different positions in the network to achieve the optimization of the feature extraction ability of the YOLOv7 network. The present invention uses the improved Focal Loss to replace the classification loss in the original network structure, increases the weight of the difficult-to-distinguish samples in the loss function, makes the loss function tend to the difficult-to-distinguish samples, and solves the problem of imbalanced white blood cell image categories, and improves the accuracy of white blood cell classification.

Description

A white blood cell classification method based on improved YOLOv7

Technical Field

The present invention relates to the technical field of medical microscopic image classification, and in particular to a leukocyte classification method based on improved YOLOv7.

Background Art

White blood cells are an important component of blood and are also the main immune cells of the human body. They play a pivotal role in maintaining immunity and phagocytizing pathogens and bacteria. Therefore, accurate identification of different types of white blood cells is of great significance for the auxiliary diagnosis of corresponding diseases. The correct classification of white blood cells is also a key step in the automatic analysis of cell images.

However, there are still some problems with the current classification and recognition technology for white blood cells. Traditional image processing methods often require manual design of features and rules, and it is difficult to accurately classify white blood cells of different scales and shapes. At the same time, traditional deep learning models often have problems with slow detection speed, low accuracy and poor generalization ability in white blood cell classification tasks. Therefore, it is necessary to develop a new white blood cell classification technology to improve the accuracy and efficiency of classification.

YOLOv7 is a deep learning model for object detection, which has many advantages over other networks. First, YOLOv7 has faster detection speed and higher accuracy. It uses a new detection method, which can greatly improve the detection speed without losing accuracy, which is of great significance for large-scale white blood cell classification. Secondly, YOLOv7 has better generalization ability and can adapt to white blood cells of different scales and shapes, thereby improving the accuracy and robustness of classification. In addition, YOLOv7 also has better scalability and flexibility, and can be easily applied to different white blood cell classification tasks.

Based on the above background technology, we proposed a patented technology for white blood cell classification based on YOLOv7. This technology uses the fast detection and high accuracy of YOLOv7, combined with the advantages of deep learning, to achieve fast and accurate classification of white blood cells of different scales and shapes. At the same time, this technology also has good generalization ability and flexibility, and can adapt to different white blood cell classification tasks. Through this technology, the accuracy and efficiency of white blood cell classification can be effectively improved, providing better support for clinical diagnosis and treatment.

Summary of the invention

1. Technical issues to be solved

In view of the shortcomings of the prior art, the present invention provides a white blood cell classification method based on an improved YOLOv7, which solves the problems of complex background, unbalanced data, and large similarity between classes of white blood cell images, reduces the number of parameters of the YOLOv7 network model, and improves the classification accuracy, which will provide objective and effective reference opinions in medical pathology diagnosis.

(II) Technical solution

In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

A white blood cell classification method based on improved YOLOv7, comprising the following steps:

S1. Collect multiple public white blood cell datasets and perform data enhancement and denoising operations on the datasets. Data enhancement uses rotation of 90°, 180°, 270° and flipping methods to increase the number of white blood cell images in the dataset to five times the original number. Noise reduction uses a combination of nonlinear filtering methods, median filtering and bilateral filtering, which can better protect image edge information while suppressing image noise.

S2. Annotate the white blood cell image obtained in step S2 using the VOC format and the Labelimg tool.

S3. Randomly divide the labeled data set into training set, test set and validation set according to the proportion, and create three folders train, test and val to place them respectively in preparation for subsequent network training and prediction.

S4. Build the basic model of YOLOv7 network structure to improve the YOLOv7 network structure.

S5. Set specific parameters of the improved YOLOv7 network structure, input image size img_size = 640 × 640, training batch size batch_size = 4, training iteration number Epoch = 100, learning rate lr = 0.0001, decay decay = 0.00001, label smoothing label-smoothing = 0.1, and use adam optimizer as optimizer. Put the improved YOLOv7 network structure with set parameters into a computer with a configured environment and use it to train the white blood cell images in the train folder generated in step S3.

S6. During training, as the number of iterations increases, the training loss and test loss slowly converge to a stable value, the difference between the two does not change, and the accuracy tends to be stable, indicating that the expected training effect has been achieved. At the same time, the classification results of the test set will also be saved in the form of pictures in the run folder. The folder is divided into two folders, train and test. After the network training is completed, the training results and weights will be output under the train folder. At this time, we select the optimal weight in the training process to test the test set to obtain the optimal test result, and finally save the result to the test folder under the run folder.

S7. Analyze the prediction results, count the misclassified results, and generate the corresponding confusion matrix heat map to facilitate the subsequent calculation of evaluation indicators.

S8. In order to verify the effectiveness of the white blood cell classification and recognition model more fairly and objectively, four indicators, namely accuracy, precision, recall and F1-score, are introduced to evaluate the model.

(III) Beneficial effects

Compared with the prior art, the present invention provides a white blood cell classification method based on an improved YOLOv7, which has the following beneficial effects:

1. The present invention replaces the four 3×3 convolution kernels in the ELAN-1 structure of the Backbone part with a lightweight GhostV2 module, thereby reducing the computational complexity and generating more feature maps, thereby reducing the computational time and improving the classification accuracy.

2. The present invention integrates SimAM and MHSA attention into the ELAN-2 structure of the Head part, so that the network pays more attention to the granular features of the cell image surface during the classification process, which significantly helps the network to distinguish between granular cells and non-granular cells.

3. The present invention adopts the improved Focal Loss to replace the cls Loss in the original YOLOv7, dynamically reduces the weight of easy-to-distinguish samples during training, and quickly focuses on those difficult-to-distinguish samples, which is equivalent to increasing the weight of difficult-to-distinguish samples in the loss function, making the loss function tend to difficult-to-distinguish samples, and at the same time solves the problem of imbalanced white blood cell image categories.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a flow chart of the method of the present invention;

Figure 2 is a diagram of the improved YOLOv7 network structure;

Figure 3 shows the five types of white blood cell images in the dataset;

Figure 4 shows some of the classification results.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example

As shown in FIGS. 1-4 , a method for classifying white blood cells based on an improved YOLOv7 network according to an embodiment of the present invention includes the following steps:

S1. Collect multiple public white blood cell datasets and perform data enhancement and denoising operations on the datasets. Data enhancement uses rotation of 90°, 180°, 270° and flipping methods to increase the number of white blood cell images in the dataset to five times the original number. Noise reduction uses a combination of nonlinear filtering methods, median filtering and bilateral filtering, which can better protect image edge information while suppressing image noise.

S2. Use the Labelimg tool to annotate the white blood cell image obtained in step S2 as neutrophils, eosinophils, basophils, monocytes and lymphocytes respectively. The VOC format is used for saving. A new folder Annotations is generated under the folder where the white blood cell image is saved to save the generated xml file.

S3. Randomly divide the labeled data set into training set, test set and validation set in the ratio of 7:1:2, and create three folders train, test and val to place them respectively in preparation for subsequent network training and prediction.

S4. Build the basic model of YOLOv7 network structure to improve the YOLOv7 network structure. The specific steps are as follows:

S4-1. First, the ELAN-1 module is improved. Four convolutions with a size of 3×3 in ELAN-1 are replaced by GhostV2 (Long-distance attention mechanism to enhance cheap operations). The original GhostV1 is composed of two stacked Ghost modules. This Ghost Block adopts a reverse bottleneck structure. The first Ghost module acts as an expansion layer to increase the number of output channels, and the second Ghost module reduces the number of channels to match the shortcut path. This structure helps to improve the abstraction ability and representation quality of features. GhostNet is a lightweight convolutional neural network, which is designed to generate more feature maps with fewer parameters. It divides the output channel into two identical parts: one part is a regular convolution, which is a normal implementation without any difference. The other part is a cheap operation, which is no longer implemented by regular convolution, but is generated by a simple linear transformation. Finally, the results of the two parts are spliced together to get the final output. GhostV2 is an improvement on GhostV1. The DFC attention branch is paralleled with the first Ghost module to enhance the extended features. The enhanced features are then sent to the second Ghost module to generate output features to capture the long-range dependencies between pixels in different spatial positions, enhancing the model's expressiveness and detail information processing capabilities. The improved ELAN-1 module replaces the first and fourth ELAN-1 modules in the Backbone part of the original YOLOv7 structure, thereby enhancing the feature extraction capabilities of the YOLOv7 network structure.

S4-2.SimAM attention mechanism is a new type of attention mechanism derived from the existing channel or spatial attention mechanism. This attention mechanism does not need to add additional parameters to derive 3D attention weights for feature maps. Most of its operations are based on the defined energy function selection, thus avoiding excessive structural adjustments. The energy function formula is as follows:

in t and _xi refer to the target neuron and other neurons of the input feature X, respectively. i refers to the index in the spatial dimension. M = H × W refers to the number of all neurons in a certain channel. _wt and _bt refer to the "weight" and "bias" of a certain neuron when it is transformed, respectively. Introducing binary labels to replace _yt and _y0 and solving the minimization of _et is equivalent to finding the linear separability of the target neuron and other neurons. The specific steps are as follows:

S4-2-1. Input feature map: SimAM will receive the input feature map as input. Assume that the size of the input feature map is H×W×C, where H and W represent the height and width of the feature map, and C represents the number of channels.

S4-2-2. Reshaping of feature maps: First, SimAM reshapes the input feature map and converts it into a two-dimensional feature matrix of size N×C, where N=H×W represents the total number of pixels in the feature map. This is done to convert spatial information into channel dimensions for subsequent attention calculations.

S4-2-3. Calculate channel attention weights: Next, SimAM calculates the attention weights of each channel. A simple calculation method is used here, which is to normalize the feature vector of each channel and then use it as the attention weight of the channel. This can make the weights of important channels higher without the need for additional parameters to learn.

S4-2-4. Apply attention weights: After calculating the attention weights of the channels, SimAM applies these weights to each channel of the input feature map. Specifically, for each channel, its corresponding attention weight is multiplied by the feature value in the channel to enhance the feature representation of important channels.

S4-2-5. Reconstruct feature map: Finally, the attention-adjusted feature map is reassembled into the original three-dimensional feature map form so that it can be passed to the subsequent network layers for feature extraction and learning.

S4-3.MHSA is a further improvement on the attention mechanism and self-attention mechanism. It can focus on different positions and features at the same time, improving the representation ability of features. In addition, the attention calculation is divided into multiple heads for parallel calculation, which also improves the calculation efficiency of the model. The specific steps are as follows:

S4-3-1. Input sequence: MHSA receives an input sequence, usually expressed as X = {x ₁ , x ₂ , x ₃ ... x _n }, where n represents the length of the sequence, and _xi represents the feature representation of the i-th element in the sequence.

S4-3-2. Generate query, key, value: For each element x _i in the input sequence, MHSA generates three sets of vectors through linear transformation: query vector Q _i , key vector K _i and value vector V _i . These vectors will be used to calculate attention weights and generate outputs.

S4-3-3. Calculate attention weights: For each query vector _Qi , MHSA calculates its similarity with all key vectors _Kj , usually using dot product or other similarity calculation methods. These similarities are then converted into attention weights through the Softmax function, indicating the degree of association between the query vector and different positions.

S4-3-4. Weighted summation: Using the calculated attention weights, MHSA performs weighted summation on the value vector _Vj to obtain the output representation of the query vector _Qi . This allows important information to receive more attention, while unimportant information is suppressed.

S4-3-5. Multi-head mechanism: MHSA usually uses multiple heads to calculate attention in parallel, and each head independently learns different query, key, and value mapping relationships. Finally, the outputs of multiple heads are concatenated or weighted summed to obtain richer and more complex representation capabilities.

S4-3-6. Residual connection and layer normalization: In each MHSA module, residual connection and layer normalization are usually used to stabilize the training process, making it easier for the model to learn effective representations. This can avoid problems such as gradient disappearance or explosion.

After the present invention integrates SimAM and MHSA into CAT in the ELAN-2 module, the two enable the YOLOv7 network to more accurately and efficiently capture and characterize the features of particles on the image surface, thereby improving the accuracy of white blood cell classification.

S4-4. Further, we use Focal Loss instead of CLS Loss, which is a dynamic scaling cross entropy loss function based on binary cross entropy CE. By introducing a dynamic scaling factor, the weight of easily distinguishable samples can be reduced during training, thereby quickly focusing on those difficult-to-distinguish samples. The following is its specific form:

make

Unify the Focal loss expression into one expression:

L _fl =-(1-p _t ) ^γ log(p _t )

It introduces a decay factor _pt , which can amplify the loss when a sample is misclassified, so that the difficult-to-classify samples occupy a larger weight in the loss function, thereby guiding the model to focus more on difficult-to-classify samples.

To solve the problem of class imbalance, I add a weight factor β _t ∈[0,1] to the formula:

Then the final balanced loss function is:

L′ _fl =-β _t (1-p _t ) ^γ tog(p _t )

S5. Set specific parameters of the improved YOLOv7 network structure, input image size imgsize = 640 × 640, training batch size batch_size = 4, training iteration number Epoch = 100, learning rate lr = 0.0001, decay decay = 0.00001, label smoothing label-smoothing = 0.1, and use adam optimizer as optimizer. Put the improved YOLOv7 network structure with set parameters into a computer with a configured environment and use it to train the white blood cell images in the train folder generated in step S3.

S6. During training, as the number of iterations increases, the training loss and test loss slowly converge to a stable value, the difference between the two does not change, and the accuracy tends to be stable, indicating that the expected training effect has been achieved. At the same time, the classification results of the test set will also be saved in the form of pictures in the run folder. The folder is divided into two folders, train and test. After the network training is completed, the training results and weights will be output under the train folder. At this time, we select the optimal weight in the training process to test the test set to obtain the optimal test result, and finally save the result to the test folder under the run folder.

S7. Analyze the prediction results, count the misclassified results, and generate the corresponding confusion matrix heat map to facilitate the subsequent calculation of evaluation indicators.

S8. In order to verify the effectiveness of the white blood cell classification and recognition model more fairly and objectively, four indicators, namely accuracy, precision, recall and F1-score, are introduced to evaluate the model.

Precision refers to the proportion of all the results predicted by the model as positive samples that are accurately predicted. The calculation formula is as follows:

Recall refers to the proportion of data that are correctly predicted among the data whose true values are positive samples. The calculation formula is as follows:

The F1-Score indicator combines the results of precision and recall. Its value range is from 0 to 1. If the value is 1, it means the model has the best output, and if the value is 0, it means the model has the worst output. Its calculation formula is as follows:

Accuracy is an indicator used to evaluate the performance of a classification model, indicating the proportion of correct classifications among all predictions. Accuracy is usually expressed as a percentage, and the higher the percentage, the better the classification performance of the model. The calculation formula is as follows:

Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (5)

1. A method of leukocyte classification based on an improved YOLOv network, comprising the steps of:

S1, collecting a leukocyte data set, and preprocessing the collected leukocyte microscopic image, wherein the preprocessing comprises noise reduction and data enhancement;

s2, labeling the images treated by the S1 into types of Neutrophils (NEU), eosinophils (EOS), basophils (BAS), monocytes (MON) and Lymphocytes (LYM) respectively;

S3, randomly mixing the white blood cell microscopic image data set processed in the step S2 according to the proportion of 7:2:1 is divided into a training set, a testing set and a verification set;

s4, improving the original YOLOv network structure.

S5, placing the training set, the test set and the verification set after the S3 is divided under the corresponding folders, operating the network to train the improved network structure, classifying the test set, and storing the final result under the run folders.

S6, analyzing the final data result, and respectively adopting Accuracy (Accuracy), precision (Precision), recall rate (Recall) and F1-score as evaluation indexes.

2. A method of leukocyte classification based on an improved YOLOv network according to claim 1, wherein: in order to eliminate the influence of noise as much as possible under the condition of maintaining the characteristics of useful information of the image, the filtering function adopted in the step S1 adopts a combined filtering mode of firstly carrying out median filtering and then carrying out bilateral filtering aiming at the leucocyte image, and the method comprises the following steps:

The median filter calculation formula:

Where k=1, 2,3, …, M is the median of the data in the observation window, x ₁,x₂,x₃…x_n is the gray value of the pixels in the neighborhood in the original image, and they are ordered by size as x _i1≤x_i2≤x_i3…≤x_in.

Bilateral filtering calculation formula:

Where i, j is the coordinate point of the current convolved pixel, k, l is the coordinate point of the neighboring pixel, w is the weighting coefficient, and is determined by the domain kernel and the value domain kernel, and is the product of these.

3. A method of leukocyte classification based on an improved YOLOv network according to claim 1, wherein: the concrete improvement steps of YOLOv network structure in step S4 are as follows:

S3-1, sequentially replacing four parallel convolution layers in a first ELAN-1 module and a fourth ELAN-1 module in a back bone part with GhostV modules; the GhostV module is based on the original GhostV1, DFC attention is added, and a simpler and easier-to-implement fully-connected layer with fixed weight is adopted to generate an attention attempt with a global receptive field.

S3-2, incorporating SimAM attention mechanisms and MHSA in the ELAN-2 module in the Head section; simAM the attention mechanism does not need to add extra parameters to the original network to deduce the attention weight for the feature map, most of operations are selected based on the defined energy function, excessive structural adjustment is avoided, and meanwhile, whether particles exist on the surface of the white blood cells or not is more concerned. MHSA is a further improvement on the attention mechanism and the self-attention mechanism, and can focus on different positions and features at the same time, so that the representation capability of the features is improved.

S3-3, in order to make the model pay more attention to the leukocyte samples which are difficult to classify so as to improve the classifying effect and simultaneously relieve the problem of unbalance of the leukocyte image categories, the cls Loss in the original Loss function is replaced by the improved Focal Loss.

4. A method of leukocyte classification based on an improved YOLOv network according to claim 1, wherein: the training set, test set and verification set of the data set in step S5 in claim 1 are placed under the designated folders of the network, the test set and verification set are transmitted to the network for training and prediction, the result is stored under the run folder, the folders are divided into a train folder and a test folder, the training result and the weight are output under the train folder after the network training is completed, at this time, the optimal weight in the training process is selected to test the test set to obtain the optimal test result, and finally the result is stored under the test folder under the run folder.

5. A method of leukocyte classification based on an improved YOLOv network according to claim 1, wherein: the improved YOLOv network structure also comprises accurate and rich evaluation indexes at the end of prediction to illustrate the advantages of the improved YOLOv network structure in the field of white blood cell classification, including Accuracy (Accuracy), precision (Precision), recall (Recall) and F1-score.