Medical image processing and analyzing method, computer device, system and storage medium
Technical Field
The invention belongs to the technical field of medical image processing, and particularly relates to a medical image processing and analyzing method, computer equipment, a system and a storage medium.
Background
Deep learning is a new research direction in the field of machine learning, and is a general term for a class of pattern analysis methods. Deep learning is the intrinsic law and expression level of the learning sample data, and the information obtained in the learning process is very helpful for the interpretation of data such as characters, images and sounds. The final aim of the method is to enable the machine to have the analysis and learning capability like a human, and to recognize data such as characters, images and sounds.
Deep learning has been widely used in medical image analysis, for example, patents such as CN202011259621.3 and CN202011618907.6 implement computer-aided diagnosis by using deep learning technology. They train a neural network model using the labeled data and then analyze the medical image data using the trained neural network model to assist in diagnosis. Deep learning has enabled a variety of purposes in medical image processing, such as classification, detection, and segmentation. However, in the deep learning model training process, a large amount of labeled data is needed, and the labeling of medical data requires strong professional knowledge. Therefore, deep learning model training often requires a professional doctor to spend a lot of time performing the data annotation function, which results in high cost of deep learning for medical image processing.
Therefore, it is very important to reduce the labeling amount of the sample in the medical image analysis. Zhou, Z et al (Zhou, Z. et al, 2017, In Proceedings of the IEEE conference on computer vision and pattern recognition, 7340) tried to reduce the cost of deep learning technology In the medical field by adding an active learning method In deep learning to reduce the amount of sample labeling, and achieved certain effect. However, the method is only oriented to the deep learning technology of image classification, and cannot be applied to the field of image detection and segmentation. Meanwhile, their active learning method is based on the sample information amount only, and the sample selection and annotation result causes serious sampling deviation and information repetition. Yang, L et al (Yang, L. et al 2017, In International reference on a dimensional image computing and computer-assisted interaction, 399-. The method reduces the sample annotation amount to a certain extent, and reduces the application cost of the depth technology in the medical field. But the convolutional neural network trained based on different samples in the method can cause selection difference, so that the difference of results is not completely dependent on the candidate samples, and the selection result is not accurate.
In a word, due to the limitation of the model modeling process, a proper deep learning method is still lacked in the prior art, and the labeling quantity of samples can be reduced aiming at various medical image analysis tasks such as classification, detection and segmentation. This leads to high cost of deep learning in the field of medical image analysis, and affects large-scale popularization and application thereof.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a medical image processing and analyzing method, computer equipment, a system and a storage medium, aiming at reducing the marking amount of medical image samples in the deep learning model modeling process, thereby obviously reducing the workload of professional doctors and reducing the cost of deep learning applied to medical image analysis.
A medical image processing method, comprising the steps of:
step 1, training based on an initial labeled medical image sample to obtain an initial deep learning model;
step 2, inputting the candidate medical image sample into the deep learning model established in the previous step, completing the forward propagation process, and converting the prediction result into an entropy diagram based on Shannon's theorem;
step 3, based on the prediction result and the entropy diagram, evaluating the total information amount and the information repetition degree of the candidate medical image samples, and selecting a batch of candidate medical image samples with the highest total non-repeated information amount;
step 4, outputting the candidate medical image samples selected in the step 3 for labeling;
step 5, combining the candidate medical image samples selected in the step 3 with all the labeled medical image samples after labeling to form a new sample set of the labeled medical image samples, and training by using the new sample set of the labeled medical image samples to obtain an intermediate deep learning model;
and 6, repeating the steps 2-5 until all the candidate medical image samples or models are traversed to reach the target performance, and obtaining the final deep learning model.
Preferably, the medical image is a CT image, MRI, ultrasound image, X-ray image, pathology slice image or bone visualization image, and the MRI is T1WI, T2WI, DWI or DTI.
Preferably, in step 1, step 5 and step 6, the initial deep learning model, the intermediate deep learning model and the final deep learning model are classification models, detection models or segmentation models of medical images.
Preferably, in step 1, step 5 and step 6, the initial deep learning model, the intermediate deep learning model and the final deep learning model are detection models or segmentation models of medical images;
in step 3, after a batch of candidate medical image samples with the highest total amount of non-repetitive information are selected, the candidate medical image samples are further selected through the following steps:
step 3a, training based on all labeled medical image samples to obtain a strong classification model;
step 3b, inputting the selected candidate medical image sample into the strong classification model to obtain a classification prediction result;
and 3c, performing regional comparison on the classification prediction result obtained in the step 3b and the training result of the deep learning model established in the previous step in the step 2, and further selecting candidate medical image samples with inconsistent comparison results.
Preferably, in step 3, the algorithm for selecting the candidate medical image samples with the highest total quantity of non-repetitive information is a greedy algorithm.
Preferably, in step 5, in the process of training the intermediate deep learning model, the training round and the learning rate are adaptively adjusted.
The invention also provides a medical image analysis method for non-diagnostic purposes, comprising the following steps:
and inputting the medical image to be analyzed into a deep learning model, analyzing the deep learning model and outputting an analysis result, wherein the deep learning model is a final deep learning model obtained after training according to the medical image processing method.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor implements the above-described deep learning-based medical image processing method or medical image analysis method for non-diagnostic purposes.
The present invention also provides a system comprising:
means for acquisition and/or input and/or storage of medical images;
the computer device described above.
The present invention also provides a computer-readable storage medium having stored thereon a computer program for implementing the above-described deep learning-based medical image processing method or medical image analysis method for non-diagnostic purposes.
In the present invention, the "initial labeled medical image sample" refers to a partial sample selected from all samples that can be used for modeling but are not labeled, and the partial samples are labeled by a professional doctor and then used for training an initial deep learning model. The "candidate medical image sample" refers to a part which is not selected as the "initial labeled medical image sample" in all samples which can be used for modeling but are not labeled, and the processing method of the invention needs to select a sample with a value as a training basis from the "candidate medical image sample".
When the total amount of information and the information repetition degree of the candidate medical image samples are evaluated, the total objective is to evaluate the information repetition among a batch of high-entropy samples and then eliminate the excessive repetition. And repeated labeling and cost waste are avoided. The total amount of information was evaluated based on entropy theory and the variability (degree of information repetition) was evaluated based on KL divergence and cross-correlation.
The classification refers to the division of lesion degree, lesion type, organ and tissue type, the detection refers to the determination of the region of the organ, tissue or lesion in the image, and the segmentation refers to the pixel-level division of the region of the organ, tissue or lesion in the image.
The term "non-diagnostic purpose" means that the analysis result outputted by the analysis method of the present invention is not the diagnosis result or health condition of the patient, but an intermediate result which requires further diagnosis by a medical professional to obtain the diagnosis result or health condition of the patient.
The invention has the following beneficial effects:
1. the invention integrates active learning into deep learning, and utilizes the evaluation results of the entropy diagram and the prediction result diagram to select, thereby realizing the purpose of reducing the sample mark amount in the deep learning model modeling process of medical image analysis. The method can effectively reduce the workload of professional doctors, reduce the application cost of deep learning in the field of medical image analysis, and is favorable for popularization and application of the medical image analysis method based on deep learning.
2. The invention can be widely adapted to the deep learning technology in the medical field, can be applied to various medical images and various deep learning models, and has extremely strong generalization.
3. In a preferred scheme, the invention utilizes a classification model to assist and guide the modeling of a detection model or a segmentation model based on the relatively easy-to-achieve characteristics of the classification task. When the deep learning model is a detection model or a segmentation model, the invention further screens the selected candidate medical image samples through a high-precision model (a strong classification model is obtained based on training of all labeled medical image samples), so that the amount of the samples needing to be labeled can be further reduced.
4. In a preferred scheme, the method adaptively adjusts the training round and the learning rate in the deep learning model modeling. A slight overfitting is formed by adjusting the training round so that the loss function does not drop for consecutive K rounds (e.g., 5-15 rounds). Therefore, the information carried in the labeled medical image sample can be extracted as much as possible, and the comprehensive analysis process of selecting the sample in the active learning process in the next iteration is facilitated. Meanwhile, the learning rate is adjusted to oscillate and attenuate, so that the adverse effect of slight overfitting on deep learning model training is overcome. Through the self-adaptive adjustment process, the influence of the training sample size change on deep learning model training in the iterative process can be eliminated, so that active learning and deep learning are better combined, and the cost of deep learning application is better reduced while the problems of over-fitting and under-fitting are overcome.
Obviously, many modifications, substitutions, and variations are possible in light of the above teachings of the invention, without departing from the basic technical spirit of the invention, as defined by the following claims.
The present invention will be described in further detail with reference to the following examples. This should not be understood as limiting the scope of the above-described subject matter of the present invention to the following examples. All the technologies realized based on the above contents of the present invention belong to the scope of the present invention.
Drawings
Fig. 1 is a flow chart of a medical image processing method based on deep learning according to the present invention.
Detailed Description
It should be noted that, in the embodiment, the algorithm of the steps of data acquisition, transmission, storage, processing, etc. which are not specifically described, as well as the hardware structure, circuit connection, etc. which are not specifically described, can be implemented by the contents disclosed in the prior art.
Embodiment 1 deep learning-based medical image processing method
The method of this embodiment is shown in fig. 1, and includes the following steps:
step 1, training based on an initial labeled medical image sample to obtain an initial deep learning model;
step 2, inputting the candidate medical image sample into the deep learning model established in the previous step, completing the forward propagation process, and converting the prediction result into an entropy diagram based on Shannon's theorem;
and 3, evaluating the total information amount and the information repetition degree of the candidate medical image samples based on the entropy diagram and the prediction result, and selecting a batch of candidate medical image samples with the highest total non-repeated information amount. Based on the mode, the method can effectively avoid selecting the sample with large absolute information amount but excessive repeated information, and avoid inefficient sample annotation caused by serious selection deviation.
The entropy diagram is an information quantity matrix obtained by converting the classification probability predicted by the model according to the Shannon theorem. For example:
1. in the face of the medical image classification task, a certain tumor image is benign or malignant. The prediction result of the whole image is a one-dimensional probability vector, and the information quantity result obtained according to the Shannon theorem is only 1 numerical value, namely the information quantity of the whole image relative to the model.
2. In the face of the medical image segmentation task, a certain brain structure is segmented (in multiple regions). The prediction result of the whole image is a 4-dimensional tensor, namely [ H, W, D, C ], wherein the first 3 dimensions are voxel positions, the 4 th dimension represents the probability that the voxels at the positions belong to a certain structure (region), similarly, the information quantity calculation is carried out on the C dimension according to the Shannon theorem, the information quantity matrix is 3 dimensions [ H, W, D ], and the value at each voxel position expresses the information quantity of the voxel relative to the model.
The entropy diagram has the following functions as a basic tool: 1) the information amount of the whole image can be measured according to the entropy diagram, for example, the simplest cumulative sum is used. 2) The information quantity distribution can be shown according to the entropy diagram, the difference of the information quantities of different positions of the sample is shown, and the difference sample is convenient to select.
When the deep learning model is a detection model or a segmentation model, after the candidate medical image samples are selected by using the greedy algorithm, the candidate medical image samples can be further selected by the following steps:
and 3a, training based on all labeled medical image samples to obtain a high-precision model, wherein the high-precision model is a strong classification model. For example, when the task of the deep learning model is to segment the brain hippocampus, the high-precision model is the brain hippocampus classification model (i.e., the probability that the prediction input is brain hippocampus).
Step 3b, inputting the candidate medical image sample selected by the greedy algorithm into the strong classification model to obtain a classification prediction result;
and 3c, performing regional comparison on the classification prediction result obtained in the step 3b and the training result of the deep learning model established in the previous step in the step 2, and further selecting candidate medical image samples with inconsistent comparison results. For example, for the above-mentioned division task of the hippocampus of the brain, the candidate medical image sample is input to the depth learning model which is not trained, and the division result is obtained, and then the division result is input to the high-precision model. If the high-precision model also considers that the input part has a high probability of being the hippocampus of the brain, the fact that the deep learning model has learned the characteristics of the candidate medical image sample can be well segmented indicates that the labeling of the candidate medical image sample is not necessary. Conversely, if the probability of the high-precision classification model considering the input as the hippocampus of the brain is low, it indicates that the candidate medical image sample needs to be labeled for learning.
Step 4, outputting the candidate medical image samples selected in the step 3 for labeling;
and 5, combining the candidate medical image samples selected in the step 3 with all the labeled medical image samples after labeling to form a new sample set of the labeled medical image samples, and training by using the new sample set of the labeled medical image samples to obtain an intermediate deep learning model. In the process of training the intermediate deep learning model, in order to solve the problems of over-fitting and under-fitting, the training round and the learning rate are adaptively adjusted based on the dynamic change of the sample data volume.
The adaptive adjustment includes adjustment of training rounds and learning rates.
The adjustment to the training round aims at controlling the sample fitting degree. In this example, the training round was controlled so that the degree of fit of the samples was slightly overfit (the loss function did not decrease for consecutive 5-15 rounds). For example: when the hardware condition is fixed (tesla V10032G × 4), when the iteration (iteration) of the model proceeds to the 4 th round, a total of 200 samples that have been picked and labeled, the deep learning training parameter batch size is 16 (determined by hardware), and the total number of training rounds (epoch) is 125. When the model iteration proceeds to round 5, the number of samples that have been picked and labeled increases to 250, and the batch size remains the same, then the round of training needs to be adjusted to 157 times as the number of samples increases.
For the adjustment of the learning rate, the purpose is to eliminate the adverse effect of slight overfitting on the parameter adjustment in the next round of deep learning model training. For example: in this embodiment, a learning rate of oscillation attenuation is adopted, in the iteration of the round, the first epochs of the model will use a larger learning rate to adjust parameters, the loss function deviates from the minimum value region of the previous round, the learning rate is gradually attenuated along with the progress of the epochs, and the global minimum point position is newly found. For example: in one iteration, 0.0005 is used as the initial learning rate (epoch _ 0), and the rate gradually decreases according to a fixed formula. The learning rate of the last epoch is half of the initial learning rate (which has been slightly overfit). In the next iteration, the global optimum of the previous round becomes the local optimum at that time, since a new sample is added. The initial learning rate setting needs to be greater than 0.00025 (e.g., 0.0004) to help the model adjust the model parameters in larger steps from the local optimum, which is a region where the model parameters are difficult to fall out of if the learning rate is maintained at 0.00025. This one iteration eventually gradually decays to 0.0002 epoch. And so on in iteration that follows.
(6) And (5) repeating the steps (2) to (5) until all the candidate medical image samples are traversed to obtain the final deep learning model.
This embodiment has a very strong generalization, and the medical images can be various medical images in the prior art, such as: CT images, MRI (including T1WI, T2WI, DWI, DTI), ultrasound images, X-ray images, pathological section images, bone visualization images. The deep learning model may be a classification model, a detection model or a segmentation model of the medical image.
Embodiment 2 method and System for medical image analysis
The embodiment provides a method and a system for analyzing a medical image. The system of the embodiment is composed of two parts connected through a data line and a data interface: a server and a computer device. The server is used for storing data of the medical image. Furthermore, as an equivalent alternative, the server may also be replaced by a device for acquiring medical images (e.g. a CT machine, an ultrasound imager, etc.) or a device for inputting medical image data.
The computer device is used for analyzing the medical image according to the following method:
inputting a medical image to be analyzed into a deep learning model, analyzing the medical image by the deep learning model, and outputting an analysis result, wherein the deep learning model is a final deep learning model obtained after training according to the method of the embodiment 1.
To illustrate further the beneficial effects of the present invention, the following experiments were performed:
experimental example 1
Data set: the Breakhis dataset contains microscopic biopsy images of benign and malignant breast tumors, obtained using magnification factors of 40X, 100X, 200X, 400X in a 3-channel RGB (red-green-blue) true color (24 bit color depth, 8 bit per color channel) color space, corresponding to objective 4X, 10X, 20X, 40X. The database consisted of 7909 images (2480 for each scale of benign tumor image and 5429 for malignant tumor) according to 3: the scale of 1 is randomly divided into a training set and a test set.
Deep learning model: the deep learning model of this experimental example is a classification model, specifically a multiscale benign and malignant breast tumor lesion classification network based on Res-net 50.
The experimental results are as follows: according to the method in the prior art, a full data set is adopted to train a deep learning model, and the classification accuracy of the trained model is 95%. After the active learning was added, the baseline effect was achieved using an average of 44.25% of the sample size (95% of the classification accuracy was used as baseline effect, 4 replicates) according to the method of example 1.
Experimental example 2
Data set: the data was derived from the ABCD neurocognitive prediction challenge (ABCD NP challenge 2019) for a total of 3736 annotated MRI (T1 WI) brain images with 1 channel image and 240x240x240 image resolution. 1000 of them were randomly selected for the experiment and tested according to 1: the scale of 1 is randomly divided into a training set and a test set.
Deep learning model: the deep learning model of the experimental example is a segmentation model, in particular to an MRI hippocampus segmentation model based on 3D U-net.
The experimental results are as follows: according to the method in the prior art, a full data set is adopted to train a deep learning model, and the segmentation dice index is 0.85. After active learning is added, the baseline effect can be achieved by using 67% of the sample amount on average (the baseline effect is achieved by using the segmentation dice index of 0.85, and the experiment is repeated for 4 times) according to the method of example 1.
It can be seen from the above embodiments that the present invention realizes the effect of reducing the amount of labeling on medical images by embedding the deep learning model into the active learning process. The method can effectively reduce the workload of professional doctors, reduce the use cost of the deep learning model, and is beneficial to large-scale popularization and application. This makes the present invention have high application value.