CN109584254B - Heart left ventricle segmentation method based on deep full convolution neural network - Google Patents

Abstract

The invention discloses a heart left ventricle segmentation method based on a deep fully convolutional neural network. The method introduces the idea of deep learning into the left ventricle segmentation of cardiac magnetic resonance short-axis images. The process is mainly divided into two parts: training and prediction. Stage: In the training stage, the preprocessed 128×128 cardiac magnetic resonance image is used as input, and the manually processed label is used as the label of the network to calculate the error. As the number of training iterations increases, the training set error and verification The set error gradually decreases; in the test phase, the data in the test set is input into the trained model, and finally the network outputs a prediction for each pixel to generate a segmentation result. The invention realizes the segmentation of cardiac magnetic resonance short-axis images from the perspective of data drive, effectively solves the time-consuming and labor-intensive problem of manually drawing contours, can overcome the shortcomings of traditional image segmentation algorithms, and realizes high-precision and high-robust left ventricle segmentation.

Description

A Heart Left Ventricle Segmentation Method Based on Deep Fully Convolutional Neural Network

technical field

The invention belongs to the technical field of medical image analysis, and in particular relates to a heart left ventricle segmentation method based on a deep fully convolutional neural network.

Background technique

In recent years, cardiovascular disease has become one of the number one killers of human life and health. With the improvement of people's living standards and the rapid development of modern medicine, early diagnosis and risk assessment of cardiovascular diseases have become important conditions for improving human living standards. In addition, with the continuous advancement of medical technology, imaging equipment capable of dynamic imaging of the heart mainly includes Magnetic Resonance Imaging (MRI), Computed Tomography (X-Ray Computer Tomography, CT) and Ultrasonic Imaging (Ultrasonic Imaging, US )Wait. Cardiac magnetic resonance imaging has good soft tissue contrast, no radioactivity, no need to inject or take tracers, and the ability to image in any plane.

However, to quantitatively analyze the overall and local functions of the heart, including clinical parameters such as ventricular volume, ejection fraction, myocardial mass, etc., also needs to rely on accurate left ventricle (LV) and right ventricle (RV) endocardium and cardiac function in short-axis images. Outline of adventitia. If these contours had to be drawn by hand, it would be a time-consuming and tedious task, and would be prone to high intra-observer and inter-observer variability. Therefore, there is a need for a fast, accurate, reproducible, and fully automated cardiac segmentation method to aid in the diagnosis of cardiovascular diseases.

For magnetic resonance cardiac images, the gray value of myocardial tissue is very close to the gray value of surrounding tissue, which brings challenges to the segmentation of the left ventricle of the heart. At present, the common segmentation algorithms in China include level set segmentation algorithm and region growing segmentation algorithm , Threshold segmentation algorithm, however, the accuracy of these segmentation algorithms is still not high, and the robustness is not strong. In recent years, with the improvement of hardware level and technology, image segmentation algorithms based on deep learning have surpassed traditional image segmentation algorithms in many fields. These include FCN architecture (Tran P V.A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI[J].2016.) and UNet architecture (Ronneberger O, FischerP, Brox T.U-Net: Convolutional Networks for Biomedical ImageSegmentation[J].2015 .) for left ventricle segmentation; the general idea behind FCN is to use the downsampling path to learn relevant features at various spatial scales, and then use the upsampling path to combine the features predicted at the pixel level, however when heart slices are smaller due to their The network does not overcome this segmentation difficulty when the size is at the apex or end-systole of the heart, since fine object information may be lost during the downscaling process at the maximization layers in the network. UNet is one of the commonly used segmentation models in medical imaging. The network obtains the segmentation map through multiple deconvolutions. In addition, the convolution layer information of the corresponding size at the front end of the network is added when the deconvolution is sampled, so that more details can be obtained. It is preserved, but the network does not pay more attention to the pixels of the segmentation boundary, and there will also be a problem of low segmentation accuracy at the apex or end-systole of the heart.

Contents of the invention

In view of the above, the present invention proposes a heart left ventricle segmentation method based on a deep fully convolutional neural network. The segmentation model used takes the full image as input and manual segmentation as a label. The network can be effectively trained end-to-end, and finally Predict each pixel of the image to realize the segmentation of the left and right ventricles, so as to overcome the shortcomings of traditional image segmentation algorithms and achieve high-precision and high-robust left ventricle segmentation.

A heart left ventricle segmentation method based on a deep fully convolutional neural network, comprising the following steps:

(1) Obtain the cardiac magnetic resonance short-axis image of the subject and manually mark the contour line of the left ventricle in the image, and construct a binary segmentation image with the same size as the cardiac magnetic resonance short-axis image. In the value segmentation image, the contour of the left ventricle of the heart and its internal pixel values are all 1, and the external pixel values of the contour are all 0;

(2) A large number of samples are obtained by using the method of step (1) for different subjects, each sample includes the short-axis cardiac magnetic resonance image of the subject and its corresponding binary segmentation image; all samples are divided into training set, validation set and test set;

(3) Use the cardiac magnetic resonance short-axis image in the training set sample as the input of the fully convolutional neural network, and the binary segmentation image as the true value label output by the fully convolutional neural network, and then train the neural network, and finally train After completion, the segmentation model of the left ventricle of the heart is obtained;

(4) Input the cardiac magnetic resonance short-axis image in the test set sample into the heart left ventricle segmentation model, and then obtain the binary segmentation image about the outline of the heart left ventricle, and then compare the binary segmentation image with the test set sample Binary segmented images for comparison.

Further, in the step (1), the cardiac magnetic resonance short-axis image of the subject is obtained, that is, the subject's heart is simultaneously positioned and imaged in three directions of coronal, sagittal, and axial by the magnetic resonance instrument, and the imaging range is from the bottom of the heart to From the root of the great vessels to the apex, from which short-axis cardiac magnetic resonance images are screened.

Further, the specific operation process of constructing the binary segmentation image in the step (1) is as follows: for the manually marked contour line of the left ventricle of the heart, it is expressed as a marked map recognizable by the neural network, that is, a map with the magnetic field of the heart A binary segmentation image of the same size as the resonance short-axis image, in which the pixels belonging to the contour line and its interior are the target class and the pixel values are all 1, and the pixels outside the contour line are the background class and the pixel values are all 0; the The ratio of the number of samples in the training set, validation set, and test set is 1:1:1.

Further, the fully convolutional neural network in the step (3) includes a residual network Resnet 50, four prediction layers P1-P4 and four deconvolution layers D1-D4, and the residual network Resnet 50 is obtained from The input to the output is sequentially composed of a convolutional layer C, a pooling layer and four residual stages L1~L4. The input of the convolutional layer C is the input of the entire neural network, and the residual stage L1 and residual Stage L4 is composed of 3 residual structures cascaded, residual stage L2 is composed of 4 residual structures, and residual stage L3 is composed of 6 residual structures. The structure is composed of three convolutional layers C1~C3 cascaded, where the input of the convolutional layer C1 and the output of the convolutional layer C3 are superimposed as the output of the residual structure; the output of the residual stage L4 and the input of the prediction layer P1 Connected, the output of the prediction layer P1 is connected to the input of the deconvolution layer D1, the output of the residual stage L3 is connected to the input of the prediction layer P2, and the output of the prediction layer P2 is superimposed with the output of the deconvolution layer D1 as a deconvolution layer The input of D2, the output of the residual stage L2 is connected to the input of the prediction layer P3, the output of the prediction layer P3 and the output of the deconvolution layer D2 are superimposed as the input of the deconvolution layer D3, the output of the residual stage L1 and the prediction The input of the layer P4 is connected, and the output of the prediction layer P4 is superimposed with the output of the deconvolution layer D3 as the input of the deconvolution layer D4, and the output of the deconvolution layer D4 is the output of the entire neural network; the prediction layer P1 ~P4 all include a 3×3 convolution kernel, the step size is 1, the padding is 1, and the activation function is Relu; the deconvolution layers D1~D4 are all equipped with convolution kernels to double the input dimension.

Further, the process of training the fully convolutional neural network in the step (3) is as follows: first, input the cardiac magnetic resonance short-axis images in the training set samples into the fully convolutional neural network one by one, and calculate the fully convolutional neural network. The loss function L between each output result of the network and the corresponding true value label, with the goal of minimizing the loss function L, continuously optimizes the parameters in the full convolutional neural network through the back propagation method, and finally establishes the full volume after the training is completed. The product neural network is the heart left ventricle segmentation model.

Further, the expression of the loss function L is as follows:

Among them: W is the width of the binary segmentation image, H is the height of the binary segmentation image, N is the number of categories and N=2, P _{(i, j, n)} is the jth row of the fully convolutional neural network output image The probability that the i-column pixel is the category n, G _(i,j,n) is the pixel value of the j-th row and the i-column in the binary segmentation image corresponding to the training set sample, category 1 indicates that the pixel is the left ventricle area of the heart, category 2 indicates Pixels are the background area.

Preferably, in the step (3), for the cardiac left ventricle segmentation model obtained after the training is completed, it is verified by using the verification set samples, and the model parameters are fine-tuned through the verification, so as to further improve the segmentation accuracy of the model.

The present invention introduces the idea of deep learning into left ventricle segmentation of cardiac magnetic resonance short-axis images, and the process is mainly divided into two stages of training and prediction: in the training stage, the preprocessed cardiac magnetic resonance image of size 128×128 is used as Input, the manually processed labels are used as the labels of the network to calculate the error. As the number of training iterations increases, the training set error and verification set error gradually decrease; in the test phase, the data in the test set is input to the trained model In , the final network outputs predictions for each pixel, generating segmentation results. The present invention realizes the segmentation of cardiac magnetic resonance short-axis images from the perspective of data drive, effectively solves the time-consuming and laborious problem of manually drawing contours, can overcome the shortcomings of traditional image segmentation algorithms, and realizes high-precision and high-robust left ventricle segmentation.

Description of drawings

Fig. 1 is a schematic structural diagram of a deep fully convolutional neural network of the present invention.

Figure 2 is a cardiac magnetic resonance short-axis cine sequence image.

Figure 3 is a sequence of labeled images of the outline of the heart's left ventricle.

Fig. 4 is a schematic diagram of the comparison between the segmentation results of the present invention and the existing segmentation models FCN and UNet on the test set.

detailed description

In order to describe the present invention more clearly, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

The present invention is based on the heart left ventricle segmentation method of deep fully convolutional neural network, and specific implementation steps are as follows:

S1. Obtain the complete cardiac magnetic resonance image of the subject, as well as the corresponding manually drawn left ventricular endocardial contour line. The short-axis image of the complete cardiac MRI of the subject only includes ordinary cine sequence images, as shown in Figure 2, that is, the coronal, sagittal, and axial positioning maps of the subject are simultaneously made by the magnetic resonance instrument, and the imaging range is from the bottom of the heart to From the root of the great vessel to the apex of the heart, we only screen short-axis cardiac images. In this embodiment, there are a total of 45 subjects.

S2. Data preprocessing.

The 128×128 image area in the middle of the image is deducted. Considering that the left ventricle is generally located in the middle area of the short-axis image, the subtracted image can reduce the influence of other tissues on the network. Normalize the contour line into a marker image that can be recognized by the neural network, and finally generate a 128×128 binary image, in which the pixels inside the contour line are the target class, and the pixels outside the contour line are background class, and process it into a label map as shown in Figure 3.

S3. Divide the data set.

The obtained short-axis magnetic resonance image data and the corresponding segmentation label data are used as a sample data set, and the data set is divided into a training set, a verification set and a test set, that is, according to a ratio of roughly 1:1:1.

S4. Build a network model.

和卷积核

(其中h_i≥h_k,w_i≥w_k)，那么K卷积I就是将卷积核K在矩阵I上滑动，在每一个滑动到的位置做矩阵点乘再求和的运算，最终得到特征矩阵

另外，有时在卷积层之后会进行池化层的操作，对结果压缩的效果，降低数据的空间尺寸，即是的计算资源耗费变少；常见的池化操作有最大池化层和平均池化层，最大池化是对局部区域取最大值，平均池化是对局部区域求取平均值。Build the network as shown in Figure 1. In this training process, a fully convolutional network is used. The idea behind the convolutional network is local connection and weight sharing. Unlike the previous perceptron network where each neuron is fully connected to all neurons in the previous layer, local connections and weight sharing can reduce a lot of parameters. Each convolutional layer is calculated by a convolution kernel of a specific size, assuming an input image

and convolution kernel

(where h _i ≥ h _k , w _i ≥ w _k ), then K convolution I is to slide the convolution kernel K on the matrix I, and do matrix point multiplication and summation at each sliding position, and finally get feature matrix

In addition, sometimes the pooling layer operation is performed after the convolutional layer, and the result compression effect reduces the space size of the data, that is, the computing resource consumption is reduced; common pooling operations include maximum pooling layer and average pooling In the layer, the maximum pooling is to take the maximum value of the local area, and the average pooling is to find the average value of the local area.

In order to deepen the depth of the network and better learn the characteristics of the image, we use the residual network as the backbone network, and the residual network uses the idea of cross-layer links. Assuming that the input of a certain neural network is x, and the expected output is H(x), if it is to learn H(x) directly through x, then the relative training will be more difficult; the residual network is the learning that transforms the target into an identity map , pass the input x to the output as the initial result, so that the output becomes H(x)=F(x)+x, the goal of this network is no longer to learn a complete output, but the target value H( The difference between x) and x, that is, F(x)=H(x)-x. This jumping structure of residuals breaks the traditional convention that the output of the n-1 layer of the traditional neural network can only be used as an input to the n layer, so that the output of a certain layer can directly cross several layers as the output of a later layer. enter.

The segmentation network is to classify images at the pixel level, which is different from the classic convolutional network that uses a fully connected layer at the end to obtain a fixed-length feature vector for classification. The segmentation network needs to use deconvolution to upsample the feature map of the last base layer to restore it to the size of the original input image, so as to complete the prediction of each pixel in the image. Considering that only the last feature layer is used for deconvolution, there will be a serious problem of information loss. The present invention adds a skip structure, so that the feature layer of the previous layer is also deconvolved and then fused with the latter layer as the final prediction. As a result, the network of the present invention utilizes a total of 4 feature layers.

The network of the present invention takes the heart image of the magnetic resonance short axis as input, and the network finally realizes the classification based on each pixel. After an MRI short-axis image enters the network, the output of the network is a prediction map of the same size as the input image, and each pixel value is the category of the pixel y ₀ ,y ₁ ,...,y _n ,. .., y _N , n∈[0,N], and then go through a softmax layer, and convert the output into a probability as shown in the following formula (1).

The network training process generally uses the cross loss function of formula (2) to calculate the gap between the predicted probability map and the label to supervise the optimization of the network parameters. In formula (2), G _{(i, j, n)} is the real label, and the smaller the value of cross entropy, the closer the two probability distributions are.

In order to better segment the left ventricle, the present invention proposes a Focal cross loss function based on formula (3), which will give more attention to pixels that are difficult to classify, and less attention to pixels that are easy to classify .

S5. Training the network.

The short-axis heart image in the training set is used as the input of the neural network, and the label image corresponding to the image is used as the ground-truth label to train the fully convolutional neural network end-to-end.

S6. Model testing.

Through learning to determine the final entire network framework, the test set is input into the network, and the final network outputs the classification results for each pixel. Compare the prediction result with the ground truth, and calculate the corresponding Dice metric (DM) index according to formula (4).

In addition, we compared the present invention with the other two segmentation models FCN and UNet, as shown in Figure 4, it can be seen that the present invention has the best effect on the test set.

The above description of the embodiments is for those of ordinary skill in the art to understand and apply the present invention. It is obvious that those skilled in the art can easily make various modifications to the above-mentioned embodiments, and apply the general principles described here to other embodiments without creative efforts. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art according to the disclosure of the present invention should fall within the protection scope of the present invention.

Claims (7)

1. A method for segmenting the left ventricle of the heart based on a deep fully convolutional neural network, comprising the steps of: (1) Obtain the cardiac magnetic resonance short-axis image of the subject and manually mark the contour line of the left ventricle in the image, and construct a binary segmentation image with the same size as the cardiac magnetic resonance short-axis image. In the value segmentation image, the contour of the left ventricle of the heart and its internal pixel values are all 1, and the external pixel values of the contour are all 0; (2) A large number of samples are obtained by using the method of step (1) for different subjects, each sample includes the short-axis cardiac magnetic resonance image of the subject and its corresponding binary segmentation image; all samples are divided into training set, validation set and test set; (3) Use the cardiac magnetic resonance short-axis image in the training set sample as the input of the fully convolutional neural network, and the binary segmentation image as the true value label output by the fully convolutional neural network, and then train the neural network, and finally train After completion, the segmentation model of the left ventricle of the heart is obtained; (4) Input the cardiac magnetic resonance short-axis image in the test set sample into the heart left ventricle segmentation model, and then obtain the binary segmentation image about the outline of the heart left ventricle, and then compare the binary segmentation image with the test set sample Binary segmented images for comparison. 2. The heart left ventricle segmentation method according to claim 1, characterized in that: in the step (1), the cardiac magnetic resonance short-axis image of the subject is obtained, that is, the subject's heart is simultaneously performed by a magnetic resonance instrument. Coronal, sagittal, and axial three-direction positioning imaging, the imaging range is from the bottom of the heart and the root of the great vessels to the apex of the heart, and the cardiac magnetic resonance short-axis images are screened out. 3. The heart left ventricle segmentation method according to claim 1, characterized in that: the specific operation process of constructing a binary segmentation image in the step (1) is: for the manually marked heart left ventricle contour line, its Expressed as a marker map recognizable by the neural network, that is, a binary segmentation image of the same size as the short-axis image of cardiac magnetic resonance, in which the pixels belonging to the contour line and its interior are the target class and the pixel values are all 1, and the contour line The external pixel points are the background class and the pixel values are all 0; the ratio of the number of samples in the training set, verification set and test set is 1:1:1. 4. The heart left ventricle segmentation method according to claim 1, characterized in that: the fully convolutional neural network in the step (3) comprises a residual network Resnet 50, four prediction layers P1～P4 and four Deconvolution layers D1-D4, the residual network Resnet 50 is sequentially composed of a convolutional layer C, a pooling layer and four residual stages L1-L4 cascaded from input to output, the convolutional layer C The input is the input of the entire neural network. Both the residual stage L1 and the residual stage L4 are composed of three residual structures cascaded, the residual stage L2 is composed of four residual structures, and the residual stage L3 It is formed by cascading six residual structures, and the residual structure is formed by cascading three convolutional layers C1 to C3, where the input of convolutional layer C1 and the output of convolutional layer C3 are superimposed as residuals The output of the structure; the output of the residual stage L4 is connected to the input of the prediction layer P1, the output of the prediction layer P1 is connected to the input of the deconvolution layer D1, the output of the residual stage L3 is connected to the input of the prediction layer P2, and the output of the prediction layer P2 The output is superimposed with the output of the deconvolution layer D1 as the input of the deconvolution layer D2, the output of the residual stage L2 is connected to the input of the prediction layer P3, and the output of the prediction layer P3 is superimposed with the output of the deconvolution layer D2 as The input of the deconvolution layer D3, the output of the residual stage L1 is connected to the input of the prediction layer P4, the output of the prediction layer P4 and the output of the deconvolution layer D3 are superimposed as the input of the deconvolution layer D4, and the deconvolution layer The output of D4 is the output of the entire neural network; the prediction layers P1-P4 all include 3×3 convolution kernels, the step size is 1, the padding is 1, and the activation function is Relu; the deconvolution layers D1-P4 D4 sets the convolution kernel to double the input dimension. 5. The heart left ventricle segmentation method according to claim 1, characterized in that: the process of training the fully convolutional neural network in the step (3) is: first the cardiac magnetic resonance short axis in the training set sample The images are input into the fully convolutional neural network one by one, and the loss function L between each output result of the full convolutional neural network and the corresponding true value label is calculated. The parameters in the network are continuously optimized, and the fully convolutional neural network established after the final training is the heart left ventricle segmentation model. 6. the heart left ventricle segmentation method according to claim 5, is characterized in that: the expression of described loss function L is as follows:

Among them: W is the width of the binary segmentation image, H is the height of the binary segmentation image, N is the number of categories and N=2, P _{(i, j, n)} is the jth row of the fully convolutional neural network output image The probability that the i-column pixel is the category n, G _(i,j,n) is the pixel value of the j-th row and the i-column in the binary segmentation image corresponding to the training set sample, category 1 indicates that the pixel is the left ventricle area of the heart, category 2 indicates Pixels are the background area. 7. heart left ventricle segmentation method according to claim 1, is characterized in that: for the cardiac left ventricle segmentation model that obtains after training is finished in described step (3), utilize verification set sample to verify it, pass verification to The model parameters are fine-tuned to further improve the segmentation accuracy of the model.

Images