CN109636808B

CN109636808B - Lung lobe segmentation method based on full convolution neural network

Info

Publication number: CN109636808B
Application number: CN201811425317.4A
Authority: CN
Inventors: 姜志强; 程国华; 何林阳; 季红丽; 宣琳娜
Original assignee: Hangzhou Jianpei Technology Co ltd
Current assignee: Hangzhou Jianpei Technology Co ltd
Priority date: 2018-11-27
Filing date: 2018-11-27
Publication date: 2022-08-12
Anticipated expiration: 2038-11-27
Also published as: CN109636808A

Abstract

The invention provides a lung lobe segmentation method based on a full convolution neural network. The technical scheme comprises the following steps: constructing a lung lobe segmentation data set; acquiring a 3D bounding box of a lung organ; preprocessing data in the lung 3D bounding box; inputting the data block into a full convolution neural network for training; and inputting the data blocks into the trained network for prediction. Due to the adoption of the full convolution neural network, end-to-end training and prediction are realized, manual intervention is not needed, and the prediction speed is high; the lung surrounding frame is segmented, so that the interference of the information outside the 3D lung surrounding frame on the lung lobe segmentation is eliminated, and the completeness and the details of the lung lobe segmentation are obviously superior to those of the traditional method; the lung lobe region segmentation can be well realized for lung CT data with obvious diseases, so that technical support is provided for further quantitative and qualitative assessment of lung lesions. Compared with the traditional algorithm, the method obviously improves the precision of lung lobe segmentation and realizes full-automatic lung lobe segmentation.

Description

Lung lobe segmentation method based on full convolution neural network

Technical Field

The invention relates to the field of medical image processing, in particular to a lung lobe segmentation method based on a full convolution neural network.

Background

In recent years, due to the rapid development of medical imaging and computer technology, it has become a great trend to better integrate computer technology into medical imaging. High-definition, high-contrast CT images are commonly used for the diagnosis of lung diseases. The observation of lung structure and function features by chest CT is an important clinical aid for various lung diseases, and in order to provide reliable diagnostic data for doctors and facilitate early detection and treatment of patient conditions, the chest CT image is usually subjected to subsequent processing, and the lung tissue image is extracted, i.e., segmented.

Currently, many segmentation methods are applied to lung segmentation, and the techniques are as follows: (1) the thresholding method is the most common lung segmentation method, although simple and fast, can not effectively remove background and trachea branches, and the threshold is difficult to determine and is often determined empirically. (2) The region growing method is a method adopted in most of works, can effectively make up for missing defects of edge tracking, but usually needs to manually select seed points, and is a semi-automatic segmentation method needing manual participation; (3) a method based on pattern classification. The method can extract the image characteristics of some data, but needs a large number of training samples, and the segmentation result has strong dependence on the samples and the characteristics and longer processing time. (4) The method based on image registration and shape model has good effect generally, but the method is influenced by training set data to cause large result variability and difficult model building, and the calculation amount is large, so that the speed is slow, and the real-time requirement of clinical application is difficult to meet.

In summary, many conventional segmentation methods are affected by various factors such as the separation steps, so that an ideal effect is difficult to segment, the robustness is not strong, improvement is needed, the segmentation speed and accuracy are improved, and the requirements of medical diagnosis on lung images are met.

Disclosure of Invention

The invention aims to provide a lung lobe segmentation method based on a full convolution neural network, aiming at improving lung lobe segmentation efficiency and improving lung lobe segmentation accuracy, and compared with a lung lobe segmentation method based on a basic traditional image algorithm, the lung lobe segmentation method based on the full convolution neural network has higher robustness.

In order to achieve the purpose, the technical scheme provided by the invention is as follows: a lung lobe segmentation method based on a full convolution neural network mainly comprises the following steps: (a) constructing a training data set: and acquiring a CT image, labeling lung lobe areas with different categories, and preprocessing data. (b) Obtaining a lung enclosure frame: in the training stage, obtaining a 3D bounding box of the lung region in the CT image according to the labeling method or the lung segmentation in the step (a); in the prediction stage, the existing lung segmentation algorithm is used for segmenting the lung region, and a 3D bounding box of the lung region in the CT image is obtained. (c) Data dicing: performing data slicing in the lung candidate 3D bounding box of the step (b), slicing the lung bounding box into a plurality of data blocks, and filling or cutting or not processing the data blocks to meet the data size requirement. (d) Training a model: and (c) providing the data block in the step (c) for a full convolution neural network to train so as to obtain a lung lobe segmentation model. (e) And (3) lung lobe segmentation: and (c) preprocessing the data by the image in the step (a), then obtaining a plurality of data blocks in the lung 3D surrounding frame by the processes in the steps (b) and (c), obtaining a lung lobe segmentation result of the data blocks by the lung lobe segmentation model in the step (D) for the data blocks, and finally obtaining the lung lobe segmentation result of the whole CT image in a data backfilling mode.

Further, in step (a), the doctor labels five regions of the lung according to the clinical anatomical structure, which are: the upper left lung lobe, the lower left lung lobe, the upper right lung lobe, the middle right lung lobe and the lower right lung lobe.

Further, in the step (a), the data is preprocessed in a way that the data is normalized in a way that a Hu value is subjected to window truncation and normalized to a range of 0 to 1 value range, and then is scaled to a range between-1 and 1; the data is interpolated such that the physical pixel spacing of the data in the three directions x, y and z is d1, d2 and d3, and d1, d2 and d3 are all numbers greater than 0, and typically the physical pixel spacing in the three directions is the same, between 0.5 and 1.4 mm.

Further, in the step (b), in the training stage, the range of the maximum value and the minimum value of the lung lobe labeling area in the x axis (horizontal axis), the y axis (vertical axis) and the z axis (vertical axis) is obtained as the standard 3D bounding box of the lung, and a candidate 3D bounding box is obtained through random offset transformation. In the prediction stage, a lung region is segmented through an existing lung segmentation model based on a 2D/3D full convolution neural network (which can also be obtained through a traditional image segmentation algorithm), and the maximum and minimum values of the segmented lung region in an x axis, a y axis and a z axis are used as the candidate lung 3D bounding boxes.

Further, in step (c), the 3D data block is obtained by performing the slicing along the sagittal plane, coronal plane or horizontal plane of the CT image, but if the video memory of the apparatus for training and prediction is sufficient, the slicing may be omitted. And in the training stage, processing such as constant filling or cutting redundant data is carried out on the data block so as to meet the data size requirement (fixed length, width and height), and the fixed length, width and height are obtained by analyzing the size of a lung bounding box of the interpolated data. In the prediction stage, the size of the data block is not strictly required, but the non-sliding window direction of the predicted data block needs to include all lung regions to ensure the integrity of the lung during prediction, in the step (c), the size of the data block to be cut can be obviously larger than that of the training stage to accelerate the prediction speed, if the size of the data block is smaller than the fixed size during training, the data block is filled to the fixed size by a constant value, otherwise, the data block is not processed.

Further, in the step (D), the convolutional neural network is a full-convolutional 3D-U type neural network, and the deep semantic features and the shallow local features are fused in a concatemate (stacking) manner. And (c) the network input is the 3D data block in the step (c), and the lung lobe segmentation result which is predicted by the network and has the same size with the input data block is output.

Compared with the prior art, the invention has the following advantages: (1) the lung lobe segmentation method has better robustness and accuracy, and the segmentation result of the model can be better and better as long as more training data are provided for training; (2) the invention realizes full-automatic lung lobe segmentation by using a neural network without manual intervention; (3) the data are segmented in the lung surrounding frame, so that the interference of data outside the lung surrounding frame is removed, the accuracy is improved, and the false positive is reduced; (4) the invention can better segment the lung lobe area for the lung CT data with obvious diseases, thereby providing basic support for further quantitative and qualitative assessment of lung lesions.

Drawings

Fig. 1 is a flowchart illustrating a lung lobe segmentation method based on a full convolution neural network.

Fig. 2 illustrates the result of the lung lobe segmentation predicted by the present invention (two-dimensional view).

Fig. 3 illustrates the result of the lung lobe segmentation predicted by the present invention (three-dimensional view).

Fig. 4 illustrates a full convolution neural network (U-network) for lung lobe segmentation according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is a flowchart of a lung lobe segmentation method according to an embodiment of the present invention. The method mainly comprises the following steps: constructing a lung lobe segmentation data set; acquiring a 3D bounding box of a lung organ; preprocessing data in the lung 3D bounding box; inputting the data block into a full convolution neural network for training; and inputting the data block into a trained network for prediction. For convenience of understanding details in the invention, the details are described by taking training a model to predict lung lobes of a segmentation result as an example.

(1) Constructing a lung lobe segmentation data set, including label labeling of five lung lobe areas and data preprocessing. The invention needs to label five lung lobe areas (left lung superior lobe, left lung inferior lobe, right lung superior lobe, right lung middle lobe and right lung inferior lobe) in pixel level, and the labeling mode is as follows: and (3) marking the data by the doctor with abundant experience through 3D slicer software, delivering the marked result to another doctor for checking, adopting the data as training data after the doctor checks and confirms that the mark is correct, and otherwise discarding the marked data or marking again. A total of 220 cases of data were annotated, and the ratio of data for diffuse lung disease with disease and diffuse lung disease without was 1: 1.

The data is preprocessed by interpolating the x, y, z (i.e. horizontal, vertical) physical pixel spacing of the data to 1.4 mm, and the Hu value clipping window ranges from-1000 to-200 and is normalized to a range of 0 to 1 value, and then scaled to between-1 and 1.

(2) Acquiring a 3D bounding box of the lung organ, including a 3D bounding box acquisition of the lung during a training phase, and a 3D bounding box acquisition of the lung during a prediction phase. In the training stage, a lung enclosure frame can be obtained through lung lobe labeling; in the prediction stage, the lung region is segmented through the existing lung segmentation model based on the 3D full convolution neural network, and finally the lung surrounding frame is obtained.

(3) And preprocessing the data in the lung 3D surrounding frame, and dividing the data into a training stage and a testing stage. In the training phase, data inside the 3D bounding box of the lung is taken, data outside the 3D bounding box is discarded, and the data block size when training the neural network is specified to be 48 × 196 × 256 (x, y, z, units: pixels in order). When in dicing, the wafer is diced along the direction of an x axis (a horizontal axis), the thickness of the dicing is 48 pixels, the dicing step length is 8 pixels, if the dicing exceeds the surrounding frame boundary, the dicing starting point is retreated to ensure that the dicing does not cut the boundary, and a data block with the thickness of 48 pixels is obtained; then, processing the y axis and the z axis of the data block, for the y axis of the data block, if the length of the y axis of the data block is less than 196, filling a constant 0 on two sides of the data along the y axis, if the length of the y axis of the surrounding box is greater than 196, randomly shearing the data along the y axis, if the length of the y axis of the surrounding box is equal to 196, not processing, and finally enabling the length of the y axis of the data block to be 196; for the z-axis of the data block, the same processing manner as the y-axis of the data block is also adopted, and the z-axis length of the data block is 256 in the final data, so that the data block with the size of 48 × 196 × 256 is finally obtained.

In the prediction stage, data inside the lung bounding box is taken, and data outside the bounding box is discarded, and due to the 3-dimensional full convolution neural network, data input into the network can be of any dimensionality. Dicing the data in the enclosing frame along the x-axis direction, wherein the dicing thickness is 48 pixels, the dicing step length is 16 pixels, and if the dicing exceeds the enclosing frame boundary, backing the starting point to prevent the dicing from cutting the boundary to obtain a data block; then processing data in the y-axis and z-axis directions of the data block, and for the y-axis, if the length of the y-axis of the data block is less than 196 pixels, filling constants 0 on two sides of the data block along the y-axis, and if the length of the y-axis of the surrounding frame is greater than or equal to 196 pixels, not performing operation, and finally obtaining the length of the y-axis of the data block which is greater than or equal to 196 pixels; and for the data z axis, performing a data processing mode similar to the data block y axis, and finally enabling the length of the data block z axis to be larger than or equal to 256 pixels.

(4) And inputting the data block into a full convolution neural network for training. The full convolution neural network for training the lung lobe segmentation model is a 3-dimensional U-shaped neural network and comprises 3 down-sampling layers (maximum pooling layer) and 3 up-sampling layers (anti-convolution layer), the middle of the full convolution neural network is connected (concat) through stacking, each down-sampling layer and each up-sampling layer are followed by 2 convolution blocks, each convolution block comprises 3-dimensional convolution (3 DConv), Batch normalization (Batch normalization), nonlinear activation (ReLU), the last layer of the full convolution neural network is a Softmax activation function, the number of channels output by the network is 6 channels, and the channels represent respectively: background, left lung superior lobe, left lung inferior lobe, right lung superior lobe, right lung middle lobe, right lung inferior lobe and other 6 areas, the optimizer of the training network is Adam, and the initial learning rate is 0.001. The output size of the network is equal to the input size.

And (3) inputting the data processed in the steps (1) to (3) into the 3-dimensional U-shaped neural network for training during training, and stopping training when the loss on the verification set is not reduced any more.

(5) And inputting the data block into a trained network for prediction. When the lung lobes of the test data are segmented, the data are preprocessed in the step (1), processed in the step (2) and the step (3) to obtain data blocks, and finally the data blocks are input into the full convolution neural network trained in the step (4) to be predicted to obtain the segmentation result of each data block. Because the blocks of the prediction data are overlapped, the obtained probability of the overlapped region of the segmentation result is fused by adding, and the category of each pixel point on the final image is obtained by argmax (namely the category with the maximum probability of 6 channels is the category of the current pixel). And finally, obtaining a lung lobe segmentation result in a backfilling mode.

The details of the above-described embodiment are only one of the preferred embodiments of the present invention, and are not intended to limit the scope of the present invention, so that those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications should also be construed as protection scope of the present invention. It is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the invention.

Claims

1. A lung lobe segmentation method based on a full convolution neural network comprises the following steps:

(a) constructing a training data set: acquiring a CT image, labeling lung lobe areas with different categories, and preprocessing data;

(b) obtaining a lung enclosure frame: in a training stage, obtaining a 3D bounding box of a lung region in a CT image according to the labeling method in the step (a); in a prediction stage, the lung region is segmented by using the existing lung segmentation algorithm to obtain a 3D (three-dimensional) bounding box of the lung region in the CT (computed tomography) image;

(c) data dicing: performing data slicing within the lung candidate 3D bounding box of step (b), slicing the lung candidate 3D bounding box into a plurality of data blocks, and padding or cropping or not processing the data blocks to meet data size requirements;

(d) training a model: providing the data block in the step (c) to a full convolution 3D-U type neural network for training to obtain a lung lobe segmentation model;

(e) and (3) lung lobe segmentation: and (c) preprocessing the data through the image in the step (a), then obtaining a plurality of data blocks in a lung 3D surrounding frame through the processes in the steps (b) and (c), obtaining lung lobe segmentation results of the data blocks through the lung lobe segmentation model in the step (D) by the data blocks, and finally obtaining the lung lobe segmentation results of the whole CT image through a data backfill mode.

2. The lung lobe segmentation method based on the full convolution neural network as claimed in claim 1, wherein: in the step (a), according to the clinical anatomical structure, the doctor labels five regions of the lung, which are respectively: the upper left lung lobe, the lower left lung lobe, the upper right lung lobe, the middle right lung lobe and the lower right lung lobe.

3. The lung lobe segmentation method based on the full convolution neural network as claimed in claim 1, wherein: in the step (a), the data is preprocessed in a mode of carrying out normalization processing on the data; the data is interpolated such that the physical pixel spacing of the data in the three x, y, z directions is d1, d2, d3, and d1, d2, d3 are all numbers greater than 0.

4. The lung lobe segmentation method based on the full convolution neural network as claimed in claim 1, wherein: in the step (b), the maximum and minimum ranges of the lung lobe labeling area in the x axis, the y axis and the z axis are obtained as a lung standard 3D surrounding frame in the training stage, a candidate 3D surrounding frame is obtained through random offset transformation, the lung area is segmented through the existing lung segmentation model based on the 2D/3D full convolution neural network in the prediction stage, and the maximum and minimum ranges of the lung area in the x axis, the y axis and the z axis obtained through segmentation are used as the lung candidate 3D surrounding frame.

5. The lung lobe segmentation method based on the full convolution neural network as claimed in claim 1, wherein: in the step (c), the CT image is cut into blocks along the sagittal plane, coronal plane or horizontal plane direction of the CT image to obtain 3D data blocks, and in the training stage, the data blocks are subjected to constant filling or cutting redundant data processing to meet the data size requirement.

6. The lung lobe segmentation method based on the full convolution neural network as claimed in claim 1, wherein: in the step (D), the full convolution 3D-U type neural network fuses deep semantic features and shallow local features through a concatenate mode, the network input is a 3D data block, and the output is a lung lobe segmentation result predicted by the network and having the same size as the input data block.