CN106780498A - Based on point depth convolutional network epithelium and matrix organization's automatic division method pixel-by-pixel - Google Patents

Abstract

The invention discloses a kind of method split automatically based on point depth convolutional network epithelium pixel-by-pixel and matrix organization, comprise the following steps：Pathological image pretreatment operation；Construction training set and test set；Build depth convolutional neural networks model (DCNN)；Image pixel point in test set is predicted, classification results are obtained.And carry out pseudo-colours according to classification results；The present invention block-based epithelium with image slices vegetarian refreshments as research object and traditional and matrix automatic segmentation algorithm are contrasted, and under identical experiment condition, the method for the present invention is more accurate, and effect is more preferable；The inventive method makes displaying while segmentation result in artwork, just clinician's direct viewing, and makes follow-up diagnosis on this basis.

Description

Automatic segmentation method of epithelial and stromal tissue based on pixel-by-pixel deep convolutional network

technical field

The invention relates to the technical field of pathological image information processing, in particular to a method for automatic segmentation of epithelial and matrix tissues based on a pixel-by-pixel deep convolution network.

Background technique

Epithelial tissue and stroma are two basic types of tissue in breast tissue. 80% of breast tumors originate from the epithelial tissue in the breast, so some scholars are now committed to applying computer-aided diagnosis systems to the heterogeneity analysis of epithelial and stromal tissues in pathological images. Automated differentiation of epithelial and stromal tissues is a prerequisite for quantifying this heterogeneity, thus enabling the separate analysis of epithelial nuclei. However, due to the complexity of pathological tissue images, it is a challenging task to successfully separate the two types of tissue.

1) Big data worthy of the name:

For a complete full-scan section of pathological tissue, its size is about 100,000×700,000 pixels, and it takes up 1.43 G of hard disk space to store it on a computer. This high-resolution, large-scale image is critical to computer hardware and image analysis. Algorithms are very challenging.

2) The types of pathological tissue structure are complex, and the morphology varies greatly

A pathological slice has many types of pathological structures with different shapes. Even if it is the same organization, its structure and shape will be varied. Therefore, it is difficult to describe it with a fixed model, and the requirements for the robustness of the model are greatly improved.

3) Different pathological grades have high tissue heterogeneity

As the grade of cancer increases, the boundaries of normal tissues are continuously eroded by cancer cells, and the boundary information between epithelial and stromal tissues becomes increasingly blurred. And the fuzzy boundary raises the accuracy requirement of the segmentation model.

4) Other challenges

The background of tissue images is complex and noisy, and there are problems with uneven staining and imaging quality.

Because the pathological images of H&E staining (hematoxylin-eosin staining) can reflect the complex morphological characteristics of pathological tissues, it is widely used in clinical practice. However, in the H&E image, not only the background is complex and the image noise is large, but also there are problems such as uneven staining and incorrect staining caused by the staining process of the slice. In addition, different scanner imaging and imaging quality and other issues. These aspects will bring great challenges to image processing and analysis algorithms.

Despite the above-mentioned challenges, there are still many scholars who have made contributions to the automatic segmentation of epithelial and stromal tissues in pathological images and promoted the development of research.

Different from traditional methods, deep learning forms more abstract high-level features by combining low-level features on the basis of large amounts of data. With the continuous deepening of deep learning and big data analysis research, people's research goals have changed from simple images to complex large-scale images. The complexity of histopathological images fits this point.

Contents of the invention

The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art, and provide a method for automatic segmentation of epithelial and stromal tissue based on pixel-by-pixel deep convolutional networks. Compared with block-based epithelial and stromal tissue segmentation methods, no matter Whether it is from the qualitative or quantitative results, the classification accuracy has been greatly improved.

The present invention adopts the following technical solutions for solving the problems of the technologies described above:

A method for automatic segmentation of epithelial and stromal tissue based on a pixel-by-pixel deep convolutional network proposed according to the present invention comprises the following steps:

Step 1. Perform preprocessing operations on all pathological images to remove the difference in color brightness between the pathological images and the pathological images;

Step 2. Randomly select part of the preprocessed pathological images as training samples, and the rest as test samples;

Step 3. According to the manually marked tissue area map, select a block from the epithelial and stromal tissue in the training sample;

Step 4, according to the manually marked tissue area map, select a block from the edge of the epithelial and matrix tissue in the training sample;

Step 5. Integrate the blocks obtained in step 3 and step 4 and randomly divide them into a training set and a test set;

Step 6. Construct a deep convolutional neural network model DCNN, which includes a convolutional layer, a pooling layer, a linear correction function activation function, a local response normalization layer, and a classifier; use the training set and test set in step 5 Set to train the deep convolutional neural network model;

Step 7, take out a pathological image in the test sample in step 2, and construct a block of Q×Q with each point in the pathological image as the center; where, Q is the size of the input size of the deep convolutional neural network;

Step 8. Input the blocks constructed in step 7 into the deep convolutional neural network model trained in step 6 to obtain classification results.

As a further optimization scheme of the method for automatic segmentation of epithelial and stromal tissue based on a pixel-by-pixel deep convolutional network according to the present invention, false color is performed according to the classification result obtained in step 8.

As a further optimization scheme of the method for automatic segmentation of epithelial and stromal tissue based on a pixel-by-pixel deep convolutional network according to the present invention, Q is 32.

As a further optimization scheme of the method for automatic segmentation of epithelial and stromal tissues based on a pixel-by-pixel deep convolutional network described in the present invention, the step 4 is specifically as follows: According to the manually labeled tissue region map, find the epithelial and stromal tissue in the training sample. The boundary line of the matrix tissue, the expansion operation is performed on the boundary line to obtain the coordinates of the points near the boundary line, and a 32×32 block is constructed with these points as the center. If the center point falls in the epithelial tissue, the block is considered as the epithelial tissue Small pieces, and vice versa, small pieces of stromal tissue.

As a further optimization scheme of the method for automatic segmentation of epithelial and stromal tissue based on a pixel-by-pixel deep convolutional network described in the present invention, a deep convolutional neural network model DCNN is constructed in the step 6, as follows:

Initialize the deep convolutional neural network with the weight matrix in the model used by Alex to successfully distinguish the CIFAR-10 data;

The specific structure of the deep convolutional neural network:

1) Convolution layer

Suppose the filter bank is Each input block of size w ^1-1 × w ^1-1 Slide through the local receptive field of the entire image through the filter of m ¹ ×m ^l , perform convolution operation with each local receptive field, and output the result; A total of filters are generated feature maps, and the size of each map is (w ^l-1 -m ^l +1)×(w ^l-1 -m ^l +1), this linear filter is expressed as in, is an m ^l × m ^l filter of an l layer, m ^l represents the size of the filter in the l layer of the network structure, is the number of filters in the l-th layer filter bank W ^l ;

2) The expression of the linear correction function activation function is as follows:

3) Pooling layer

The operation of the pooling layer is to perform a downsampling pyramid operation after the convolutional feature map of the previous layer, and extract its maximum value or average value as the feature value of the next layer within the range of the local receptive field. After the nonlinear operation , the feature map size of the image becomes:

Among them, s is the size of the pooling layer operation;

4) Local response normalization layer

Used for partial subtraction and division and normalization;

5) Output layer

The last layer of the entire network is the output layer, and the output layer is a classifier. The input of the classifier is the last layer of the neural network, and the output of the classifier is the number of categories. In the deep convolutional neural network, the softmax classification of the two classifications The logistic regression model of the device is:

Among them, x is the feature vector of the sample, T is the transpose symbol, and θ is the parameter;

The input of the Softmax classifier is the output of the last layer of the DCNN network, and the parameter θ of the Softmax classifier is obtained by minimizing the following loss function J(θ);

Among them, m is the number of samples, y ⁽ⁱ⁾ is the i-th sample label, x ⁽ⁱ⁾ is the feature vector of the i-th sample, and k is the number of categories;

θ stands for all model parameters as follows:

in, It is the parameter used when it is classified into the jth class, and it is also the jth line of all model parameters of θ, 0<j<k+1 and j is an integer;

According to the obtained Softmax parameter θ, each image block obtained through the sliding window will first be forwarded by DCNN to obtain the feature vector x ⁽ⁱ⁾ , and then sent to the logistic regression model to obtain a probability between 0 and 1 value, the category of the final image patch for:

Wherein, e is a natural base, k=2, It is the parameter used when it is classified into the l-th category, and it is also the l-th row among all model parameters of θ.

Compared with the prior art, the present invention adopts the above technical scheme and has the following technical effects:

(1) under the same experimental conditions, the detection accuracy rate of the inventive method is higher than the segmentation method accuracy rate based on pixel blocks;

(2) The present invention aims at classifying each pixel, avoiding the problem that different types of pixels that exist in the segmentation based on pixel blocks are divided into one block;

(3) the method of the present invention is aimed at edge tissue, adopts the method for mirror image edge pixel to expand edge, thereby they are classified;

(4) The method of the present invention displays the segmentation result on the original image at the same time, so that the clinician can directly watch it and make a follow-up diagnosis on this basis.

Description of drawings

Figure 1 is a structural diagram of a deep volume and a neural network.

Figure 2 is the overall flow chart of the deep volume and neural network experiment; where (a) is the original H&E pathological image; (b) is the 32x32 small block taken out from (a) through the sliding window; (c) is the The small block is input into the entire deep convolutional neural network (schematic diagram), and the classification result is obtained; (d) performs pseudo-color dyeing on the center point pixel of the small block in (b) according to the classification result in (c); (e ) is the result obtained when all the sliding blocks of the entire image are colored, as the segmentation result.

Fig. 3 is a schematic diagram of the method for taking out small pieces of tissue edge in the present invention; wherein, (a) is the original H&E pathological image; (b) is the result manually marked by the pathologist (dark gray is the epithelium, light gray is the stroma, Black is the area of no concern); (c) According to manual labeling, the dividing line between epithelium and stroma is obtained and expanded; (d) Random points are taken in the area where the dividing line is located, and a small block is constructed around this point; (e ) is a small piece of matrix; (f) is a small piece of epithelium.

Figure 4 is the pseudo-color results of different models for the segmentation of epithelial and stromal tissues in pathological images; among them, (a) is the original pathological image, (b) is manually marked by pathologists, and (c) is the original pathological image. The method based on the pixel-by-pixel and deep convolutional neural network proposed by the invention; (d)-(i) are SW-SVM, SW-SMC, Ncut-SVM, Ncut-SMC, SLIC-SVM, DCNN-SLIC-SMC respectively The obtained pseudo-color segmentation result map.

Fig. 5a is a comparison of the ROC curves of the method of the present invention and the existing segmentation method based on pixel blocks on the NKI dataset.

Fig. 5b is a comparison of the ROC curves of the method of the present invention and the existing segmentation method based on pixel blocks on the VGH dataset.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

Step 1, pathological image preprocessing operation, remove the color brightness difference between the image and the image;

This method preselects a pathological image as the target image, and other pathological images will have the same color distribution as the target image after color normalization. The specific method is to convert the target image and the pathological image to be standardized from the RGB color space to the LAB color space, perform a linear transformation on the gray value of each pixel of the three channels, and then convert the linearly transformed LAB color space to be standardized The pathological image is restored to the RGB color space, so that the pathological image to be standardized and the target image have the same color distribution.

Step 2, take out part of pathological images as training samples, and the rest as test samples;

Randomly select pictures in the data, while ensuring that the training samples are completely separated from the test samples.

Step 3. Select image blocks from inside the epithelium and stromal tissue according to expert annotation;

In the pathological image, all the pixel points belong to the image block of epithelial tissue or stromal tissue. Regarding the selection of tissue image blocks, clinicians with professional pathological knowledge will mark tissue areas in large slice images, and the program will select square image blocks with a side length of 32 pixels from these marked areas. Among them, the block selected in the epithelial tissue is used as a positive sample, and the block selected in the stromal tissue is used as a negative sample.

Step 4. According to expert annotation, select image blocks from the edge of epithelial and stromal tissue;

According to expert annotation, find the boundary between epithelial and stromal tissue in the training sample, and perform an expansion operation on the boundary line to obtain the coordinates of points near the boundary line. A small block of 32×32 is constructed centering on these points, and if the central point falls in the epithelial tissue, the small block is considered as a small block of epithelial tissue. On the contrary, it is a small piece of matrix tissue;

Step 5. Integrate the small blocks obtained in step 3 and step 4 and randomly divide them into a training set and a test set;

The data obtained in steps 3 and 4 were integrated by random screening, in which the ratio of small pieces inside the tissue: small pieces at the edge of the tissue was roughly 1:4.

Step 6. Construct a deep convolutional neural network model (DCNN), which includes a convolutional layer, a linear correction function activation function, a pooling layer, a local response normalization layer and a final classifier;

Deep convolutional neural network (DCNN) is a kind of artificial neural network. Its weight sharing network structure makes it more similar to biological neural network, which reduces the complexity of the network model and the number of weights. This advantage is more obvious when the input of the network is a multi-dimensional image, so that the image can be directly used as the input of the network, avoiding the complicated feature extraction and data reconstruction process in the traditional recognition algorithm. A deep convolutional network is a multi-layer perceptron specially designed to recognize two-dimensional shapes. This network structure is highly invariant to translation, scaling, tilting, or other forms of deformation.

The performance of the deep convolutional neural network model depends to a certain extent on the training samples and the initial neural network weights. The method of random initialization is easy to fall into a local optimum, so the weight matrix in the model used by the well-known scholar Alex to successfully distinguish CIFAR-10 data is used here to initialize the deep convolutional neural network of the present invention.

The specific structure of the deep convolutional neural network is introduced below.

1) Convolution layer

Suppose the filter bank is Every (in ) is an m ^l ×m ^l filter of an l layer, is the number of filters in the filter bank W ^l of layer l. Each input block of size w ^l-1 × w ^l-1 Slide through the local receptive field of the whole image through the filter of m ^l × m ^l , perform convolution operation with each local receptive field, and output the result. A total of filters are generated feature maps, and the size of each map is (w ^l-1 -m ^l +1)×(w ^l-1 -m ^l +1), this linear filter can be simply expressed as

2) ReLu activation function

In order to imitate the working principle of human brain neurons, and to better fit and represent our data information, the feature map obtained after each layer of linear filtering must be activated by a nonlinear activation function. The Relu activation function is used here, and the expression is as follows:

Compared with the traditional sigmod activation function, the Relu activation function is not saturated, and can converge more quickly when the training gradient drops, thereby speeding up the training speed of the entire network.

3) Pooling layer (S)

The operation of the Pooling layer is to perform a downsampling pyramid operation after the convolution feature map of the previous layer, and extract its maximum value (or average value) as the feature value of the next layer within the local receptive field range, so in the pooling layer The layer has no parameters, and only needs to do a nonlinear operation. The reason for this is that in a meaningful image, the information of the local area is redundant, and what we need to do is to extract the energy Characteristic that represents and reflects its maximum response. After the pooling operation, the feature map size of the image becomes:

where s is the size of the pooling operation.

4) Local response normalization layer

This module mainly performs local subtractive and divisive normalizations and normalization. It will force the adjacent features in the feature map to compete locally, and it will also force the same spatial position of different feature maps. features to compete. Performing a subtraction normalization operation at a given position is actually the value of the position minus the weighted value of each pixel in the neighborhood. The weight is to distinguish the different effects of the distance from the position. The weight can be determined by a Gaussian weighted window to determine. The division normalization actually first calculates the weighted sum of each feature map in the neighborhood of the same spatial position, and then takes the mean value of all feature maps, and then the value of each feature map at this position is recalculated as the The value of the point is divided by max (the mean, the value of the weighted sum of the point's neighborhood in the map). The denominator represents the weighted standard deviation in the same spatial neighborhood of all feature maps. In fact, for an image, it is mean and variance normalization, that is, feature normalization. This one is actually inspired by computational neuroscience models. Local response normalization imitates the lateral inhibition mechanism of the biological nervous system layer by layer, and creates a competition mechanism for the activities of local neurons, so that the response is relatively large and the value is relatively large, improving the generalization ability of the model. The implementation method is to perform a subtraction normalization operation at each given position, which is actually the value of the position minus the weighted value of each pixel in the neighborhood. The weight is to distinguish the different effects of different distances from the position. Values can be determined by a Gaussian weighted window.

5) Output layer

The last layer of the entire network is the output layer, and the output layer is a classifier. The input of the classifier is the last layer of the neural network, and the output of the classifier is the number of categories. In the deep convolutional neural network, the softmax classification of the two classifications The logistic regression model of the device is:

Among them, the training set consists of m labeled samples: {(x ⁽¹⁾ ,y ⁽¹⁾ ),…,(x ^m ,y ^m )}, x is the feature vector of the sample, T is the transposed symbol, θ is the parameter;

The input of the Softmax classifier is the output of the last layer of the DCNN network, and the parameter θ of the Softmax classifier is obtained by minimizing the following loss function J(θ);

Among them, m is the number of samples, y ⁽ⁱ⁾ is the i-th sample label, and x ⁽ⁱ⁾ is the feature vector of the i-th sample;

For convenience, the symbol θ is used here to represent all model parameters, as follows:

Among them, the subscript θ is the specific category, the superscript T is the transposition symbol, and k is the total number of categories;

According to the obtained Softmax parameter θ, each image block obtained through the sliding window will first be forwarded by DCNN to obtain the feature vector x ⁽ⁱ⁾ , and then sent to the logistic regression model to obtain a probability between 0 and 1 value, the category of the final image patch for:

Wherein, e is a natural base, and k=2. is the parameter used when classifying as the jth class, and it is also the jth line of all model parameters of θ. It is the parameter used when it is classified into the l-th category, and it is also the l-th row among all model parameters of θ.

Step 7, take out a pathological image in the test sample in step 2, and construct a small block of 32×32 with each point in the image as the center;

Taking each pixel as the center, take 15 pixels up and 16 pixels down to form a small block of 32×32. For the edge organization, in order to facilitate block selection, the method of mirroring edge pixels is adopted to expand the edge, so as to classify them;

Step 8. Input the small block in step 7 into the pre-trained deep convolutional neural network to obtain the classification result. And perform pseudo-coloring according to the classification results;

Input the small block taken out in step 7 into the deep convolutional neural network model trained in step 6, and get the final output result. If the result is 0, it is considered that the central pixel of the small block is an epithelial tissue pixel, and it is dyed dark gray. If the result is 1, it is considered that the central pixel of the small block is a stromal tissue pixel, and it is dyed light gray. At the same time find the location of the black area in the expert mark and color the same location in the false color result black.

In order to facilitate the public to understand the technical solution of the present invention, a specific embodiment is given below.

In this embodiment, the technical solution provided by the present invention is applied to the breast cancer tissue image set stained with hematoxylin and eosin (H&E). The method of the present invention is tested in two databases, which are respectively: the data provided by the Netherlands Cancer Institute (NKI) and Vancouver General Hospital (VGH). It included 157 pathological images (NKI, 106; VGH, 51) with epithelial and stromal tissues manually labeled by expert pathologists. Each image was cropped from an H&E-stained breast cancer tissue microarray (TMA) at 20× optical resolution, with an image size of 1128×720.

In this embodiment, the tissue feature extraction part adopts a deep convolutional neural network, and the classification part is a softmax classifier. In order to verify the effectiveness of the epithelial and stromal tissue segmentation method based on the pixel-by-pixel deep convolutional network of the present invention, other methods were compared. Several common epithelial and stromal tissue segmentation methods that use deep convolutional neural networks to extract small block features, including SW-SVM (sliding window + support vector machine classification), SW-SMC (sliding window + softmax classification), Ncut-SVM (normalized graph cut + support vector machine classification), Ncut-SMC (normalized graph cut + softmax classification), SLIC-SVM (simple linear iterative clustering + support vector machine classification), SLIC-SMC (simple linear iterative clustering + softmax Classification).

Step 1, pathological image preprocessing operation, remove the color brightness difference between the image and the image;

This method preselects a pathological image as the target image, and other pathological images will have the same color distribution as the target image after color normalization. The specific method is to convert the target image and the pathological image to be standardized from the RGB color space to the LAB color space, perform a linear transformation on the gray value of each pixel of the three channels, and then convert the linearly transformed LAB color space to be standardized The pathological image is restored to the RGB color space, so that the pathological image to be standardized and the target image have the same color distribution.

Pixel gray value linear change formula: defined here are the mean square error and mean value of all pixel gray values in each channel of LAB, respectively. Target is the target image, original is the image before normalization, and mapped is the normalized image.

Step 2, take out part of pathological images as training samples, and the rest as test samples;

Randomly select pictures in the data, while ensuring that the training samples are completely separated from the test samples.

Step 3. Select image blocks from inside the epithelium and stromal tissue according to expert annotation;

In the pathological image, all the pixel points belong to the image block of epithelial tissue or stromal tissue. Regarding the selection of tissue image blocks, clinicians with professional pathological knowledge will mark tissue areas in large slice images, and the program will select square image blocks with a side length of 32 pixels from these marked areas. Among them, the block selected in the epithelial tissue is used as a positive sample, and the block selected in the stromal tissue is used as a negative sample.

Step 4. According to expert annotation, select image blocks from the edge of epithelial and stromal tissue;

As shown in the figure, according to the expert annotation ((b) in Figure 3), the boundary between the epithelial and stromal tissues in the training sample is found, and the morphological expansion operation is performed on the boundary line ((c) in Figure 3) to obtain the expanded boundary. From this the coordinates of the points belonging to the boundary line are obtained. A small block of 32×32 is constructed centering on these points, and if the central point falls in the epithelial tissue, the small block is considered as a small block of epithelial tissue ((f) in FIG. 3 ). On the contrary, it is a small piece of stromal tissue ((e) in Figure 3); for a better display effect, the original image ((a) in Figure 3) and the expanded boundary ((c) in Figure 3) The image is fused to obtain a schematic diagram of the boundary ((d) in Figure 3).

Step 5. Integrate the small blocks obtained in step 3 and step 4 and randomly divide them into a training set and a test set;

The data obtained in steps 3 and 4 were integrated by random screening, in which the ratio of small pieces inside the tissue: small pieces at the edge of the tissue was roughly 1:4. The sample size is shown in Table 1.

Table 1 Number of training samples

Step 6. Construct a deep convolutional neural network model (DCNN), which includes a convolutional layer, a linear correction function activation function, a pooling layer, a local response normalization layer and a final classifier;

For the convolutional neural network, the framework used in the present invention is the very popular Caffe framework at present. The network structure is shown in Figure 1:

The first layer uses 32 convolution kernels (conv) to perform convolution operations on the image (convolution kernel size Kernel size=5; step size Stride=1; image edge mirror fill pixel Pad=2;).

The second layer uses the maximum pooling method to down-sample the convolution result (pooling kernel size Kernelsize=3; step size Stride=2; image edge mirror padding pixel Pad=0;).

Then use the ReLU activation function with Local Response Normalization (LRN).

The third layer uses 32 convolution kernels to perform convolution operations on the image (convolution kernel size Kernel size=5; step size Stride=1; image edge mirror padding pixel Pad=2;).

Then use the ReLU activation function.

The fourth layer uses the maximum pooling method to down-sample the convolution result (pooling kernel size Kernel size=3; step size Stride=2; image edge mirror padding pixel Pad=0;).

Local response normalization was then used.

The fifth layer uses 64 convolution kernels to perform convolution operations on the image (convolution kernel size Kernel size=5; step size Stride=1; image edge mirror padding pixel Pad=2;).

Then use the ReLU activation function.

The sixth layer uses the maximum pooling method to down-sample the convolution result (pooling kernel size Kernel size=3; step size Stride=2; image edge mirror padding pixel Pad=0;).

The seventh layer uses 64 fully connected units (ip) to perform fully connected operations with the previous layer.

The eighth time outputs the classification result and the loss value compared with the real value.

Step 7, take out a pathological image in the test sample in step 2, and construct a small block of 32×32 with each point in the image as the center;

Taking each pixel as the center, take 15 pixels up and 16 pixels down to form a small block of 32×32. For the edge organization, in order to facilitate block selection, the method of mirroring edge pixels is adopted to expand the edge, so as to classify them;

Step 8. Input the small block in step 7 into the pre-trained deep convolutional neural network to obtain the classification result. And perform pseudo-coloring according to the classification results;

Input the small block taken out in step 7 into the deep convolutional neural network model trained in step 6, and get the final output result. If the result is 0, it is considered that the central pixel of the small block is an epithelial tissue pixel, and it is dyed dark gray. If the result is 1, it is considered that the central pixel of the small block is a stromal tissue pixel, and it is dyed light gray. At the same time find the location of the black area in the expert mark and color the same location in the false color result black.

Figure 2 is the overall flow chart of the deep volume and neural network experiment; where (a) is the original H&E pathological image; (b) is the 32x32 small block taken out from (a) through the sliding window; (c) is the The small block is input into the entire deep convolutional neural network (schematic diagram), and the classification result is obtained; (d) performs pseudo-color dyeing on the center point pixel of the small block in (b) according to the classification result in (c); (e ) is the result obtained when all the sliding blocks of the entire image are colored, as the segmentation result.

In order to verify the effectiveness of the segmentation of epithelial and stromal tissue based on the pixel-by-pixel deep convolutional network of the present invention, several other common segmentations of epithelial and stromal tissue based on pixel blocks that use deep convolutional neural networks to extract small block features are compared. Methods, including SW-SVM (sliding window + support vector machine classification), SW-SMC (sliding window + softmax classification), Ncut-SVM (normalized graph cut + support vector machine classification), Ncut-SMC (normalized graph cut + softmax classification), SLIC-SVM (simple linear iterative clustering + support vector machine classification), SLIC-SMC (simple linear iterative clustering + softmax classification).

Figure 4 shows the pseudo-color results of different models for the segmentation of epithelial and stromal tissues in pathological images. Among them, (a) in Figure 4 is the original pathological image; (b) in Figure 4 is manually labeled by pathologists, in which the dark gray part represents the epithelial tissue, the light gray part represents the stromal tissue, and the black part represents the stromal tissue. Part of it is the background area, that is, the area that is not concerned; (c) in Figure 4 is a method based on pixel-by-pixel and deep convolutional neural network proposed in this chapter; (d-i) in Figure 4 are SW-SVM, The pseudo-color segmentation results obtained by SW-SMC, Ncut-SVM, Ncut-SMC, SLIC-SVM, and DCNN-SLIC-SMC, where the dark gray represents the area where the classifier classification result is epithelial tissue, and the light gray represents the classifier classification result is the area of stromal tissue, and the black area is the background area that is not concerned.

It can be seen from the results that the algorithm proposed in the present invention has a very high similarity with the result of expert marking, which has obvious advantages, excluding the background area, that is, the black area in the expert marking.

In order to quantitatively represent the experimental results, the derived parameters in the confused matrix (confused Matrix) and the ROC curve are used to compare the experimental results.

TP represents true positives, that is, the number of pixels marked by experts as epithelial tissue, which the classifier considers to be epithelial tissue;

FP represents false positives, that is, the number of pixels that experts mark as stromal tissue and the classifier considers to be epithelial tissue;

FN represents false negatives, that is, the number of pixels that the expert marks as epithelial tissue and the classifier considers as stromal tissue.

TN means true negative, that is, the number of pixels that experts mark as stromal tissue, and the classifier considers it to be stromal tissue;

The calculation formulas of the derived parameters of the confusion matrix are shown in Table 2, true positive rate (TPR), true negative rate (TNR), positive predictive value (PPV), negative predictive value (NPV), false positive rate (FPR), false Negative Rate (FNR), False Discovery Rate (FDR), Accuracy (ACC), F1 Score (F1), and Matthews Correlation Coefficient (MCC) are all evaluations derived from the parameters in the above four confusion matrices index. Among them, ACC, F1, and MCC are indicators for evaluating the comprehensive ability of the model.

Table 2 Derived parameter formula of confusion matrix

Table 3 below shows the quantitative evaluation (%) of the segmentation results of different models, where the bold font is the optimal value in the index.

table 3

A graph showing a ROC curve is called a "ROC graph". When comparing multiple learners, if the ROC curve of one learner completely "wraps" the curve of another learner, it can be asserted that the performance of the former is better than the latter; if the ROC curve of two learners occurs It is difficult to say whether the performance of the two is better or worse. A more reasonable criterion is to compare the area under the ROC curve, that is, AUC (Area Under ROC Curve). It can be seen from the definition that the AUC value can be obtained by summing the area of each part under the ROC curve. The larger the AUC value, the better the effect. Figure 5a shows the ROC curves of the segmentation effects of several methods on the NKI dataset, and Figure 5b shows the ROC curves of the segmentation effects of several methods on the VGH dataset from the AUC value. The automatic epithelium and matrix segmentation of the product network is better than the block-based automatic epithelium and matrix segmentation.

Claims (5)

1. A method for automatically segmenting epithelial and stromal tissues based on a pixel-by-pixel deep convolution network is characterized by comprising the following steps:

step 1, preprocessing all pathological images to remove color and brightness differences between the pathological images;

step 2, randomly selecting a part of preprocessed pathological images as training samples, and taking the rest pathological images as test samples;

step 3, selecting blocks from the interior of the epithelium and stroma tissues in the training sample according to the artificially marked tissue region diagram;

step 4, selecting blocks from the edges of the epithelial and stromal tissues in the training sample according to the artificially marked tissue region diagram;

step 5, integrating the blocks obtained in the step 3 and the step 4 and randomly dividing the blocks into a training set and a testing set;

step 6, constructing a deep convolutional neural network model DCNN, wherein the model comprises a convolutional layer, a pooling layer, a linear correction function activation function, a local response normalization layer and a classifier; training the deep convolutional neural network model by adopting the training set and the test set in the step 5;

step 7, taking out a pathological image in the test sample in the step 2, and constructing a block of Q multiplied by Q by taking each point in the pathological image as a center; wherein Q is the size of the input size of the deep convolutional neural network;

and 8, inputting the block constructed in the step 7 into the deep convolutional neural network model trained in the step 6 to obtain a classification result.

2. The method according to claim 1, wherein the pseudo-color is performed according to the classification result obtained in step 8.

3. The method of claim 1, wherein Q is 32.

4. The method for automatically segmenting epithelial and stromal tissues based on the pixel-by-pixel deep convolutional network as claimed in claim 3, wherein the step 4 is as follows: according to the artificially labeled tissue region diagram, boundary lines of epithelial tissues and stromal tissues in a training sample are found, the boundary lines are subjected to expansion operation to obtain coordinates of points near the boundary lines, a 32x32 block is constructed by taking the points as centers, if a central point falls in the epithelial tissues, the block is regarded as an epithelial tissue small block, and otherwise, the block is regarded as a stromal tissue small block.

5. The method according to claim 4, wherein a deep convolutional neural network model DCNN is constructed in the step 6, and specifically comprises the following steps:

initializing a deep convolutional neural network by adopting a weight matrix in a model used when Alex successfully distinguishes CIFAR-10 data;

the specific structure of the deep convolutional neural network is as follows:

1) convolutional layer

Assume a filter bank ofEach input having a size w^l-1×w^l-1OfThrough m^l×m^lThe filter slides over the local receptive field of the whole image, performs convolution operation with each local receptive field and outputs a result;co-generation of filtersA feature map, and each map has a size of (w)^l-1-m^l+1)×(w^l-1-m^l+1), this linear filtering is denoted asWherein,is one m of one layer^l×m^lM of the filter^lRepresenting the size of the filter in the l-th layer of the network structure, is the l-th layer filter group W¹The number of filters in (1);

2) the linear correction function activation function is expressed as follows:

$x^l=f\left(g_k^l\right)=max(0,g_k^l);$

3) pooling layer

The operation of the pooling layer is to perform a down-sampling pyramid operation after the convolution feature mapping of the previous layer, extract the maximum value or the average value thereof as the feature value of the next layer in the local receptive field range, and after the nonlinear operation, the feature map size of the image becomes:

$\frac{(w^{l-1}-m^l+1)}{s}\&times;\frac{(w^{l-1}-m^l+1)}{s}$

where s is the size of the pooling layer operation;

4) local response normalization layer

The method is used for local subtraction and division and normalization;

5) output layer

The last layer of the whole network is an output layer, the output layer is a classifier, the input of the classifier is the last layer of the neural network, the output of the classifier is the number of classes, and in the deep convolutional neural network, the logistic regression model of the two-class Softmax classifier is as follows:

$h_{\mathrm{\&theta;}}\left(x\right)=\frac{1}{1+\exp (-{\mathrm{\&theta;}}^{T}x)};$

wherein x is a feature vector of the sample, T is a transposed symbol, and θ is a parameter;

the input of the Softmax classifier is the output of the last layer of the DCNN, and the parameter theta of the Softmax classifier is obtained by minimizing the following loss function J (theta);

$J\left(\mathrm{\&theta;}\right)=-\frac{1}{m}\&lsqb;\underset{i=1}{\overset{m}{\&Sigma;}}{y}^{\left(i\right)}log{h}_{\mathrm{\&theta;}}\left(x^{\left(i\right)}\right)+(1-y^{\left(i\right)})log(1-h_{\mathrm{\&theta;}}(x^{\left(i\right)}\left)\right)\&rsqb;$

where m is the number of samples, y⁽ⁱ⁾For the ith sample, x⁽ⁱ⁾The feature vector of the ith sample is obtained, and k is the number of categories;

θ represents all model parameters, as follows:

$\mathrm{\&theta;}=\left[\begin{array}{c}{\mathrm{\&theta;}}_1^T\\ {\mathrm{\&theta;}}_2^T\\ ...\\ {\mathrm{\&theta;}}_k^T\end{array}\right]$

wherein,is the parameter used when classifying as the jth class, and is also the jth line, 0, of all model parameters of theta<j<k +1 and j is an integer;

according to the obtained parameter theta of Softmax, each image block obtained through the sliding window is subjected to forward propagation of DCNN to obtain a feature vector x⁽ⁱ⁾Then sending the image data to a logistic regression model to obtain a probability value between 0 and 1, and finally obtaining the category of the image blockComprises the following steps:

$\hat{i}=\arg \max p(y^{\left(i\right)}=j|x^{\left(i\right)};\mathrm{\&theta;});$

$p(y^{\left(i\right)}=j|x^{\left(i\right)};\mathrm{\&theta;})=\frac{e^{{\mathrm{\&theta;}}_j^T{x}^{\left(i\right)}}}{{\mathrm{\&Sigma;}}_{l=1}^k{e}^{{\mathrm{\&theta;}}_l^T{x}^{\left(i\right)}}}$

wherein e is a natural base number, k is 2,is the parameter used for classification into the l-th class, and is the l-th line of all model parameters θ.