CN115294157A - Pathological image processing method, model and equipment - Google Patents

Abstract

The invention discloses a pathological image processing method, which comprises the steps of constructing a cell map in a received pathological image, wherein the cell map comprises cell structure information; inputting the cytogram into a trained pathological image processing model to obtain a fusion characteristic H _O For predicting expression of gene markers. The pathological image processing model integrates a atlas convolution neural network GNN model and a convolution neural network CNN model, and the GNN model is used for extracting low-dimensional features H from the cytogram _G The CNN model is used for extracting low-dimensional features H of an image layer from the pathological image _I Characteristic H _G And H _I The fusion characteristic is obtained after splicing of one fusion layer _O 。

Description

A pathological image processing method, model and device

technical field

The invention belongs to the field of biology and pathology, and in particular relates to a pathological image processing method, model and equipment.

Background technique

In biology and pathology, structural information in cell morphology, such as cell aggregation, distance between cells, etc., reflect the expression of genetic markers (such as microsatellite instability in colorectal cancer) . However, there is little mention about how to use this feature for gene marker prediction.

Contents of the invention

One of the embodiments of the present invention is a pathological image processing method, which constructs a cell map in the received pathological image, and the cell map contains cell structure information;

The cell map is input into the trained pathological image processing model, and the fusion feature is obtained as HO for predicting the expression of gene markers.

The embodiment of the present invention explicitly extracts the cell structure information in the pathological image in the form of a graph, and uses a graph convolutional neural network to extract geometric information features from the graph. This feature was fused with features extracted from pathological images by a convolutional neural network to predict the expression of gene markers.

The present invention proposes a fusion model of graph convolution and convolutional neural network for gene marker prediction of pathological images. In 3 pathological datasets, namely colorectal cancer pathological images predicting microsatellite instability, gastric cancer predicting microsatellite instability (MSI) and gastric cancer predicting programmed cell death-ligand 1 (PD-L1) expression pathology dataset , have achieved significant experimental improvements, and the accuracy of classification has been significantly improved compared with the pathological image processing model that only uses convolutional neural networks.

Description of drawings

The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are shown by way of illustration and not limitation, in which:

Fig. 1 is a schematic structural diagram of a pathological image processing model according to one embodiment of the present invention.

Detailed ways

The problem to be solved by the invention is how to use the graph convolutional neural network to extract cell structural information from histopathological images to assist the convolutional neural network in gene marker prediction.

According to one or more embodiments, a pathological image processing method includes two stages, the first stage extracts structured graph information from pathological images, and the second stage provides a graph and image fusion model (ie graph convolution and convolutional neural network fusion model).

The first stage is to construct a cell map from the pathological image: use a pre-trained nucleus segmentation neural network (such as UNet, TransUNet, etc.) to segment the nucleus region from the histopathological image image, and each segmented nucleus region constitutes a map A node of , denoted as v _j , where the subscript j is a number for the nucleus.

Next, the weights of edges between graph nodes are calculated based on the distance between the centroids of the nucleus regions. The weight w _ij of the edge between graph nodes v _j and v _i is calculated as follows:

Among them, d(v _i , v _j ) represents the distance between the cell nucleus v _j and the center of gravity of v _i , d _c represents the critical distance, which determines the maximum range of cell distance, and is a hyperparameter, which is determined by pathological images depends on the magnification. The distance here is in pixels, which may need to be converted according to the magnification of the image. For example, when the actual physical distance is 50 microns. Conversions are made for different magnifications. For example, at 20x magnification (0.5 μm/pixel), the value of dc is 40 pixels.

否则权重为0，即这两个节点之间没有链接的边。每个图节点v_j的节点特征由这块细胞核区域的预定义的radiomics特征确定，取决于下游任务调整radiomics特征的需要。在这个阶段，构造的图刻画了病理图像中细胞和细胞之间的形态学几何结构，用于下一个阶段图卷积神经网络的输入。When the distance between the center of gravity of the cell is less than d _c , the weight is

Otherwise the weight is 0, i.e. there is no link between these two nodes. The node characteristics of each graph node v _j are determined by the predefined radiomics characteristics of this nuclear region, depending on the needs of downstream tasks to adjust the radiomics characteristics. At this stage, the constructed graph characterizes the morphological geometry of cells and between cells in the pathological image, which is used as the input of the graph convolutional neural network in the next stage.

In the second stage, a fusion model of graph convolutional neural network (GNN) and convolutional neural network (CNN). First, train a GNN model (the structure of the model is not limited, such as but not limited to GCN, GIN) to extract a low-dimensional feature H _G that describes the geometric structure of the pathological image from the cell map constructed in the first stage, where the following The mark G represents that this feature describes the geometric information of the pathological image. That is, the GNN model takes the cell graph as input, and obtains a one-dimensional feature output H _G after the graph convolution operation. At the same time, train a CNN model (the structure of the model is not limited, such as but not limited to DenseNet, ResNet) is used to extract the low-dimensional feature H _I of the image level from the pathological image, where the subscript I represents that this feature describes the information of the image level , that is, the CNN model takes the pathological image as input, and obtains a one-dimensional feature output H _I after the convolution operation.

Because _HG and _HI are extracted from the cell map extracted from the pathological image and the pathological image itself is extracted by GNN and CNN respectively, they describe different features and are complementary. Therefore, the two parts of low-dimensional features are spliced first, and then the spliced features are further fused through a fusion layer for the prediction of downstream gene expression tasks . Among them, the fusion layer contains two strategies: a learnable MLP layer or a learnable Transformer layer. The fusion feature obtained after the splicing of H _G and HI is recorded as _HO , where the subscript _O indicates that this feature is a fusion feature. The overall model structure is shown in Figure 1.

MLP-based fusion layer model: The MLP fusion strategy uses the iteration of multiple MLP units _MLPBlock to obtain the expression of the fusion feature HO:

H _O ＝MLPBlock(...(MLPBlock(H _C ))) (2)

Wherein, H _C is a feature after H _G and H _I are linked, that is, H _C =concat[H _I , H _G ], and concat is a link operation.

The expression of MLPBlock is as follows,

MLPBlock(H)＝Dropout(ReLU(Linear(H))) (3)

Among them, Linear is a learnable linear layer, ReLU is the ReLU activation function, and Dropout is the Dropout regularization layer.

Transformer-based fusion layer model: Transformer-based fusion strategy is expressed as follows:

H _O ＝Pooling(TransBlock(...(TransBlock(H _C )))) (4)

Wherein, H _C is a feature after H _G and H _I are linked, that is, H _C =concat[H _I , H _G ], and concat is a link operation.

Pooling is the pooling layer, and TransBlock is the basic unit of transformer, expressed as follows:

TransBlock(H) = ResidualPreNorm(MLPBlock, ResidualPreNorm(MHSA, H)) (5)

MHSA is a multi-headed self-attention mechanism layer (Multi-headed self attention), and ResidualPreNorm is a residual pre-LayerNorm layer. The definition of MLPBlock is the same as above.

Experimental comparison results of the present invention.

In this experiment, using a subset of the public TCGA pathological image dataset, we constructed colorectal cancer pathological images to predict microsatellite instability (negative vs positive), gastric cancer to predict microsatellite instability (negative vs positive), and a private collection of gastric cancer Data predict PDL1 expression. The accuracy of the classification of the present invention improves the AUC by more than 6.0% compared with only the convolutional neural network (including ResNet, DenseNet and MobileNetV3).

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may also be electrical, mechanical or other forms of connection.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of software products, and the computer software products are stored in a storage medium In, several instructions are included to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes. .

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalents within the technical scope disclosed in the present invention. Modifications or replacements shall all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. A pathological image processing method, characterized in that, comprising steps, Construct a cell map in the received pathological image, which contains cell structure information; The cell map was input into a trained pathological image processing model, and the fusion feature was obtained as _HO , which was used to predict the expression of gene markers. 2. The pathological image processing method according to claim 1, wherein the method for constructing a cell map comprises, Use a pre-trained nucleus segmentation model to segment the nucleus region from the pathological image, and each segmented nucleus region constitutes a node of the cell map, denoted as v _j , where the subscript j is the nucleus in a pathological image a number of Calculate the weight of the edge between the nodes of the graph according to the distance between the centers of gravity of the cell nucleus area. The calculation formula (1) of the weight w _ij of the edge between any two nodes v _j and v _i in the graph is as follows:

Among them, d(v _i , v _j ) represents the 2-dimensional Euclidean distance between the nucleus v _j and the center of gravity of v _i , and d _c represents the critical distance, which determines the maximum range of cell distance, determined by the specific data set and pathological image magnification decision, When the distance between the centers of gravity of the cell nuclei is less than d _c , the weight w _ij is

Otherwise the weight w _ij is 0, i.e. there is no linking edge between these two nodes. 3. pathological image processing method according to claim 1, is characterized in that, pathological image processing model has merged graph convolutional neural network GNN model and convolutional neural network CNN model, A GNN model is used to extract low-dimensional features _HG from the cell map, The CNN model is used to extract the low-dimensional feature H _I of the image level from the pathological image, After the features H _G and HI are _spliced by a fusion layer, the fusion feature is _HO . 4. pathological image processing method according to claim 3, is characterized in that, described fusion layer is based on MLP fusion strategy, adopts the iteration of a plurality of MLP units _MLPBlock , obtains the expression of fusion feature H0, H _O ＝MLPBlock(...(MLPBlock(H _C ))) (2) Among them, H _C is the feature after H _G and H _I are linked, that is, H _C = concat[H _I , H _G ], concat is a link operation, The expression of MLPBlock is as follows, MLPBlock(H)＝Dropout(ReLU(Linear(H))) (3) Among them, Linear is a learnable linear layer, ReLU is the ReLU activation function, and Dropout is the Dropout regularization layer. 5. pathological image processing method according to claim 3, is characterized in that, described fusion layer is based on Transformer fusion strategy, and its fusion strategy based on Transformer is expressed as follows, H _O ＝Pooling(TransBlock(...(TransBlock(H _C )))) (4) Among them, H _C is the feature after H _G and H _I are linked, that is, H _C = concat[H _I , H _G ], concat is the link operation, Pooling is the pooling layer, and TransBlock is the basic unit of the transformer, expressed as follows: TransBlock(H)＝ResidualPreNorm(MLPBlock, ResidualPreNorm(MHSA,H)) (5) MHSA is the multi-headed self-attention mechanism layer (Multi-headed self-attention), and ResidualPreNorm is the residual pre-Layer Norm layer, that is, the Layer Norm layer is placed before. 6. A pathological image processing model, characterized in that, the pathological image processing model has merged a graph convolutional neural network GNN model and a convolutional neural network CNN model, The GNN model is used to extract low-dimensional features H _G from the cytogram of pathological images, The CNN model is used to extract low-dimensional features H _I at the image level from pathological images, After the features H _G and HI are _spliced by a fusion layer, the fusion feature is _HO . 7. A pathological image processing device, characterized in that the device includes a memory; and a processor coupled to the memory, the processor configured to execute instructions stored in the memory, the processor performing the following operations: Construct a cell map in the received pathological image, which contains cell structure information; The cell map was input into the trained pathological image processing model, and the fusion feature was obtained as _HO for predicting the expression of gene markers. 8. A storage medium on which a computer program is stored, wherein when the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images