CN114565919A - Tumor microenvironment spatial relationship modeling system and method based on digital pathological image - Google Patents

Abstract

The invention discloses a system and method for modeling the spatial relationship of tumor microenvironment based on digital pathological images. The system includes: an image staining normalization module, which is used to determine the pixel distribution type of the pathological image, so as to perform color normalization on the staining distribution change to obtain a staining normalized image; a structural region segmentation module, which is used for the staining normalized image, using weak supervision. The deep learning model detects the region of interest, and then segmentes to obtain the target structure region; the cell detection module is used to extract various types of cell information from the target structure region; the spatial relationship building module is used to use a multi-layer graph to model multiple layers The network is used to characterize the co-spatial distribution among various types of cells, and the multi-layer graph is clustered to obtain a quantitative model of the spatial distribution. The invention can accurately reveal the correlation between the heterogeneity in the tumor and the spatial distribution law of cells and tissues in the tumor microenvironment, and provide a new quantitative analysis idea for the tumor evolution mechanism.

Description

A system and method for modeling the spatial relationship of tumor microenvironment based on digital pathological images

technical field

The invention relates to the technical field of medical image processing, and more particularly, to a system and method for modeling the spatial relationship of tumor microenvironment based on digital pathological images.

Background technique

Tumor tissue is a complex structure composed of cancer cells and surrounding non-cancer cells (such as stromal cells and lymphocytes) forming a tumor microenvironment, and its spatial heterogeneity is very complex. However, existing methods rely only on simple distance measurements, cell density statistics or The clustering method cannot achieve full expression. In addition, due to the abundance of cell types in the tumor microenvironment, the spatial organization relationship of cells constructed by using graph neural network cannot be applied to fully automatic and comprehensive quantitative analysis of the spatial organization distribution of multiple cell types at the same time. Therefore, it is necessary to study a new method for topological clustering of multi-layer networks oriented to multi-cell types.

The tumor microenvironment controls the formation, development, metastasis and drug resistance of solid tumors. It is the result of the interaction between tumor cells and non-tumor cells and tissues such as stromal cells and immune cells to produce anti-tumor immune responses. Strong clinical and experimental evidence supports the importance of the tumor microenvironment in cancer progression and in mediating drug resistance. However, the relationship of complex anatomical structures and local microenvironments to metabolic and immune responses remains to be further explored. Routine qualitative or semi-quantitative parametric visual inspection by pathologists can hardly capture the interaction between the tumor and its microenvironment, so computational analysis of digital pathology images is used to decipher the properties of the tumor microenvironment, especially the spatial heterogeneity within the tumor , not only provides a new way of thinking to solve the problem of tumor microenvironment analysis, but more importantly, it can dig out potential biomarkers related to cancer treatment, so as to design the most suitable precision medicine treatment plan for patients.

With the development of digital pathology panoramic imaging and deep learning-based pathological image processing algorithms, computational pathology can not only assist pathologists to examine patients' histological data in a high-throughput, quantitative and objective manner, but also utilize automatic detection algorithms to obtain histological data. Various types of cells are used to construct a spatial relationship map of cells in the tumor, and combined with spatial analysis methods to achieve accurate evaluation of tumor treatment response and prognosis. The initial spatial analysis research work of tumor microenvironment based on digital pathology usually uses clustering algorithms to perform spatial localization and morphometric measurements of cell features extracted from digital pathology images to describe the relationship between the spatial distribution pattern of immune cells and disease. . For example, the identification of tumor-infiltrating lymphocytes (TILs) and the segmentation of tumor necrosis regions are first realized by using convolutional neural networks, and then the affine clustering algorithm is used to model the spatial patterns of infiltrating lymphocytes, and then the corresponding clusters are extracted. Class signatures describe spatial patterns of TILs, revealing associations between TIL patterns and immune subtypes, tumor types, immune cell debris, and patient survival. Other studies have used Euclidean distance to measure cell distribution density to quantify the spatial relationship between cancer and microenvironment components, and to explore its clinical significance. These studies demonstrate that the use of image analysis can go beyond sample cell counts for spatial analysis of the tumor microenvironment based on spatial distance. In order to make better use of high-level spatial distribution information, KunHuang et al. adopted the topological space modeling of deep learning features based on delaunay triangulation graphs, firstly using stacked autoencoder network to learn high-level semantic features of cells, and then using K-means The spatial patterns of nuclei were obtained by clustering. Finally, the statistical method of edge histogram confirmed that the spatial topological features of the renal tumor microenvironment were significantly correlated with survival. It was also verified that in terms of survival prediction, topological features were compared with clinical features and cell morphological features. Has better performance. Guanghua Xiao et al. used a cell statistical density method to construct a spatial organization map of the region, used a deep convolutional network to automatically identify cell types, and finally calculated two spatially distributed features to predict the survival of lung cancer patients.

To sum up, the tumor microenvironment is very complex and has spatial heterogeneity. Existing methods cannot fully express only simple distance measurement, cell statistics or clustering methods. Although the latest research attempts to use graphs to construct spatial organization relationships. , but the spatial analysis of the graph still relies on the artificial extraction of the number of adjacent nodes, the edge histogram and other characteristics of the graph, and can only analyze the relationship between several simple cells. Due to the abundance of cell types in the tumor microenvironment, it is difficult for current spatial analysis methods to simultaneously perform a fully automatic and comprehensive analysis of the spatial organization distribution of various types of components (including structures such as blood vessels, lymphocytes, stromal cells and other different types of cells). Quantitative analysis.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a system and method for modeling the spatial relationship of tumor microenvironment based on digital pathological images, which is a new technical solution for multi-cell type multi-layer network topology clustering.

According to a first aspect of the present invention, a system for modeling the spatial relationship of tumor microenvironment based on digital pathological images is provided. The system includes:

Image staining standardization module: used to determine the pixel distribution type of the pathological image, and color-standardize the staining distribution changes according to the overall distribution of each pixel of the pathological image to obtain a staining-standardized image;

Structural region segmentation module: for the dyed standardized image, the weakly supervised deep learning model is used to detect the region of interest, and then segmented to obtain the target structural region;

Cell detection module: used to extract various types of cell information from the obtained target structure area;

Spatial relationship building module: used to model a multi-layer network using a multi-layer graph to characterize the co-spatial distribution between multiple types of cells, and perform cluster analysis on the multi-layer graph to obtain a quantitative model of spatial distribution, wherein the The spatial distribution quantitative model is used to quantitatively characterize the interaction between tumor cells and the tumor microenvironment. The multi-layer graph contains intra-layer relationships and inter-layer interactions, nodes in the same layer represent cells of the same type, and connections between different layers represent Spatial connections between different types of cells or structures.

According to a second aspect of the present invention, a method for modeling the spatial relationship of tumor microenvironment based on digital pathological images is provided. The method includes the following steps:

Step S1: Determine the pixel distribution type of the pathological image, and perform color standardization on the staining distribution change according to the overall distribution of each pixel of the pathological image to obtain a staining standardized image;

Step S2: For the dyed standardized image, use a weakly supervised deep learning model to detect a region of interest, and then segment to obtain a target structure region;

Step S3: extracting various types of cell information from the obtained target structure region;

Step S4: using a multi-layer graph to model a multi-layer network to characterize the co-spatial distribution among various types of cells, and performing cluster analysis on the multi-layer graph to obtain a spatial distribution quantitative model, wherein the spatial distribution The quantitative model is used to quantitatively characterize the interaction between tumor cells and the tumor microenvironment. The multi-layer graph contains intra-layer relationships and inter-layer interactions, nodes in the same layer represent cells of the same type, and connections between different layers represent different types Spatial connections between cells or structures.

Compared with the prior art, the advantage of the present invention is that, due to the richness and spatial heterogeneity of tumor cell types, various components of the tumor microenvironment have strong spatial correlation with cancer cells at the same time. The multi-component topological spatial mathematical model of tumor cells and tumor microenvironment can reveal the correlation between intratumor heterogeneity and the spatial distribution of tumor microenvironment cells and tissues, and provide a new quantitative analysis idea for tumor evolution mechanism.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is an architectural diagram of a system for modeling the spatial relationship of tumor microenvironment based on digital pathological images according to an embodiment of the present invention;

FIG. 2 is a flowchart of a method for modeling the spatial relationship of tumor microenvironment based on digital pathological images according to an embodiment of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

Referring to Figure 1, the provided system for modeling the spatial relationship of tumor microenvironment based on digital pathological images includes an image staining standardization module, a structural region segmentation module, a cell detection module and a spatial relationship building module. The staining standardization module is used to solve the problem of inconsistent color distribution in different slices; the structural region segmentation module is used to combine the multi-scale imaging characteristics of pathological images to achieve high-resolution segmentation of lesion regions and structures through a regularizable weakly supervised learning method; cells The detection module is used to detect and identify various types of cells in clustered small targets; the spatial relationship building module is used to realize the identification of immune cell types through image registration algorithm, and to construct multi-type structure-cells and tumor cells through multi-layer graph network. The topological spatial relationship model between them can realize the quantitative analysis of the tumor microenvironment.

The functions and specific embodiments of each module will be introduced hereinafter.

(1) Image staining standardization module

The distribution of stained pixels in pathological images usually conforms to a Gaussian distribution or a skewed normal distribution, which can be determined by a self-supervised algorithm based on parameter estimation of the distribution model, where the probability density function (PDF) of the multivariate skewed normal distribution is:

是一个对称矩阵的平方根，

φ_d(·；u,∑)为带有μ均值向量的d变量高斯分布的PDF和协方差矩阵，(·)是指φ_d(·；u,∑))为标准单变量高斯分布的累积分布函数。多元的混合高斯分布的概率密度函数为：In the formula, θ=(μ ^T , a ^T , λ ^T ) ^T is the unknown parameter vector, a is the element of the upper triangular part of the matrix Σ,

is the square root of a symmetric matrix,

φ _d (·;u,∑) is the PDF and covariance matrix of the d-variable Gaussian distribution with μ-mean vector, (·) means that φ _d (·;u,∑)) is the accumulation of the standard univariate Gaussian distribution Distribution function. The probability density function of the multivariate mixture Gaussian distribution is:

且0≤π_d≤1。

为选择的第k个分布的先验概率，密度

为给定第k个分布时x的概率。where the parameter π _d is called the mixing coefficients,

and 0≤π _d ≤1.

is the prior probability of the selected k-th distribution, the density

is the probability of x given the kth distribution.

In one embodiment, the Jarque-Bera test can be used to confirm the actual distribution model type of the pixels of the pathological image, such as the test based on the skewness and kurtosis of the pixel data, and the parameters of the distribution model are further checked. When solving, the parameters of the model can be estimated and updated by the method of depth convolution, so as to obtain the overall distribution of each pixel of the pathological image, and finally realize the color standardization of the image to be analyzed through the staining distribution change model.

Due to various differences in the preservation solutions, dyes, and production processes of different manufacturers, as well as differences in digital scanners, the color of pathological images will change significantly. Standardization of staining can solve the color distribution of pathological images caused by staining operations, staining conditions or equipment images. inconsistencies, thereby improving the results identified by subsequent analysis.

(2) Structural area segmentation module

The structural region segmentation module obtains the structural region reservation segmentation map by performing regularization encoding, regularization decoding, weakly supervised learning, and region-of-interest detection through multi-colored standardized images in sequence.

Specifically, for building weakly supervised segmentation models, one of the key issues is to extract enough key feature codes with limited information to effectively assist segmentation. The features of key regions in the data are constructed according to the image-level labels, irrelevant redundant information and noise are removed, and the original data information is abstracted into two categories of data matrices. The target structure information is a low-rank matrix, and the redundant and the noise information are sparse matrices, solve the two matrices respectively, and finally obtain the feature information of the target structure data. This method is used in the multi-scale pathological image segmentation model, and a regularizer is designed for the above loss for model training. For example, let the image be I and its label be Y, and let f _θ (I) be the output of the θ-parameterized segmentation network, the corresponding optimization problem can be trained using a convolutional neural network with joint regularization loss, expressed as:

是真实值与预测值之间的损失，R(S)是正则化损失，参数S＝f_θ(I)∈[0,1]^|Ω|×K，即网络生成的K通道的softmax分割结果，λ和μ是设定的相应项权重参数。in,

is the loss between the true value and the predicted value, R(S) is the regularization loss, and the parameter S=f _θ (I)∈[0,1] ^|Ω|×K , which is the softmax segmentation result of the K channel generated by the network , λ and μ are the corresponding item weight parameters set.

In order to integrate the feature information of images at different magnifications into the learning process of the algorithm, the input of the model can fuse the image information of multiple magnifications to realize different attention to cells at high magnifications and tissues at medium and low magnifications, so as to fully consider the data samples The specificity and generality of learning data are key features. At the same time, it simulates the diagnosis process of clinical pathologists, and gives different attention weights to image features of different magnifications, so as to fully consider the data features of images with different magnifications. For example, the optimization function for the corresponding multi-rate regularization loss is:

Among them, I _d represents the image input under the d magnification, f _{θ, η} represents the feature calculation under the θ parameter and the attention weight of η; in addition, η is mainly calculated by Softmax (f _θ (I)). The above parameters are learned and optimized through a deep convolutional network, which has the ability to efficiently extract features, so that the model can learn data priors better and faster.

After the parametric model completes the learning, according to the identified target category, the weighted category activation map is obtained by fusing the features of multiple scales, and the target tissue and structural area can be obtained after post-processing.

Weakly supervised learning can replace pixel-wise ground-truth labels with more readily available ground-truth labels, thereby reducing data labeling costs and improving the efficiency of image segmentation. The structural region segmentation network can use various types such as AlexNet, VGG, GoogleNet, ResNet, etc., which is not limited in the present invention.

(3) Cell detection module

Due to the large-scale nature of pathological images, after obtaining the structural region of interest of cancer and delivering the key features of the discriminative properties of cancer and non-cancer tissues (ie, the key discriminant matrix, which corresponds to the matrix according to the high-dimensional features of different types of images) The probabilistic distance is obtained by calculation, and the tumor area is used as the benchmark reference, that is, the distance of the cancer is short, and the distance of others is far). It is necessary to extract different types of cell information from this target structural area. However, due to the large number of cells and the proportion of It is a huge challenge whether the algorithm can detect accurately.

In one embodiment, according to the difference and correlation between the cells and the structure itself, the transformation network based on self-attention realizes the encoding and decoding calculation of each cell and structure of the image, and performs fusion analysis in combination with the key discriminant matrix, as follows:

First, convert the series of feature maps X extracted by the convolutional network into visual tokens T, which are expressed as:

T=SOFTMAX _HW (XW _A ) ^T X (5)

W_A为可学习的权重且

H、W、C分别代表的特征的各向维度，L代表视觉标记T的个数，且L<<HW。in,

W _A is a learnable weight and

H, W, and C represent the respective dimensions of the features, L represents the number of visual markers T, and L<<HW.

After obtaining the visual label T, the self-attention transformation is used to model the dependencies between T, and projected to the dimension of the normal feature map, combined with the pre-order key discriminant matrix G, which is expressed as:

X _out =X _in +SOFTMAX _L ((X _in W _Q )(TW _K ) ^T )T+G (6)

Among them, X _in represents the image features obtained during the detection of tumor regions in multi-scale pathological images, X _out represents the final output result of the cell detection module, W _Q and W _K are respectively learnable weight parameters. A large amount of data is learned to realize the identification and localization of different types of cells and structures.

(4) Spatial relationship building blocks

The spatial relationship building module sequentially performs image dicing, image feature encoding, low-magnification rigid registration, high-magnification non-rigid registration, multi-layer graph network construction, graph embedding dimensionality reduction, acquisition of point cloud distribution data, continuous coherence modeling and features Cluster analysis and other processes, and finally obtain a quantitative model of spatial distribution. The following focuses on multi-layer graph network construction and feature clustering analysis.

Specifically, in order to analyze the co-spatial distribution of expression between multiple types of cells to quantitatively characterize the interaction between tumor cells and the tumor microenvironment, in one embodiment, a multi-layer graph is used to model a multi-layer network. A layer graph is a set of weighted adjacency matrices of a single-layer graph containing intra-layer relationships and inter-layer interactions. The specific implementation includes multi-layer network based on multi-layer graph and clustering calculation for multi-layer network, so as to finally realize the construction of spatial distribution expression model between tumor cells and multi-components of tumor microenvironment.

1) Modeling of spatial higher-order relationships of multi-layer networks

是边的集合。图G中点的总数为n＝|V|。ω:

是一个边权重函数，边e_uv∈E的权重表示为ω_uv，邻接矩阵A是一个对称矩阵，即A_ij＝A_ji，表示每个节点是否有连接关系，也就是不同类型细胞节点的信息。A single-layer graph network is defined as: G = (V, E, ω), where V is the set of nodes,

is the set of edges. The total number of points in graph G is n=|V|. ω:

is an edge weight function, the weight of edge e _uv ∈ E is expressed as ω _uv , the adjacency matrix A is a symmetric matrix, that is, A _ij =A _ji , indicating whether each node has a connection relationship, that is, the information of different types of cell nodes .

由不重叠的m层组成，每一层都由邻接矩阵为A_i,i＝1,…,m的加权图G_i建模。集合A＝{A₁,A₂,…,A_m}中的元素称为层内矩阵，表示单层内的连接，即层内连接。对于两个图之间联系的建模，G_k和G_l以及它们的邻接矩阵可分别表示为，A_k和A_l(k,l＝1,2,…,m；k≠l)，其代表了两个相关图的节点之间一对一的对称内部连接。通过这种方式，可以获得一个跨层邻接矩阵的集合C_p＝{A_l,k,k≠l}，表示不同层的节点之间的边，p代表联系图的数量。According to the definition based on a single layer graph, a multi-layer network can be constructed

It consists of non-overlapping m layers, each layer is modeled by a weighted graph G _i with adjacency matrix A _i , i=1,...,m. The elements in the set A= _{ A ₁ , A ₂ , . For modeling the relationship between two graphs, G _k and G _l and their adjacency matrices can be expressed as, respectively, A _k and A _l (k,l=1,2,...,m; k≠l), which Represents a one-to-one symmetric internal connection between nodes of two correlation graphs. In this way, a set of cross-layer adjacency matrices C _p ={A _l,k ,k≠l} can be obtained, representing edges between nodes in different layers, and p represents the number of connection graphs.

具有一个连接跨层节点的层间连接集合

对于边

有u∈V(G_k)以及v∈V(G_l)，且k≠l。定义的多层网络

的超邻接矩阵具有一个块矩阵结构：In summary, a multi-layer network

A collection of inter-layer connections with one connecting nodes across layers

for edge

There are u∈V(G _k ) and v∈V(G _l ), and k≠l. Defined Multilayer Network

The superadjacency matrix of has a block matrix structure:

The diagonal elements in set A are intra-layer matrices, and the off-diagonal elements A _kl (k,l=1,2,...,m; k≠l) represent the connection between nodes in layer G _k and nodes in layer G _l interlayer connections. In one embodiment, nodes at the same layer are defined to represent cells of the same type, and connections between different layers represent spatial connection relationships between cells or structures of different types. Taking the vascular structure and tumor cells as examples, the establishment of the relationship between the cell-structure layer can be based on the size of the spatial distance to obtain the value of the off-diagonal element A _kl between the layers. The tumor or immune cells close to the blood vessel have a strong connection with the structural layer, and vice versa. There is a weak connection; the diagonal elements are the intra-layer matrix and are also obtained by the Euclidean distance between cells.

After building a multi-layer graph network, considering that extracting meaningful information from a complex network requires a lot of computation and memory, to solve these two problems, the network is transformed into a low-dimensional space through node embedding and preserves its structural information , such as using graph embedding to achieve dimensionality reduction.

2), a multi-layer network topology analysis method based on persistent graph clustering

In order to infer tumor evolution conclusions from the node embeddings of multiple cell types and cancer cells in the tumor microenvironment, it is necessary to perform clustering calculations on the node embeddings. By forming clusters based on shape dynamics, it is helpful to discover persistent node clusters with similar patterns, in one embodiment, the concept of topological data analysis (TDA) is introduced into complex multi-layer network topology analysis .

的图G_j。如果将阈值改为∈₁<∈₂<…<∈_n得到图的分层嵌套序列

称为“网络过滤”。以广泛应用的单纯复形Vietoris–Rips(VR)复形为例，阈值v_j处的VR复形定义为

借助于网络过滤，采用评估网络拓扑归纳的变化来检测大范围阈值∈_j上的持久性特征，其目标就是检测超过不同阈值∈的持久性特征，而这种持久性特征就是内在空间组织分布的特征。这种持续性图聚类算法能够获得更准确的聚类结果。Assuming a weighted graph G, if you choose a threshold ∈ _j > 0 and keep only edges whose weights satisfy ω _uv ≤ ∈ _j , an adjacency matrix can be obtained as

Graph G _j . If the threshold is changed to ∈ ₁ < ∈ ₂ <…< ∈ _n we get a hierarchically nested sequence of graphs

It's called "network filtering". Taking the widely used simplicial complex Vietoris–Rips (VR) complex as an example, the VR complex at the threshold _vj is defined as

With the help of network filtering, we use the evaluation of changes in network topology induction to detect persistent features over a wide range of thresholds ∈ _j . The goal is to detect persistent features that exceed different thresholds ∈, and such persistent features are inherently spatially organized and distributed. feature. This persistent graph clustering algorithm can obtain more accurate clustering results.

To sum up, most current clustering methods of multi-layer networks embed graphs into Euclidean space based on graph decomposition, and do not explicitly consider local graph geometry and topology. The clustering method is to perform clustering computations on multi-layer networks in an unsupervised situation from the perspective of data shape similarity recorded at multiple resolutions. To quantify the shape dynamics of multilayer networks at evolutionarily similar scales, the multi-lens tool of TDA is introduced into the clustering computation, the core idea of which is that if the local neighborhoods of two points are similar in shape at all resolution scales, then They are close enough to each other to cluster into a cluster. Therefore, persistent graph clustering utilizes the distance function and local spatial information around points, and more accurate clustering results can be obtained for multi-layer graph networks.

Correspondingly, the present invention also provides a method for modeling the spatial relationship of tumor microenvironment based on digital pathological images, which is used to realize the functions of each module in the above system. For example, the method includes: step S110, determining the pixel distribution type of the pathological image, performing color standardization on the changes of the staining distribution according to the overall distribution of each pixel of the pathological image, and obtaining a staining standardized image; step S120, for the staining standardized image, using The weakly supervised deep learning model detects the region of interest, and then segmentes to obtain the target structure region; step S130, extracts various types of cell information from the obtained target structure region; The co-spatial distribution among various types of cells is characterized, and the multi-layer graph is clustered to obtain a quantitative model of the spatial distribution. The spatial distribution quantitative model is used to quantitatively characterize the interaction between tumor cells and the tumor microenvironment, and the multi-layer graph includes intra-layer relationships and inter-layer interactions. The connections represent the spatial connections between different types of cells or structures.

To sum up, with respect to the prior art, the present invention has at least the following technical effects:

1), designed a multi-scale pathological image fast calculation method based on weakly supervised learning of learnable regularization constraint encoding and decoding, aiming at the computational difficulty of a single pathological image exceeding one billion pixels and the incomplete information utilization under different magnification scales, Combining weak supervision and deep learning technology, it does not need to rely on large-scale data annotation and makes full use of cross-scale information to achieve rapid detection of lesion regions of interest and accurate positioning of cell nuclei in digital panoramic pathological images.

2) Combining the differences and correlations between cells and structures themselves, the self-attention transformation network is used to realize the encoding and decoding of each cell and structure, thereby realizing the rapid detection and accurate identification of clustered multi-type small target cells.

3) The topological space modeling method of tumor microenvironment based on persistent graph clustering further realizes the quantitative calculation of pathological diagnosis indicators. Conventional distance or statistical methods are difficult to realize the spatial expression analysis of complex tumor microenvironment. The present invention introduces the concept of topological data analysis into the complex multi-layer network clustering calculation, and proposes a topological space construction of persistent graph clustering. The model method can reveal the correlation between intratumor heterogeneity and the spatial distribution of cells and tissues in the tumor microenvironment, and provide a new idea for quantitative analysis of tumor evolution mechanism.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code written in any combination, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (10)

1. A tumor microenvironment spatial relationship modeling system based on digital pathology images, comprising:

an image staining normalization module: the method comprises the steps of determining the pixel distribution type of a pathological image, and carrying out color standardization on dyeing distribution change according to the overall distribution condition of each pixel of the pathological image to obtain a dyeing standardized image;

a structural region segmentation module: the system is used for detecting the region of interest by utilizing a weak supervision deep learning model aiming at the dyeing standardized image, and further segmenting to obtain a target structure region;

a cell detection module: extracting a plurality of types of cell information from the obtained target structure area;

a spatial relationship construction module: the method is used for adopting a multilayer graph modeling multilayer network to represent the co-spatial distribution among multiple types of cells, and performing cluster analysis on the multilayer graph to obtain a spatial distribution quantitative model, wherein the spatial distribution quantitative model is used for quantitatively representing the interaction between tumor cells and a tumor microenvironment, the multilayer graph comprises an in-layer relation and an interlayer interaction, nodes on the same layer represent the same type of cells, and the connection among different layers represents the spatial connection relation among different types of cells or structures.

2. A tumor microenvironment spatial relationship modeling method based on digital pathological images comprises the following steps:

step S1: determining the pixel distribution type of the pathological image, and carrying out color standardization on the dyeing distribution change according to the overall distribution condition of each pixel of the pathological image to obtain a dyeing standardized image;

step S2: aiming at the dyeing standardized image, detecting an interested region by using a weak supervision deep learning model, and further segmenting to obtain a target structure region;

step S3: extracting various types of cell information from the obtained target structure area;

step S4: the method is used for adopting a multilayer graph modeling multilayer network to represent the co-spatial distribution among multiple types of cells, and performing cluster analysis on the multilayer graph to obtain a spatial distribution quantitative model, wherein the spatial distribution quantitative model is used for quantitatively representing the interaction between tumor cells and a tumor microenvironment, the multilayer graph comprises an in-layer relation and an interlayer interaction, nodes on the same layer represent the same type of cells, and the connection among different layers represents the spatial connection relation among different types of cells or structures.

3. The method of claim 2, wherein the input of the weakly supervised deep learning model fuses image information of multiple magnifications, and the training process employs multiple-magnification regularization loss as an optimization target, expressed as:

wherein, I_dRepresenting the image input at d magnification, f_θ，ηDenotes feature calculation under attention weight of η according to Softmax (f) under θ parameter_θ(I) Is calculated) to perform a calculation of,

is the loss between true and predicted values, R (S) is the regularization loss, parameter S ═ f_θ(I)∈[0，1]^|Ω|×KK denotes the number of channels, λ and μ are set weight parameters, I denotes an image, and Y denotes a label corresponding to the image.

4. The method according to claim 2, wherein step S3 includes the sub-steps of:

converting the extracted feature map X of the target structure region into a visual marker T;

and modeling the dependency relationship between T by using self-attention transformation, and projecting the dependency relationship to the dimension of the normal feature map, wherein the dimension is represented as:

X_out＝X_in+SOFTMAX_L((X_inW_Q)(tW_K)^t)T+G

where G is the key discrimination matrix, W_QAnd W_KIs a weight parameter, X_inRepresenting image features, X, obtained in the detection of a tumor region in a multi-scale pathological image_outRepresenting the output result;

and according to the characteristic relation of the constructed image, recognizing and positioning different types of cells are realized through data learning.

5. The method of claim 2, wherein the multi-layer map comprises non-overlapping m layers, each layer being defined by an adjacency matrix as A_iWeighted graph G of 1, …, m_iModeling, set a ═ a₁，A₂，…，A_mThe elements in the are called intra-layer matrices, representing intra-layer connections; for the modeling of the connection between two graphs, G_kAnd G_lAnd their adjacency matrices are denoted as A_kAnd A_lWhich represents a one-to-one symmetric internal connection between the nodes of two related graphs, set C of cross-layer adjacency matrices_p＝{A_l，kAnd k ≠ l), which represents an edge between nodes of different layers, and p represents the number of contact maps, wherein k, l ≠ 1,2, …, m, k ≠ l.

6. The method of claim 5, wherein the method is applied to a multi-layer network constructed from a multi-layer graph

Inter-layer connection set with one connection cross-layer node

To the edge

With u e V (G)_k) And V ∈ V (G)_l) And k ≠ l, said multilayer network

The super-adjacency matrix of (a) has a block matrix structure represented by:

wherein in set AIs an intralayer matrix, and the off-diagonal elements A_kl(k, l ═ 1,2, …, m; k ≠ l) denotes that G is substituted_kNode in layer and G_lInter-layer connections where nodes in a layer are connected.

7. The method of claim 6, wherein for inter-layer connections, inter-layer off-diagonal element A is obtained based on spatial distance size_klFor the intralayer matrix, the value of the diagonal element is obtained by the euclidean distance between the cells.

8. The method of claim 5, further comprising reducing dimensions of the multi-layer network by using graph embedding, and clustering the multi-layer network after dimension reduction according to the shape similarity of local neighborhoods of two points on all resolution scales.

9. The method of claim 2, wherein the type of pixel distribution of the pathology image is determined using the Jacobian test.

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 2 to 9.

Images