CN116530965A - Glioma patient prognosis lifetime prediction method and glioma patient prognosis lifetime prediction system based on multi-mode images - Google Patents

Abstract

The invention discloses a method and system for predicting the prognosis and survival period of glioma patients based on multi-modal images, and relates to the technical field of data processing. The key points of the technical scheme are: inputting multi-modal magnetic resonance image data into a decision tree to determine the target object tumor grade; match the cov-Split transformer model according to the tumor grade; extract global features from multimodal MRI data and nuclear magnetic spectrum data through the cov-Split transformer model; linearize the global features and add the target object After the age information is fully connected, the predicted survival period of the target object is obtained. The present invention can extract the features of the image in depth, expand the receptive field, add position information, and avoid the loss of features to a certain extent; and it does not need to supplement the geometric characteristics of the tumor, making the extracted features lighter and improving the accuracy of prediction. precision.

Description

Method and system for predicting prognosis and survival of glioma patients based on multimodal imaging

technical field

The present invention relates to the technical field of data processing, more specifically, it relates to a method and system for predicting the prognosis and survival period of glioma patients based on multimodal images.

Background technique

Multi-modal Magnetic Resonance Image (mMRI) is currently the most commonly used medical image in clinical diagnosis, and it is also often used in the treatment and postoperative tracking of glioma. Compared with other imaging devices, MRI can display richer and clearer details of brain structure, and among its multiple modalities (T1w, T2w, T1wce, Flair), each highlights different parts of the tissue structure and can provide complementary information.

It is recorded in the prior art that the MR image features of tumor enhancement areas and non-enhancement areas are extracted from T1 enhancement-weighted images, and the geometric characteristics of the tumor are used as a supplement to the image features to realize the prediction and analysis of patient survival. However, when the existing technology analyzes the prognosis and survival of glioma patients, it mainly uses the tumor area obtained by segmentation to extract features from the tumor area again, and the method of first segmenting and then extracting features, the accuracy of feature selection Depending on the accuracy of segmentation, inaccurate and incomplete segmentation can easily lead to errors in classification feature extraction. In addition, the existing survival prediction methods have the limitation that the tumor information rich in images cannot be fully utilized, such as the loss of information and location information caused by the limitation of the receptive field, so the geometric characteristics of the tumor are selected for supplementation. The number of features based on radiomics is large, and all of them are used for prediction, which often leads to over-fitting of the model and low accuracy of predictive analysis results.

Therefore, how to study and design a method and system for predicting the prognosis and survival of glioma patients based on multimodal imaging that can overcome the above defects is an urgent problem to be solved.

Contents of the invention

In order to solve the deficiencies in the prior art, the object of the present invention is to provide a method and system for predicting the prognosis and survival of glioma patients based on multi-modal imaging, using the cov-Split transformer model to obtain information from multi-modal magnetic resonance imaging data and nuclear magnetic spectrum Data extraction features can extract image features in depth, expand the receptive field, add location information, and avoid the loss of features to a certain extent; and do not need to supplement the geometric characteristics of the tumor, making the extracted features lighter and improving the quality of life. Prediction accuracy.

Above-mentioned technical purpose of the present invention is achieved through the following technical solutions:

In the first aspect, a method for predicting the prognosis and survival of glioma patients based on multimodal imaging is provided, including the following steps:

Obtain multimodal magnetic resonance image data and nuclear magnetic spectrum data of the target object;

Determine the tumor grade of the target subject after inputting multimodal magnetic resonance image data into a decision tree;

Match the corresponding pre-built cov-Split transformer model according to the tumor grade;

Extract global features from multimodal magnetic resonance image data and nuclear magnetic spectrum data through the cov-Split transformer model;

Linearize the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the prognosis and survival period of the target object.

The present invention adopts the cov-Split transformer model to extract features from multi-modal magnetic resonance image data and nuclear magnetic spectrum data, can extract image features for depth, expand the receptive field at the same time, add position information, and avoid feature loss to a certain extent ; and the combination of the features in the nuclear magnetic spectrum data and the morphological features provided by the MRI image can more effectively reflect the tumor development of the patient without supplementing the geometric characteristics of the tumor, making the extracted features lighter and improving the prediction accuracy .

Further, the multimodal magnetic resonance image data includes four modality data of T1w, T2w, T1wce and Flair.

Further, the decision tree classifies tumor grades according to biochemical characteristics.

Further, the biochemical characteristics include IDH, ATRX, 1p/19q, CDKN2A/B, TERT_EGFR, H3.3G34R/V and H3 K27M.

Further, the tumor grade includes four grades of glioma WHO1, WHO2, WHO3 and WHO4.

Further, the cov-Split transformer model includes 2D convolution block, pooling block, 3 Bottleneck0, 1 Bottleneck1, 2 matrix addition function blocks, segmentation label module, linear projection module and Transformer Encoder module.

Further, the data import module is used to receive input multimodal magnetic resonance image data and nuclear magnetic spectrum data;

The 2D convolution block is used to project the three-dimensional multimodal magnetic resonance image data to two dimensions;

The pooling block is used to perform pooling processing on data;

The Bottleneck0 consists of three convolutional layers;

The Bottleneck1 consists of three convolutional layers;

The matrix addition function block is used to perform an addition operation on convolution results

The segmentation labeling module is used to divide the image into nine small squares of equal size and label them;

The linear projection module is used to linearly project images;

The Transformer Encoder module is used to model high-dimensional global features.

In the second aspect, a system for predicting the prognosis and survival of glioma patients based on multimodal imaging is provided, including:

A data acquisition module, configured to acquire multimodal magnetic resonance image data and nuclear magnetic spectrum data of a target object;

A grade classification module, used to input the multimodal magnetic resonance imaging data into the decision tree to determine the tumor grade of the target object;

The model matching module is used to match the corresponding pre-built cov-Split transformer model according to the tumor grade;

The feature extraction module is used to extract the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data respectively through the cov-Split transformer model;

The predictive analysis module is used to linearize the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the prognosis and survival period of the target object.

In the third aspect, a computer terminal is provided, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the program, the computer terminal described in any one of the first aspects is implemented. A method for predicting the prognosis and survival of glioma patients based on multimodal imaging.

In the fourth aspect, there is provided a computer-readable medium, on which a computer program is stored, and the computer program is executed by a processor to realize the multimodal image-based glioma diagnosis described in any one of the first aspects. Patient prognosis survival prediction method.

Compared with the prior art, the present invention has the following beneficial effects:

1. The method for predicting the prognosis and survival of glioma patients based on multimodal images provided by the present invention uses the cov-Split transformer model to extract features from multimodal magnetic resonance image data and nuclear magnetic spectrum data, and can extract features of images in depth. At the same time, the receptive field is expanded, and the position information is added, which avoids the loss of features to a certain extent; and there is no need to supplement the geometric characteristics of the tumor, which makes the extracted features lighter and improves the prediction accuracy;

2. The present invention uses a decision tree to logically judge the biochemical characteristics of the target object, and predetermines the tumor level to be targeted, so as to match the corresponding cov-Split transformer model, so that the accuracy and reliability of feature extraction are higher, which is beneficial to Enhance the accuracy of patient prognosis and survival prediction;

3. The present invention also considers the age information when predicting the patient's prognosis and survival period, and more comprehensively considers the influencing factors of survival period prediction.

Description of drawings

The drawings described here are used to provide a further understanding of the embodiments of the present invention, constitute a part of the application, and do not limit the embodiments of the present invention. In the attached picture:

Fig. 1 is the flow chart among the embodiment 1 of the present invention;

Fig. 2 is a flow chart of dividing tumor grades in Example 1 of the present invention;

Fig. 3 is a network structure diagram of the cov-Split transformer model in Embodiment 1 of the present invention, a is the cov part, b is the Split part, and c is the transformer part;

Fig. 4 is a system block diagram in Embodiment 2 of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the examples and accompanying drawings. As a limitation of the present invention.

Embodiment 1: A method for predicting the prognosis and survival of glioma patients based on multimodal imaging, as shown in Figure 1, includes the following steps:

Step S1: Acquiring multimodal magnetic resonance image data and nuclear magnetic spectrum data of the target object;

Step S2: after inputting the multimodal magnetic resonance image data into the decision tree, the tumor grade of the target object is determined;

Step S3: matching the corresponding pre-built cov-Split transformer model according to the tumor grade;

Step S4: extracting the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data respectively through the cov-Split transformer model;

Step S5: Linearize the global features in the multimodal magnetic resonance imaging data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the predicted survival period of the target object.

It should be noted that MRS is a technology that can non-invasively observe the metabolism and biochemical changes of living tissues. The nuclear magnetic spectrum data obtained by MRS technology can provide more accurate chemical composition and content of tumor sites, which can be combined with the morphological features provided by MRI images. , can more effectively reflect the tumor development of patients.

However, the present invention uses a new cov-Split transformer model to extract global features in multimodal magnetic resonance image data and nuclear magnetic spectrum data, and extracts more comprehensive features in depth without supplementing image geometric features, so that the extracted The features are relatively lightweight, which improves the prediction accuracy.

Multimodal magnetic resonance image data includes four modal data of T1w, T2w, T1wce and Flair.

The present invention uses a decision tree to logically judge the biochemical characteristics of the target object, and predetermines the tumor level to be targeted, thereby matching the corresponding cov-Split transformer model, making the accuracy and reliability of feature extraction higher, which is beneficial to enhance the patient's Accuracy of Prognostic Survival Prediction. Specifically, the decision tree divides tumor grades based on one or more biochemical characteristics, and the biochemical characteristics include but are not limited to IDH, ATRX, 1p/19q, CDKN2A/B, TERT_EGFR, H3.3G34R/V and H3 K27M.

As shown in Figure 2, for example considering IDH, ATRX, 1p/19q, CDKN2A/B, TERT_EGFR, H3.3 G34R/V, and H3 K27M biochemical characteristics, tumor grades can be divided into glioma WHO1, WHO2, WHO3 and WHO4 has four levels, and the models of different tumor levels are trained based on the sample data of the corresponding level.

As shown in Figure 3, the cov-Split transformer model includes 1 2D convolution block, 1 pooling block, 3 Bottleneck0, 1 Bottleneck1, 2 matrix addition function blocks, 1 segmentation label module, 1 linear Projection module and 1 Transformer Encoder module.

Among them, the data import module is used to receive the input multi-modal magnetic resonance image data and nuclear magnetic spectrum data; the 2D convolution block is used to project the three-dimensional multi-modal magnetic resonance image data to two-dimensional; the pooling block is used to For pooling data, it can speed up the calculation speed and prevent overfitting; Bottleneck0 is composed of three convolutional layers; Bottleneck1 is composed of three convolutional layers; matrix addition function block is used for convolution results Carry out the addition operation segmentation labeling module, which is used to divide the image into nine small squares of equal size and label; the linear projection module, which is used to linearly project the image; the TransformerEncoder module, which is used to model high-dimensional global features.

In general, the input of the cov-Split transformer model is 3x224x224, the output of the cov part is 512x28x28, and the 2D projection processing is 512x784, and the 512x784 is input into the Split part, divided, input into the liner, and its position information and feature linearization are input into the transformer encoder. Output 512x784, input linearization layer, output 512x1x1, add age, fully connect output lifetime.

Example 2: A system for predicting the prognosis and survival of glioma patients based on multimodal images, which is used to implement the method for predicting the prognosis and survival of glioma patients based on multimodal images described in Example 1, as shown in the figure 4, including data acquisition module, class division module, model matching module, feature extraction module and predictive analysis module.

Among them, the data acquisition module is used to obtain multimodal magnetic resonance image data and nuclear magnetic spectrum data of the target object; the level division module is used to input the multimodal magnetic resonance image data into the decision tree to determine the tumor level of the target object; The model matching module is used to match the corresponding pre-built cov-Split transformer model according to the tumor grade; the feature extraction module is used to extract the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data respectively through the cov-split transformer model; The predictive analysis module is used to linearize the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the prognosis and survival period of the target object.

Working principle: The present invention uses the cov-Split transformer model to extract features from multimodal magnetic resonance image data and nuclear magnetic spectrum data, which can extract image features for depth, expand the receptive field at the same time, and add position information, avoiding to a certain extent The loss of features; and there is no need to supplement the geometric characteristics of the tumor, which makes the extracted features lighter and improves the accuracy of prediction; in addition, the decision tree is used to logically judge the biochemical characteristics of the target object, and the tumor level to be targeted is determined in advance. In order to match the corresponding cov-Split transformer model, the accuracy and reliability of feature extraction are higher, which is conducive to enhancing the accuracy of patient prognosis and survival prediction; in addition, age information is also considered when predicting patient prognosis and survival. Factors affecting survival prediction are considered more comprehensively.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above specific implementation manners have further described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation modes of the present invention, and are not used to limit the protection scope of the present invention. Within the spirit and principles of the present invention, any modifications, equivalent replacements, improvements, etc., shall be included in the protection scope of the present invention.

Claims (10)

1. The method for predicting the prognosis and survival period of patients with glioma based on multimodal imaging is characterized in that it comprises the following steps: Obtain multimodal magnetic resonance image data and nuclear magnetic spectrum data of the target object; Determine the tumor grade of the target subject after inputting multimodal magnetic resonance image data into a decision tree; Match the corresponding pre-built cov-Split transformer model according to the tumor grade; Extract global features from multimodal magnetic resonance image data and nuclear magnetic spectrum data through the cov-Split transformer model; Linearize the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the prognosis and survival period of the target object. 2. The method for predicting the prognosis and survival of glioma patients based on multimodal images according to claim 1, wherein the multimodal magnetic resonance imaging data includes four modalities of T1w, T2w, T1wce and Flair data. 3 . The method for predicting the prognosis and survival of glioma patients based on multimodal images according to claim 1 , wherein the decision tree classifies tumor grades based on biochemical characteristics. 4 . 4. The method for predicting the prognosis and survival of glioma patients based on multimodal imaging according to claim 1, wherein the biochemical characteristics include IDH, ATRX, 1p/19q, CDKN2A/B, TERT_EGFR, H3. 3 G34R/V and H3 K27M. 5 . The method for predicting the prognosis and survival of glioma patients based on multimodal images according to claim 1 , wherein the tumor grades include four grades of gliomas WHO1, WHO2, WHO3 and WHO4. 6. The method for predicting the prognosis and survival of glioma patients based on multimodal images according to claim 1, wherein the cov-Split transformer model includes 2D convolution blocks, pooling blocks, 3 Bottleneck0, 1 Bottleneck1, 2 matrix addition function blocks, segmentation label module, linear projection module and TransformerEncoder module. 7. The method for predicting the prognosis and survival of glioma patients based on multimodal images according to claim 5, wherein the data import module is used to receive input multimodal magnetic resonance image data and nuclear magnetic spectrum data; The 2D convolution block is used to project the three-dimensional multimodal magnetic resonance image data to two dimensions; The pooling block is used to perform pooling processing on data; The Bottleneck0 consists of three convolutional layers; The Bottleneck1 consists of three convolutional layers; The matrix addition function block is used to perform an addition operation on convolution results The segmentation labeling module is used to divide the image into nine small squares of equal size and label them; The linear projection module is used to linearly project images; The Transformer Encoder module is used to model high-dimensional global features. 8. A system for predicting the prognosis and survival of patients with glioma based on multimodal imaging, characterized in that it includes: A data acquisition module, configured to acquire multimodal magnetic resonance image data and nuclear magnetic spectrum data of a target object; A classification module for determining the tumor grade of the target object after inputting the multimodal magnetic resonance imaging data into the decision tree; The model matching module is used to match the corresponding pre-built cov-Split transformer model according to the tumor grade; The feature extraction module is used to extract the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data respectively through the cov-Split transformer model; The predictive analysis module is used to linearize the global features in the multimodal magnetic resonance image data and nuclear magnetic spectrum data, and add the age information of the target object to perform full connection to predict the prognosis and survival period of the target object. 9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, characterized in that, when the processor executes the program, it realizes any one of claims 1-7 The method for predicting the prognosis and survival of glioma patients based on multimodal imaging described in the item. 10. A computer-readable medium, on which a computer program is stored, wherein the computer program is executed by a processor to realize the multi-modal image-based glue film according to any one of claims 1-7. Survival prediction method for tumor patients.