CN111968742A - Cross-modal prediction system and method for lung cancer gene mutation - Google Patents

Abstract

The invention provides a cross-modal prediction system and a cross-modal prediction method for lung cancer gene mutation, which relate to the field of neural networks and comprise the following steps: the acquisition module is used for acquiring computed tomography images, digital pathological images, real gene mutation types and real tumor mutation loads of patients before receiving targeted therapy and immunotherapy; the marking module is used for marking the computed tomography image and the digital pathological image to obtain a corresponding marked image; the training module is used for taking the labeled images of computed tomography and digital pathology as input, taking the real gene mutation type and the real tumor mutation load as output, and training to obtain a lung cancer gene mutation prediction model; and the prediction module is used for inputting the computed tomography image and the digital pathological image of the patient to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation type and the predicted tumor mutation load. The invention can accurately predict the lung cancer gene type and the tumor mutation load, and provide doctors to evaluate the curative effects of targeted therapy and immunotherapy.

Description

A cross-modal prediction system and method for gene mutation in lung cancer

technical field

The invention relates to the field of neural networks, in particular to a cross-modality prediction system and method for gene mutation of lung cancer.

Background technique

Lung cancer is the leading cause of cancer-related death worldwide. In recent years, with the development of molecular biology and the advancement of gene sequencing technology, the treatment of lung cancer has developed from the traditional treatment mode based on clinical features and pathological typing to the era of precise treatment based on gene and molecular changes, driving gene-guided targeting Treatment and immunotherapy targeting immune checkpoints offer new possibilities for lung cancer patients. However, in unselected treatment populations, only a minority of patients benefit from it. Therefore, accurate analysis of gene mutation status, clear therapeutic targets, and screening of potential benefit groups are crucial for personalized and precise treatment of lung cancer.

At present, the detection of lung cancer gene mutation mainly relies on the invasive operation of needle biopsy, and the subsequent sequencing process requires a lot of time and high cost, which cannot meet the needs of large-scale clinical promotion. In addition, due to the high heterogeneity of lung cancer, a small number of specimens obtained by puncture cannot reflect the overall characteristics of the tumor, and cannot achieve a comprehensive and accurate assessment of tumor gene mutation status.

Radiomics models based on patient CT images can predict lung cancer driver gene mutations, but most of them use a single modality and a single time series of tumor imaging data, which cannot reflect the dynamic change process of tumors and their microenvironment, and cannot resolve regional differences in tumor imaging characteristics. In addition, the model is limited to gene mutation prediction of lung cancer, fails to evaluate the survival benefit of targeted therapy and immunotherapy, and has limited guiding value for precise treatment of lung cancer patients, which has great limitations in clinical application. .

Therefore, how to non-invasively and efficiently analyze gene mutation characteristics, so as to accurately and dynamically guide targeted therapy and immunotherapy strategies, is an urgent problem to be solved in the field of lung cancer precision therapy.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the present invention provides a cross-modal prediction system for lung cancer gene mutation, which specifically includes:

an acquisition module, used for respectively acquiring computed tomography images, digital pathology images, real gene mutation types and real tumor mutation loads corresponding to each of the real gene mutation types of several lung cancer patients before receiving targeted therapy and immunotherapy;

a labeling module, connected to the acquisition module, for labeling the lesion area in the computed tomography image and the digital pathology image respectively to obtain a computed tomography labeling image and a digital pathology labeling image;

A training module, which is respectively connected to the acquisition module and the labeling module, is configured to use the CT annotated image and the digital pathology annotated image of each lung cancer patient as input, and use the corresponding real gene mutation type as input. and the real tumor mutation load as output, and train a lung cancer gene mutation prediction model;

A prediction module, connected to the training module, for inputting the computed tomography image and the digital pathological image of the lung cancer patient to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation type of the lung cancer patient to be predicted And the corresponding predicted tumor mutation load, as reference data for doctors to give targeted therapy and immunotherapy opinions.

Preferably, the collection module includes:

a first acquisition unit, configured to acquire the computed tomography images of each of the lung cancer patients;

a second acquisition unit, configured to acquire a pathological slice of each of the lung cancer patients, and scan the pathological slice to obtain the digital pathological image;

A third collection unit, configured to perform gene sequencing on each of the lung cancer patients to obtain the real gene mutation type and the real tumor mutation load corresponding to each of the real gene mutation types.

Preferably, the training module includes:

The grouping unit is configured to use the computed tomography annotated image, the digital pathology annotated image, the real gene mutation type and the real tumor mutation load of each lung cancer patient as a data set, and according to each The data set forms a training set and a validation set;

A training unit, connected to the grouping unit, configured to take the CT annotated image and the digital pathology annotated image in the training set as input, and take the real gene mutation type and the real tumor mutation load as output , training to obtain an initial model of lung cancer gene mutation;

A verification unit, which is respectively connected to the grouping unit and the training unit, is configured to perform parameter optimization and adjustment on the initial model of lung cancer gene mutation according to the verification set to obtain the lung cancer gene mutation prediction model.

Preferably, the training module further includes a test unit, which is respectively connected to the grouping unit and the verification unit, and the test unit includes:

a first subunit, configured to form a test set according to each of the data sets;

The second subunit is connected to the first subunit, and is used for inputting each of the CT annotated images and the digital pathology annotated images in the test set into the lung cancer gene mutation prediction model to obtain corresponding gene mutations Type prediction results and tumor mutation burden prediction results;

The third subunit is connected to the second subunit, and is configured to process a first prediction accuracy rate according to the prediction result of each gene mutation type and the corresponding real gene mutation type, and obtain a first prediction accuracy rate according to each of the tumor mutation loads The prediction result and the corresponding real tumor mutation load are processed to obtain a second prediction accuracy rate, which is used as reference data for doctors to give advice on targeted therapy and immunotherapy.

Preferably, the receiver operating characteristic curve and the area under the curve are processed to obtain the first prediction accuracy rate and the second prediction accuracy rate.

Preferably, all the data sets are divided according to a preset ratio to form the training set, the verification set and the test set, and the preset ratio is 3:1:1.

A cross-modal prediction method for lung cancer gene mutation is applied to the above-mentioned cross-modal prediction system for lung cancer gene mutation, and the cross-modal prediction method for lung cancer gene mutation specifically includes the following steps:

Step S1, the cross-modality prediction system for lung cancer gene mutation respectively collects computed tomography images, digital pathology images, real gene mutation types and each type of real gene mutation before receiving targeted therapy and immunotherapy. The true tumor mutation burden corresponding to the gene mutation type;

Step S2, the cross-modality prediction system for lung cancer gene mutation respectively annotates the lesion area in the computed tomography image and the digital pathology image to obtain a computed tomography annotated image and a digital pathology annotated image;

Step S3, the cross-modal prediction system for lung cancer gene mutation takes the CT annotated image and the digital pathology annotated image of each lung cancer patient as input, and uses the corresponding real gene mutation type and all The real tumor mutation load is used as the output, and the lung cancer gene mutation prediction model is obtained by training;

Step S4, the cross-modal prediction system for lung cancer gene mutation inputs the computed tomography image and the digital pathology image of the lung cancer patient to be predicted into the lung cancer gene mutation prediction model to obtain the lung cancer patient to be predicted. The predicted gene mutation type and the corresponding predicted tumor mutation load are used as reference data for doctors to give targeted therapy and immunotherapy opinions.

Preferably, the step S1 includes:

Step S11, the cross-modality prediction system for lung cancer gene mutation collects the computed tomography images of each of the lung cancer patients;

Step S12, the cross-modality prediction system for lung cancer gene mutation obtains the pathological slices of each lung cancer patient, and scans the pathological slices to obtain the digital pathological image;

Step S13, the cross-modal prediction system for lung cancer gene mutation performs gene sequencing on each of the lung cancer patients to obtain the true gene mutation type and the true tumor mutation load corresponding to each true gene mutation type.

Preferably, the step S3 includes:

Step S31, the cross-modality prediction system for lung cancer gene mutation annotates the computed tomography scan image, the digital pathology annotated image, the real gene mutation type and the real tumor mutation load of each lung cancer patient. as a data set, and form a training set and a verification set according to each of the data sets;

Step S32, the cross-modality prediction system for lung cancer gene mutation uses the computed tomography annotated image and the digital pathology annotated image in the training set as input, and uses the real gene mutation type and the real tumor mutation as input. The load is used as the output, and an initial model of lung cancer gene mutation is obtained by training;

Step S33, the cross-modal prediction system for lung cancer gene mutation performs parameter optimization and adjustment on the initial model of lung cancer gene mutation according to the validation set, to obtain the lung cancer gene mutation prediction model.

Preferably, the step S3 further includes a process of obtaining the model prediction accuracy, which specifically includes:

Step A1, the cross-modal prediction system for lung cancer gene mutation forms a test set according to each of the data sets;

Step A2, the cross-modality prediction system for lung cancer gene mutation inputs each of the CT annotated images and the digital pathology annotated images in the test set into the lung cancer gene mutation prediction model to obtain the corresponding gene mutation type prediction results and tumor mutation burden prediction results;

Step A3, the cross-modal prediction system for lung cancer gene mutation obtains a first prediction accuracy rate according to the prediction result of each gene mutation type and the corresponding real gene mutation type, and predicts the tumor mutation load according to each The result is processed with the corresponding real tumor mutation load to obtain a second prediction accuracy rate, which is used as reference data for doctors to give targeted therapy and immunotherapy opinions.

The above-mentioned technical scheme has the following advantages or beneficial effects:

This solution integrates radiomics and pathomics to achieve cross-modal prediction, and achieves accurate prediction of mutated genotype and tumor mutation load through a lung cancer gene mutation prediction model. The efficacy of immunotherapy can provide personalized guidance for precise treatment strategies for lung cancer patients.

Description of drawings

1 is a schematic structural diagram of a cross-modality prediction system for lung cancer gene mutation in a preferred embodiment of the present invention;

2 is a schematic flowchart of a method for cross-modality prediction of lung cancer gene mutation in a preferred embodiment of the present invention;

3 is a schematic flowchart of a data collection process of a method for cross-modality prediction of lung cancer gene mutation in a preferred embodiment of the present invention;

4 is a schematic flowchart of a model training process of a method for cross-modality prediction of lung cancer gene mutation in a preferred embodiment of the present invention;

5 is a schematic flowchart of a process of obtaining model accuracy of a method for cross-modal prediction of lung cancer gene mutation in a preferred embodiment of the present invention;

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment, and other embodiments may belong to the scope of the present invention as long as it conforms to the gist of the present invention.

In a preferred embodiment of the present invention, based on the above-mentioned problems in the prior art, a cross-modal prediction system for lung cancer gene mutation is provided, as shown in Figure 1, which specifically includes:

The acquisition module 1 is used to respectively acquire the computed tomography images, digital pathology images, real gene mutation types and real tumor mutation loads corresponding to each real gene mutation type of several lung cancer patients before receiving targeted therapy and immunotherapy;

The labeling module 2 is connected to the acquisition module 1, and is used for labeling the lesion area in the computed tomography image and the digital pathology image respectively to obtain the computed tomography scanning image and the digital pathology labeling image;

The training module 3, which is connected to the acquisition module 1 and the labeling module 2 respectively, is used to take the CT annotated images and digital pathology annotated images of each lung cancer patient as input, and the corresponding real gene mutation type and real tumor mutation load as the output. Obtain lung cancer gene mutation prediction model;

The prediction module 4 is connected to the training module 3, and is used for inputting the computed tomography images and digital pathology images of the lung cancer patients to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation types of the lung cancer patients to be predicted and the corresponding predicted tumor mutation load, As a reference data for doctors to give targeted therapy and immunotherapy opinions.

Specifically, in this example, 325 lung cancer patients who received targeted therapy or immunotherapy in Shanghai Pulmonary Hospital and underwent lung cancer genotype and tumor mutation load detection were included, and the computed tomography images of these lung cancer patients were collected through acquisition module 1 , digital pathology image, real gene mutation value and real tumor mutation load; and through the annotation module 2, the lesion area in the computed tomography image and the digital pathology image are respectively annotated to obtain the computed tomography annotated image and the digital pathology annotated image, and then The annotated images of computed tomography and digital pathology are used as the input of the training model in training module 3, and the corresponding real gene mutation type and real tumor mutation load are used as the output of the training model, and then the lung cancer gene mutation prediction model is obtained by training. In the embodiment, cross-modal prediction is realized by acquiring computed tomography images and digital pathology images, which effectively improves the prediction accuracy of the lung cancer gene mutation prediction model; the computed tomography images and digital pathology images of lung cancer patients to be predicted are input into the prediction module 4. The lung cancer gene mutation prediction model obtains the predicted gene mutation type and the corresponding predicted tumor mutation load of the lung cancer patients to be predicted; the predicted gene mutation type is used to characterize the effectiveness of targeted therapy, and the predicted tumor mutation load is used to characterize the effectiveness of immunotherapy. Therefore, doctors can give treatment advice based on the predicted gene mutation type and predicted tumor mutation burden.

The efficacy of the model can be externally verified by performing survival analysis on patients receiving targeted therapy and immunotherapy: according to the predicted gene mutation type and the corresponding predicted tumor mutation load of the lung cancer patients to be predicted, the lung cancer patients to be predicted can be calculated by deep learning algorithms Targeted therapy benefit score and immunotherapy benefit score; and according to the median value of the target therapy benefit score for lung cancer patients to be predicted, the treatment benefit stratification of the patients to be predicted, the target therapy benefit score is higher than the target therapy benefit score. The patients with the median value were divided into the high targeted therapy group, the patients with targeted therapy benefit scores not higher than the median value were divided into the targeted therapy low group; The median value is used to stratify the treatment benefit of the patients to be predicted, and the patients with the immunotherapy benefit score higher than the median value are divided into the high immunotherapy group, and the patients with the immunotherapy benefit score not higher than the median value are divided into the immunotherapy treatment group. Low grouping; K-M survival analysis and Cox regression analysis were used to analyze the overall survival and progression-free survival of patients with targeted therapy high grouping, targeted therapy low grouping, immunotherapy high grouping and immunotherapy low grouping patients respectively. Patients with high allocation to treatment and high allocation to immunotherapy had better overall survival and progression-free survival, thus demonstrating the predictive validity of the lung cancer gene mutation prediction model.

In a preferred embodiment of the present invention, the collection module 1 includes:

The first acquisition unit 11 is used for acquiring computed tomography images of each lung cancer patient;

The second acquisition unit 12 is configured to acquire pathological slices of each lung cancer patient, and scan the pathological slices to obtain digital pathological images;

The third collection unit 13 is configured to perform gene sequencing on each lung cancer patient to obtain the real gene mutation type and the real tumor mutation load corresponding to each real gene mutation type.

Specifically, in this embodiment, the first collection unit 11 collects the computed tomography images of each lung cancer patient and the second collection unit 12 collects the digital pathological images to obtain the input of the lung cancer gene mutation prediction model; through the third collection Unit 13 collects the real gene mutation type and the real tumor mutation load to obtain the output of the lung cancer gene mutation prediction model, which lays the foundation for the construction of the lung cancer gene mutation prediction model.

In a preferred embodiment of the present invention, the training module 3 includes:

The grouping unit 31 is used to use the computed tomography annotated image, digital pathology annotated image, real gene mutation type and real tumor mutation load of each lung cancer patient as a data set, and form a training set and a validation set according to each data set ;

The training unit 32 is connected to the grouping unit 31, and is used for taking the CT annotated image and digital pathology annotated image in the training set as input, and using the real gene mutation type and the real tumor mutation load as the output, and training to obtain an initial model of lung cancer gene mutation;

The verification unit 33 is respectively connected to the grouping unit 31 and the training unit 32, and is configured to optimize and adjust the parameters of the initial model of the lung cancer gene mutation according to the verification set, so as to obtain the prediction model of the lung cancer gene mutation.

Specifically, in this embodiment, the grouping unit 31 is set to combine the computed tomography annotated image, digital pathology annotated image, real gene mutation type and real tumor mutation load of each lung cancer patient into one data set, and some data are combined into one data set. The data in the set is divided into a training set and a verification set; the initial model of lung cancer gene mutation is obtained by training according to the data in the training set by the training unit 32, and the parameters of the initial model of lung cancer gene mutation are optimized and adjusted by the verification unit 33 according to the data in the verification set , to obtain a lung cancer gene mutation prediction model.

In a preferred embodiment of the present invention, the training module 3 further includes a test unit 34, which is respectively connected to the grouping unit 31 and the verification unit 33, and the test unit 34 includes:

The first subunit 341 is used to form a test set according to each data set;

The second subunit 342 is connected to the first subunit 341, and is used for inputting each CT scan annotated image and digital pathology annotated image in the test set into the lung cancer gene mutation prediction model to obtain the corresponding gene mutation type prediction result and tumor mutation load prediction result;

The third subunit 343 is connected to the second subunit 342, and is configured to obtain a first prediction accuracy rate according to the prediction results of each gene mutation type and the corresponding real gene mutation type, and to obtain a first prediction accuracy rate according to the prediction results of each gene mutation type and the corresponding real gene mutation load prediction results. The tumor mutation burden is processed to obtain a second prediction accuracy rate, which is used as reference data for doctors to give targeted therapy and immunotherapy opinions.

In a preferred embodiment of the present invention, the receiver operating characteristic curve and the area under the curve are used to obtain the first prediction accuracy rate and the second prediction accuracy rate.

Specifically, in this embodiment, the receiver operating characteristic curve and the area under the curve are used to evaluate the first accuracy rate and the second accuracy rate, and the receiver operating characteristic curve and the area under the curve can be used to easily and intuitively pass Graphical observation and analysis of the first accuracy rate and second accuracy rate can be used to make judgments with the naked eye; the receiver operating characteristic curve combines sensitivity and specificity in a graphical way, which can accurately reflect the relationship between specificity and sensitivity. This makes it more objective and accurate to evaluate the first accuracy rate and the second accuracy rate.

In a preferred embodiment of the present invention, all data sets are divided according to a preset ratio to form a training set, a verification set and a test set, and the preset ratio is 3:1:1.

Specifically, in this embodiment, the training set includes data of 195 patients undergoing examination, and both the validation set and the test set include data of 65 patients undergoing examination.

A cross-modal prediction method for lung cancer gene mutation is applied to the above-mentioned cross-modal prediction system for lung cancer gene mutation. As shown in Figure 2, the cross-modal prediction method for lung cancer gene mutation specifically includes the following steps:

Step S1, the cross-modality prediction system for lung cancer gene mutation collects computed tomography images, digital pathology images, real gene mutation types, and the corresponding data of each real gene mutation type of several lung cancer patients before receiving targeted therapy and immunotherapy, respectively. True tumor mutational burden;

Step S2, the cross-modality prediction system for lung cancer gene mutation respectively annotates the lesion area in the computed tomography image and the digital pathology image to obtain the computed tomography annotated image and the digital pathology annotated image;

Step S3, the cross-modality prediction system for lung cancer gene mutation takes the computed tomography annotated image and digital pathology annotated image of each lung cancer patient as input, and uses the corresponding real gene mutation type and real tumor mutation load as output, and trains to obtain lung cancer gene mutation prediction model;

Step S4, the cross-modality prediction system for lung cancer gene mutation inputs the computed tomography image and digital pathology image of the lung cancer patient to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation type of the lung cancer patient to be predicted and the corresponding predicted tumor mutation load , as reference data for doctors to give targeted therapy and immunotherapy opinions.

In a preferred embodiment of the present invention, as shown in FIG. 3 , step S1 includes:

Step S11, the cross-modality prediction system for lung cancer gene mutation collects computed tomography images of each lung cancer patient;

Step S12, the cross-modality prediction system for lung cancer gene mutation obtains the pathological slices of each lung cancer patient, and scans the pathological slices to obtain digital pathological images;

Step S13, the cross-modal prediction system for lung cancer gene mutation performs gene sequencing on each lung cancer patient to obtain the real gene mutation type and the real tumor mutation load corresponding to each real gene mutation type.

In a preferred embodiment of the present invention, as shown in FIG. 4 , step S3 includes:

Step S31, the cross-modality prediction system for lung cancer gene mutation takes the CT annotated image, digital pathology annotated image, real gene mutation type and real tumor mutation load of each lung cancer patient as a data set, and forms a data set according to each data set. training set and a validation set;

Step S32, the cross-modality prediction system for lung cancer gene mutation takes the CT annotated image and digital pathology annotated image in the training set as input, and uses the real gene mutation type and real tumor mutation load as output, and trains to obtain an initial model of lung cancer gene mutation ;

Step S33, the cross-modality prediction system for lung cancer gene mutation optimizes and adjusts the parameters of the initial lung cancer gene mutation model according to the validation set to obtain a lung cancer gene mutation prediction model.

In a preferred embodiment of the present invention, step S3 further includes a process of obtaining the model prediction accuracy, as shown in FIG. 5 , specifically including:

Step A1, the cross-modal prediction system for lung cancer gene mutation forms a test set according to each data set;

Step A2, the cross-modality prediction system for lung cancer gene mutation inputs each CT scan annotated image and digital pathology annotated image in the test set into the lung cancer gene mutation prediction model to obtain the corresponding gene mutation type prediction result and tumor mutation load prediction result;

Step A3, the cross-modal prediction system for lung cancer gene mutation processes according to the prediction result of each gene mutation type and the corresponding real gene mutation type to obtain a first prediction accuracy rate, and according to the prediction result of each tumor mutation load and the corresponding real tumor mutation load After processing, a second prediction accuracy rate is obtained, which is used as reference data for doctors to give opinions on targeted therapy and immunotherapy.

The above are only preferred embodiments of the present invention, and are not intended to limit the embodiments and protection scope of the present invention. For those skilled in the art, they should be able to realize that all equivalents and obvious substitutions made by using the contents of the description and the drawings are obvious. The solutions obtained by the changes of the above should be included in the protection scope of the present invention.

Claims (10)

1. A cross-modal prediction system for lung cancer gene mutation is characterized by specifically comprising:

the system comprises an acquisition module, a detection module and a control module, wherein the acquisition module is used for respectively acquiring computed tomography images, digital pathological images, real gene mutation types and real tumor mutation loads corresponding to each real gene mutation type of a plurality of lung cancer patients before receiving targeted therapy and immunotherapy;

the marking module is connected with the acquisition module and is used for marking the focus areas in the computed tomography image and the digital pathology image respectively to obtain a computed tomography marking image and a digital pathology marking image;

the training module is respectively connected with the acquisition module and the labeling module and is used for taking the computed tomography labeling image and the digital pathology labeling image of each lung cancer patient as input, taking the corresponding real gene mutation type and the real tumor mutation load as output, and training to obtain a lung cancer gene mutation prediction model;

and the prediction module is connected with the training module and used for inputting the computed tomography image and the digital pathological image of the lung cancer patient to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation type and the corresponding predicted tumor mutation load of the lung cancer patient to be predicted, and the predicted gene mutation type and the corresponding predicted tumor mutation load are used as reference data for doctors to give targeted therapy and immunotherapy opinions.

2. The system of claim 1, wherein the collection module comprises:

the first acquisition unit is used for acquiring the computed tomography image of each lung cancer patient;

the second acquisition unit is used for acquiring pathological sections of the lung cancer patients and scanning the pathological sections to obtain the digital pathological images;

and the third acquisition unit is used for carrying out gene sequencing on each lung cancer patient to obtain the real gene mutation type and the real tumor mutation load corresponding to each real gene mutation type.

3. The system of claim 1, wherein the training module comprises:

a grouping unit, configured to use the computed tomography labeling image, the digital pathology labeling image, the true gene mutation type, and the true tumor mutation load of each lung cancer patient as a data set, and form a training set and a verification set according to each data set;

the training unit is connected with the grouping unit and used for taking the computed tomography labeling image and the digital pathology labeling image in the training set as input, taking the real gene mutation type and the real tumor mutation load as output and training to obtain a lung cancer gene mutation initial model;

and the verification unit is respectively connected with the grouping unit and the training unit and is used for carrying out parameter optimization adjustment on the lung cancer gene mutation initial model according to the verification set to obtain the lung cancer gene mutation prediction model.

4. The system of claim 3, wherein the training module further comprises a testing unit connected to the grouping unit and the verification unit, respectively, the testing unit comprising:

a first subunit, configured to form a test set according to each data set;

the second subunit is connected with the first subunit and is used for inputting each computed tomography labeling image and the digital pathology labeling image in the test set into the lung cancer gene mutation prediction model to obtain a corresponding gene mutation type prediction result and a tumor mutation load prediction result;

and the third subunit is connected with the second subunit and used for processing according to each gene mutation type prediction result and the corresponding real gene mutation type to obtain a first prediction accuracy rate and processing according to each tumor mutation load prediction result and the corresponding real tumor mutation load to obtain a second prediction accuracy rate which is used as reference data for doctors to give targeted therapy and immunotherapy opinions.

5. The system of claim 4, wherein the first prediction accuracy and the second prediction accuracy are obtained by processing a characteristic curve of the subject and an area under the curve.

6. The system of claim 4, wherein the training set, the validation set, and the testing set are formed by dividing all the data sets according to a predetermined ratio, the predetermined ratio being 3: 1: 1.

7. a cross-modal lung cancer gene mutation prediction method is applied to the lung cancer gene mutation prediction system of any one of claims 1 to 6, and specifically comprises the following steps:

step S1, the lung cancer gene mutation cross-modal prediction system respectively collects the computed tomography images, digital pathological images, real gene mutation types and real tumor mutation loads corresponding to each real gene mutation type of a plurality of lung cancer patients before receiving targeted therapy and immunotherapy;

step S2, the cross-modal prediction system of the lung cancer gene mutation labels the focus areas in the computed tomography image and the digital pathology image respectively to obtain a computed tomography labeling image and a digital pathology labeling image;

step S3, the lung cancer gene mutation cross-modal prediction system takes the computed tomography labeling image and the digital pathology labeling image of each lung cancer patient as input, takes the corresponding real gene mutation type and the real tumor mutation load as output, and trains to obtain a lung cancer gene mutation prediction model;

step S4, the cross-modal lung cancer gene mutation prediction system inputs the computed tomography image and the digital pathological image of the lung cancer patient to be predicted into the lung cancer gene mutation prediction model to obtain the predicted gene mutation type and the corresponding predicted tumor mutation load of the lung cancer patient to be predicted, and the predicted gene mutation type and the corresponding predicted tumor mutation load are used as reference data for doctors to give targeted therapy and immunotherapy opinions.

8. The method of claim 7, wherein the cross-modal prediction of lung cancer gene mutation is performed,

the step S1 includes:

step S11, the cross-modal lung cancer gene mutation prediction system collects the computed tomography images of the lung cancer patients;

step S12, the cross-modal lung cancer gene mutation prediction system acquires pathological sections of the lung cancer patients and scans the pathological sections to obtain the digital pathological images;

step S13, the cross-modal lung cancer gene mutation prediction system performs gene sequencing on each lung cancer patient to obtain the true gene mutation types and the true tumor mutation loads corresponding to each of the true gene mutation types.

9. The method for cross-modal prediction of lung cancer gene mutation according to claim 7, wherein the step S3 comprises:

step S31, the cross-modal lung cancer gene mutation prediction system takes the computed tomography labeling image, the digital pathology labeling image, the real gene mutation type and the real tumor mutation load of each lung cancer patient as a data set, and forms a training set and a verification set according to the data sets;

step S32, the cross-modal prediction system of the lung cancer gene mutation takes the computed tomography labeling image and the digital pathology labeling image in the training set as input, takes the real gene mutation type and the real tumor mutation load as output, and trains to obtain a lung cancer gene mutation initial model;

and step S33, the cross-modal lung cancer gene mutation prediction system performs parameter optimization adjustment on the lung cancer gene mutation initial model according to the verification set to obtain the lung cancer gene mutation prediction model.

10. The method for cross-modal prediction of lung cancer gene mutation of claim 9, wherein the step S3 further comprises a process for obtaining model prediction accuracy, specifically comprising:

step A1, the cross-modal system for lung cancer gene mutation prediction forms a test set according to each data set;

step A2, the cross-modal system for lung cancer gene mutation prediction inputs each computed tomography labeling image and the digital pathology labeling image in the test set into the lung cancer gene mutation prediction model to obtain a corresponding gene mutation type prediction result and a tumor mutation load prediction result;

step A3, the cross-modal lung cancer gene mutation prediction system obtains a first prediction accuracy rate by processing according to each gene mutation type prediction result and the corresponding real gene mutation type, and obtains a second prediction accuracy rate by processing according to each tumor mutation load prediction result and the corresponding real tumor mutation load, and the second prediction accuracy rate is used as reference data for doctors to give targeted therapy and immunotherapy opinions.

Images