CN117371038B - Distributed medical image artificial intelligence model evaluation method and device - Google Patents

Abstract

The present invention discloses a distributed medical imaging artificial intelligence model evaluation method and device, the method includes storing medical imaging evaluation data to respective client servers; registering pre-trained models and corresponding pre-trained model files on a central server; pre-setting evaluation tasks and their corresponding evaluation indicators, sending relevant pre-trained models to be evaluated to designated client servers, the client servers access pre-trained model files and evaluation indicators and evaluate the pre-trained models to be evaluated based on the medical imaging evaluation data stored therein, and after the evaluation is completed, sending the comprehensive evaluation result information to the evaluator, the corresponding model developer and the designated hospital for review and analysis. The present invention is based on the idea of distributed and privacy computing, realizes the computable invisibility of data, fully protects patient privacy, can better deal with the long-tail distribution problem of medical imaging data, and improves the robustness and reliability of medical imaging artificial intelligence models in real medical scenarios.

Description

A distributed medical imaging artificial intelligence model evaluation method and device

Technical Field

The present invention relates to the field of intelligent medical technology, and in particular to a distributed medical imaging artificial intelligence model evaluation method and device.

Background technique

Artificial intelligence medical imaging systems have achieved good results in the auxiliary diagnosis, prediction and prognosis of many diseases. However, the process from research results to implementation of artificial intelligence medical imaging systems faces a series of unique challenges such as technology, ethics and supervision. Medical data has characteristics different from data in other fields: high privacy protection requirements, strict legal and regulatory supervision, high difficulty in labeling, obvious data island phenomenon, small data volume, obvious long-tail distribution phenomenon, etc. The European Union Health Technology Assessment Network EUnetHTA has developed 9 common medical and health technology assessment model fields; based on the EUnetHTA framework, Odense University Hospital in Denmark has developed 9 value assessment model fields for medical imaging artificial intelligence MAS-AI. A significant difference between the MAS-AI framework and the EUnetHTA framework is that it emphasizes "AI model development, performance and verification" in the "technology" dimension. These guiding frameworks provide a higher-level reference, but do not provide specific evaluation model development practices. The performance evaluation of medical imaging artificial intelligence systems is a consensus challenge in the field of artificial intelligence medical imaging.

Traditional AI model evaluation methods are generally centralized, that is, model developers use evaluation data locally to conduct model evaluation, and then deploy the model in the production environment. Factors such as changes in data distribution, data quality, and environmental changes in the production environment will lead to model performance degradation. This situation is particularly evident in the operation of AI medical imaging systems, because in clinical scenarios, medical imaging data has a long-tail distribution characteristic, which is mainly manifested in two aspects: ① In medical imaging data, some common diseases may appear more frequently, while many rare diseases appear less frequently; ② Larger hospitals accumulate more medical imaging data, while relatively smaller hospitals accumulate less data, and these data cannot be shared and circulated between different hospitals. The long-tail distribution of data brings challenges to AI medical imaging analysis. Effectively dealing with the long-tail distribution of medical imaging data is very important for improving the clinical utility of medical imaging AI models. In dealing with the problem of long-tail distribution of data, special methods or techniques are generally required to process rare disease data or small sample data sets, so that the model can also perform well in these cases to ensure the reliability and stability of the model in clinical scenarios. In addition, medical imaging data has extremely high privacy protection and regulatory requirements and cannot be shared or circulated between different medical institutions, which makes it difficult to use traditional centralized methods to conduct medical imaging artificial intelligence model evaluation.

Summary of the invention

In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

The first aspect of the present invention provides a distributed medical imaging artificial intelligence model evaluation method, comprising: each hospital uploads and stores private medical imaging evaluation data to its own client server; each model developer registers a pre-trained model on a central server, and uploads and stores the pre-trained model files corresponding to each pre-trained model to the central server; the evaluator presets an evaluation task and an evaluation indicator corresponding to the evaluation task on the central server, sends the pre-trained model to be evaluated related to the evaluation task to the client server of the designated hospital, and then starts the evaluation task, the client server of the designated hospital accesses the pre-trained model file and the evaluation indicator and evaluates the pre-trained model to be evaluated based on the medical imaging evaluation data stored therein; after the evaluation is completed, the central server sends the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis.

Furthermore, before each hospital uploads and stores the private medical image evaluation data to its own client server, it also includes pre-processing the medical image evaluation data of the hospital.

Furthermore, each hospital only has management authority over its own private medical imaging assessment data.

Furthermore, the pre-trained model file includes the model name, model type and backbone network structure used in training of the pre-trained model.

Furthermore, the pre-trained model file supports multiple formats when it is uploaded to the central server. Furthermore, the central server and the client server communicate via a virtual private network VPN, and the central server and the client server are respectively designed with a B/S architecture.

Furthermore, the evaluator reviews the pre-trained models registered by each model developer on the central server.

Furthermore, the comprehensive evaluation result information includes the evaluation data used, the test parameters and the evaluation result index values.

Furthermore, the top three pre-trained models in terms of performance in the comprehensive evaluation results of pre-trained models were selected, and federated learning was initiated separately, and supervised training of the pre-trained models was performed using the local medical imaging optimization data of each hospital.

A second aspect of the present invention provides a distributed medical image artificial intelligence model evaluation device, the device comprising:

The data management module is used by each hospital to upload and store private medical imaging evaluation data to their respective client servers;

The model management module is used by each model developer to register the pre-trained model on the central server and upload the pre-trained model file corresponding to the pre-trained model to the central server;

An evaluation management module is used for the evaluator to preset an evaluation task and an evaluation index corresponding to the evaluation task on a central server, and to initiate the evaluation task after sending a pre-trained model to be evaluated related to the evaluation task to a client server of a designated hospital. The client server of the designated hospital accesses the pre-trained model file and the evaluation index and evaluates the pre-trained model to be evaluated based on the medical image evaluation data stored therein;

The visualization module is used for sending the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis after the evaluation is completed.

Beneficial effects of the present invention

The distributed medical imaging artificial intelligence model evaluation method and device provided by the present invention have the following three advantages:

(1) Using private medical imaging data from real hospital scenarios to evaluate medical imaging artificial intelligence models, while ensuring that data does not leave the hospital, the data can be calculated but not seen, fully protecting the privacy of patients;

(2) Based on the idea of privacy computing, the value of real-scene data is fully utilized. Through the feedback of real data, the model performance is optimized and the clinical utility of medical imaging models in medical environments is improved;

(3) Better deal with the long-tail distribution problem of medical imaging data, and further improve the robustness and reliability of medical imaging artificial intelligence models in real medical scenarios. Even if some smaller medical institutions have less accumulated medical imaging data, they can benefit from the final medical imaging artificial intelligence model through the distributed medical imaging artificial intelligence model evaluation method proposed in the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the process flow of the distributed medical imaging artificial intelligence model evaluation method according to the present invention;

FIG2 is a schematic diagram of the functional structure of the distributed medical imaging artificial intelligence model evaluation device according to the present invention.

Detailed ways

In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts in the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

The display screen is used to display the user interface of each application.

In addition, those skilled in the art will appreciate that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

Embodiment 1

As shown in FIG1 , this embodiment provides a distributed medical image artificial intelligence model evaluation method, comprising the following steps:

S1, each hospital uploads and stores private medical imaging evaluation data to its own client server;

It should be noted that the hospital is always the holder of the medical imaging evaluation data it owns. The hospital's private medical imaging evaluation data is only stored on the client server within each hospital. The hospital only has management authority over its own private medical imaging evaluation data. The hospital can access the pre-trained model files issued by the central server and use the private medical imaging evaluation data to evaluate the performance of the issued pre-trained model; in addition, before uploading the private medical imaging evaluation data to its own client server, each hospital can also pre-process the medical imaging evaluation data. The above-mentioned medical imaging evaluation data includes the type of medical imaging (such as CT, X-ray), data volume, organs involved in the imaging, and basic information on related diseases.

S2, each model developer registers a pre-trained model on the central server, and uploads and stores the pre-trained model files corresponding to each pre-trained model to the central server;

It should be noted that the pre-trained model file includes the model name, model type, backbone network structure used in training, etc. of the pre-trained model. The pre-trained model file supports multiple formats when uploading to the central server. There are two types of model developers, one is the developer of the pre-trained model, and the other is the developer of a specific task model based on the pre-trained model. Each model developer has no authority to access the medical imaging evaluation data in the client server of each hospital throughout the process.

S3, the evaluator presets the evaluation task and the evaluation index corresponding to the evaluation task on the central server, sends the pre-trained model to be evaluated related to the evaluation task to the client server of the designated hospital, and then starts the evaluation task. The client server of the designated hospital accesses the pre-trained model file and the evaluation index and evaluates the pre-trained model to be evaluated based on the medical image evaluation data stored therein;

It should be noted that the central server and the client server communicate through a virtual private network VPN. The central server and the client server respectively adopt a B/S (browser/server mode) architecture design, which consists of independent and separate front and back ends. This highly modular design is easy to maintain and deploy. When the model evaluation is initialized, it is assumed that there is an evaluator user. The evaluator has administrator privileges and can manage the registration of hospitals and model developers and the corresponding privileges of the two. The evaluator can see the medical imaging evaluation data stored by the hospital on the client server and the pre-trained model registered by the model developer on the central server and its corresponding pre-trained model file. After each model developer registers the pre-trained model it has developed on the central server, the evaluator reviews the above pre-trained model. After the review is passed, the evaluator can carry out the performance test of the pre-trained model to be evaluated in its management interface. In this embodiment, by setting up an evaluator, it can be ensured that the evaluation of each pre-trained model is carried out fairly, impartially and independently.

S4, after the evaluation is completed, the central server sends the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis.

It should be noted that the comprehensive evaluation result information of the model includes the evaluated model file, the evaluation data used, the test parameters and the evaluation result index values. Each hospital can only obtain the individual evaluation results of the pre-trained model to be evaluated based on its private medical imaging evaluation data. The model developer can obtain the individual evaluation results of each hospital, and can also obtain the average evaluation results of the joint evaluation of multiple hospitals.

Furthermore, in order to optimize the performance of the pre-trained model, the top three pre-trained models in the comprehensive evaluation results of the pre-trained model were selected, and federated learning was initiated separately. The pre-trained models were supervised and trained using the local medical imaging optimization data of each hospital (this part of the data is not the medical imaging evaluation data). After obtaining the fine-tuned pre-trained model, the hospital's medical imaging evaluation data was used to evaluate it again, continuously promoting the development of the pre-trained model performance. In the daily operation of the pre-trained model, a data flywheel is formed, that is, new patient data is added to assist the evaluation and fine-tuning of the pre-trained model, and the fine-tuned pre-trained model can assist more clinical use.

Regarding the evaluation result index value, this embodiment takes a medical image classification problem as an example for explanation. Assuming that a medical image is used to determine whether a patient has a certain disease, that is, to determine the negative and positive of the disease, then this is a classification problem. The result of the classification problem can be represented by a confusion matrix, as shown in Table 1 below:

Table 1

in,

TP: True Positive, the true value is positive, the predicted value is positive, and the prediction is correct;

FP: False Positive, the true value is negative, the predicted value is positive, and the prediction is wrong;

FN: False Negative, the true value is positive, the predicted value is negative, and the prediction is wrong;

TN: True Negative, the true value is negative, the predicted value is negative, and the prediction is correct.

Based on Table 1, the evaluation result index values used in this embodiment are as follows:

Accuracy: The percentage of correct predictions in the total samples.

Accuracy=(TP+TN)/(TP+TN+FP+FN);

Precision: The probability of samples predicted to be positive being actually positive.

Precision = TP/(TP+FP);

Recall: The probability of samples predicted to be positive among samples that are actually positive.

Recall = TP/(TP+FN);

Specificity: Also known as True Negative Rate (TNR), it refers to the proportion of true negative samples in actual negative samples.

TNR = TN / (FP + TN);

Sensitivity: Also known as True Positive Rate (TPR), it refers to the proportion of true positive samples in actual positive samples.

TPR = TP/(TP+FN);

F1: It is the harmonic mean of precision and recall. Its purpose is to find a balance between precision and recall. It takes both into account and maximizes them at the same time.

F1＝2×(Precision×Recall)/(Precision+Recall);

AUC (Area Under Curve) is the area under the receiver operating characteristic (ROC) curve and the coordinate axis. The value range of AUC is between 0.5 and 1. The closer AUC is to 1, the higher the authenticity of the detection method. When it is equal to 0.5, the authenticity is the lowest and it has no application value.

Traditionally, the above 7 evaluation indicators are widely used in non-distributed scenarios. This embodiment adopts a distributed mechanism to calculate the above evaluation indicators. Taking the AUC indicator as an example, hospitals A, B, and C used their own medical imaging evaluation data (these data are data for the same disease) to perform an evaluation task on the pre-trained model (called model A) registered by model developer A on the central service. After the task is executed, each hospital obtains an AUC value, which are AUC _a , AUC _b , and AUC _c respectively. The comprehensive evaluation result AUC _{mol_a} for model A finally obtained by the evaluator is as follows:

Similarly, hospitals A, B, and C use their own medical imaging evaluation data to perform evaluation tasks on the pre-trained model registered by model developer B (called model B). After the task is completed, each hospital obtains an AUC value, which are AUC _a ′, AUC _b ′, and AUC _c ′ respectively. The comprehensive evaluation results of model B obtained by the evaluator are as follows:

The distributed evaluation mechanism used in this embodiment, on the one hand, ensures that hospital patient data "does not leave the hospital", thus protecting the privacy of patients; on the other hand, it provides real-scene data for the evaluation of medical imaging artificial intelligence models, and provides great value for further optimizing the performance of medical imaging artificial intelligence models. This distributed evaluation method is of great significance to the development of the entire medical imaging artificial intelligence application.

Embodiment 2

As shown in FIG2 , an embodiment of the present invention further provides a distributed medical image artificial intelligence model evaluation device, comprising:

The data management module is used by each hospital to upload and store private medical imaging evaluation data to their own client servers. The main function of this module is to meet the needs of hospitals in managing their medical imaging evaluation data.

The model management module is used by each model developer to register the pre-trained model on the central server and upload the pre-trained model file corresponding to the pre-trained model to the central server. The main function of this module is to meet the needs of model developers to manage their pre-trained models;

An evaluation management module is used for the evaluator to preset the evaluation task and the evaluation index corresponding to the evaluation task on the central server, and to initiate the evaluation task after sending the pre-trained model to be evaluated related to the evaluation task to the client server of the designated hospital. The client server of the designated hospital accesses the pre-trained model file and the evaluation index and evaluates the pre-trained model to be evaluated based on the medical image evaluation data stored therein. The main function of this module is to satisfy the neutral evaluator to manage the evaluation task and the corresponding evaluation index;

The visualization module is used for the central server to send the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis after the evaluation is completed. The main function of this module is to monitor the evaluation task process and compare and analyze the evaluation results.

Furthermore, the device also includes a system management module, the main function of which is to meet the evaluator's system management needs such as account management and authority management.

Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (10)

1. A distributed medical imaging artificial intelligence model evaluation method, characterized by comprising: Each hospital uploads and stores private medical imaging evaluation data to its own client server; Each model developer registers a pre-trained model on the central server, and uploads and stores the pre-trained model files corresponding to each pre-trained model to the central server; The evaluator presets the evaluation task and the evaluation index corresponding to the evaluation task on the central server, sends the pre-trained model to be evaluated related to the evaluation task to the client server of the designated hospital, and then starts the evaluation task. The client server of the designated hospital accesses the pre-trained model file and the evaluation index and evaluates the pre-trained model to be evaluated based on the medical image evaluation data stored therein; After the evaluation is completed, the central server will send the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis; Among them, the comprehensive evaluation result information of the pre-trained model to be evaluated includes the evaluated model file, the evaluation data used, the test parameters and the evaluation result indicator value. Each hospital can only obtain the individual evaluation results of the pre-trained model to be evaluated based on its private medical imaging evaluation data. The model developer can obtain the individual evaluation results of each hospital, and can also obtain the average evaluation results of multiple hospitals' joint evaluation. 2. The model evaluation method as described in claim 1 is characterized in that before each hospital uploads and stores the private medical imaging evaluation data to its own client server, it also includes pre-processing the medical imaging evaluation data of the hospital. 3. The model evaluation method as described in claim 1 is characterized in that each hospital only has management authority over its own private medical imaging evaluation data. 4. The model evaluation method as described in claim 1 is characterized in that the pre-trained model file includes the model name, model type and backbone network structure used in training of the pre-trained model. 5. The model evaluation method as described in claim 1 is characterized in that the pre-trained model file supports multiple formats when uploading to the central server. 6. The model evaluation method as described in claim 1 is characterized in that the central server and the client server communicate with each other through a virtual private network VPN, and the central server and the client server respectively adopt a B/S architecture design. 7. The model evaluation method as described in claim 1 is characterized in that the evaluator reviews the pre-trained models registered by each model developer on the central server. 8. The model evaluation method as described in claim 1 is characterized in that the comprehensive evaluation result information includes the evaluation data used, test parameters and evaluation result indicator values. 9. The model evaluation method as described in claim 1 is characterized in that the pre-training models ranked in the top three in terms of performance in the comprehensive evaluation results of the pre-training models are selected, and federated learning is initiated separately, and the pre-training models are supervised and trained using the local medical imaging optimization data of each hospital. 10. A distributed medical image artificial intelligence model evaluation device, characterized by comprising: The data management module is used by each hospital to upload and store private medical imaging evaluation data to their respective client servers; The model management module is used by each model developer to register the pre-trained model on the central server and upload the pre-trained model file corresponding to the pre-trained model to the central server; An evaluation management module is used for the evaluator to preset an evaluation task and an evaluation index corresponding to the evaluation task on a central server, and to initiate the evaluation task after sending a pre-trained model to be evaluated related to the evaluation task to a client server of a designated hospital. The client server of the designated hospital accesses the pre-trained model file and the evaluation index and evaluates the pre-trained model to be evaluated based on the medical image evaluation data stored therein; The visualization module is used for sending the comprehensive evaluation result information of the pre-trained model to be evaluated to the evaluator, the corresponding model developer and the designated hospital for review and analysis after the evaluation is completed. Among them, the comprehensive evaluation result information of the pre-trained model to be evaluated includes the evaluated model file, the evaluation data used, the test parameters and the evaluation result indicator value. Each hospital can only obtain the individual evaluation results of the pre-trained model to be evaluated based on its private medical imaging evaluation data. The model developer can obtain the individual evaluation results of each hospital, and can also obtain the average evaluation results of multiple hospitals' joint evaluation.