CN112506797B - A performance testing method for medical image recognition system - Google Patents

Abstract

The invention discloses a performance test method of a medical image recognition system, which comprises the following steps that 1) a multi-class image test data generation module comprises a confrontation sample generation network and an entity and background recombination method; 2) the multi-angle test module is based on system stability, reliability and safety performance; 3) and a model decision evaluation module. The invention realizes the generation of multi-class image test data and multi-angle complete system test, finally completes the decision evaluation of the medical image recognition system, and has wide future application prospect.

Description

A performance test method for medical image recognition system

technical field

The invention belongs to the technical field of performance analysis of a medical image recognition system, in particular to a performance testing method for a medical image recognition system.

Background technique

Medical image recognition system plays an important role in clinical medical diagnosis. It has greatly changed the clinical diagnosis mode and promoted the development of clinical medicine. Intelligent medical image recognition is a process of analyzing and processing scanned images and surgical videos of commonly used medical imaging technologies such as X-ray, computed tomography, and magnetic resonance imaging based on artificial intelligence technology. Reconstruction and registration, intelligent surgical video analysis, etc. At present, the research in this field has made some progress and is gradually moving towards clinical application. Therefore, the evaluation and testing of the performance of the medical image recognition system is particularly important for the development of clinical medicine in the future. For the first time, FERET set a performance benchmark for the recognition algorithm and defined a series of evaluation standards, which greatly promoted the development of recognition technology. profound influence. Although there are some test schemes for general image recognition systems, no test schemes for medical image recognition systems have been proposed. Moreover, since the recognition technology at that time was not yet mature, the recognition systems involved in the FERET evaluation were mostly prototype systems in university laboratories, and the recognition effect was not very satisfactory.

In recent years, there has been a growing demand for test methods that analyze recognition models and automate performance analysis. With the rapid development of deep learning technology, the performance indicators of medical image recognition systems have also been rapidly improved, greatly improving the recognition efficiency, so how to test the performance of these models needs to be solved urgently. For the medical image recognition system in the test, it is necessary to consider the visual authenticity of the generated test image, so the adversarial sample generation network and the entity and background reorganization method are proposed to fully ensure the authenticity of the generated samples; Therefore, a multi-angle test scheme is proposed, and the adversarial samples are applied to medical images to achieve better analysis of the medical recognition model.

SUMMARY OF THE INVENTION

The present invention provides a performance testing method for a medical image recognition system to solve the problems in the prior art.

In order to achieve the above object, the present invention adopts the following technical solutions:

A performance testing method for a medical image recognition system, the performance testing method includes: a multi-category image test data generation module, wherein the multi-category image test data generation module includes an adversarial sample generation network and an entity and background reorganization method; Test module, the multi-angle test module includes performance test, reliability test and safety test; decision evaluation module, the decision evaluation module analyzes the input test results, judges the model performance, and gives a detailed test report ;

The network inputs a set of pictures to be classified and recognized, and the pictures are input into the multi-category image test data generation module for the first time, and after image augmentation, the model is input for classification, and the classification results are input into the multi-angle test module, which is a multi-angle test module. The learning results of the model are tested, and the results are transmitted to the decision evaluation module. The decision evaluation module analyzes the input test results, judges the performance of the model, and gives a detailed test report.

Further, the adversarial sample generation network and the entity and background reorganization method include adversarial augmentation using multi-loss mixed adversarial camouflage,

表示为：Multiple loss functions

Expressed as:

表示对抗性损失、

表示用于样式生成的样式损失、

表示用于保存源图像内容的内容损失、

表示用于确保扩展样本的平滑度的平滑度损失；Among them: λ represents the strength of confrontation,

represents the adversarial loss,

represents the style loss used for style generation,

Indicates the content loss used to preserve the content of the source image,

represents the smoothness loss used to ensure smoothness of the expanded samples;

The user defines the existing image, the target attack area and the expected target style, generates the desired style in the desired area, adds additional physical fitness training to the generated extended samples at each step;

The style distance between two images is defined by the difference in the style representation of the two images,

为特征距离，l是风格层特征，S_l是提取风格表示的风格层集，

是风格样式的特征提取器，

是从

的一组样式层提取的深层特征的Gram矩阵，x^s是风格参考图像，x′是生成的对抗样本；in:

is the feature distance, l is the style layer feature, S _l is the style layer set for extracting style representation,

is a style-style feature extractor,

From

A set of Gram matrices of deep features extracted by the style layer, x ^s is the style reference image, and x′ is the generated adversarial example;

所生成参考样式中的增强图像的内容与原始图像的内容非常不同；具体如下，style loss for style generation

The content of the enhanced image in the generated reference style is very different from the content of the original image; as follows,

是内容损失，t是内容层特征，c_t是提取内容表示的内容层集，

是内容层的特征提取器，x是原始图像，x′是生成的对抗样本；in:

is the content loss, t is the content layer feature, c _t is the content layer set from which the content representation is extracted,

is the feature extractor of the content layer, x is the original image, and x′ is the generated adversarial example;

The smoothness of the enhanced image is improved by reducing the variation between adjacent pixels; for the enhanced image, the smoothness loss is defined as follows,

Among them: x _{i, j} is the pixel value at the coordinate of the adversarial sample (i, j), x _{i+1, j} is the pixel value at the coordinate of the original image (i+1, j), x _{i, j+1} is the original image Pixel values at the image (i,j+1) coordinates:

使用以下交叉熵损失：For adversarial loss

Use the following cross-entropy loss:

Among them: p _yadv () and p _y () are the target model F respectively (F refers to the objective function of the general machine model, for example, the objective function F of vgg is fc8, from which the probability output corresponding to 1000 classes can be obtained) to the label yadv (the class of the adversarial example) and the probability output of y (the class of the original image).

Real-world conditions are introduced into the generation of augmented examples as follows:

是变换的集合；通过根据原始图像x和背景图像o，生成的增强样本对于人类观察者来说基本是合法的；where: o is a random background image sampled in the physical world, T is a random transformation for rotation, resizing and color shifting,

is the set of transformations; by relying on the original image x and the background image o, the generated augmented samples are basically legal to a human observer;

The target background reorganization and augmentation uses the segmentation algorithm Mask R-CNN to segment the target from the background, uses the interpolation algorithm to supplement the blank part of the background with pixels, and finally randomly combines the target and the background to achieve image augmentation.

Further, the performance test in the multi-angle test module includes different angles: the recognition accuracy rate Accuracy judgment, the recognition loss value Loss judgment judgment and the transformation relationship judgment; the accuracy rate and the loss value Loss judgment are both before and after augmentation. The difference between the accuracy rate of the model output Accuracy and the recognition loss value Loss is obtained by subtracting the recognition accuracy difference percentage Δacc before and after expansion, and the difference percentage of recognition loss before and after expansion Δloss;

的被图像识别系统分类标签，S_i为原测试图像

的置信分数；C_i′为结合蜕变关系利用

合成的新测试图像

的分类标签，S_i′为结合蜕变关系利用

合成的新测试图像

的置信分数，那么蜕变关系表述为：The transformation test is defined as: C _i is the original test image

is classified by the image recognition system, and S _i is the original test image

The _confidence score of

Synthesized new test image

The classification label of , S _i ′ is used in combination with the metamorphic relationship

Synthesized new test image

The confidence score of , then the transformation relationship is expressed as:

C _i =C' _i andΔS=|S _i -S' _i |<c (7)

Where: c is a hyperparameter, 0<c<100, c is set to 50, and ΔS is the difference between the confidence scores before and after expansion.

Further, the reliability test in the multi-angle test module is a robustness test. Under the condition that the original image x satisfies the confidence level guarantee, it can be immune to attacks within the radius R of the norm sphere:

指任意，ε为引入的噪声，B(x；R)为噪声集合，R为范数球半径，R为一个无线接近0的值，x是原始图像；Among them: z() is the loss function, g() is the objective function to be optimized,

Refers to any, ε is the noise introduced, B(x; R) is the noise set, R is the radius of the norm sphere, R is a wireless value close to 0, and x is the original image;

The final robustness accuracy (robacc) is defined as:

The security test in the multi-angle test module is a model invariance test. A random image is selected, a perturbation of a pixel is selected using one of the four methods described below, and the sensitivity of the network to this perturbation is measured. The first method is The Crop method randomly selects a square in the original image and resize the square to 224x224px, then we translate the square by one pixel diagonally to create a second image, which is translated by A single pixel is different from the first image; the second method is the "Embedding" method, which first reduces the image to a minimum size of 100px, while maintaining the aspect ratio, and embeds it at a random position within the 224x224px image , while filling the rest of the image with black pixels, then shifting the embedding position by a single pixel, creating two identical images again, up to a single pixel shift; in the third method, the image is first scaled down to the smallest size is 100px while maintaining aspect ratio and embed it at random position inside 224x224px image, then use simple repair algorithm (each black pixel is replaced by weighted average of non-black pixels in its neighborhood), fourth The first method is the same as the second protocol, first shrink the image so that its minimum size is 100px, but instead of moving the embed position, we keep the embed position the same and change the size of the embed image by a single pixel (for example, from size 100x100px Change to size 101x101px).

Further, in the security test, two methods are used to measure the sensitivity as the invariance test of the model. The first one is called P (Top-1 Change), which is the change of the TOP-1 prediction of the network after single-pixel perturbation. the probability of Absolute value change (ie the class with the highest probability in the first of the two frames).

Further, the decision evaluation module analyzes the input test results and judges the performance of the model [accuracy after expansion, recognition loss after Loss expansion, poor recognition accuracy before and after Δacc expansion, poor recognition loss before and after Δloss expansion, and CR model is robust. Rodiness (characterized by robacc), difference in confidence scores before and after ΔS expansion, TOP-1 prediction probability of P(Top-1Change) network changes after single-pixel perturbation, MAC mean absolute change)], and gives detailed In the test report, when we compare the performance of multiple recognition models, a large number of individual performance indicators are often too complicated for users, making it difficult for users to make reasonable judgments. Therefore, we consider the comprehensive impact of different indicators on the recognition system. In the design of performance indicators, a comprehensive performance indicator CM (Composite Value) is defined to reflect the comprehensive performance of different identification systems; the formula is as follows:

Among them: CM _i represents the comprehensive performance value of the ith identification system, ω _j represents the weight of the jth performance index value of the cloud service, max(M _j ) represents the maximum value of the jth performance index in multiple identification systems, min (M _j ) represents the minimum value of the j-th performance index in multiple recognition systems, M _ij represents the j-th performance index value of the ith recognition system, and N represents the total number of performance index values of the recognition system. By using formula (2 *max(M _j )-M _ij )/(2*max(M _j )-min(M _j )) normalizes the value of M _ij to the [0, 1] interval. It can be seen that the larger the value of CM, the better the comprehensive performance of the recognition system.

For some identification system performance indicators, such as Loss, P(Top-1 Change), MAC, etc., the smaller the value M _ij is, the larger the CM comprehensive performance index value is, and the values of these performance indicators can be directly substituted into the above formula, For indicators such as recognition accuracy, the larger the value, the better the recognition performance, but directly substituting the formula will cause the CM value to decrease, which is not in line with expectations, so we need to process the values of these performance indicators, using ( 1-M _ij ) to replace the Mi _ij value in the formula.

Further, the decision evaluation module will eventually output the results of the multi-angle test module test, and generate a corresponding test report form, as shown in Table 1-3. Due to the different requirements of different task scenarios, the test system will also give corresponding suggestions.

Compared with the prior art, the present invention has the following beneficial effects:

The invention realizes multi-category image test data generation and multi-angle complete system test, finally completes the decision evaluation of the medical image recognition system, and has wide application prospects in the future.

Description of drawings

Fig. 1 is the frame flow chart of the present invention;

Fig. 2 is a flow chart of confrontation augmentation in the present invention;

Fig. 3 is a flow chart of target background reorganization and augmentation in the present invention;

Fig. 4 is a decision evaluation module in the present invention.

Detailed ways

The present invention will be further described below in conjunction with the examples.

A performance test method for a medical image recognition system, as shown in Figure 1, the performance test method includes: a multi-category image test data generation module, a multi-angle test module and a decision evaluation module, the multi-category image test data generation module The module includes an adversarial sample generation network and entity, and a background reorganization method; the multi-angle testing module includes performance testing, reliability testing and security testing; a decision evaluation module, the decision evaluation module analyzes the input test results and judges Model performance, and give a detailed test report;

The network inputs a set of pictures to be classified and recognized, and the pictures are input into the multi-category image test data generation module for the first time, and after image augmentation, the model is input for classification, and the classification results are input into the multi-angle test module, which is a multi-angle test module. The learning results of the model are tested, and the results are transmitted to the decision evaluation module. The decision evaluation module analyzes the input test results, judges the performance of the model, and gives a detailed test report.

是对抗强度λ与对抗性损失

的乘积、用于样式生成的样式损失

用于保存源图像内容的内容损失

和用于确保扩展样本的平滑度的平滑度损失

的组合。The adversarial sample generation network and entity and background reorganization method, considering the characteristics of medical images and the visual authenticity of the generated test images, use the adversarial sample generation joint entity and background reorganization scheme. Adversarial augmentation uses multi-loss adversarial camouflage, a technique that can generate new augmented images that look legitimate to a human observer without relying on large amounts of data to train generative networks. Our goal is to develop a mechanism to generate extended samples with custom styles, image enhancement using style transformation techniques, and image concealment using adversarial attack techniques. The final multiple loss function

is the adversarial strength λ and the adversarial loss

The product of , the style loss for style generation

Content loss for saving source image content

and the smoothness loss used to ensure smoothness of the expanded samples

The combination.

表示为：Multiple loss functions

Expressed as:

表示对抗性损失、

表示用于样式生成的样式损失、

表示用于保存源图像内容的内容损失、

表示用于确保扩展样本的平滑度的平滑度损失；Among them: λ represents the strength of confrontation,

represents the adversarial loss,

represents the style loss used for style generation,

Indicates the content loss used to preserve the content of the source image,

represents the smoothness loss used to ensure smoothness of the expanded samples;

As shown in Figure 2, an overview of adversarial augmentation methods is shown. The user defines the existing image, the target attack area, and the expected target style to generate the desired style in the desired area, as shown on the right side of Figure 2. To make the expanded samples robust to various environmental conditions (including lighting, rotation, etc.), additional physical adaptation training is added to the generated expanded samples at each step;

The style distance between two images is defined by the difference in the style representation of the two images,

为特征距离，l是风格层特征，S_l是提取风格表示的风格层集，

是风格样式的特征提取器，

是从

的一组样式层提取的深层特征的Gram矩阵，x^s是风格参考图像，x′是生成的对抗样本；in:

is the feature distance, l is the style layer feature, S _l is the style layer set for extracting style representation,

is a style-style feature extractor,

From

A set of Gram matrices of deep features extracted by the style layer, x ^s is the style reference image, and x′ is the generated adversarial example;

所生成参考样式中的增强图像的内容与原始图像的内容非常不同；具体如下，style loss for style generation

The content of the enhanced image in the generated reference style is very different from the content of the original image; as follows,

是内容损失，t是内容层特征，c_t是提取内容表示的内容层集，

是内容层的特征提取器，x是原始图像，x′是生成的对抗样本；in:

is the content loss, t is the content layer feature, c _t is the content layer set from which the content representation is extracted,

is the feature extractor of the content layer, x is the original image, and x′ is the generated adversarial example;

The smoothness of the enhanced image is improved by reducing the variation between adjacent pixels; for the enhanced image, the smoothness loss is defined as follows,

Where: x′ _{i, j} is the pixel value at the coordinate of the adversarial sample (i, j), x _{i+1, j} is the pixel value at the coordinate of the original image (i+1, j), x _{i, j+1} is Pixel values at the (i,j+1) coordinates of the original image:

使用以下交叉熵损失：For adversarial loss

Use the following cross-entropy loss:

Among them: p _yadv () and p _y () are the target model F respectively (F refers to the objective function of the general machine model, for example, the objective function F of vgg is fc8, from which the probability output corresponding to 1000 classes can be obtained) to the label yadv (the class of the adversarial example) and the probability output of y (the class of the original image).

To make adversarial image samples achievable in the real world, we model real-world conditions during the generation of augmented samples. Since real-world environments often involve fluctuations in conditions such as viewpoint movement, image noise, and other natural transitions, we use a series of adjustments to accommodate these different conditions. In particular, we use a technique similar to expected over-transformation (EOT). Our goal is to improve the adaptability of augmented samples to different physical conditions. Therefore, we consider transformations used to simulate fluctuations in physical world conditions, including rotation, scaling, color shifting (to simulate lighting changes), and random backgrounds. Here, real-world conditions are introduced into the generation of augmented examples as follows:

是变换的集合；通过根据原始图像x和背景图像o，生成的增强样本对于人类观察者来说基本是合法的；where: o is a random background image sampled in the physical world, T is a random transformation for rotation, resizing and color shifting,

is the set of transformations; by relying on the original image x and the background image o, the generated augmented samples are basically legal to a human observer;

Target background reorganization and augmentation The segmentation algorithm Mask R-CNN is used to segment the target from the background, and the interpolation algorithm is used to supplement the blank part of the background with pixels. Finally, the target and background are randomly combined to achieve image augmentation. The overall method framework is shown in Figure 3. Show.

The performance test in the multi-angle test module includes different angles: recognition accuracy rate Accuracy judgment judgment, recognition loss value Loss judgment judgment and transformation relationship judgment; Accuracy rate and loss value Loss judgment are all output from the model before and after augmentation The difference between the recognition accuracy before and after expansion, Δacc, and the difference in recognition loss before and after expansion, Δloss, are obtained by subtracting the accuracy rate Accuracy and the recognition loss value Loss;

的被图像识别系统分类标签，S_i为原测试图像

的置The transformation test is defined as: C _i is the original test image

is classified by the image recognition system, and S _i is the original test image

set

合成的新测试图像

的分类标签，S_i′为结合蜕变关系利用

合成的新测试图像

的置信分数，那么蜕变关系表述为：trust score; C _i ′ is used for the combined metamorphosis relationship

Synthesized new test image

The classification label of , S _i ′ is used in combination with the metamorphic relationship

Synthesized new test image

The confidence score of , then the transformation relationship is expressed as:

C _i =C ^′ _i andΔS=|S _i -S ^′ _i |<c (7)

Where: c is a hyperparameter, 0<c<100, c is set to 50, and ΔS is the difference between the confidence scores before and after expansion.

The reliability test in the multi-angle test module is a certified robustness test. Under the condition that the original image x satisfies the confidence guarantee, it is immune to attacks within the radius R of the norm sphere:

指任意，ε为引入的噪声，B(x；R)为噪声集合，R为范数球半径，R为一个无线接近0的值，x是原始图像；Among them: z() is the loss function, g() is the objective function to be optimized,

Refers to any, ε is the noise introduced, B(x; R) is the noise set, R is the radius of the norm sphere, R is a wireless value close to 0, and x is the original image;

The final robustness accuracy (robacc) is defined as:

The security test in the multi-angle test module is a model invariance test. A random image is selected, a perturbation of a pixel is selected using one of the four methods described below, and the sensitivity of the network to this perturbation is measured. The first method is The Crop method randomly selects a square in the original image and resize the square to 224x224px, then we translate the square by one pixel diagonally to create a second image, which is translated by A single pixel is different from the first image; the second method is the "Embedding" method, which first reduces the image to a minimum size of 100px, while maintaining the aspect ratio, and embeds it at a random position within the 224x224px image , while filling the rest of the image with black pixels, then shifting the embedding position by a single pixel, creating two identical images again, up to a single pixel shift; in the third method, the image is first scaled down to the smallest size is 100px while maintaining aspect ratio and embed it at random position inside 224x224px image, then use simple repair algorithm (each black pixel is replaced by weighted average of non-black pixels in its neighborhood), fourth The first method is the same as the second protocol, first shrink the image so that its minimum size is 100px, but instead of moving the embed position, we keep the embed position the same and change the size of the embed image by a single pixel (for example, from size 100x100px Change to size 101x101px).

In the security test, two methods are used to measure the sensitivity as the invariance test of the model. The first one is called P (Top-1 Change), which is the probability that the TOP-1 prediction of the network will change after a single pixel perturbation; The second, called "Mean Absolute Change" (MAC), measures the mean absolute change in the probability computed by the network after one pixel perturbation of the top class (i.e. the class with the highest probability in the first of two frames) (i.e. the class with the highest probability in the first of the two frames).

As shown in Figure 4, the decision evaluation module analyzes the input test results to judge the performance of the model [accuracy after expansion, recognition loss after Loss expansion, poor recognition accuracy before and after Δacc expansion, and poor recognition loss before and after Δloss expansion, The robustness of the CR model (characterized by robacc), the difference in confidence scores before and after ΔS expansion, the probability that the TOP-1 prediction of the P(Top-1 Change) network will change after a single-pixel perturbation, the MAC mean absolute change)], and A detailed test report is given. When comparing the performance of multiple recognition models, a large number of individual performance indicators are often too complicated for users, making it difficult for users to make reasonable judgments. Therefore, different indicators are used for the synthesis of the recognition system. The impact is taken into account in the design of performance indicators, and then a comprehensive performance indicator CM (Composite Value) is defined to reflect the comprehensive performance of different identification systems; the formula is as follows:

Among them: CM _i represents the comprehensive performance value of the ith identification system, ω _j represents the weight of the jth performance index value of the cloud service, max(M _j ) represents the maximum value of the jth performance index in multiple identification systems, min (M _j ) represents the minimum value of the j-th performance index in multiple recognition systems, M _ij represents the j-th performance index value of the ith recognition system, and N represents the total number of performance index values of the recognition system. By using formula (2 *max(M _j )-M _ij )/(2*max(M _j )-min(M _j )) normalizes the value of M _ij to the [0, 1] interval. It can be seen that the larger the value of CM, the better the comprehensive performance of the recognition system.

For some identification system performance indicators, such as Loss, P(Top-1 Change), MAC, etc., the smaller the value M _ij is, the larger the CM comprehensive performance index value is, and the values of these performance indicators can be directly substituted into the above formula, For indicators such as recognition accuracy, the larger the value, the better the recognition performance, but directly substituting the formula will cause the CM value to decrease, which is not in line with expectations, so we need to process the values of these performance indicators, using ( 1-M _ij ) to replace the Mi _ij value in the formula.

The decision evaluation module will eventually output the results of the multi-angle test module test, and generate the corresponding test report form, as shown in Table 1-3. Due to the different requirements of different task scenarios, the test system will also give corresponding suggestions.

Table 1. Performance Metrics Report

Table 2. Security Metrics Report

Table 3. Model stability index report and model comprehensive performance report

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (7)

1. A performance test method for a medical image recognition system is characterized by comprising the following steps: the performance test method comprises the following steps: the multi-class image test data generation module comprises a confrontation sample generation network and an entity and background recombination method; the multi-angle test module comprises a performance test, a reliability test and a safety test; the decision evaluation module analyzes the input test result, judges the performance of the model and gives a detailed test report;

inputting a group of pictures to be classified and identified by a network, inputting the pictures into the multi-class image test data generation module for the first time, inputting the pictures into the model for classification after image augmentation, inputting the classification result into the multi-angle test module, testing the learning result of the model by the multi-angle test module, transmitting the result to the decision evaluation module, analyzing the input test result by the decision evaluation module, judging the performance of the model, and giving a detailed test report;

The confrontational sample generation network and entity, the background reorganization method comprises confrontation augmentation usage multi-loss mixed confrontation camouflage,

multiple loss function

Expressed as:

wherein: λ represents the antagonistic strength,

Indicating a loss of resistance,

Representing a style loss for style generation,

Representing a content loss for preserving source image content,

Representing a smoothness penalty for ensuring smoothness of the extended sample;

defining an existing image, a target attack area and an expected target pattern by a user, generating a required pattern in the required area, and adding additional physical adaptation training to the generated extended sample in each step;

the style distance between two images is defined by the difference in the style representation of the two images,

wherein:

as a feature distance, l is a style level feature, S_lIs a collection of style layers from which a style representation is extracted,

is a feature extractor for a style or style,

is from

A set of pattern layers of the extracted deep layer features of the Gram matrix, x^sIs a stylistic reference image, x' is the generated countermeasure sample;

pattern loss for pattern generation

The content of the enhanced image in the generated reference pattern is very different from the content of the original image; specifically, as follows, the following description will be given,

Wherein:

is content loss, t is a content layer characteristic, c_tIs a set of content layers that extracts the content representation,

is a feature extractor for the content layer, x is the original image, x' is the generated countermeasure sample;

improving the smoothness of the enhanced image by reducing the variation between adjacent pixels; for the enhanced image, the smoothness penalty is defined as,

wherein: x'_i，jTo combat the pixel value at the sample (i, j) coordinate, x_i+1，jIs the pixel value, x, at the (i +1, j) coordinate of the original image_i，j+1For the pixel value at the original image (i, j +1) coordinate:

for the loss of antagonism

Using the following crossesFork entropy loss:

wherein: p is a radical of_yadv() And p_y() Respectively outputting the probability of the target model F to labels yadv and y, wherein yadv is the category of the confrontation sample, and y is the category of the original image;

realistic conditions are introduced into the generation process of the augmented example, as follows:

wherein: o is a random background image sampled in the physical world, T is a random transformation of rotation, resizing and color shift,

is a set of transformations;

and the target background is recombined and expanded, the target is segmented from the background by using a segmentation algorithm Mask R-CNN, pixels are supplemented to a blank part in the background by using an interpolation algorithm, and finally the target and the background are randomly combined to realize the image expansion.

2. The performance testing method for medical image recognition systems according to claim 1, wherein:

the performance test in the multi-angle test module comprises different angles: judging and judging the identification Accuracy Accuracy, judging and judging the identification Loss value Loss and judging the metamorphic relation; judging the Accuracy Accuracy and the Loss value Loss, wherein the Accuracy Accuracy and the Loss value Loss are obtained by subtracting the Accuracy Accuracy and the identification Loss value Loss output by the models before and after the augmentation to obtain the identification Accuracy difference percentage delta acc before and after the augmentation and the identification Loss difference percentage delta Loss before and after the augmentation;

the disintegration test is defined as: c_iFor the original test image

Is classified by the image recognition system, S_iFor the original test image

A confidence score of; c_i' use for association with metamorphic relations

Synthesized new test image

Class label of S_i' use for association with metamorphic relations

Synthesized new test image

The confidence score of (a) is determined,

the metamorphic relationship is expressed as:

C_i＝C′_i and ΔS＝|S_i-S′_i|＜c (7)

wherein: c is the hyperparameter, 0< c <100, c is set to 50, Δ S is the difference in confidence scores before and after expansion.

3. The performance testing method for medical image recognition system according to claim 1, wherein:

the reliability test in the multi-angle test module is a robustness verified robustness test, and under the condition that an original image x meets confidence degree guarantee, immune attack can be carried out in a norm sphere radius R:

Wherein: z () is lossA missing function, g () is the objective function to be optimized,

the method is characterized in that the method is arbitrary, epsilon is introduced noise, B (x; R) is a noise set, R is a norm sphere radius, and x is an original image;

the final robustness accuracy robac is defined as:

4. the performance testing method for medical image recognition system according to claim 1, wherein:

the security test in the multi-angle test module is a model invariance test, a random image is selected, the disturbance of a pixel is selected by using one of four methods described below, and then the sensitivity of the network to the disturbance is measured, the first method is a Crop method, a square is randomly selected in an original image, the size of the square is adjusted to 224x224px, then the diagonal of the square is translated by one pixel to create a second image, and the image is different from the first image by translating a single pixel; the second method, Embedding, is the Embedding method, which first reduces the image to a minimum size of 100px while maintaining the aspect ratio and embeds it at a random location within the 224x224px image while filling the rest of the image with black pixels, then shifts the Embedding location by a single pixel, and creates two identical images again until a single pixel is shifted; in the third method, the image is first reduced to a minimum size of 100px while maintaining the aspect ratio and embedded at random locations within the 224x224px image, then a simple repair algorithm is used, i.e., each black pixel is replaced by a weighted average of the non-black pixels in its neighborhood, and the fourth method is the same as the second method, the image is first reduced to a minimum size of 100px without moving the embedded location, but keeping the embedded location unchanged, and changing the size of the embedded image by a single pixel.

5. The performance testing method for medical image recognition systems of claim 4, wherein:

in the security test, sensitivity is measured as an invariance test of a model by two methods, the first is the probability that TOP-1 of the network predicts the Change after a single-pixel disturbance, which is called P, i.e. Top-1 Change, and the second is the average absolute Change of the probability calculated by the network after a pixel disturbance of the TOP class, which is called MAC.

6. The performance testing method for medical image recognition systems according to claim 1, wherein:

the decision evaluation module analyzes the input test result, and judges the model performance, namely the identification Accuracy after Accuracy expansion, the identification Loss after Loss expansion, the identification Accuracy difference before and after delta acc expansion, the identification Loss difference before and after delta Loss expansion, the CR model robustness is represented by robac, the confidence score difference before and after delta S expansion, the probability that the TOP-1 of a P network changes after single-pixel disturbance is predicted, the MAC average absolute change is given, and a test report is given; the formula is as follows:

Wherein: CM (compact message processor)_iRepresenting the integrated performance value, ω, of the i-th identification system_jWeight, max (M) representing jth performance index value of cloud service_j) Represents the maximum value of the jth individual performance index, min (M), in multiple recognition systems_j) Representing the minimum value of the j-th performance indicator, M, in a plurality of recognition systems_ijJ (th) representing i (th) recognition systemThe performance indicator value, N, represents the total number of identifying system performance indicator values, by using the formula (2 max (M)_j)-M_ij)/(2*max(M_j)-min(M_j) ) to M_ijIs normalized to [0, 1 ]]An interval.

7. The performance testing method for medical image recognition system according to claim 1, wherein:

and the decision evaluation module finally outputs various results tested by the multi-angle test module and generates a corresponding test report form.

Images