CN118247224A - A breast cancer pathology image analysis method based on deep learning - Google Patents

Abstract

The invention discloses a breast cancer pathological image analysis method based on deep learning. Firstly, constructing an improved EFFICIENTNET-B0 network model, and carrying out model optimization through a disclosed breast cancer clinical sample data set until a loss function converges to obtain a trained image analysis model; and analyzing the breast cancer pathological image to be identified by using the trained image analysis model to generate a clear epithelial tissue image and a clear mesenchymal tissue image. The method can more accurately identify and classify the epithelial tissues and the mesenchymal tissues in the pathological images of the breast cancer, and better help doctors to understand and read the medical images, so that the accuracy of breast cancer diagnosis is improved, the working efficiency is improved, and meanwhile, important assistance can be provided for predicting the development and the recovery of the breast cancer, and important references are provided for the doctors to formulate personalized and efficient treatment schemes.

Description

A breast cancer pathology image analysis method based on deep learning

Technical Field

The present invention relates to the field of deep learning technology, and more specifically to a breast cancer pathology image analysis method based on deep learning.

Background Art

Artificial intelligence technology is developing rapidly, showing unique advantages and being widely used in various fields. Among them, various algorithms represented by deep learning are the current research hotspots. Deep learning is an important subfield of artificial intelligence, which aims to simulate the connection of neurons in the human brain so that computers can better recognize the relationship between information and context. In recent years, deep learning methods based on neural networks have achieved breakthrough results in many tasks such as image recognition, speech recognition, and natural language processing. Deep learning models have been widely used to solve various problems with their high flexibility and effectiveness. Deep learning is one of the most promising methods in the field of computer science in the future, and it continues to expand into new application areas.

With the further development of computer science and medicine, deep learning has played an important role in breast cancer pathology image recognition. The continuous breakthroughs of this technology in image recognition, classification and image segmentation provide a better way for early detection and treatment of breast cancer. The core challenges of breast cancer pathology image recognition include the localization of cell tissues and branches, the identification of cell changes, and the differentiation of benign and malignant tumors. In the past, these tasks relied on the experience and knowledge of pathologists, but the emergence of deep learning has broken this limitation. Because deep learning can automatically analyze images, it can significantly reduce workload, improve efficiency and accuracy, thereby reducing the impact of human factors. With the increasing application of deep learning in medical image analysis, this technology has also played an important role in breast cancer pathology image recognition. Deep learning provides a new direction for the early detection and treatment of breast cancer.

Researchers used deep learning to segment the cell nuclei of breast cancer tissue and achieved high accuracy, indicating the great potential of deep learning in breast cancer pathology image recognition. Then some scholars used unsupervised deep learning methods to identify breast cancer tissue sections, further improving research in this field. The role of deep learning in breast cancer screening and diagnosis is increasingly valued. In the screening and diagnosis of breast cancer patients, it is crucial to detect important landmark features as early as possible. These features may be very small and difficult to be recognized by the naked eye, but they can be recognized by the convolutional neural network (CNN) in the deep learning model. The detection of subtle changes in the key landmark characteristics of breast cancer through the convolutional neural network (CNN) can increase the possibility of early diagnosis. The existing technology proposes a model called "deep residual learning", which successfully incorporates these tiny features into the mathematical model for predicting the risk of breast cancer recurrence, greatly improving the accuracy of the prediction. On the other hand, deep learning also plays an important role in breast cancer treatment decisions, including the decision on surgical intervention and the choice of treatment options, because the tumor needs to be accurately assessed during surgery. At the same time, the use of deep learning to identify cells transported to lymph nodes in breast cancer patients has made important breakthroughs in clinical practice. Deep learning has also been applied to personalized treatment of patients, providing important help in designing the most suitable treatment plan for patients by analyzing the molecular subtypes of breast cancer.

Studies have shown that breast cancer is related to epithelial-mesenchymal transition (EMT). EMT is a biological process in which cells transform between epithelial and mesenchymal states. This process plays an important role in biological development, tissue repair, and the occurrence of malignant tumors. As early as the 1980s, the EMT process had been studied by biologists, and it was found that EMT is one of the mechanisms for the occurrence and development of malignant tumors. In breast cancer, the EMT process is closely related to many important biological characteristics such as cancer invasion, migration, and drug resistance. The transformation from epithelial cells to more invasive mesenchymal cells can promote cancer cells to migrate through the basement membrane to distant organs and form metastatic lesions. Since the EMT process is related to the invasiveness and metastasis of breast cancer, studying EMT is of great significance for the treatment of breast cancer.

Deep learning has shown great potential in identifying and predicting the EMT process of breast cancer. Deep neural networks (DNNs) can be used to extract key features from complex genomic data, which enables researchers to identify and understand the potential mechanisms and transformation patterns that affect EMT-related genes. In order to improve the accuracy of this research method, some scholars have proposed a new deep learning model that optimizes the structure of DNNs through reinforcement learning algorithms, enhancing the accuracy of the model in predicting EMT status. Furthermore, some scholars have developed an automated method based on deep learning that can determine the state of EMT by performing complex image analysis on breast cancer tissue sections. This method not only greatly improves the efficiency of EMT status judgment, but also reduces the risk of misjudgment due to manual pathology reading.

In summary, deep learning has shown great potential in the study of breast cancer EMT. It is expected that in the future, with the continuous development and optimization of technology, this technology will play a greater role in the prediction, diagnosis and treatment of breast cancer. However, the study of breast cancer EMT still faces many challenges. EMT is not an either-or process. In many cases, cells may only be in a partial EMT state, that is, a state between epithelium and mesenchyme. This makes the study of this process complicated and difficult, and this difficulty is very likely to delay the early detection and treatment of breast cancer.

Therefore, how to more accurately identify and classify epithelial tissue and mesenchymal tissue in breast cancer pathological images and provide support for doctors to improve the accuracy of breast cancer diagnosis is a technical problem that technicians in this field urgently need to solve.

Summary of the invention

In view of this, the present invention provides a breast cancer pathology image analysis method based on deep learning, which solves the problems existing in the background technology.

In order to achieve the above object, the present invention provides the following technical solutions:

A breast cancer pathology image analysis method based on deep learning, comprising the following steps:

Build an improved EfficientNet-B0 network model and optimize the model using a public breast cancer clinical sample dataset until the loss function converges to obtain a trained image analysis model;

The trained image analysis model is used to analyze the breast cancer pathology images to be identified and generate clear epithelial tissue images and mesenchymal tissue images.

Optionally, the public breast cancer clinical sample dataset adopts the TCGA (The Cancer Genome Atlas) dataset;

Optionally, the improved EfficientNet-B0 network model is based on the EfficientNet model, and the network structure is optimized by increasing the number of network layers, the number of channels and the number of parameters, including a backbone network, a first module, Block1, Block2, Block3, Block4, Block5, a second module and an output layer connected in sequence;

The backbone network includes an input layer, a first scaling layer, a normalization layer, a first zero-filling layer, a first two-dimensional convolutional layer, a first batch of normalization layers, and a first activation function layer connected in sequence;

Block1 includes a second module, a third module and a first stacking layer connected in sequence, wherein the second module is also connected to the first stacking layer;

Block2 includes a second module, a third module and a second stacking layer connected in sequence, and the first stacking layer in Block1 is connected to the second module and the second stacking layer in Block2 respectively;

Block3 includes a second module, a third module, a third stacking layer, a third module and a fourth stacking layer connected in sequence, the second stacking layer in Block2 is connected to the second module and the third stacking layer in Block3 respectively, and the third stacking layer is connected to the fourth stacking layer;

Block4 includes a second module, a third module, a fifth superimposed layer, a third module and a sixth superimposed layer connected in sequence, and the fourth superimposed layer in Block3 is connected to the second module, the fifth superimposed layer and the sixth superimposed layer in Block4 respectively;

Block5 includes a second module, a third module, a seventh superimposed layer, a third module, an eighth superimposed layer, a ninth superimposed layer, a third module and a tenth superimposed layer connected in sequence, and the sixth superimposed layer in Block4 is respectively connected to the second module, the seventh superimposed layer, the eighth superimposed layer, the ninth superimposed layer and the tenth superimposed layer in Block5.

Optionally, the first module includes a first depth-wise separable convolutional layer, a second batch normalization layer, and a second activation function layer connected in sequence.

Optionally, the second module includes a second depthwise separable convolutional layer, a third batch normalization layer, a third activation function layer, a second zero-filling layer, a third depthwise separable convolutional layer, a fourth batch normalization layer and a fourth activation function layer, which are connected in sequence.

Optionally, the third module includes a global average pooling layer, a second scaling layer, a second two-dimensional convolution layer and a third two-dimensional convolution layer.

Optionally, the model is optimized using the Adam algorithm, and the loss function is a binary cross entropy loss function.

Optionally, the trained image analysis model is used to analyze the breast cancer pathology image to be identified, as follows:

First, the breast cancer pathology image to be identified is converted from the KFB file format to the SVS file format, the cropped image is converted to the RGB format and scaled to half of the original size, the scaled width and height are obtained and the scaled image conversion data type is stored; the scaled image is processed by sliding windows, and the width and height of the scaled image are traversed by two loops respectively; the number of non-white pixels in each image block is calculated to determine whether the non-white pixel threshold is met. When the number of non-white pixels is greater than the threshold λ, the image block is flipped, rotated, and normalized in turn and input into the trained image analysis model for prediction to obtain an array _g1 , which represents the prediction result of each transformed image block; then the prediction results in _g1 are weighted averaged and accumulated to the corresponding position of the array, and the number of times each pixel is processed is recorded by accumulating the mask matrix, and the prediction result of each pixel is divided by the number of times the pixel is processed to obtain the final normalized prediction result _g2 ; next, the normalized value in _g2 is scaled to the range of 0 to 255, and the image within this range is regarded as a valid image. If _g2 >255 will be regarded as noise and discarded; then the effective image array is converted into image I ₁ , and image I ₁ is converted into grayscale image I ₂ , thus completing the conversion of the SVS image file into a PNG file; the PNG file includes clear epithelial tissue images and mesenchymal tissue images.

Optionally, the method further includes: using a tissue enhancement visual fusion algorithm to generate a visualized thermal overlay image, specifically including the following steps:

Read SVS image file and obtain SVS image data;

Read the PNG file and resize the PNG image to be the same as the SVS image;

Use Gaussian filter to smooth PNG images;

Map the smoothed PNG image values to the color space to obtain a color heat map;

Apply color thresholding and create a mask to remove the black background from the heat map;

The processed thermal map and SVS image are fused at a preset ratio to generate the final visualized thermal coverage image.

Optionally, the method further comprises evaluating the image analysis result by using a comprehensive grayscale threshold intelligent determination algorithm, specifically comprising the following steps:

Calculate the sum of all pixels of a PNG image;

Calculate the area of the PNG image, which is the width multiplied by the height of the image;

Calculate the total number of pixels with pixel values greater than or equal to 1 and the area they occupy in the PNG image;

Calculate the ratio of epithelial tissue area or interstitial tissue area to the total tissue area in each PNG image and

In the formula, It represents the ratio of epithelial tissue area to total area in the i-th image, Total_sum(i) represents the number of epithelial tissue pixels in the i-th image, and Total_area(i) represents the total number of pixels in the i-th image. If there are N images, the value range of i is: 1≤i≤N;

It represents the ratio of the mesenchymal tissue area to the total area in the kth image, Total_area_1(k) represents the total number of pixels in the kth image, and Total_sum_1(k) represents the number of mesenchymal tissue pixels in the kth image. If there are M images, the value range of k is: 1≤k≤M.

Set a ratio threshold r, Compared with the ratio threshold r, if Then _Xi = 1, otherwise _Xi = 0; similarly, if Then Y _k =1, otherwise Y _k =0;

Calculate separately and Take the ratio of the quantities 1 and save them as L _e and L _m :

The beneficial effects of the present invention are as follows:

It can be seen from the above technical solution that compared with the prior art, the present invention provides a breast cancer pathology image analysis method based on deep learning, which can more accurately identify and classify epithelial tissue and mesenchymal tissue in breast cancer pathology images, better help doctors understand and interpret medical images, thereby improving the accuracy of breast cancer diagnosis and improving work efficiency. At the same time, it can provide important help for predicting the development and recovery of breast cancer, and also provide an important reference for doctors to formulate personalized and efficient treatment plans.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

FIG1 is a flow chart of a breast cancer pathology image analysis method based on deep learning provided by the present invention;

FIG2 is a schematic diagram of a composite network structure provided by the present invention;

FIG3 is a structural diagram of an improved EfficientNet-B0 network model provided by the present invention;

FIG. 4( a )-FIG 4( b ) are schematic diagrams of slice information of original images used in experiments provided by the present invention;

FIG5 is an epithelial tissue image corresponding to the original image in FIG4 provided by the present invention;

FIG6 is a mesenchymal image corresponding to the original image in FIG4 provided by the present invention;

FIG7 is a mesenchyme image after color conversion provided by the present invention;

FIG8 is a visualized heat map provided by the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The embodiment of the present invention discloses a method for analyzing breast cancer pathology images based on deep learning, as shown in FIG1 , comprising the following steps:

Acquire microscope images based on breast cancer tissue sections and perform preprocessing to construct a training set;

Build an improved EfficientNet-B0 network model and optimize the model based on the training set until the loss function converges to obtain a trained image analysis model;

The trained image analysis model is used to analyze the breast cancer pathology images to be identified and generate clear epithelial tissue images and mesenchymal tissue images.

Next, the process shown in FIG. 1 is described in detail to provide a deeper understanding of the present technical solution.

1. Experimental pathological images

The pathological images used in the experiment were derived from breast cancer cases in a university hospital. Breast cancer pathological image data are mainly provided by the hospital's pathology department. These data cover microscope images of breast tissue sections after surgery or biopsy, including conventional HE staining and related immunohistochemical staining, which are used to observe the morphological characteristics of tumor cells and the expression of biological markers. In addition, pathological images contain rich cellular microscopic information, such as cell size, shape, arrangement, and lesion conditions, which is very important for pathologists to diagnose and analyze diseases. These breast cancer pathological image data provide valuable resources for breast cancer research. By applying advanced technologies such as deep learning, more help can be provided for the diagnosis and treatment of breast cancer.

2. Image acquisition

The case images obtained in this embodiment are from breast cancer surgery cases admitted to a university affiliated hospital from 2015 to 2022. A total of 525 slice images were obtained, all of which were digitally scanned to obtain high-definition digital images in KFB format. The size of each image is more than 100MB, and the image pixels are all more than 10000 pixels × 10000 pixels.

Image acquisition refers to obtaining microscopic images from breast cancer tissue sections. First, affected breast tissue is obtained from the patient. Then, these tissues are prepared into thin slices and stained, such as conventional hematoxylin-eosin (HE) staining and immunohistochemical staining, to observe the morphology of tumor cells and the expression of biological markers. Next, these stained sections are scanned using a microscope and high-resolution images are captured, which will be used for further analysis.

3. Model construction

In view of the characteristics of breast cancer pathology images, this embodiment selects EfficientNet as the basic network model. The EfficientNet network model significantly surpasses the previous network in terms of effect, parameter quantity and speed. The main design of this network model adopts a composite model, and the composite network is shown in Figure 2.

The composite network structure in Figure 2 takes into account the three dimensions of the network (depth, width, and resolution) at the same time. Depth refers to the depth of the model, that is, the number of layers of the network. The deeper the network, the larger the receptive field, and the stronger the extracted feature semantics; width refers to the size of the network feature dimension (that is, the number of channels). The larger the feature dimension, the stronger the representation ability of the model; resolution refers to the input image size of the network, that is, height times width (H*W). The larger the input image resolution, the more conducive it is to extract finer-grained features. Depth and width affect the parameter size and speed of the network model, but resolution only affects the model speed, because the larger the resolution, the greater the amount of calculation. The EfficientNet composite network model in this embodiment increases the width, depth, and input resolution of the network on the basis of the traditional model, and achieves higher accuracy and efficiency than other networks through balanced scaling of network width, depth, and resolution.

Although increasing the depth of the network can extract richer and more complex high-level semantic features, too deep a network will face problems such as gradient vanishing and difficulty in training. Increasing the width of the network can obtain more fine-grained features and is easier to train, but it is difficult for a network with a large width and shallow depth to learn deeper features. Increasing the image resolution of the input network can obtain more fine-grained feature templates. The higher the image resolution, the more details can be seen, which can improve the resolution ability. However, for very high input resolutions, the accuracy gain will also decrease, and large-resolution images will increase the amount of computation for the network. Therefore, we cannot emphasize only one of the three dimensions, but need to strike a balance between the three dimensions.

Therefore, in view of the large input image, this embodiment is based on the EfficientNet model, increases the number of network layers, increases the number of channels and the number of parameters to optimize the network structure, and obtains an improved EfficientNet-B0 network model after weighing the pros and cons. Its structure is shown in FIG3 , including: a backbone network, a first module, Block1, Block2, Block3, Block4, Block5, a second module and an output layer connected in sequence; specifically:

The first layer of the EfficientNet-B0 network model is the backbone network, which has 32 channels and includes the input layer, the first scaling layer, the normalization layer, the first zero-filling layer, the first two-dimensional convolution layer, the first batch of normalization layers, and the first activation function layer. The convolution kernel size of the convolution layer is 3×3, and the resolution is 224×224 (height×width).

The second layer of the EfficientNet-B0 network model, i.e. the first module, has 16 channels and includes the first depthwise separable convolutional layer, the second batch normalization layer, and the second activation function layer connected in sequence; the convolution kernel size of the convolutional layer is k3×3, and the resolution is 112×112 (height×width);

The third layer of the EfficientNet-B0 network model is Block1, which has 24 channels and includes a second module, a third module and a first stacking layer connected in sequence, wherein the second module is also connected to the first stacking layer; the second module includes a second depth-separable convolution layer, a third batch normalization layer, a third activation function layer, a second zero-filling layer, a third depth-separable convolution layer, a fourth batch normalization layer and a fourth activation function layer connected in sequence; the third module includes a global average pooling layer, a second scaling layer, a second two-dimensional convolution layer and a third two-dimensional convolution layer, wherein the convolution kernel size of the convolution layer is k3×3 and the resolution is 112×112 (height×width);

The fourth layer of the EfficientNet-B0 network model is Block2, which has 40 channels and includes the second module, the third module, and the second stacking layer connected in sequence. The first stacking layer in Block1 is connected to the second module and the second stacking layer in Block2 respectively. The convolution kernel size of the convolution layer is k5×5, and the resolution is 56×56 (height×width).

The fifth layer of the EfficientNet-B0 network model is Block3, which has 80 channels and includes the second module, the third module, the third stacked layer, the third module and the fourth stacked layer connected in sequence. The second stacked layer in Block2 is connected to the second module and the third stacked layer in Block3 respectively, and the third stacked layer is connected to the fourth stacked layer. The convolution kernel size of the convolution layer is k3×3, and the resolution is 28×28 (height×width).

The sixth layer of the EfficientNet-B0 network model is Block4, which has 112 channels and includes the second module, the third module, the fifth stacking layer, the third module and the sixth stacking layer connected in sequence. The fourth stacking layer in Block3 is connected to the second module, the fifth stacking layer and the sixth stacking layer in Block4 respectively; the convolution kernel size of the convolution layer is k5×5, and the resolution is 14×14 (height×width);

The seventh layer of the EfficientNet-B0 network model is Block5, which has 192 channels and includes the second module, the third module, the seventh superposition layer, the third module, the eighth superposition layer, the ninth superposition layer, the third module and the tenth superposition layer connected in sequence. The sixth superposition layer in Block4 is connected to the second module, the seventh superposition layer, the eighth superposition layer, the ninth superposition layer and the tenth superposition layer in Block5 respectively. Among them, the convolution kernel size of the convolution layer is k5×5, and the resolution is 14×14 (height×width);

The eighth layer of the EfficientNet-B0 network model, i.e. the second module, has 320 channels and includes the second depthwise separable convolutional layer, the third batch normalization layer, the third activation function layer, the second zero-filling layer, the third depthwise separable convolutional layer, the fourth batch normalization layer, and the fourth activation function layer, which are connected in sequence; the convolution kernel size of the convolutional layer is k3×3, and the resolution is 7×7 (height×width);

The ninth layer of the EfficientNet-B0 network model is the output layer, which has 1280 channels, contains a 1×1 convolution operation kernel, and has a resolution of 7×7 (height×width).

According to the above structure, it can be seen that EfficientNet-B0 has been optimized in many aspects such as network units, structural design, and regularization methods. Its performance and efficiency have been greatly improved compared to traditional networks. It is currently a model structure with high efficiency and accuracy, and is ready for subsequent training.

4. Model training

(1) Training Dataset

The source of the dataset is pathological images downloaded from the public dataset TCGA library as a training dataset.

(2) Loss Function

The main research purpose of this embodiment is to distinguish epithelial cells and stromal cells in breast cancer pathology images. Therefore, a binary cross entropy loss function is used as a loss function.

The binary cross entropy loss function is suitable for binary classification problems, where each sample has only two possible categories. In this embodiment, epithelial cells and mesenchymal cells can be regarded as two categories, and the data labels are binarized, with epithelial cells marked as 1 and mesenchymal cells marked as 0.

The calculation formula of the binary cross entropy loss function is as follows:

Loss(p,y)＝-[y*log(p)+(1-y)*log(1-p)];

Among them, y is the true label (the label after binarization, the value is 0 or 1), p is the predicted probability of the model, the value range is [0,1], and log is the natural logarithm.

For a batch of training samples, the loss function is calculated as:

Loss(p,y)＝-1/Q*∑(yj*log(pj)+(1-yj)*log(1-pj));

Where Q is the number of samples in the batch, and j represents the jth sample. When the sample is a positive sample (y=1), the loss function focuses on the size of p, making p as close to 1 as possible. When the sample is a negative sample (y=0), the loss function focuses on the size of 1-p, making 1-p as close to 1 as possible. In this way, by minimizing the loss function, the model's predicted probability p can match the true label y as much as possible. The smaller the calculated result of the loss function, the closer the model's predicted result is to the true label, the better the performance of the model, and the better it can distinguish epithelial cells and stromal cells in breast cancer pathology images.

(3) Optimization algorithm and parameter setting

Since the model used in this embodiment has a large number of parameters and needs to converge quickly and optimize performance during the training process, according to the characteristics of the Adam algorithm, choosing the Adam algorithm is a reasonable choice.

The Adam algorithm is suitable for deep learning models with a large number of parameters. It can adaptively adjust the learning rate and combine the concept of momentum to improve the convergence speed and stability of the model. Its adaptive characteristics make it less likely to fall into local optimality, and the momentum mechanism also accelerates the training process.

The main idea of the Adam optimization algorithm is to adaptively adjust the learning rate of each parameter by calculating the first-order moment estimate (mean) and second-order moment estimate (variance) of the gradient. By adjusting the adaptive learning rate, Adam can apply different learning rates to different parameters to improve convergence speed and stability.

The update formula of the Adam algorithm is as follows:

Wherein, m is the first-order moment estimate (mean) of the gradient, v is the second-order moment estimate (variance) of the gradient, g is the current gradient, and θ is the parameter of the model. α is the learning rate, which is 0.001 in this embodiment. _{β 1} and β ₂ are hyperparameters for controlling the decay rate of the first-order moment and the second-order moment, which are 0.9 and 0.999 respectively in this embodiment. ε is a very small constant for numerical stability, which is 10 ^-8 . The Adam algorithm updates the parameters by calculating the first-order and second-order moment estimates of the gradient. The first-order moment estimate is equivalent to the momentum term in the momentum method, and the second-order moment estimate is equivalent to the adjustment of the adaptive learning rate, which enables the Adam algorithm to apply different learning rates to different parameters, so as to better adapt to the gradient characteristics of different parameters. In addition, the batch size is set to 256, L1 regularization, and the number of iterations is 1000 when training the model. The specific settings will be adjusted and optimized according to the actual situation, and the best parameter combination will be selected through experiments and verification.

5. Image Prediction

The prediction dataset comes from 525 pathological images collected from a university affiliated hospital.

Preprocessing: Preprocessing is a certain degree of processing of the image before in-depth analysis to reduce noise, eliminate irrelevant information and enhance image features.

① Format conversion: The collected image file is converted into a format suitable for computer processing. Since the KFB file obtained in this embodiment is not suitable for computer processing, file format conversion is required. First, the KFB file format is converted into the SVS file format. The converted SVS file format is suitable for computer processing.

② Image enhancement: The main steps to achieve image data enhancement in this embodiment include: image reading and data enhancement, such as: adjusting to a uniform size; randomly generating transformation parameters such as rotation angle and translation distance according to parameters; processing image edges according to the filling mode; performing affine transformation on the image pixel value matrix according to the transformation parameters; performing random flipping or scaling operations on the image; standardizing the image pixel values; returning the transformed image batches and labels to the model training. By randomly transforming some samples in each iteration, the amount of training data can be effectively increased, the generalization ability of the model can be improved, overfitting can be avoided, and efficient image data enhancement can be achieved.

③ Predict EM tissue typing: Since the original images in this embodiment are very large, it is necessary to segment the region of interest in the image from the background, remove irrelevant information, and retain the area containing only tumor cells and surrounding normal tissues. The main steps for implementing EM tissue typing in this embodiment are: dividing the original image into multiple overlapping slices according to the specified slice size and overlap size; calculating the number of non-white pixels in each slice to determine whether the non-white pixel threshold is met. If it is met, it is considered that the slice contains valid information and needs to be processed later. The improved EfficientNet-B0 network model is used to predict each slice that meets the conditions. The prediction results are averaged and added to the segmentation result matrix. At the same time, the mask matrix is accumulated, and the number of times each pixel is processed is recorded. Finally, the segmentation result matrix is divided by the mask matrix to achieve the average of the results, and the segmentation results are saved as picture output. By dividing the image into overlapping small slices, and then independently predicting each slice and superimposing the results, slice-based EM tissue typing is achieved. This method can handle large medical images well.

The images after EM tissue typing will continue to be processed to further optimize the prediction results. For example, the tissue enhancement visual fusion algorithm used in this embodiment visualizes the predicted segmentation results, making the segmentation results more intuitive and convenient for manual inspection and analysis.

6. Experiment

(1) Experimental data and experimental settings

FIG4 is one of the original images used in the experiment. The overall image information and the complete image shape can be obtained from FIG4(a). FIG4(b) is a detail image after local magnification, from which the pathological area of the slice information can be clearly observed.

The experiment was conducted using PyTorch V3.9, and the model was trained on an Nvidia Tesla V100 GPU. This example uses the EfficientNet-B0 network model, the optimizer is Adam, the learning rate is set to 0.001, the batch size is 256, and a total of 10 epochs are performed during the training process. The loss function is the cross entropy loss function. The computer performance used in the experiment: the processor is Intel(R) Core(TM) i5-8500 CPU@3.00GHz, the RAM is 8.00GB, the operating system is 64-bit, the processor is based on x64, and the Linux system.

(2) Experimental process

The original pathological images used in the experiment are KFB files, a file format specifically used to store digital pathological images. Since pathological images are large in size, the KFB format supports the storage of such large-size images, and also supports the storage of color pathological images in RGB three-channel color mode. In addition, KFB files can divide images into multiple levels or layers, and the image resolution of each level is different, which allows users to view images of different resolutions as needed. Although KFB files have many of the above advantages, their reading and processing require specific software provided by specific companies, which limits their compatibility and convenience, and is not convenient for subsequent deep learning algorithms to identify KFB files. Therefore, in this experiment, KFB files are preferentially converted to SVS files. SVS files not only have the advantages of KFB files, but are also more open and compatible, and can be easily read and processed in a variety of different software and environments. Therefore, compared with KFB files, the openness and compatibility of SVS files are their main advantages. It can be widely used in various pathological image analysis work, which is also the reason why KFB files are preferentially converted to SVS files in this experiment.

A series of image processing operations are performed on each transformed SVS image file. First, an SVS image file is opened, and then the total size of the image file is obtained, an area of interest is cropped from the image, and then the cropped image is converted to RGB format and scaled to half of the original size; then the scaled width and height are obtained, and the scaled image conversion data type is stored for subsequent processing.

Next, the image is processed by sliding window, and two loops are used to traverse the width and height of the image respectively. For each image block, the number of non-white pixels is first calculated. When the number of non-white pixels is greater than the threshold λ, the image block is successively flipped, rotated, normalized, and other transformations are input into the trained image analysis model for prediction, and an array g ₁ is obtained, which represents the prediction results of each transformed image. Then the prediction results in g ₁ are weighted averaged to obtain the final normalized prediction result g ₂ , which can eliminate the influence of multiple processing of the overlapping parts of the image blocks, making the prediction results of the model more accurate. Next, the normalized value in g ₂ is scaled to the range of 0 (black) to 255 (white), and the image within this range is regarded as a valid image (if g ₂ >255, it will be regarded as noise and discarded), and then the valid image array is converted and resized to obtain a new image as shown in Figure 5, that is, the conversion of the SVS image file into a PNG file is completed.

Grayscale image in Fig. 5 is changed from the original image in Fig. 4. Downsampling filter was used in the process of obtaining the image. When the image needs to be reduced, better results can be obtained by using this filter, because it can carry out anti-aliasing to the image, i.e., eliminate or reduce the jagged edges that may appear after the image is reduced. Image in Fig. 5 is exactly the image of the epithelial tissue needed. By comparing these two images, it can be found that not only the shapes are similar, but also the epithelial tissue cells are effectively extracted. From the comparison of these two images, it can be explained that the neural network model of the present embodiment and a series of image processing methods can accurately obtain the epithelial tissue image from the original image, which provides reliable data support for early detection and diagnosis of breast diseases.

Next, we perform an inversion operation on Figure 5, inverting each pixel in the image, that is, changing the color value of each pixel from bright to dark and from dark to bright, so that we can achieve visual enhancement, feature extraction, and data enhancement. In grayscale mode, black will become white and white will become black, so that we can get a new image Figure 6 that is the opposite of Figure 5.

FIG6 is the mesenchymal image corresponding to the original image. By comparing FIG6 with FIG4, it can be found that not only are their images almost identical in shape, but also FIG6 is the mesenchymal tissue extracted from FIG4. The comparison of the two images further confirms that the network model, algorithm and data processing method of the present embodiment can accurately extract the mesenchymal image from the original image, which provides reliable data support for early detection and accurate diagnosis of breast diseases.

In order to obtain a higher quality mesenchymal image, Figure 6 is further processed by color conversion in this experiment. The specific processing logic is to convert white pixels (the values of the R, G, and B channels are all 255) and the pixels with transparency greater than or equal to 128 into black, and the pixels of other colors remain unchanged. In this way, the dark area in the original image can be converted into a bright area, so as to better extract the target object or specific area. After further processing, the new image is obtained as shown in Figure 7. By comparing Figure 7 with Figure 6, it can be found that the features and details in the image can be more clearly observed in Figure 7, and the edges, contours or other features in the original image can be highlighted. In subsequent image processing tasks, it helps to separate the foreground target from the background in the image, and it is easier to extract the target object based on the brightness difference of the image. This is very useful for feature extraction, shape analysis and subsequent processing in tasks such as image segmentation and target detection.

(3) Experimental evaluation

In order to evaluate the experimental effect, two methods are used in this embodiment for experimental evaluation. The first method is to evaluate the effect by using a tissue enhancement visual fusion algorithm, and the second method is to evaluate the effect by using a comprehensive gray threshold intelligent judgment algorithm. The following introduces these two evaluation methods respectively.

Method 1: Tissue Enhanced Visual Fusion Algorithm

In actual medical diagnosis, grayscale images are not conducive to observation. Therefore, in order to facilitate the observer to better understand and analyze the specific features of the original image, so that such features can be intuitively seen in the original image, the grayscale PNG thermal map image is converted into a transparent color overlay image, and then the color overlay image is fused with the original SVS full image to generate a new visual thermal map image. The thermal map can be used to highlight specific areas of the SVS image, so that the characteristics, trends and anomalies of the data can be more intuitively observed and analyzed.

Step 1: Read the SVS image and obtain SVS image data. The SVS file used in this embodiment is converted from the KFB file and contains a large amount of biological tissue information;

Step 2: Read the PNG image and resize it to match the SVS image. The purpose of resizing it to the same size as the SVS image is to be able to effectively fuse the two later.

Step 3: Applying a Gaussian filter to smooth the PNG image can make the heat map smoother. Gaussian filtering can be used to reduce the noise and details of the image, making the image smoother and avoiding sharp edges in the final fused image.

Step 4: Map the smoothed PNG image values to the color space to obtain a color heat map, which can make the information of the heat map more intuitive and easy to understand;

Step 5: Apply color threshold and create mask. This step is to remove the black background in the heat map and keep only useful information. When fusion is performed, the background will not affect the SVS image, and only useful heat map information will be displayed on the SVS image.

Step 6: Fuse images and save results. This step is to fuse the processed heat map and SVS image according to the preset ratio to generate the final visualized thermal overlay image. The new visualized heat map is shown in Figure 8.

A heat map is a visualization method that uses color changes to represent the size or density of data. Comparing the original image Figure 4 and the heat map Figure 8, it can be found that the features that are difficult to observe intuitively in Figure 4 are presented in an intuitive way in Figure 8, allowing the observer to better understand and diagnose the disease. The image in Figure 8 contains both the original information of the SVS image and the characteristic information of the heat map. The distribution, concentration, and changes of the data can be quickly seen through the depth of color. At the same time, the heat map can also highlight the high-value or low-value areas in the data, and quickly find the key areas or abnormal points of the data.

Therefore, the heat map generated by the tissue-enhanced visual fusion algorithm in this embodiment represents the distribution of specific features in the original image (such as cell density, severity of the disease, etc.), so that features that were originally difficult to observe intuitively are displayed in an intuitive manner, allowing observers to better understand and analyze the original image, providing data support for the diagnosis of the disease, which is a great contribution of this technical solution.

Method 2: Comprehensive grayscale threshold intelligent judgment algorithm

This embodiment extracts quantitative features from breast cancer pathology images, mainly using the following two ratio features to represent the ratio of the epithelial tissue area or the interstitial tissue area in each PNG image to the total tissue area. The calculation formulas for these two ratios are:

In the formula, It represents the ratio of epithelial tissue area to total area in the i-th image, Total_sum(i) represents the number of epithelial tissue pixels in the i-th image, and Total_area(i) represents the total number of pixels in the i-th image. If there are N images, the value range of i is: 1≤i≤N;

R _mk represents the ratio of the mesenchymal tissue area to the total area in the kth image, Total_area_1(k) represents the total number of pixels in the kth image, and Total_sum_1(k) represents the number of mesenchymal tissue pixels in the kth image. If there are M images, the value range of k is: 1≤k≤M.

Step 1: Read the PNG image;

Step 2: Calculate the sum of all pixels of the PNG image (`Total_sum` or Total_sum_1);

Step 3: Calculate the area of the PNG image (`Total_area`), which is the width of the image multiplied by the height;

Step 4: Calculate the total number of pixels with pixel values greater than or equal to 1 and the area they occupy in the PNG image;

Step 5: Save these calculation results in Excel tables (as shown in Table 1 and Table 2), and calculate the proportions according to the above formulas and In this embodiment, a total of 56 image data were calculated, and 28 valid Data and 28 valid data;

Step 6: Set a ratio threshold r. Compared with the threshold, if Then _Xi = 1, otherwise _Xi = 0; similarly, if Then Y _k =1, otherwise Y _k =0;

Step 7: Calculate according to the following formula and Take the ratio of the quantity 1 and save _Le and _Lm :

Table 1 The ratio data of 28 epithelial tissue areas and total area obtained in the experiment

Table 2 The ratio of the area of 28 mesenchymal tissues to the total area obtained in the experiment

In the comprehensive gray threshold intelligent judgment algorithm, the higher the _Le value, the more accurate the recognition of epithelial tissue, and the higher the _Lm value, the more accurate the recognition of interstitial tissue. It is hoped that the higher the _Le and Lm values are, the better. However, usually, _Le and _Lm cannot simultaneously obtain the highest value. In fact, they are contradictory to each other, that is, if one value is increased, the other value will be reduced. However, if _we want to obtain the best recognition performance, we need to obtain the best balance between _Le and _Lm . Therefore, in this embodiment, in order to obtain the best balance between _Le and _Lm , a genetic algorithm is used to select the optimal threshold r.

The main reason for using genetic algorithms is that they are suitable for optimization problems, especially in parameter optimization and combinatorial optimization. Genetic algorithms can provide effective solutions. In this embodiment, complex, multi-parameter problems are particularly suitable for genetic algorithms to solve.

After calculation by genetic algorithm, the optimal threshold value r = 0.0336 was obtained, and _Le = 0.8548 and _Lm = 0.8125 were calculated under this threshold value. The experimental results obtained show that the method in the technical solution can balance _Le and _Lm , and the recognition rates of epithelial tissue and interstitial tissue are both over 80%, which proves that the model and method in the present invention have excellent performance, far exceeding other existing methods.

In order to detect breast cancer and its risk of disease earlier, the present invention proposes an innovative deep neural network algorithm specifically for analyzing breast cancer medical images. The core function of this algorithm is to accurately distinguish epithelial tissue and stromal tissue from pathological images, thereby providing key data support for early diagnosis.

After a series of experimental tests, the model and method of the present invention successfully generated clear images of epithelial tissue and mesenchymal tissue, and presented good visualization effects in the thermal map. The results of quantitative evaluation showed that the model in this embodiment achieved 0.8548 and 0.8125 in the recognition accuracy of epithelial tissue and mesenchymal tissue, respectively. This achievement not only demonstrates the efficiency of the network model and the effectiveness of the processing method, but also provides important technical support for the early detection and accurate diagnosis of breast cancer.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A breast cancer pathology image analysis method based on deep learning, characterized in that it comprises the following steps: Build an improved EfficientNet-B0 network model and optimize the model using a public breast cancer clinical sample dataset until the loss function converges to obtain a trained image analysis model; The trained image analysis model is used to analyze the breast cancer pathology images to be identified and generate clear epithelial tissue images and mesenchymal tissue images. 2. According to the deep learning-based breast cancer pathology image analysis method of claim 1, it is characterized in that the public breast cancer clinical sample dataset adopts the TCGA dataset. 3. According to claim 1, a method for analyzing breast cancer pathology images based on deep learning is characterized in that the improved EfficientNet-B0 network model is based on the EfficientNet model, and the network structure is optimized by increasing the number of network layers, the number of channels and the number of parameters, including a backbone network, a first module, Block1, Block2, Block3, Block4, Block5, a second module and an output layer connected in sequence; The backbone network includes an input layer, a first scaling layer, a normalization layer, a first zero-filling layer, a first two-dimensional convolutional layer, a first batch of normalization layers, and a first activation function layer connected in sequence; Block1 includes a second module, a third module and a first stacking layer connected in sequence, wherein the second module is also connected to the first stacking layer; Block2 includes a second module, a third module and a second stacking layer connected in sequence, and the first stacking layer in Block1 is connected to the second module and the second stacking layer in Block2 respectively; Block3 includes a second module, a third module, a third stacking layer, a third module and a fourth stacking layer connected in sequence, the second stacking layer in Block2 is connected to the second module and the third stacking layer in Block3 respectively, and the third stacking layer is connected to the fourth stacking layer; Block4 includes a second module, a third module, a fifth superimposed layer, a third module and a sixth superimposed layer connected in sequence, and the fourth superimposed layer in Block3 is connected to the second module, the fifth superimposed layer and the sixth superimposed layer in Block4 respectively; Block5 includes a second module, a third module, a seventh superimposed layer, a third module, an eighth superimposed layer, a ninth superimposed layer, a third module and a tenth superimposed layer connected in sequence, and the sixth superimposed layer in Block4 is respectively connected to the second module, the seventh superimposed layer, the eighth superimposed layer, the ninth superimposed layer and the tenth superimposed layer in Block5. 4. A breast cancer pathology image analysis method based on deep learning according to claim 3, characterized in that the first module includes a first depth-separable convolutional layer, a second batch normalization layer and a second activation function layer connected in sequence. 5. According to a deep learning-based breast cancer pathology image analysis method according to claim 4, it is characterized in that the second module includes a second depth-separable convolutional layer, a third batch normalization layer, a third activation function layer, a second zero-filling layer, a third depth-separable convolutional layer, a fourth batch normalization layer and a fourth activation function layer connected in sequence. 6. A breast cancer pathology image analysis method based on deep learning according to claim 5, characterized in that the third module includes a global average pooling layer, a second scaling layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer. 7. A breast cancer pathology image analysis method based on deep learning according to any one of claims 1-6, characterized in that the model optimization adopts the Adam algorithm and the loss function is a binary cross entropy loss function. 8. The method for analyzing breast cancer pathology images based on deep learning according to claim 1 is characterized in that the breast cancer pathology images to be identified are analyzed using a trained image analysis model, specifically as follows: First, the breast cancer pathology image to be identified is converted from the KFB file format to the SVS file format, the cropped image is converted to the RGB format and scaled to half of the original size, the scaled width and height are obtained and the scaled image conversion data type is stored; the scaled image is processed by sliding windows, and the width and height of the scaled image are traversed by two loops respectively; the number of non-white pixels in each image block is calculated to determine whether the non-white pixel threshold is met. When the number of non-white pixels is greater than the threshold λ, the image block is flipped, rotated, and normalized in turn and input into the trained image analysis model for prediction to obtain an array _g1 , which represents the prediction result of each transformed image block; then the prediction results in _g1 are weighted averaged and accumulated to the corresponding position of the array, and the number of times each pixel is processed is recorded by accumulating the mask matrix, and the prediction result of each pixel is divided by the number of times the pixel is processed to obtain the final normalized prediction result _g2 ; next, the normalized value in _g2 is scaled to the range of 0 to 255, and the image within this range is regarded as a valid image. If _g2 >255 will be regarded as noise and discarded; then the effective image array is converted into image I ₁ , and image I ₁ is converted into grayscale image I ₂ , thus completing the conversion of the SVS image file into a PNG file; the PNG file includes clear epithelial tissue images and mesenchymal tissue images. 9. A method for analyzing breast cancer pathology images based on deep learning according to claim 8, characterized in that the method further comprises: using a tissue enhancement visual fusion algorithm to generate a visualized thermal overlay image, specifically comprising the following steps: Read SVS image file and obtain SVS image data; Read the PNG file and resize the PNG image to be the same as the SVS image; Use Gaussian filter to smooth PNG images; Map the smoothed PNG image values to the color space to obtain a color heat map; Apply color thresholding and create a mask to remove the black background from the heat map; The processed thermal map and SVS image are fused at a preset ratio to generate the final visualized thermal coverage image. 10. A breast cancer pathology image analysis method based on deep learning according to claim 8, characterized in that the method further comprises evaluating the image analysis results using a comprehensive grayscale threshold intelligent judgment algorithm, specifically comprising the following steps: Calculate the sum of all pixels of a PNG image; Calculate the area of the PNG image, which is the width multiplied by the height of the image; Calculate the total number of pixels with pixel values greater than or equal to 1 and the area they occupy in the PNG image; Calculate the ratio of epithelial tissue area or interstitial tissue area to the total tissue area in each PNG image and In the formula, It represents the ratio of epithelial tissue area to total area in the i-th image, Total_sum(i) represents the number of epithelial tissue pixels in the i-th image, and Total_area(i) represents the total number of pixels in the i-th image. If there are N images, the value range of i is: 1≤i≤N; represents the ratio of the mesenchymal tissue area to the total area in the kth image, Total_area_1(k) represents the total number of pixels in the kth image, and Total_sum_1(k) represents the number of mesenchymal tissue pixels in the kth image. If there are M images, the value range of k is: 1≤k≤M; Set a ratio threshold r, Compared with the ratio threshold r, if Then _Xi = 1, otherwise _Xi = 0; similarly, if Then Y _k =1, otherwise Y _k =0; Calculate separately and Take the ratio of the quantities 1 and save them as L _e and L _m :