CN116385393A - Convolutional neural network model, pathological image recognition system, method, equipment and medium - Google Patents
Convolutional neural network model, pathological image recognition system, method, equipment and medium Download PDFInfo
- Publication number
- CN116385393A CN116385393A CN202310348734.8A CN202310348734A CN116385393A CN 116385393 A CN116385393 A CN 116385393A CN 202310348734 A CN202310348734 A CN 202310348734A CN 116385393 A CN116385393 A CN 116385393A
- Authority
- CN
- China
- Prior art keywords
- module
- pathological
- image
- neural network
- feature
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/69—Microscopic objects, e.g. biological cells or cellular parts
- G06V20/698—Matching; Classification
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
- G06T2207/10061—Microscopic image from scanning electron microscope
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30096—Tumor; Lesion
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
本申请提供了一种卷积神经网络模型、病理图像的识别系统、方法、设备及介质，其中，卷积神经网络模型，所述卷积神经网络模型至少包括依次连接的第一模块、第二模块、第三模块、第四模块、第五模块和第六模块；所述第一模块，用于对接收到的目标特征图的通道数进行降值处理；所述第二模块和所述第三模块，分别依次用于对降值处理后的通道数进行第一转换，并对所述目标特征图的尺寸进行降维处理；所述第四模块，用于基于所述第一转换后的通道数进行第二转换；所述第五模块，用于获取所述第二转换后每个通道平面的特征向量；所述第六模块，用于根据所述特征向量输出分类结果。
The present application provides a convolutional neural network model, a pathological image recognition system, method, device and medium, wherein, the convolutional neural network model, the convolutional neural network model at least includes sequentially connected first modules, second module, the third module, the fourth module, the fifth module and the sixth module; the first module is used to devalue the number of channels of the received target feature map; the second module and the first module The three modules are respectively used to perform a first conversion on the number of channels after the devaluation process, and perform dimensionality reduction processing on the size of the target feature map; the fourth module is used to perform the first conversion based on the first converted The second conversion is performed on the number of channels; the fifth module is used to obtain the feature vector of each channel plane after the second conversion; the sixth module is used to output a classification result according to the feature vector.
Description
Technical Field
The present disclosure relates to the field of information technologies, and in particular, to a convolutional neural network model, a pathological image recognition system, a method, a device, and a medium.
Background
Pathological diagnosis is a gold standard for tumor diagnosis, which generally requires observation of imaging of pathological tissue sections of cells by a microscope to determine the type, arrangement, degree of differentiation, cytologic type or histological origin of the pathological tissue, conditions of surgical margin, and metastasis of lymph nodes, presence or absence of cancer plugs in blood vessels, presence or absence of invasion of nerves, and the like.
In the related technology, a pathologist needs to combine the whole pathological image to identify micro-features and macro-features, so as to further obtain a conclusion of tumor diagnosis according to the identification result. Along with the rapid development of the related technology of artificial intelligence, feature identification of pathological images by using a convolutional neural network model is possible, and the combination of the convolutional neural network and the pathological images can obviously improve the working efficiency of pathologists.
However, the inventors found that there are at least the following technical problems in the related art: for images such as pathological images, which need to be comprehensively identified by combining micro features and macro features, the images cannot be identified by using the existing convolutional neural network model.
Disclosure of Invention
An object of the present application is to provide a convolutional neural network model, a system, a method, a device and a medium for identifying pathological images, which are at least used for solving the technical problem that the existing convolutional neural network model cannot be used for identification in the related art.
To achieve the above object, some embodiments of the present application provide a convolutional neural network model, which includes at least a first module, a second module, a third module, a fourth module, a fifth module, and a sixth module connected in sequence; the first module is used for carrying out the reduction processing on the channel number of the received target feature map; the second module and the third module are respectively and sequentially used for performing first conversion on the channel number after the reduction processing and performing dimension reduction processing on the dimension of the target feature map; the fourth module is configured to perform a second conversion based on the first converted channel number; the fifth module is configured to obtain a feature vector of each channel plane after the second conversion; and the sixth module is used for outputting a classification result according to the feature vector.
Some embodiments of the present application further provide a system for identifying a pathological image, where the system includes an image segmentation unit, a feature extraction unit, an image feature generation unit, and an image input unit; the image segmentation unit is used for segmenting the pathological image into a plurality of pathological image blocks; the feature extraction unit is used for extracting pathological features of the pathological image blocks; the image feature generation unit is used for generating a pathological feature map corresponding to the pathological image according to the pathological feature; the image input unit is used for inputting the pathological feature map into the pre-trained convolutional neural network model according to any one of the above, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
Some embodiments of the present application further provide a method for identifying a pathological image, which is applied to the system as described above, and the method includes: dividing the pathological image into a plurality of pathological image blocks; extracting pathological features of the pathological image blocks; generating a pathological feature map corresponding to the pathological image according to the pathological feature; inputting the pathological feature map into the pre-trained convolutional neural network model according to any one of the above, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
Some embodiments of the present application also provide an electronic device, the device comprising: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the method as described above.
Some embodiments of the present application also provide a computer readable medium having stored thereon computer program instructions executable by a processor to implement the method as described above.
Compared with the prior art, the convolutional neural network model provided by the embodiment of the application can be suitable for processing the target feature map obtained after preprocessing the target image which needs to consider the micro factors and the macro factors in the image, and the specific structure of the convolutional neural network model is redesigned.
Drawings
Fig. 1 is a schematic structural diagram of a convolutional neural network model according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of another convolutional neural network model according to an embodiment of the present disclosure;
FIG. 3 is a flowchart illustrating a convolution module according to an embodiment of the present disclosure;
FIG. 4 is a flowchart illustrating a convolution block processing according to an embodiment of the present disclosure;
FIG. 5 is a flowchart illustrating a process for a down convolutional block according to an embodiment of the present application;
fig. 6 is a schematic diagram of slicing the pathology image into pathology image blocks according to an embodiment of the present application;
fig. 7 is a flowchart of a method for identifying a pathological image according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
The following terms are used herein, see table 1.
TABLE 1
Example 1
The embodiment of the application provides a convolutional neural network model, which at least comprises a first module, a second module, a third module, a fourth module, a fifth module and a sixth module which are sequentially connected, as shown in fig. 1.
The first module is used for carrying out the reduction processing on the channel number of the received target feature map;
the second module and the third module are respectively and sequentially used for performing first conversion on the channel number after the reduction processing and performing dimension reduction processing on the dimension of the target feature map;
the fourth module is configured to perform a second conversion based on the first converted channel number;
the fifth module is configured to obtain a feature vector of each channel plane after the second conversion;
and the sixth module is used for outputting a classification result according to the feature vector.
Specifically, the target feature map specifically includes: and preprocessing a target image of which the micro factors and macro factors are required to be considered, so as to obtain a target feature map. In some examples, the preprocessing may include: and cutting the target image into a plurality of image blocks, extracting feature vectors of the image blocks, and generating a target feature map corresponding to the target image according to the feature vectors. Further, the generating the target feature map corresponding to the target image according to the feature vector may include: and acquiring the characteristic length of the image block, and splicing according to the characteristic vector of the image block and the characteristic length to generate a target characteristic diagram corresponding to the target image. The target feature map has a larger channel value and a smaller size than an image that can be processed by a general convolutional neural network model in the related art. Other details of the target feature map may be found in the description of the third embodiment, and will not be described in detail herein.
It will be appreciated that in the related art, since the target image belongs to the macro image, a larger memory is occupied, for example, the pathological image needs to occupy a larger memory, which is typically above several hundred Megabytes (MB), even several gigabytes (G). If the whole sample image corresponding to the target image is input to the convolutional neural network model at one time, the video memory capacity of the common GPU is far insufficient, so that training and learning of the convolutional neural network model cannot be performed in computer engineering; if the whole sample image is split into a plurality of sample image blocks and then is independently learned, because the association relation among the sample image blocks is not considered, the lack of macroscopic consideration on the whole sample image can lead to a narrow conclusion of 'see only trees and not forest'. Therefore, if the existing convolutional neural network model in the related art is used, the accuracy of feature recognition of the pathological image is low. In the embodiment of the application, the feature recognition of 1 pathological image is converted into the feature recognition of 1 pathological feature map, so that the data size of the pathological image is reduced by thousands of times. For example, if the pathology image is segmented into pathology image blocks of 2048×2048, the feature length of which is 2048, the dimension reduction rate is 2048×2048×3/(2048×4) =1536 times. Therefore, when the convolutional neural network model is used for carrying out feature recognition on the pathological feature map, the consumption of GPU video memory can be greatly reduced while the accuracy of recognition is ensured.
In the embodiment of the present application, the target image is taken as a pathological image as an example. However, it will be understood by those skilled in the art that other types of images that need to consider micro-factors and macro-factors in the images may also be used as inputs to the convolutional neural network model provided in embodiments of the present application, and the types of the target images are not specifically limited in the embodiments of the present application.
In the embodiment of the present application, a pathological feature map of the pathological image is used as an input of the convolutional neural network model. Since the dimension of the pathological feature map is m×n×k. Wherein K represents a characteristic length of each pathological image block and is a fixed value. In reality, the pathological images have different sizes, so that the values of M and N corresponding to the different pathological images are different and are not fixed values. Convolutional neural networks as they are known in the art cannot be handled if they are used. This is because the number of channels of an image that can be processed by the convolutional neural network in the related art is 1 or 3, wherein 1 corresponds to a gray image and 3 corresponds to a color RGB image, and the size of the image that can be processed by the convolutional neural network in the related art is the width x height of the image, which is much larger than the pathological feature map, and the feature content in the image can be completely preserved after scaling the image. In summary, in order to expand the receptive field of the pathological image and combine the micro-feature and macro-feature of the pathological image, the convolutional neural network model provided by the embodiment of the application is provided by using the characteristics of unfixed size, multiple channels and smaller size of the input pathological feature map.
The convolutional neural network model provided by the embodiment of the application is suitable for pathological images. Specifically, firstly, carrying out a reduction treatment on the channel number of a received target feature map through a first module, for example, reducing the channel number of the pathological feature map to 1/4 of the original channel number, then sequentially carrying out a first conversion on the channel number after the reduction treatment through a second module and a third module, carrying out a dimension reduction treatment on the dimension of the target feature map, and reducing the dimension of the pathological feature map to 1/2 of the original dimension; after the fourth module performs the second conversion based on the number of channels after the first conversion, the fifth module can obtain the feature vector of each channel plane after the second conversion, so that the sixth module outputs a classification result according to the feature vector.
The sixth module is a classifier, and the classifier may belong to two classes or may belong to multiple classes, which is not specifically limited herein. When the target image is a pathological image, if the classifier belongs to a second classification, the classifier can be suitable for screening scenes aiming at a specific tumor, such as screening of papillary thyroid cancer, and the result of the second classification is negative or positive; if the classifier belongs to multiple classifications, the classifier can be suitable for a scene of screening for multiple tumors, and the result of multiple classifications can be tumor classification/tumor grading and the like.
Compared with the related art, the convolutional neural network model provided by the embodiment of the application can be suitable for processing the target feature map obtained after preprocessing the target image which needs to consider the micro factors and the macro factors in the image, and the specific structure of the convolutional neural network model is redesigned.
Example two
The embodiment of the application provides a specific implementation mode of a convolutional neural network model according to the first embodiment. See fig. 1 and 2.
The first module comprises a convolution operation layer, a normalization layer and a nonlinear activation function layer;
the second module and the third module respectively comprise a convolution module and a descending convolution block;
the fourth module comprises the convolution module, the normalization layer and the nonlinear activation function layer;
the fifth module comprises a global maximum pooling layer and a full connection layer;
the sixth module is a classifier.
For ease of understanding, the structure of the convolutional neural network model will be described herein with reference to an example in which the number of channels of the input pathological feature map is 2048, and whether or not the pathological image is papillary carcinoma is identified by 2 classification.
A first module: by adopting 1 two-dimensional convolution kernel of 3×3, the channel number of the pathological feature map is reduced from 2048 to 512, and then sequentially passes through a normalization layer BN layer and a nonlinear activation function layer ReLU layer, wherein the BN layer can improve the stability of the convolution neural network model, and the ReLU layer can improve the nonlinear learning capacity of the convolution neural network model.
A second module: the channel number of the pathological feature map is increased from 512 to 1024 by adopting 1 convolution module ConvGroup and 1 descending convolution block reduce_block, and then the channel number of the pathological feature map is decreased from 1024 to 512, and the descending convolution block reduce_block is used for halving the width and height dimensions of the pathological feature map and reducing the dimension in the feature dimension.
And a third module: the third module and the second module are identical in structure and function. The depth of the convolutional neural network model can be increased, and the fitting capacity of the convolutional neural network model is improved.
A fourth module: and (3) after the channel number of the pathological feature map is increased from 512 to 1024 by adopting 1 convolution module ConvGroup, sequentially processing the normalization layer BN layer and the nonlinear activation function layer ReLU layer. Similarly, the BN layer may promote stability of the convolutional neural network model, and the ReLU layer may promote nonlinear learning ability of the convolutional neural network model.
A fifth module: and taking the maximum value of each channel plane of the pathological feature map by adopting 1 two-dimensional global maximum pooling layer GlobalMaxPool2d, wherein the dimension of the pathological feature map is M multiplied by N multiplied by K, so that 1 feature vector with 1024 length can be obtained, and then the feature vector with 1024 length is reduced to 2 by a full-connection layer Linear, so that the sixth module can conveniently conduct two-classification.
It can be understood that before the convolutional neural network finally enters the sixth module to output the identification result, the fifth module sequentially adopts global maximum pooling of channels and a full-connection layer with a fixed size, so that the convolutional neural network can be matched with the input of pathological feature maps with different sizes; the size of the convolution neural network model gradually decreases the size amplitude of the pathological feature map in the processing process of the pathological feature map due to the fact that the channel number of the pathological feature map is large and the size of the channel number is small.
A sixth module: may be a Softmax classifier. For example, the classification result of the pathological feature map can be output through a Softmax classifier, and the classification result is the identification result. The recognition result may be, for example, a probability of positive or negative. The normalization formula of the Softmax classifier is as follows:
in the training stage of the convolutional neural network model, training can be performed based on a plurality of pathological sample images. a, a ij Representing the probability that the ith pathology sample image belongs to the jth class; z ij Representing that the ith pathology sample image corresponds to a feature value on the jth class; c represents the total number of classifications. Similarly, in the use stage of the convolutional neural network model, training may be performed based on 1 batch of pathological sample images, that is, a plurality of pathological sample images, which will not be described herein.
In some examples, for each pathology image output of length 2, its vector format may be: "probability that the pathology image is a normal pathology image; the pathological image is the probability of a papillary carcinoma pathological slice. In practical application, a class corresponding to the Index of the maximum classification probability value of the vector array may be used as the classification result of the pathological image.
Further, in some embodiments of the present application, the convolution module may include a convolution block and a feature channel stitching module; the convolution block comprises the normalization layer, the nonlinear activation function layer and the two-dimensional convolution operation layer which are sequentially connected.
As shown in fig. 3, a process flow diagram of the convolution module is shown. The 1 convolution module ConvGroup comprises 2 convolution blocks. For the input feature x, the convolution module can generate a feature Y1 through a convolution block conv_block, then the feature x and the feature Y1 are spliced in the feature channel dimension through a feature channel splicing module, and the spliced output feature is marked as a feature x||Y1; and then, using the characteristic x I Y1 as an input characteristic, generating a characteristic Y2 by using another convolution block conv_block, and splicing the characteristic x, the characteristic Y1 and the characteristic Y2 in the characteristic channel dimension by a characteristic channel splicing module, wherein the characteristic output after splicing is marked as the characteristic x I Y1I Y2.
As shown in fig. 4, is a process flow diagram of the convolution block. Each convolution block comprises a normalization layer BN layer, a nonlinear activation function layer ReLU layer and a two-dimensional convolution operation layer Conv2d layer which are sequentially connected, wherein the two-dimensional convolution operation layer Conv2d layer can specifically adopt a 3 multiplied by 3 convolution kernel.
In some embodiments of the present application, the down convolution block includes the normalization layer, the two-dimensional convolution operation layer, and an average pooling layer connected in sequence.
As shown in fig. 5, a flowchart of the process of the down convolutional block is shown. The descending convolution block reduce_block is specifically a channel/size descending block, and comprises a normalization layer BN layer, a two-dimensional convolution operation layer Conv2d layer and a two-dimensional average pooling layer AvgPool2d layer which are sequentially connected. The Conv2d layer of the two-dimensional convolution operation layer can specifically adopt a 1 multiplied by 1 convolution kernel for reducing the number of dimension channels; the two-dimensional average pooling layer AvgPool2d is used for reducing the characteristic amplitude of the dimension-reducing pathological feature map, namely halving the length and width of the corresponding pathological feature map.
Compared with the related art, the specific structure of the convolutional neural network model is redesigned in order to adapt to the processing of the target feature map obtained after the target image of micro factors and macro factors in the image is required to be considered.
In practical application, a thyroid pathology image is used as a data set to identify and predict whether the thyroid pathology image is papillary cancer, and the identification result is a negative or positive 2-classification result. The total number of data sets is 1245, and each thyroid pathology image has a negative label or a positive label, wherein the number of positive images is 1019, and the number of negative images is 226. According to 7:3 (training set: validation/test set, 871:374) ratio to randomly divide the number of samples, training the convolutional neural network model to obtain a confusion matrix of the prediction results as shown in table 2 below:
TABLE 2
The corresponding indices are shown in table 3 below:
TABLE 3 Table 3
Compared with other algorithms, the convolutional neural network model provided by the embodiment of the application has higher recognition accuracy, and is shown in the following table 4:
TABLE 4 Table 4
Example III
The third embodiment of the present application is an application embodiment provided according to any one or any several of the first to second embodiments. In the implementation of the present application, the target image is specifically a pathological image.
Some embodiments of the present application provide a system for identifying a pathological image, where the system includes an image segmentation unit, a feature extraction unit, an image feature generation unit, and an image input unit;
the image segmentation unit is used for segmenting the pathological image into a plurality of pathological image blocks;
the feature extraction unit is used for extracting pathological features of the pathological image blocks;
the image feature generation unit is used for generating a pathological feature map corresponding to the pathological image according to the pathological feature;
the image input unit is configured to input the pathological feature map to a pre-trained convolutional neural network model according to any one of the first to second embodiments, so that the convolutional neural network model performs feature recognition on the pathological feature map, and a recognition result is obtained.
The respective units described above are specifically described below.
In some examples, the pathological slice of the cell tissue can be scanned by a microscope, so that a pathological image occupying a large memory is generated, and the pathological image scanned by the microscope is in svs format and can be opened and browsed by using tool software such as ASAP. Then, the image segmentation unit may segment the pathology image to obtain m×n pathology image blocks, where M represents the number of rows and N represents the number of columns, as shown in fig. 6. The size of the pathological image block can be 512×512, 1024×1024, 2048×2048, or the like. In some examples, the image segmentation unit may be an OpenSlide library interface, for example, the OpenSlide library interface may be used to segment a svs format pathology image.
For example, assuming that the pathological image has a size of w×h pixels, the image segmentation unit may segment the pathological image into pathological image blocks of 2048×2048, the number of lines
Column number
In some examples, portions of the pathology image having fewer rows and/or columns than 2048 may be filled with blanks. It will be appreciated that +.>
The symbology is rounded up.
In some examples, the pathological features of the pathological image blocks may be extracted by a feature extraction unit. The feature extraction unit may be a feature extraction model trained based on a convolutional neural network such as res net disclosed in the related art. For example, the ResNet50 convolutional neural network disclosed in the related technology can be utilized to delete the last fully connected layer of the ResNet50 convolutional neural network and all layers after the fully connected layer, and further training is performed to obtain a feature extraction model, wherein the implementation of the feature extraction model mainly comprises a training stage and a using stage.
Specifically, during the training phaseThe method comprises the steps that a transfer learning technology can be used, a transfer ResNet convolutional neural network takes weight parameters which are determined based on an image set of the ImageNet as a pre-training model, and then a large number of pathological image blocks are input into the pre-training model for fine adjustment learning, so that a feature extraction model for extracting pathological features of the pathological image blocks is obtained. Wherein, in some examples, before inputting a large number of pathological image blocks into the pre-training model for fine-tuning learning, a simple label classification can be performed manually on whether a focus area exists in the large number of pathological image blocks; in some other examples, the feature extraction model may also be trained using an unsupervised learning mode, as embodiments of the present application are not specifically limited. In the using stage, a plurality of pathological image blocks of the pathological image can be input into the feature extraction model, so that the feature extraction model performs feature extraction on each pathological image block to obtain feature vectors for representing the pathological features, for example, the feature vectors are F, F epsilon R 2048 Where R represents a rational number and 2048 represents the dimension of feature extraction using a res net50 convolutional neural network.
Further, in some embodiments of the present application, the image feature generating unit may include an acquiring module and a stitching module; the acquisition module is used for acquiring the characteristic length of the pathological image block; and the splicing module is used for splicing the pathological image blocks and the characteristic length to generate a pathological feature map corresponding to the pathological image.
For example, the obtaining module may determine the feature length of each pathological image block according to the dimension of feature extraction performed by the res net convolutional neural network, for example, the dimension of feature extraction performed by using the res net50 convolutional neural network is 2048, and the value of the feature length of each pathological image block is 2048; and then the splicing module splices according to the size of the pathological image blocks and the characteristic length of each pathological image block to generate a pathological feature map corresponding to the pathological image. For example, if the dimension of the pathological image block is m×n, the dimension of the pathological feature map is m×n×k. Wherein K represents a characteristic length of each of the pathological image blocks.
After the preprocessing of the image segmentation unit, the feature extraction unit and the image feature generation unit, each pathological image may form 1 corresponding m×n×k-dimensional pathological feature map, where K is a fixed value, and as mentioned above, for example, the dimension of feature extraction by using a res net50 convolutional neural network is 2048, the value of the feature length of each pathological image block is 2048. In reality, each pathology image has different sizes, so that the values of M and N are correspondingly different from each other. By sample statistics from the actual thyroid pathology image, the values of M and N are mainly distributed between [10,60], typically m+.n.
Further, the image input unit may input the pathological feature map with the dimension of mxnxk to a pre-trained convolutional neural network model according to any one of the first to second embodiments, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
It is easy to find that, compared with the related art, the recognition system for the pathological image provided by the embodiment of the application can perform feature recognition on the pathological image, and the pathological features of the pathological image blocks are extracted by dividing the pathological image into a plurality of pathological image blocks, so that the requirements for microscopic features such as local cell nucleus morphology and the like in the pathological image are met; by generating a pathological feature map corresponding to the pathological image according to the pathological features, the requirements of macroscopic characteristics such as cell growth vigor and the like of certain areas in the pathological image are met; the feature recognition of 1 pathological image is converted into the feature recognition of 1 pathological feature image, so that the data volume of the pathological image is reduced by thousands of times. For example, if the pathology image is segmented into pathology image blocks of 2048×2048, the feature length of which is 2048, the dimension reduction rate is 2048×2048×3/(2048×4) =1536 times. Therefore, when the convolutional neural network model is used for carrying out feature recognition on the pathological feature map, the consumption of GPU video memory can be greatly reduced while the accuracy of recognition is ensured.
Example IV
Some embodiments of the present application provide a method for identifying a pathological image, which is applied to the system according to the third embodiment, and the method may include the following steps, as shown in fig. 7:
step S101, segmenting the pathological image into a plurality of pathological image blocks;
step S102, extracting pathological features of the pathological image blocks;
step S103, generating a pathological feature map corresponding to the pathological image according to the pathological feature;
step S104, inputting the pathological feature map to a pre-trained convolutional neural network model according to any one of the above embodiments, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
It is to be noted that, in this embodiment of the present application, the implementation details in the third embodiment are the same as those applicable thereto, and in order to avoid repetition, the details are not repeated here.
In addition, the embodiment of the application further provides an electronic device, the structure of which is shown in fig. 7, the device includes a memory 11 for storing computer readable instructions and a processor 12 for executing the computer readable instructions, where the computer readable instructions, when executed by the processor, trigger the processor to execute the virtual content distribution method.
The methods and/or embodiments of the present application may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. The above-described functions defined in the method of the present application are performed when the computer program is executed by a processing unit.
It should be noted that, the computer readable medium described in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowchart or block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more computer readable instructions executable by a processor to implement the steps of the methods and/or techniques of the various embodiments of the present application described above.
In a typical configuration of the present application, the terminals, the devices of the services network each include one or more processors (CPUs), graphics Processors (GPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium which can be used to store information that can be accessed by a computing device.
In addition, the embodiment of the application also provides a computer program which is stored in the computer equipment, so that the computer equipment executes the method for executing the control code.
It should be noted that the present application may be implemented in software and/or a combination of software and hardware, for example, using Application Specific Integrated Circuits (ASIC), a general purpose computer or any other similar hardware device. In some embodiments, the software programs of the present application may be executed by a processor to implement the above steps or functions. Likewise, the software programs of the present application (including associated data structures) may be stored on a computer readable recording medium, such as RAM memory, magnetic or optical drive or diskette and the like. In addition, some steps or functions of the present application may be implemented in hardware, for example, as circuitry that cooperates with the processor to perform various steps or functions.
It will be evident to those skilled in the art that the present application is not limited to the details of the foregoing illustrative embodiments, and that the present application may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the apparatus claims can also be implemented by means of one unit or means in software or hardware. The terms first, second, etc. are used to denote a name, but not any particular order.
Claims (9)
1. The convolutional neural network model at least comprises a first module, a second module, a third module, a fourth module, a fifth module and a sixth module which are connected in sequence;
the first module is used for carrying out the reduction processing on the channel number of the received target feature map;
the second module and the third module are respectively and sequentially used for performing first conversion on the channel number after the reduction processing and performing dimension reduction processing on the dimension of the target feature map;
the fourth module is configured to perform a second conversion based on the first converted channel number;
the fifth module is configured to obtain a feature vector of each channel plane after the second conversion;
and the sixth module is used for outputting a classification result according to the feature vector.
2. The convolutional neural network model of claim 1, wherein,
the first module comprises a convolution operation layer, a normalization layer and a nonlinear activation function layer;
the second module and the third module respectively comprise a convolution module and a descending convolution block;
the fourth module comprises the convolution module, the normalization layer and the nonlinear activation function layer;
the fifth module comprises a global maximum pooling layer and a full connection layer;
the sixth module is a classifier.
3. The convolutional neural network model of claim 2, wherein the convolutional module comprises a convolutional block and a characteristic channel stitching module;
the convolution block comprises the normalization layer, the nonlinear activation function layer and the two-dimensional convolution operation layer which are sequentially connected.
4. The convolutional neural network model of claim 2, wherein the falling convolution block comprises the normalization layer, the two-dimensional convolution operation layer, and an average pooling layer connected in sequence.
5. A pathological image recognition system, which is characterized by comprising an image segmentation unit, a feature extraction unit, an image feature generation unit and an image input unit;
the image segmentation unit is used for segmenting the pathological image into a plurality of pathological image blocks;
the feature extraction unit is used for extracting pathological features of the pathological image blocks;
the image feature generation unit is used for generating a pathological feature map corresponding to the pathological image according to the pathological feature;
the image input unit is configured to input the pathological feature map to a pre-trained convolutional neural network model according to any one of claims 1 to 4, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
6. The system of claim 5, wherein the image feature generation unit comprises an acquisition module and a stitching module;
the acquisition module is used for acquiring the characteristic length of the pathological image block;
and the splicing module is used for splicing the pathological image blocks and the characteristic length to generate a pathological feature map corresponding to the pathological image.
7. A method of identifying a pathology image, applied to the system of claim 5 or 6, said method comprising:
dividing the pathological image into a plurality of pathological image blocks;
extracting pathological features of the pathological image blocks;
generating a pathological feature map corresponding to the pathological image according to the pathological feature;
inputting the pathological feature map into a pre-trained convolutional neural network model according to any one of claims 1 to 4, so that the convolutional neural network model performs feature recognition on the pathological feature map to obtain a recognition result.
8. An electronic device, the device comprising:
one or more processors; and
a memory storing computer program instructions that, when executed, cause the processor to perform the method of claim 7.
9. A computer readable medium having stored thereon computer program instructions executable by a processor to implement the method of claim 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310348734.8A CN116385393B (en) | 2023-04-04 | 2023-04-04 | Convolutional neural network model, pathological image recognition system, method, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310348734.8A CN116385393B (en) | 2023-04-04 | 2023-04-04 | Convolutional neural network model, pathological image recognition system, method, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116385393A true CN116385393A (en) | 2023-07-04 |
| CN116385393B CN116385393B (en) | 2025-02-18 |
Family
ID=86962911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310348734.8A Active CN116385393B (en) | 2023-04-04 | 2023-04-04 | Convolutional neural network model, pathological image recognition system, method, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116385393B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112233117A (en) * | 2020-12-14 | 2021-01-15 | 浙江卡易智慧医疗科技有限公司 | New coronary pneumonia CT detects discernment positioning system and computing equipment |
| KR102283265B1 (en) * | 2020-12-31 | 2021-07-29 | 가천대학교 산학협력단 | Method and apparatus for colorectal disease classification using scaled dilation in convolutional neural network |
| CN113906472A (en) * | 2019-04-04 | 2022-01-07 | 普雷萨根私人有限公司 | Method and system for selecting embryos |
| US20220157048A1 (en) * | 2019-02-08 | 2022-05-19 | Singapore Health Services Pte Ltd | Method and System for Classification and Visualisation of 3D Images |
| US20220230302A1 (en) * | 2019-06-24 | 2022-07-21 | Zhejiang University | Three-dimensional automatic location system for epileptogenic focus based on deep learning |
| CN115035074A (en) * | 2022-06-17 | 2022-09-09 | 重庆大学 | Cervical epithelial tissue pathological image identification method based on global spatial perception network |
| CN115830001A (en) * | 2022-12-22 | 2023-03-21 | 抖音视界有限公司 | Intestinal image processing method and device, storage medium and electronic equipment |
-
2023
- 2023-04-04 CN CN202310348734.8A patent/CN116385393B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220157048A1 (en) * | 2019-02-08 | 2022-05-19 | Singapore Health Services Pte Ltd | Method and System for Classification and Visualisation of 3D Images |
| CN113906472A (en) * | 2019-04-04 | 2022-01-07 | 普雷萨根私人有限公司 | Method and system for selecting embryos |
| US20220230302A1 (en) * | 2019-06-24 | 2022-07-21 | Zhejiang University | Three-dimensional automatic location system for epileptogenic focus based on deep learning |
| CN112233117A (en) * | 2020-12-14 | 2021-01-15 | 浙江卡易智慧医疗科技有限公司 | New coronary pneumonia CT detects discernment positioning system and computing equipment |
| KR102283265B1 (en) * | 2020-12-31 | 2021-07-29 | 가천대학교 산학협력단 | Method and apparatus for colorectal disease classification using scaled dilation in convolutional neural network |
| CN115035074A (en) * | 2022-06-17 | 2022-09-09 | 重庆大学 | Cervical epithelial tissue pathological image identification method based on global spatial perception network |
| CN115830001A (en) * | 2022-12-22 | 2023-03-21 | 抖音视界有限公司 | Intestinal image processing method and device, storage medium and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| AMIRREZA MAHBOD等: "Breast Cancer Histological Image Classification Using Fine-Tuned Deep Network Fusion", PROCEEDINGS OF INTERNATIONAL CONFERENCE ON IMAGE ANALYSIS& RECOGNITION, 31 December 2018 (2018-12-31), pages 754 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116385393B (en) | 2025-02-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11373305B2 (en) | Image processing method and device, computer apparatus, and storage medium | |
| CN114283164B (en) | Breast cancer pathological section image segmentation prediction system based on UNet3+ | |
| US20170076448A1 (en) | Identification of inflammation in tissue images | |
| JP7739442B2 (en) | Image clustering method, device, computer device, and computer program | |
| CN118710952B (en) | Hierarchical prediction method and hierarchical prediction system for grape leaf disease degree through cross-level feature fusion | |
| KR102913198B1 (en) | Frequency based image forgery location detection method and system through self-attention and multi-task learning | |
| CN118053022A (en) | Skin image classification method, device, chip and terminal | |
| CN118967706B (en) | An image segmentation method based on medical hyperspectral image segmentation network | |
| CN117437423A (en) | Weakly supervised medical image segmentation method and device based on SAM collaborative learning and cross-layer feature aggregation enhancement | |
| CN117911757A (en) | Medical image classification method based on biased label learning model | |
| CN113724267B (en) | A method and device for segmenting tumors in breast ultrasound images | |
| CN114782355A (en) | Gastric cancer digital pathological section detection method based on improved VGG16 network | |
| CN115908363B (en) | Tumor cell statistics method, device, equipment and storage medium | |
| CN113033371A (en) | CSP model-based multi-level feature fusion pedestrian detection method | |
| CN120451188B (en) | Weak supervision cell nucleus segmentation method based on wavelet differential convolution and region expansion | |
| KR20240052193A (en) | Method and Apparatus for Analyzing Digital Pathological Image Based on Multi-scale Vision Transformer | |
| WO2023014789A1 (en) | System and method for pathology image analysis using a trained neural network and active learning framework | |
| CN118762041B (en) | A CT image classification method based on image segmentation and RU classifier and its application | |
| CN115641317A (en) | Dynamic knowledge backtracking multi-instance learning and image classification method for pathological images | |
| CN116385393B (en) | Convolutional neural network model, pathological image recognition system, method, equipment and medium | |
| CN120708868A (en) | A cervical cell lesion detection method based on hypergraph convolutional network | |
| CN115861604B (en) | Cervical tissue image processing method, cervical tissue image processing device, computer equipment and storage medium | |
| US20260024316A1 (en) | Object detection and image segmentation for eukaryotic cells | |
| CN119810831A (en) | An automatic blood cell detection method based on classification strategy | |
| CN117218094A (en) | Methods, equipment and media for obtaining characteristic information of pathological images |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |