TWI767506B - Image recognition method, training method and equipment of recognition model - Google Patents

Abstract

The embodiment of the application provides an image recognition method, a training method for recognition model, and related devices and equipment, wherein, the image recognition method includes: obtaining the pathological image to be identified; using the detection sub model in the recognition model to carry out target detection of the pathological image to be recognized, obtaining the detection area containing the target cells in the pathological image to be recognized; and using the points in the recognition model The first classification of the detection region is carried out by the class sub model, and the category of the target cell is obtained.

Description

Image recognition method, training method and equipment of recognition model

The present invention relates to the technical field of artificial intelligence, in particular to an image recognition method, a training method and equipment for a recognition model.

With the development of artificial intelligence technologies such as neural networks and deep learning, training neural network models and using the trained neural network models to meet relevant business needs in the medical field are gradually gaining popularity.

In the related business needs, due to the serious shortage of domestic cytopathologists, artificial intelligence technology is used to assist in the identification of pathological images to screen target cells such as diseased cells. important meaning. In view of this, how to accurately and efficiently identify target cells in pathological images has become an urgent problem to be solved.

Embodiments of the present invention provide an image recognition method, a training method and equipment for a recognition model.

An embodiment of the present invention provides an image recognition method, comprising: acquiring a pathological image to be recognized; using a detection sub-model in a recognition model to perform target detection on the pathological image to be recognized, to obtain a detection area containing target cells in the pathological image to be recognized ; Use the classification sub-model in the recognition model to perform the first classification process on the detection area to obtain the classification of the target cell.

Therefore, by using the detection sub-model in the recognition model to perform target detection on the acquired pathological image to be recognized, the detection area containing the target cells in the to-be-recognized pathological image is obtained, and then the analysis sub-model in the recognition model is used to detect the target. The first classification process of regional inspection is to obtain the type of target cells, and then the detection of target cells can be carried out first, and then the classification of target cells can be carried out to separate detection and classification, so that target cells in pathological images can be accurately and efficiently identified.

In some embodiments of the present invention, using the detection sub-model in the recognition model to perform target detection on the pathological image to be recognized, and obtaining the detection area containing the target cells in the pathological image to be recognized includes: using the first part of the detection sub-model to be recognized The pathological image is subjected to a second classification process to obtain an image classification result of the pathological image to be identified, wherein the image classification result is used to indicate whether the pathological image to be identified contains target cells; if the image classification result indicates the pathological image to be identified If the image contains target cells, the second part of the detection sub-model is used to perform region detection on the pathological image to be recognized to obtain a detection region containing the target cells.

Therefore, by performing the second classification process on the pathological image to be recognized by the first part of the detection sub-model, an image classification result of the pathological image to be recognized is obtained, and the image classification result is used to indicate whether the pathological image to be recognized contains target cells , when the image classification result indicates that the pathological image to be identified contains target cells, the second part of the detection sub-model is used to perform region detection on the pathological image to be identified to obtain the detection area containing the target cells, so the detection of target cells can be achieved. Dynamic detection improves the efficiency of target cell identification.

In some embodiments of the present invention, after performing the second classification process on the pathological image to be recognized by using the first part of the detection sub-model to obtain the image classification result of the pathological image to be recognized, the method further includes: if the image classification result represents If the pathological image to be recognized does not contain target cells, the first part outputs a detection result prompt that the pathological image to be recognized does not contain target cells.

Therefore, when the image classification result indicates that the pathological image to be recognized does not contain target cells, the first part outputs a detection result prompt that the pathological image to be recognized does not contain target cells, so the dynamic detection of target cells can be realized, and the target cells can be improved. identification efficiency.

In some embodiments of the present invention, using the detection sub-model in the recognition model to perform target detection on the pathological image to be recognized, and obtaining the detection area containing the target cells in the pathological image to be recognized further includes: using the third part of the detection sub-model Feature extraction is performed on the pathological image to be recognized to obtain image features of the pathological image to be recognized.

Therefore, by extracting the features of the pathological image to be recognized in the third part of the detection sub-model, the image features of the pathological image to be recognized can be obtained, so that the pathological image to be recognized can be processed first, and then the detection sub-model can be used on this basis. The model performs other processing, so it can help to improve the running efficiency of the model.

In some embodiments of the present invention, using the first part of the detection sub-model to perform a second classification process on the pathological image to be recognized to obtain an image classification result of the pathological image to be recognized, including: using the first part of the detection sub-model to compare the image The second classification process is performed on the image features to obtain an image classification result of the pathological image to be recognized.

Therefore, using the first part of the detection sub-model to perform the second classification process on the image features extracted by the third part to obtain the image classification result of the pathological image to be recognized, which can improve the accuracy of the classification process.

In some embodiments of the present invention, using the second part of the detection sub-model to perform region detection on the pathological image to be recognized to obtain a detection region containing the target cells, comprising: using the second part of the detection sub-model to perform region detection on image features Detected to obtain a detection area containing the target cells.

Therefore, using the second part of the detection sub-model to perform region detection on image features to obtain a detection region containing target cells can help to improve the accuracy of target cell identification.

In some embodiments of the present invention, the first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; wherein the feature extraction network includes a deformable convolution layer, global information at least one of the enhancement mods.

Therefore, by setting the feature extraction network to include a deformable convolutional layer, the accuracy of identifying polymorphic target cells can be improved, and by setting the feature extraction network to include at least one of the global information enhancement modules, It is beneficial to obtain long-distance and dependent features, and is beneficial to improve the accuracy of target cell identification.

In some embodiments of the present invention, using the classification sub-model in the recognition model to perform a first classification process on the detection area to obtain the category of the target cell, including: using the classification sub-model to perform feature extraction on the detection area of the pathological image to be recognized, The image features of the detection area are obtained; the first classification processing is performed on the image features of the detection area to obtain the category of the target cells.

Therefore, by performing feature extraction on the detection area of the pathological image to be recognized, the image features of the detection area are obtained, and the first classification processing is performed on the image features of the detection area to obtain the category of target cells, which can help improve the efficiency of classification processing. efficiency.

In some embodiments of the present invention, the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to indicate the diseased degree of the target cell.

Therefore, the target cell includes any one of a single diseased cell and a diseased cell cluster, which can help to identify a single diseased cell and a diseased cell cluster, and the type of the target cell is used to indicate the degree of disease of the target cell, which is conducive to achieving the target Lesion grading of cells.

An embodiment of the present invention provides a training method for an identification model, the identification model includes a detection sub-model and a classification sub-model, and the training method includes: acquiring a first sample image and a second sample image, wherein the first sample image is The actual area corresponding to the target cell is marked, and the actual category of the target cell is marked in the second sample image; the detection sub-model is used to perform target detection on the first sample image, and it is obtained that the first sample image contains the target cell and use the classification sub-model to perform the first classification process on the second sample image to obtain the predicted category of the target cell; based on the actual area and the predicted area, determine the first loss value of the detection sub-model, and based on the actual category and The category is predicted, and the second loss value of the classification sub-model is determined; the parameters of the detection sub-model and the classification sub-model are adjusted correspondingly by using the first loss value and the second loss value.

Therefore, in the training process, the detection of target cells can be carried out first, and then the classification of target cells can be carried out, and the detection and classification can be separated, so as to solve the problem of unbalanced sample data categories, which can help to improve the accuracy of the model obtained by training. Therefore, it can help to improve the accuracy and efficiency of target cell identification.

In some embodiments of the present invention, using the detection sub-model to perform target detection on the first sample image, and obtaining the predicted region containing the target cells in the first sample image includes: performing a second step on the first sample image. The classification process is performed to obtain an image classification result of the first sample image, wherein the image classification result is used to indicate whether the first sample image contains target cells; if the image classification result indicates that the first sample image contains If the target cell is included, the region detection is performed on the first sample image to obtain a predicted region including the target cell.

Therefore, in the training process, when the image classification result indicates that the first sample image contains target cells, the region detection is performed on the first sample image to obtain the predicted area containing the target cells, which can enhance the model to identify positive and negative The ability of the sample to reduce the probability of false detection is conducive to improving the accuracy of the model obtained by training, which can help to improve the accuracy of target cell recognition.

In some embodiments of the present invention, the detection sub-model is used to perform target detection on the first sample image to obtain a predicted region containing target cells in the first sample image, and the classification sub-model is used to perform target detection on the second sample image. Like performing the first classification process to obtain the predicted category of the target cell, the method further includes: performing data enhancement on the first sample image and the second sample image; and/or, combining the first sample image and the second sample image The primitive values in the sample image are normalized; the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to represent the diseased degree of the target cell.

Therefore, by enhancing the data of the first sample image and the second sample image, the sample diversity can be improved, which is beneficial to avoid overfitting and improve the generalization performance of the model; The primitive values in the sample image are normalized, which can help to improve the convergence speed of the model; the target cell includes any one of a single diseased cell or a diseased cell cluster, and the category of the target cell is used to represent the target cell's The degree of lesion can help to identify a single lesion cell and a cluster of lesion cells, and the category of the target cell is used to represent the lesion degree of the target cell, which is beneficial to achieve the lesion classification of the target cell.

An embodiment of the present invention provides an image recognition device, including: an image acquisition module, an image detection module and an image classification module, the image acquisition module is configured to acquire pathological images to be recognized; the image detection module The group is configured to use the detection sub-model in the recognition model to perform target detection on the pathological image to be recognized, and obtain the detection area containing the target cells in the pathological image to be recognized; the image classification module is configured to use the classification sub-model in the recognition model. A first classification process is performed on the detection area to obtain the category of the target cell.

An embodiment of the present invention provides a training device for an identification model. The identification model includes a detection sub-model and a classification sub-model. The training device for the identification model includes: an image acquisition module, a model execution module, a loss determination module, and a parameter adjustment module , the image acquisition module is configured to acquire a first sample image and a second sample image, wherein the first sample image is marked with the actual area corresponding to the target cell, and the second sample image is marked with the target The actual category of the cell; the model execution module is configured to use the detection sub-model to perform target detection on the first sample image, obtain the predicted area containing the target cell in the first sample image, and use the classification sub-model to detect the second sample. The image is subjected to the first classification process to obtain the predicted category of the target cell; the loss determination module is configured to determine the first loss value of the detection sub-model based on the actual area and the predicted area, and determine the classification sub-model based on the actual category and the predicted category. The parameter adjustment module is configured to use the first loss value and the second loss value to correspondingly adjust the parameters of the detection sub-model and the classification sub-model.

An embodiment of the present invention provides an electronic device, including a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, so as to implement the image recognition method in one or more of the foregoing embodiments, or The training method of the recognition model in one or more of the above embodiments is implemented.

An embodiment of the present invention provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the image recognition method in one or more of the foregoing embodiments is implemented, or one or more of the foregoing embodiments are implemented. The training method of the recognition model in .

An embodiment of the present invention provides a computer program, including computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the program for implementing one or more of the foregoing embodiments. The image recognition method, or the training method of the recognition model in one or more of the above embodiments.

In the above scheme, by using the detection sub-model in the recognition model to perform target detection on the acquired pathological image to be recognized, the detection area containing the target cells in the to-be-recognized pathological image is obtained, and then the analysis sub-model in the recognition model is used to detect the target cells. The detection area is inspected by the first classification process to obtain the type of the target cells, and then the detection of the target cells can be carried out first, and then the classification of the target cells can be carried out, and the detection and classification can be separated, so that the target cells in the pathological image can be accurately and efficiently identified. .

The solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

In the following description, for the purpose of illustration rather than limitation, specific details, such as specific system structures, interfaces, and technologies, are set forth in order to provide a thorough understanding of the embodiments of the present invention.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the character "/" in this text generally indicates that the contextually related objects are in an "or" relationship. Also, "multiple" herein means two or more than two.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an image recognition method provided by an embodiment of the present invention. Specifically, the following steps may be included.

Step S11: Obtain the pathological image to be identified.

The pathological images to be identified may include, but are not limited to: cervical pathological images, liver pathological images, and kidney pathological images, which are not limited herein.

Step S12: Use the detection sub-model in the recognition model to perform target detection on the pathological image to be recognized, and obtain a detection area containing the target cells in the pathological image to be recognized.

The recognition model includes a detection sub-model. In a specific implementation scenario, the detection sub-model can use the Faster RCNN (Region with Convolutional Neural Networks) network model. In another specific implementation scenario, the detection sub-model can also use Fast RCNN, YOLO (You Only Look Once), etc., which is not limited here.

Use the detection sub-model to detect the pathological image to be recognized, and obtain the detection area containing the target cells in the pathological image to be recognized. For example, by detecting the cervical pathological image, the detection area containing the squamous epithelial cells in the cervical pathological cells is obtained; Alternatively, the pathological image of the liver is detected to obtain the detection area containing the diseased cells in the pathological image of the liver. When the pathological image to be identified is another image, the same can be deduced, and no examples are provided here. In an implementation scenario, the detection area can be specifically represented by the center coordinates of a rectangle containing the target cells and the length and width of the rectangle. The primitive coordinates (50, 60) are the center, a rectangle with a length of 10 and a width of 20. In addition, it can also be represented by the center coordinate of a rectangle containing the target cell and the ratio of the length and width of the rectangle to a preset rectangle. , for example, the preset rectangle can be a rectangle with a length of 10 and a width of 20, then (50, 60, 1, 1) can be used to represent a pathological image that is located in the pathological image to be recognized with the primitive coordinates (50, 60) as The center is a rectangle with a length of 10 and a width of 20, which is not limited here.

In some embodiments of the present invention, the pathological image to be recognized may also be an image that does not contain target cells. In this case, the detection sub-model in the recognition model is used to perform target detection on the pathological image to be recognized. Since the detection area is not obtained , it can output a prompt that the pathological image to be recognized does not contain target cells, thereby eliminating the steps of subsequent classification processing and improving the efficiency of the model. For example, the prompt that the cervical pathological image does not contain squamous epithelial cells can be directly output, and other pathological images can be deduced in the same way, which will not be exemplified here.

In some embodiments of the present invention, please refer to FIG. 2 , which is a schematic state diagram of an image recognition method provided by an embodiment of the present invention. As shown in FIG. 2 , the pathological image to be recognized is a cervical pathological image, and the pathological image to be recognized is subjected to target detection by the detection sub-model in the recognition model, and two detection areas containing target cells are obtained.

Step S13: Use the classification sub-model in the recognition model to perform a first classification process on the detection area to obtain the type of the target cell.

The identification model may also include a classification sub-model, and in a specific implementation scenario, the classification sub-model may adopt the EfficientNet network model. In another specific implementation scenario, the classification sub-model may also use ResNet, MobileNet, etc., which is not limited here.

The classification sub-model in the recognition model is used to classify the detection area, and the category of the target cell can be obtained. Specifically, in order to improve the classification efficiency, the classification sub-model can be used to perform feature extraction on the detection area of the pathological image to be recognized to obtain the detection area. The first classification process is performed on the image features of the detection area to obtain the category of the target cell. For example, the image features of the detection area can be pooled and fully connected to obtain the category of the target cell, which will not be repeated here.

In some embodiments of the present invention, in order to achieve grading of the target cells, the category of the target cells may represent the degree of the target cells' lesions. Taking the image of the lesion to be identified as a cervical pathological image as an example, the target cells may specifically include but are not limited to the following categories: High-grade Squamous Intraepithelial Lesion (HSIL), mild squamous cell epithelium Low-grade Squamous Intraepithelial Lesion (LSIL), Atypical Squamous Cells of Undetermined Significance (ASC-US), Atypical Squamous Cells Squamous Cells-cannot exclude HSIL, ASC-H). When the pathological image to be identified is other pathological images, the same can be deduced, and no examples are given here. In an implementation scenario, the target cell may include any one of a single diseased cell or a diseased cell cluster, so that a single diseased cell or a diseased cell cluster can be identified.

In some embodiments of the present invention, please continue to refer to FIG. 2 , the classification sub-model performs classification processing on the two detection areas detected by the detection sub-model respectively, and obtains the categories of the target cells contained in the two detection areas: wherein The target cells in one test area were high-grade squamous intraepithelial neoplasia (HSIL), and the target cells in the other were atypical squamous cells for which high-grade intraepithelial neoplasia cannot be excluded (ASC-H).

In some embodiments of the present invention, the classification sub-model may further perform a first classification process on the detection area to obtain the category of the target cell and its confidence level, where the confidence level indicates that the real category of the target cell is the same as the category predicted by the model. Credibility, the higher the confidence, the higher the credibility. Please continue to refer to Figure 2. The classification sub-models respectively classify the detection areas to obtain the category and confidence of the target cells. The target cells in one of the detection areas are high-grade squamous cell intraepithelial neoplasia (HSIL), and It has a confidence level of 0.97 (ie, 97% confidence), and the target cells in the other detection area are atypical squamous cells (ASC-H) that cannot be excluded from high-grade intraepithelial neoplasia, and its confidence level is 0.98 ( i.e. 98% confidence).

In the above scheme, by using the detection sub-model in the recognition model to perform target detection on the acquired pathological image to be recognized, the detection area containing the target cells in the to-be-recognized pathological image is obtained, and then the analysis sub-model in the recognition model is used to detect the target cells. The detection area is inspected by the first classification process to obtain the type of the target cells, and then the detection of the target cells can be carried out first, and then the classification of the target cells can be carried out, and the detection and classification can be separated, so that the target cells in the pathological image can be accurately and efficiently identified. .

Please refer to FIG. 3 , which is a schematic flowchart of an image recognition method provided by an embodiment of the present invention. Specifically, the following steps may be included.

Step S31: Acquire the pathological image to be identified.

For details, please refer to the relevant steps in the foregoing embodiments.

Step S32: Use the first part of the detection sub-model to classify the pathological image to be recognized, and obtain an image classification result of the pathological image to be recognized.

The image classification result is used to indicate whether the target cell is included in the pathological image to be identified. Specifically, "0" may be used to indicate that the target cell is not included in the pathological image to be identified, and "1" may be used to indicate the pathological image to be identified. The target cells are included, and are not limited here.

In some embodiments of the present invention, the first part of the detection sub-model is a global classification network, and the global classification network is a neural network model including neurons. Different from the classification sub-model in the foregoing embodiments, the global classification network is The path is used to perform binary classification processing on the pathological image to be recognized, and obtain an image classification result of whether the pathological image to be recognized contains target cells. In a specific implementation scenario, in order to distinguish it from the classification process of the classification sub-model, the classification process of detecting the first part of the sub-model may be called the second classification process, which is not limited herein.

Step S33: Determine whether the image classification result indicates that the target cell is included in the pathological image to be identified, if so, go to step S34, otherwise go to S36.

According to the image classification result, it is judged whether the pathological image to be recognized contains target cells. If it contains target cells, the pathological image to be recognized can be processed in the next step. The classification process containing the target cells is separated from the detection area that specifically detects the target cells, so that the operating efficiency of the model can be further improved, thereby improving the efficiency of target cell recognition in the image.

Step S34: Use the second part of the detection sub-model to perform region detection on the pathological image to be recognized, to obtain a detection region containing the target cells.

In some embodiments of the present invention, the second part of the detection sub-model is an image detection network, and the image detection network is a neural network model including neurons. Taking the detection sub-model using Faster RCNN as an example, the first The second part can be an RPN (Region Proposal Networks) network. When the detection sub-model is another network model, it can be deduced by analogy, and no examples will be given here.

In some embodiments of the present invention, please refer to FIG. 2 , which is a schematic state diagram of an image recognition method provided by an embodiment of the present invention. As shown in FIG. 2 , the pathological image to be recognized is a cervical pathological image, and the pathological image to be recognized is subjected to target detection by the detection sub-model in the recognition model, and two detection areas containing target cells are obtained.

In some embodiments of the present invention, in order to improve the accuracy of target cell identification. The third part of the detection sub-model can also be used to perform feature extraction on the pathological image to be recognized to obtain the image features of the pathological image to be recognized. Specifically, the third part can be a feature extraction network. In some embodiments of the present invention , the feature extraction network may be a ResNet101 network, or the feature extraction network may also be a ResNet50 network, etc., which is not limited here. In some embodiments of the present invention, in order to improve the accuracy of identifying polymorphic target cells, the feature extraction network may include a deformable convolution layer, the deformable convolution is based on the location information used in the space, Further displacement adjustments are made to achieve feature extraction for cells of different shapes. In some embodiments of the present invention, in order to obtain long-distance features with dependencies, thereby improving the accuracy of target cell identification, the feature extraction network may further include a global information enhancement module. Please refer to FIG. 4. FIG. 4 is a schematic state diagram of an image recognition method provided by an embodiment of the present invention. After the feature extraction of the pathological image to be recognized, the first part of the detection sub-model can be used to classify the image features. , obtain the image classification result of the pathological image to be recognized, and when the image classification result indicates that the pathological image to be recognized contains target cells (that is, when the image classification result is positive), the second part of the detection sub-model is used to The image features are subjected to region detection to obtain a detection region containing target cells for subsequent classification processing. For details, reference may be made to the relevant steps in this embodiment, which will not be repeated here.

Step S35: Use the classification sub-model in the recognition model to classify the detection area to obtain the category of the target cell.

For details, please refer to the relevant steps in the foregoing embodiments.

In some embodiments of the present invention, please refer to FIG. 2 , which is a schematic state diagram of an image recognition method provided by an embodiment of the present invention. As shown in FIG. 2 , the pathological image to be recognized is a cervical pathological image, and the pathological image to be recognized is subjected to target detection by the detection sub-model in the recognition model, and two detection areas containing target cells are obtained.

Step S36: The first part outputs a detection result prompt that the pathological image to be identified does not contain target cells.

When the image detection result indicates that the pathological image to be recognized does not contain target cells (that is, when the image classification result is negative), no further processing is required, so that the pathological image to be recognized that does not contain target cells can be directly output The detection result hints (that is, the hints that the result is negative) can improve the running efficiency of the model, thereby improving the efficiency of target cell recognition in the image.

Different from the foregoing embodiments, the second classification process is performed on the pathological image to be recognized by the first part of the detection sub-model, and the image classification result of the pathological image to be recognized is obtained, and the image classification result is used to represent the pathological image to be recognized. Whether the target cell is included or not, when the image classification result indicates that the target cell is included in the pathological image to be recognized, the second part of the detection sub-model is used to detect the region of the pathological image to be recognized to obtain the detection area containing the target cell, so it can be Realize the dynamic detection of target cells and improve the efficiency of target cell identification.

Please refer to FIG. 5. FIG. 5 is a schematic flowchart of a training method for a recognition model provided by an embodiment of the present invention. In the embodiment of the present invention, the recognition model may specifically include a detection sub-model and a classification sub-model, and may specifically include the following steps .

Step S51: Acquire a first sample image and a second sample image.

In this embodiment of the present invention, an actual area corresponding to the target cell is marked in the first sample image, and the actual area can be represented by the center coordinates of a rectangle containing the target cell and the length and width of the rectangle. For example, (50, 60, 10, 20) represents a rectangle with a length of 10 and a width of 20 in the first sample image with the primitive point (50, 60) as the center. The second sample image is marked with the actual category of the target cell. In some embodiments of the present invention, the actual category of the target cell is used to represent the degree of pathological change of the target cell. Taking the second sample image as a cervical pathological image as an example, the target cells may specifically include but are not limited to the following categories: high-grade squamous cell intraepithelial neoplasia (HSIL), mild squamous cell intraepithelial neoplasia (LSIL), Atypical squamous cells of undetermined significance (ASC-US), atypical squamous cells with high-grade intraepithelial neoplasia cannot be excluded (ASC-H). When the pathological image to be identified is other pathological images, the same can be deduced, and no examples are given here. In some embodiments of the present invention, the target cell may include any one of a single diseased cell or a diseased cell cluster, so that the single diseased cell or the diseased cell cluster can be identified.

In some embodiments of the present invention, the first sample image and the second sample image are pathological images, for example, including but not limited to: cervical pathological images, liver pathological images, and kidney pathological images. Taking the first sample image and the second sample image as cervical pathological images as an example, the target cells may be squamous epithelial cells. When the first sample image and the second sample image are other pathological images, it can be deduced in the same way, and will not be exemplified one by one here.

In some embodiments of the present invention, data enhancement can also be performed on the acquired ground sample image and the second sample image, thereby improving sample diversity, helping to avoid overfitting and improving the generalization performance of the model. In a specific implementation scenario, data enhancement may be performed by operations including but not limited to: random cutting, random rotation, random flip, color perturbation, gamma correction, and Gaussian noise.

In some embodiments of the present invention, the primitive values in the first sample image and the second sample image may also be normalized, so as to improve the convergence speed of the model. In some embodiments of the present invention, the first mean and the first variance of the primitive values of all the first sample images may be counted first, and then the primitive values in each first sample image are used to subtract the first mean and the first variance. A mean value is divided by the first variance, so that each first sample image is normalized; and the second mean and second variance of the primitive values of all the second sample images can be counted, and then Each second image is normalized by subtracting the second mean from the primitive values of each second sample image, and then dividing by the second variance.

Step S52: use the detection sub-model to perform target detection on the first sample image, obtain a predicted area containing the target cells in the first sample image, and use the classification sub-model to perform a first classification process on the second sample image, Get the predicted class of the target cell.

The detection sub-model may use Faster RCNN, and for details, reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here. The prediction area can be represented by the center coordinates of a rectangle and the length and width of the rectangle. For example, (70, 80, 10, 20) can be used to represent a location in the first sample image with the primitive point (70, 80) as the center , a prediction area with a length of 10 and a width of 20. The prediction area can also be represented by the center coordinates of a rectangle and the ratio of the length and width of the rectangle to the length and width of the preset rectangle. For example, you can set a preset rectangle, preset The length of the rectangle is 10 and the width is 20, then (70, 80, 1, 1) can be used to indicate that a rectangle is located in the first sample image with (70, 80) as the center of the image, the length is 10 and the width is 20 forecast area. The classification sub-model may use the EfficientNet network model, and for details, reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here.

In some embodiments of the present invention, in order to improve the ability of the model to identify positive and negative samples and realize dynamic prediction, so as to improve the operating efficiency of the model, the detection sub-model is used to perform target detection on the first sample image, and the first sample is obtained. In the process of including the predicted area of the target cell in the image, a second classification process may also be performed on the first sample image to obtain an image classification result of the first sample image, wherein the image classification result is used to represent the first sample image. Whether a sample image contains target cells, if the image classification result indicates that the first sample image contains target cells, the region detection is performed on the first sample image to obtain the predicted area containing the target cells. Referring to the relevant steps in the foregoing embodiments, details are not repeated here. In addition, the detection sub-model may also include a first part and a second part. The first part is configured to perform classification processing on the first sample image to obtain an image classification result of whether the first sample image contains target cells. The second part It is configured to perform region detection on the first sample image when the target cell is included in the first sample image to obtain a predicted region including the target cell. For details, please refer to the relevant steps in the foregoing embodiments, which will not be repeated here. . In addition, the detection sub-model may further include a third part configured to perform feature extraction on the first sample image to obtain the image features of the first sample image, so that the first part performs feature extraction on the image features to obtain the first sample image. The image classification result of a sample image, the second part performs region detection on the image features, and obtains the predicted region containing the target cells. Specifically, the first part can be a global classification network, the second part is an image detection network, and the third part is a feature extraction network, wherein the feature extraction network includes a deformable convolution layer, a global information enhancement module For at least one, specific reference may be made to the relevant steps in the foregoing embodiments, which will not be repeated here.

Step S53: Determine the first loss value of the detection sub-model based on the actual area and the predicted area, and determine the second loss value of the classification sub-model based on the actual category and the predicted category.

In some embodiments of the present invention, a mean square error loss function, a cross entropy loss function, or the like may be used to determine the first loss value of the detection sub-model. In some embodiments of the present invention, a cross-entropy loss function may be used to determine the second loss value of the classification sub-model, which will not be repeated here.

Step S54: Using the first loss value and the second loss value, correspondingly adjust the parameters of the detection sub-model and the classification sub-model.

Specifically, gradient descent optimization methods such as stochastic gradient descent, exponential average weighting, Adam, etc. can be used to adjust the parameters of the detection sub-model and the classification sub-model, which will not be repeated here.

In addition, the first sample image and the second sample image can also be divided into multiple batches, and the detection sub-model and the classification sub-model can be trained in a mini-batch training method. train. In some embodiments of the present invention, a training end condition can also be set, and when the training end condition is satisfied, the training can be ended. Specifically, the training end conditions may include but are not limited to: the number of repeated operations of the training is greater than or equal to a preset threshold (for example, 100 times, 500 times, etc.); the first loss value and the second loss value are less than a preset loss threshold value, And no longer decrease; the performance of the model obtained by using a verification data set to verify the detection sub-model and the classification sub-model is no longer improved, which is not limited here.

In the above solution, during the training process, the target cells can be detected first, and then the target cells can be classified, and the detection and classification can be separated, so as to solve the problem of unbalanced sample data categories, which can help to improve the accuracy of the model obtained by training. Therefore, it can help to improve the accuracy and efficiency of target cell identification.

Please refer to FIG. 6 , which is a schematic structural framework diagram of an image recognition apparatus 60 according to an embodiment of the present invention. The image recognition device 60 includes an image acquisition module 61, an image detection module 62 and an image classification module 63. The image acquisition module 61 is configured to acquire pathological images to be recognized; the image detection module 62 is configured to In order to use the detection sub-model in the recognition model to perform target detection on the pathological image to be recognized, the detection area containing the target cells in the pathological image to be recognized is obtained; The detection area is subjected to the first classification process to obtain the category of the target cell.

In the above scheme, by using the detection sub-model in the recognition model to perform target detection on the acquired pathological image to be recognized, the detection area containing the target cells in the to-be-recognized pathological image is obtained, and then the analysis sub-model in the recognition model is used to detect the target cells. The detection area is inspected by the first classification process to obtain the type of the target cells, and then the detection of the target cells can be carried out first, and then the classification of the target cells can be carried out, and the detection and classification can be separated, so that the target cells in the pathological image can be accurately and efficiently identified. .

In some embodiments of the present invention, the image detection module 62 includes a first part sub-module configured to use the first part of the detection sub-model to perform a second classification process on the pathological image to be recognized to obtain a map of the pathological image to be recognized The image classification result, wherein the image classification result is used to indicate whether the target cell is included in the pathological image to be identified, and the image detection module 62 further includes a second sub-module configured to indicate the pathology to be identified in the image classification result When the target cell is included in the image, the second part of the detection sub-model is used to perform region detection on the pathological image to be recognized, so as to obtain the detection region including the target cell.

Different from the foregoing embodiments, the second classification process is performed on the pathological image to be recognized by the first part of the detection sub-model, and the image classification result of the pathological image to be recognized is obtained, and the image classification result is configured to represent the pathological image to be recognized. Whether the target cell is included or not, when the image classification result indicates that the target cell is included in the pathological image to be recognized, the second part of the detection sub-model is used to detect the region of the pathological image to be recognized to obtain the detection area containing the target cell, so it can be Realize the dynamic detection of target cells and improve the efficiency of target cell identification.

In some embodiments of the present invention, the image detection module 62 further includes a result prompting sub-module, configured to output the first part of the pathological image to be identified when the image classification result indicates that the target cell is not included in the pathological image to be identified The detection result indicates that the target cells are not included in the image.

Different from the foregoing embodiments, after using the first part of the detection sub-model to perform the second classification process on the pathological image to be recognized, and obtaining the image classification result of the pathological image to be recognized, the method further includes: if the image classification result indicates the pathological image to be recognized. If the image does not contain target cells, the first part outputs a detection result prompt that the pathological image to be identified does not contain target cells.

In some embodiments of the present invention, the image detection module 62 further includes a third part sub-module configured to perform feature extraction on the pathological image to be recognized by using the third part of the detection sub-model to obtain the pathological image to be recognized. image features.

Different from the foregoing embodiments, the feature extraction of the pathological image to be recognized is performed through the third part of the detection sub-model to obtain the image features of the pathological image to be recognized, so that the pathological image to be recognized can be processed first, and then the subsequent steps are based on this. The detection sub-model is used for other processing, so it can be beneficial to improve the running efficiency of the model.

In some embodiments of the present invention, the first part sub-module is specifically configured to use the first part of the detection sub-model to perform a second classification process on the image features to obtain an image classification result of the pathological image to be recognized.

Different from the foregoing embodiments, the first part of the detection sub-model is used to perform the second classification process on the image features extracted by the third part to obtain the image classification result of the pathological image to be recognized, which can improve the accuracy of the classification process.

In some embodiments of the present invention, the second part of the sub-module is specifically configured to use the second part of the detection sub-model to perform region detection on the image features to obtain a detection region containing the target cells.

Different from the foregoing embodiments, the second part of the detection sub-model is used to perform region detection on image features to obtain a detection region containing target cells, which can help improve the accuracy of target cell identification.

In some embodiments of the present invention, the first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; wherein the feature extraction network includes a deformable convolution layer, global information at least one of the enhancement mods.

Different from the foregoing embodiments, by setting the feature extraction network to include a deformable convolution layer, the accuracy of identifying polymorphic target cells can be improved. At least one of them can be beneficial to obtain long-distance features with dependencies, and is beneficial to improve the accuracy of target cell identification.

In some embodiments of the present invention, the image classification module 63 includes a feature extraction sub-module, which is configured to use the classification sub-model to perform feature extraction on the detection area of the pathological image to be recognized to obtain the image features of the detection area. The image classification module 63 includes a classification processing sub-module, which is configured to perform a first classification process on the image features of the detection area to obtain the type of the target cell.

Different from the foregoing embodiments, by performing feature extraction on the detection region of the pathological image to be recognized, the image features of the detection region are obtained, and the image features of the detection region are subjected to the first classification processing to obtain the category of the target cell, which can be beneficial to Improve the efficiency of classification processing.

In some embodiments of the present invention, the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to indicate the diseased degree of the target cell.

Different from the foregoing embodiments, the target cell includes any one of a single diseased cell and a diseased cell cluster, which can help identify a single diseased cell and a diseased cell cluster, and the category of the target cell is used to indicate the degree of disease of the target cell, It is beneficial to achieve lesion grading of target cells.

Please refer to FIG. 7. FIG. 7 is a schematic structural framework diagram of an apparatus 70 for training a recognition model according to an embodiment of the present invention. The recognition model includes a detection sub-model and a classification sub-model, and the training device 70 of the recognition model includes an image acquisition module 71, a model execution module 72, a loss determination module 73 and a parameter adjustment module 74, and the image acquisition module 71 is configured In order to obtain the first sample image and the second sample image, wherein the first sample image is marked with the actual area corresponding to the target cell, and the second sample image is marked with the actual category of the target cell; the model executes The module 72 is configured to use the detection sub-model to perform target detection on the first sample image, obtain a predicted area containing the target cell in the first sample image, and use the classification sub-model to perform a first classification on the second sample image processing to obtain the predicted category of the target cell; the loss determination module 73 is configured to determine the first loss value of the detection sub-model based on the actual area and the predicted area, and determine the second loss value of the classification sub-model based on the actual category and the predicted category ; The parameter adjustment module 74 is configured to use the first loss value and the second loss value to correspondingly adjust the parameters of the detection sub-model and the classification sub-model.

In the above solution, during the training process, the target cells can be detected first, and then the target cells can be classified, and the detection and classification can be separated, so as to solve the problem of unbalanced sample data categories, which can help to improve the accuracy of the model obtained by training. Therefore, it can help to improve the accuracy and efficiency of target cell identification.

In some embodiments of the present invention, the model execution module 72 includes an initial classification sub-module configured to perform a second classification process on the first sample image to obtain an image classification result of the first sample image, wherein , the image classification result is used to indicate whether the first sample image contains target cells, the model execution module 72 includes a region detection sub-module, and is configured to indicate that the first sample image contains target cells in the image classification result When , the region detection is performed on the first sample image to obtain a predicted region containing the target cells.

Different from the previous embodiments, in the training process, when the image classification result indicates that the first sample image contains target cells, the region detection is performed on the first sample image to obtain the predicted area containing the target cells, which can Enhancing the ability of the model to identify positive and negative samples and reducing the probability of false detection is conducive to improving the accuracy of the model obtained by training, thereby improving the accuracy of target cell identification.

In some embodiments of the present invention, the training device 70 for the recognition model further includes a data enhancement module configured to perform data enhancement on the first sample image and the second sample image.

Different from the foregoing embodiments, by performing data enhancement on the first sample image and the second sample image, the sample diversity can be improved, which is beneficial to avoid overfitting and improve the generalization performance of the model .

In some embodiments of the present invention, the training device 70 for the recognition model further includes a normalization processing module configured to normalize the primitive values in the first sample image and the second sample image.

Different from the foregoing embodiments, by normalizing the primitive values in the first sample image and the second sample image, the convergence speed of the model can be improved.

In some embodiments of the present invention, the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to indicate the diseased degree of the target cell.

Different from the foregoing embodiments, the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to represent the diseased degree of the target cell, which can help to identify a single diseased cell and a diseased cell cluster, and The category of target cells is used to indicate the degree of disease of the target cells, which is beneficial to achieve the classification of the target cells.

Please refer to FIG. 8 . FIG. 8 is a schematic structural framework diagram of an electronic device 80 provided by an embodiment of the present invention. The electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps of any of the above image recognition method embodiments, or to implement any of the above Steps in an embodiment of the training method of the recognition model. In a specific implementation scenario, the electronic device 80 may include, but is not limited to, a microcomputer and a server. In addition, the electronic device 80 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.

Specifically, the processor 82 is configured to control itself and the memory 81 to implement the steps of any of the above image recognition method embodiments, or to implement any of the above recognition model training method embodiments. The processor 82 may also be referred to as a central processing unit (Central Processing Unit, CPU). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA) Or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 82 may be commonly implemented by an integrated circuit die.

The above solution can accurately and efficiently identify target cells in pathological images.

Please refer to FIG. 9 . FIG. 9 is a schematic structural framework diagram of a computer-readable storage medium 90 according to an embodiment of the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be run by the processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned image recognition method embodiments, or to implement any of the above-mentioned training method embodiments of the recognition model. step.

The above solution can accurately and efficiently identify target cells in pathological images.

An embodiment of the present invention provides a computer program, including computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes any of the diagrams provided in the embodiment of the present invention to implement Like the recognition method, or any training method of the recognition model provided by the embodiments of the present invention.

In the several embodiments provided by the present invention, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the device implementations described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other divisions. For example, units or elements may be combined or integrated. to another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed over network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention are essentially or contribute to the prior art, or all or part of the technical solutions can be embodied in the form of software products, and the computer software products are stored in a storage medium. , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or CD, etc. medium.

Industrial Applicability Embodiments of the present invention provide an image recognition method, a training method for a recognition model, and related devices and equipment, wherein the image recognition method includes: acquiring a pathological image to be recognized; using a detection sub-model in the recognition model to identify the pathological image Perform target detection to obtain a detection area containing target cells in the pathological image to be identified; use the classification sub-model in the recognition model to perform a first classification process on the detection area to obtain the target cell category. According to the image recognition method of the embodiment of the present invention, the target cells in the pathological image can be accurately and efficiently recognized.

60: Image recognition device 61: Image acquisition module 62: Image detection module 63: Image Classification Module 70: Training device for recognition model 71: Image acquisition module 72: Model execution module 73: Loss Determination Module 74: Parameter adjustment module 80: Electronic equipment 81: Memory 82: Processor 90: Computer-readable storage media 901: Program command S11~S13: Steps S31~S36: Steps S51~S54: Steps

1 is a schematic flowchart of an image recognition method provided by an embodiment of the present invention; 2 is a schematic state diagram of an image recognition method provided by an embodiment of the present invention; 3 is a schematic flowchart of an image recognition method provided by an embodiment of the present invention; 4 is a schematic state diagram of an image recognition method provided by an embodiment of the present invention; 5 is a schematic flowchart of a training method for an identification model provided by an embodiment of the present invention; 6 is a schematic structural diagram of an image recognition apparatus provided by an embodiment of the present invention; 7 is a schematic structural framework diagram of a training device for a recognition model provided by an embodiment of the present invention; 8 is a schematic structural framework diagram of an electronic device provided by an embodiment of the present invention; FIG. 9 is a schematic structural framework diagram of a computer-readable storage medium provided by an embodiment of the present invention.

S11~S13: Steps

Claims (14)

An image recognition method, comprising: acquiring a pathological image to be recognized; using a detection sub-model in a recognition model to perform target detection on the pathological image to be recognized, to obtain a detection area containing target cells in the pathological image to be recognized ; Utilize the classification sub-model in the recognition model to carry out the first classification process to the detection area to obtain the category of the target cell; Describe the use of the detection sub-model in the recognition model to target the pathological image to be recognized Detecting to obtain a detection area containing target cells in the pathological image to be identified, including: classifying the pathological image to be identified; When the target cells are detected, the detection sub-model in the recognition model is used to perform target detection on the pathological image to be recognized, and a detection area containing the target cells is obtained. The image recognition method according to claim 1, wherein the classifying the to-be-recognized pathological image comprises: using the first part of the detection sub-model to perform a second step on the to-be-recognized pathological image classification processing to obtain an image classification result of the pathological image to be identified, wherein the image classification result is used to indicate whether the target cell is included in the pathological image to be identified; When the classification result indicates that the target cell is contained in the pathological image to be identified, the detection sub-model in the identification model is used Performing target detection on the to-be-recognized pathological image to obtain a detection area containing the target cell includes: if the image classification result indicates that the to-be-recognized pathological image contains the target cell, using the The second part of the detection sub-model performs region detection on the pathological image to be identified to obtain a detection region containing the target cells. The image recognition method according to claim 2, wherein a second classification process is performed on the pathological image to be recognized in the first part of using the detection sub-model to obtain a map of the pathological image to be recognized. After obtaining the image classification result, the method further includes: if the image classification result indicates that the target cell is not included in the pathological image to be identified, the first part outputs that the pathological image to be identified does not contain the target cell. The detection result of the target cell is prompt. The image recognition method according to claim 2 or 3, wherein the target detection is performed on the pathological image to be recognized by using a detection sub-model in the recognition model, and it is obtained that the pathological image to be recognized contains target cells The detection area further includes: using the third part of the detection sub-model to perform feature extraction on the to-be-recognized pathological image to obtain image features of the to-be-recognized pathological image. The image recognition method according to claim 4, wherein the second classification process is performed on the pathological image to be recognized by using the first part of the detection sub-model to obtain the image of the pathological image to be recognized Classification results, including: using the first part of the detection sub-model to perform a first step on the image features The second classification process is performed to obtain the image classification result of the pathological image to be recognized. The image recognition method according to claim 4, wherein the using the second part of the detection sub-model to perform region detection on the pathological image to be identified to obtain a detection region containing the target cells, comprising: Using the second part of the detection sub-model to perform region detection on the image features, to obtain a detection region containing the target cells. The image recognition method according to claim 4, wherein the first part is a global classification network, the second part is an image detection network, and the third part is a feature extraction network; The feature extraction network includes at least one of a deformable convolution layer and a global information enhancement module. The image recognition method according to claim 1 or 2, wherein the performing a first classification process on the detection area by using the classification sub-model in the recognition model to obtain the category of the target cell includes: using The classification sub-model performs feature extraction on the detection area of the pathological image to be recognized to obtain the image features of the detection area; performs a first classification process on the image features of the detection area to obtain the The class of target cells. The image recognition method according to any one of claims 1 to 3, wherein the target cell includes any one of a single diseased cell and a diseased cell cluster, and the category of the target cell is used to represent the target degree of cell disease. A kind of training method of recognition model, described recognition model package Including a detection sub-model and a classification sub-model, the method includes: acquiring a first sample image and a second sample image, wherein the first sample image is marked with an actual area corresponding to the target cell, so the The second sample image is marked with the actual category of the target cell; the detection sub-model is used to perform target detection on the first sample image, and the predicted area containing the target cell in the first sample image is obtained. , and use the classification sub-model to perform a first classification process on the second sample image to obtain the predicted category of the target cell; based on the actual area and the predicted area, determine the first classification of the detection sub-model a loss value, and based on the actual category and the predicted category, determine the second loss value of the classification sub-model; use the first loss value and the second loss value to correspondingly adjust the detection sub-model and parameters of the classification sub-model; performing target detection on the first sample image by using the detection sub-model to obtain a predicted region containing target cells in the first sample image, including: The first sample image is subjected to classification processing; when the obtained image classification result indicates that the target cell is included in the first sample image, the detection sub-model is used to classify the first sample image. Like performing target detection, a predicted region containing the target cell is obtained. The training method according to claim 10, wherein the classifying the first sample image includes: performing a second classification process on the first sample image to obtain the first sample image The image classification result of this image, wherein the image classification result is used for Indicates whether the target cell is contained in the first sample image; when the obtained image classification result indicates that the target cell is contained in the first sample image, using the detection sub-model to Performing target detection on the first sample image to obtain a predicted region containing the target cell includes: if the image classification result indicates that the target cell is contained in the first sample image, Region detection is performed on the first sample image to obtain a predicted region containing the target cell. The training method according to claim 10 or 11, wherein, when the target detection is performed on the first sample image by using the detection sub-model, a target cell containing target cells in the first sample image is obtained. Before obtaining the predicted category of the target cell, the method further includes: performing a first classification process on the second sample image by using the classification sub-model to obtain the predicted category of the target cell. Data enhancement is performed on the second sample image; and/or, the primitive values in the first sample image and the second sample image are normalized; the target cells include single diseased cells, diseased cells In any of the clusters, the category of the target cell is used to indicate the degree of disease of the target cell. An electronic device, comprising a mutually coupled memory and a processor, the processor is configured to execute program instructions stored in the memory to implement the image recognition method described in any one of claim 1 to 9, or The training method of the recognition model according to any one of claim 10 to 12. A computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, realize the image recognition method described in any one of claim items 1 to 9, or any one of claim items 10 to 12. The training method of the recognition model described above.

Images