CN110598652A - Fundus data prediction method and device - Google Patents

Abstract

The present invention provides a fundus data prediction method and equipment, the method comprising: acquiring a fundus image; using a machine learning model to predict the center position of the macula, the center position of the optic disc, and the size of the optic disc in the fundus image, and the machine learning model is in A heat map is generated during the prediction process, and the position of the center of the macula and the center of the optic disc are determined according to the pixel values of the heat map.

Description

Fundus data prediction method and device

technical field

The invention relates to the field of ophthalmology image detection, in particular to a fundus data prediction method and equipment.

Background technique

In the medical field, the macula is located at 0.35cm on the temporal side of the optic disc of the fundus and slightly below it, in the optical center of the human eye, and is the projection point of the vision axis. The macula is located in the center of the retina, where a large number of visual function cells are concentrated. Abnormalities in the macular area often directly lead to the decline of visual ability. If the lesions in the macular area are not detected and treated in time, the chance of blindness will be greatly increased.

At present, machine learning has been widely used in the medical field, especially machine learning technology represented by deep learning has been widely concerned in the field of medical imaging. In terms of fundus image detection, deep learning technology has been used in the detection of glaucoma, diabetic retinopathy and other diseases, and achieved good results.

However, because the morphological features of the macula and optic disc in fundus images often vary greatly due to the degree of lesion, it is difficult for machine learning technology to accurately segment the boundaries of images such as the macula and optic disc. Therefore, the recognition result of the existing technology is a rough detection Although it can ensure that the detection frame contains all the content of the optic disc or macula to a certain extent, the detection result is still not accurate enough, which will affect the subsequent abnormal detection conclusion.

Contents of the invention

In view of this, the present invention provides a fundus data prediction method, including:

Obtain fundus images;

A machine learning model is used to predict the predicted macular center position, optic disc center position and optic disc size in the fundus image. The machine learning model generates a heat map during the prediction process. The macular center position and the optic disc center position are based on the determined The pixel value of the above heat map is determined.

Optionally, the heat map is a feature map output by the last layer of the neural network used to extract fundus image features in the machine learning model.

Optionally, using a machine learning model to predict the central position of the macula and the central position of the optic disc in the fundus image includes:

Obtain the two peaks in the heat map;

determining that the two peaks correspond to two pixels in the fundus image;

The central position of the macula and the central position of the optic disc are determined according to the pixel values of the two pixel points.

The present invention also provides a fundus data prediction model training method, comprising:

Acquiring training data, the training data includes marking the fundus image of the central position of the macula, the central position of the optic disc and the optic disc area, wherein the marking content of the optic disc area is used to determine the size of the optic disc;

The machine learning model is trained by using the training data, so that it can predict the position of the center of the macula, the position of the center of the optic disc and the size of the optic disc according to the input fundus image.

Optionally, the training process uses the following loss function:

Loss=Lp+λLwh,

Among them, Lp represents the difference between the predicted macular center position and optic disc center position and the macular center position and optic disc center position in the training data, Lwh represents the difference between the predicted optic disc size and the optic disc size in the training data, λ is the weight, 0<λ <1.

Optionally,

Among them, Fmap(x, y) is the value of the pixels in the feature map extracted during the prediction process, Heatmap(x, y) is the value of the pixels in the heat map, and N is the number of pixels in the feature map , x, y are the coordinates of the pixel, Ω is the set of pixel positions in the feature map and the heat map.

Optionally,

Where (x _m0 , y _m0 ) is the center position of the macula in the training data, (x _d0 , y _d0 ) is the center position of the optic disc in the training data, σ is the size information of the optic disc in the training data, (x, y) is the thermal force The coordinates of the pixels in the image.

Optionally,

in is the predicted disc size, S _d is the disc size in the training data, and the disc size includes the length information and/or width information of the disc label frame.

Correspondingly, the present invention also provides a fundus data prediction device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores information that can be executed by the one processor. Instructions, the instructions are executed by the at least one processor, so that the at least one processor executes the above fundus data prediction method.

Correspondingly, the present invention also provides a fundus data prediction model training device, including: at least one processor; and a memory connected to the at least one processor; wherein, the memory stores information that can be used by the one processor Executable instructions, the instructions are executed by the at least one processor, so that the at least one processor executes the above fundus data prediction model training method.

According to the method for predicting fundus data provided by the present invention, combining the advantages of neural network algorithm and regression can accurately locate the center of the optic disc and macula in the fundus image. Predict the heat map that integrates the position information of the optic disc and macula, and implicitly fuse the prior information of the distance and position between the optic disc and the macula into the model to improve the accuracy of detection. The solution provided by the present invention can obtain the size of the optic disc while predicting the center of the macula and the optic disc, which can provide an important basis for doctors to determine the treatment plan.

The thermal map obtained according to the scheme provided by the present invention can be applied to abnormal detection of fundus images, and the attention selection mechanism is used to superimpose it with the corresponding area of the original image to give relatively larger areas of the optic disc and macula relative to the center. Attention, so as to better simulate the actual diagnosis method of clinicians, that is, the closer to the center of the optic disc and macula, the more critical the lesion appears, and more attention needs to be given, thereby improving the accuracy of abnormal detection operations.

Description of drawings

In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the specific embodiments or prior art. Obviously, the accompanying drawings in the following description The drawings show some implementations of the present invention, and those skilled in the art can also obtain other drawings based on these drawings without creative work.

Figure 1 is a fundus image;

Fig. 2 is a schematic diagram of the heat map obtained from the fundus image and its combination with the original image in the embodiment of the present invention;

Fig. 3 is a schematic diagram of training data in an embodiment of the present invention.

Detailed ways

The technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Apparently, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

In addition, the technical features involved in the different embodiments of the present invention described below may be combined with each other as long as there is no conflict with each other.

An embodiment of the present invention provides a fundus data prediction method, which can be executed by electronic devices such as computers and servers. In this method, a machine learning model is used to recognize images, and the machine learning model may be neural networks of various types and structures. The method comprises the steps of:

S1A, acquiring a fundus image. In order to improve the recognition efficiency, preprocessing is performed on the obtained fundus photos in this embodiment, and a fundus image as shown in FIG. 1 is obtained. Preprocessing includes but not limited to image enhancement, boundary removal and other operations. In other embodiments, these preprocessing may not be performed, and the fundus photos collected by the fundus photographing equipment may be used directly.

S2A, Prediction of macular center location, optic disc center location, and optic disc size in fundus images using a machine learning model. The machine learning model in this embodiment is used for regression. The model is trained by using a large number of sample fundus images marked with the central position of the macula, the central position of the optic disc and the size of the optic disc. The training scheme of the model will be introduced in detail below.

The machine learning model in this embodiment first extracts the feature information of the fundus image, and then obtains the above three pieces of information according to the feature information. These feature information belong to the intermediate results of the regression process rather than the final results. The feature extraction part of the model can use stacked hour-glass modules, or the feature extraction part of YoloV3, DSOD, or a structure similar to U-Net. The feature information obtained in this step is the two-dimensional feature map (Feature map) output by the last layer of the feature extraction network. The last layer can be a convolutional layer, a pooling layer, or the like.

The machine learning model in this embodiment generates a heat map during the prediction process, which is a visual processing of feature information. Taking Figure 2 as an example, the feature information extracted during the prediction process of the machine learning model can be obtained to generate a heat map (also known as a heat map, HeatMap) on the left side as shown in Figure 2. In an optional embodiment, it can be based on The feature map generates a heat map, or the feature map is a heat map. For example, in the prediction process, the output of the feature extraction network is normalized as a feature map, so that the pixel value of each point ranges from 0 to 255, and the heat map can be obtained. In the heat map, the value of the pixel closer to the center of the optic disc and macula is larger, and the value of the pixel farther away from the center is smaller.

As shown in Figure 2, the heat (value) generated at locations that have a greater impact on the prediction results is higher than that at other locations. Combining the heat map with the fundus image (on the right) it can be seen more clearly that the closer to the center of the optic disc and macula, the higher the value of the heat map, the higher the attention of the model, so the model can be based on the pixel value of the heat map To determine the center of the macula and the center of the optic disc.

The central position of the macula and the central position of the optic disc in the prediction result can be represented by coordinates of pixels. There are many ways to express the size of the optic disc. For example, the visual area of the optic disc can be regarded as a circular area, and the radius of the circular area can be given here, or the optic disc can be regarded as a square or rectangular area, and the length of the square area can be given here. and/or width etc.

Figure 2 is only a visualization result shown for illustration. In actual application, the heat map and its combination with the original image can be displayed to the user, prompting the user to focus on this area. It is also possible not to perform visualization processing, and the heat map will be used as a kind of auxiliary data for subsequent classification of fundus images or segmentation of regions of interest.

According to the fundus data prediction method provided by the embodiment of the present invention, combining the advantages of neural network algorithm and regression can accurately locate the center of the optic disc and macula in the fundus image. Predict the heat map that integrates the position information of the optic disc and macula, and implicitly fuse the prior information of the distance and position between the optic disc and the macula into the model to improve the accuracy of detection.

For diseases in the optic disc area, the lesion area is generally concentrated in the range of one optic disc diameter of the optic disc. For diseases in the macular area, the severity of the disease is generally measured by the distance between the lesion area and the center of the macula, such as diabetic macular edema, where the most important grading index is the distance between the edema lesion and the center of the macula. In order to better determine the relative distance, the ratio of the distance between the fovea and the lesion to the diameter of the optic disc is generally used as a measure. For some overall fundus diseases, whether it affects the macular area and optic disc area is an important basis for measuring the degree of visual impairment. The solution provided by the present invention can obtain the size of the optic disc while predicting the center of the macula and the optic disc, which can provide an important basis for doctors to determine the treatment plan.

In addition, the obtained heat map can be applied to abnormal detection of fundus images. Using the attention selection mechanism, by superimposing it with the corresponding area of the original image, the relative center area of the optic disc and macula is given relatively greater attention, thus To better simulate the actual diagnosis method of clinicians, that is, the closer to the center of the optic disc and macula, the more critical the lesions appear, and more attention needs to be given, thereby improving the accuracy of abnormal detection operations.

In a preferred embodiment, the machine learning model obtains the central position of the macula and the central position of the optic disc in the following manner: obtain two peaks in the heat map; determine that the two peaks correspond to two pixels in the fundus image; The pixel value of the point determines the location of the center of the macula and the center of the optic disc.

The peak location of the heatmap is most likely the center of the macula and optic disc. There are two peak positions in the feature map output by the feature extraction network, which are the center of the macula and the center of the optic disc. In the original fundus image, the center of the macula is darker than the center of the optic disc, that is, its pixel value is lower than that of the center of the optic disc. Therefore, according to the pixel values in the original fundus image corresponding to the two peak positions, the macula and the center of the optic disc can be distinguished.

The present invention also provides a fundus data prediction device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores instructions executable by the one processor, so The instructions are executed by the at least one processor, so that the at least one processor executes the above fundus data prediction method.

An embodiment of the present invention provides a method for training a fundus data prediction model, which can be used to train the machine learning model used in the above embodiments, and the method can be executed by electronic devices such as computers and servers. This method comprises the steps:

S1B, acquire training data, the training data includes the fundus image marked with the macular center 31 position, the optic disc center 32 position and the optic disc area 33 as shown in Figure 3, wherein the marked content of the optic disc area 33 is used to determine the size of the optic disc. In this implementation, the optic disc region 33 is marked as a square, and its side length is the size of the optic disc, which can be regarded as the diameter of the optic disc. In other embodiments, circles or other shapes may also be used to mark this area.

In practical applications, a large amount of the above training data should be obtained. If the actual number of fundus images is not large enough, a data enhancement module can be added to generate training data based on the actual fundus images. The data enhancement module can use random flip, mirror image, rotation, translation, random Add noise, blur, improve contrast, adjust color space, etc., and perform data amplification based on the original fundus image. The principle is to simulate as much as possible the shapes that may appear in the natural photographing of the fundus map, so that the generated fundus map is consistent with the actual photographed fundus map.

According to the above expansion method, multiple transformed images can be obtained as training data based on one actually collected fundus image, which can significantly increase the amount of training data, thereby optimizing the performance of the machine learning model.

S2B, using the above training data to train the machine learning model, so that it can predict the center position of the macula, the center position of the optic disc and the size of the optic disc according to the input fundus image. Certain convergence conditions should be set during training, and a large number of sample images as shown in Figure 3 should be used to make the predicted results of the machine learning model consistent with the actual marked content.

Specifically, an appropriate loss function should be set to measure the difference between the predicted result and the labeled content (actual data), and corresponding penalties should be given according to the difference so that the model can optimize its own parameters until the predicted result of the model is consistent with the labeled content. The gap is small enough and remains stable. The loss function can include two parts, one part is used to measure the difference between the predicted center point of the optic disc and macula and the actual data, and the other part is used to measure the difference between the predicted optic disc size and the actual size. These two parts can be biased, according to Set up the actual application scenarios.

According to the fundus data prediction model training method provided by the embodiment of the present invention, combining the advantages of neural network algorithm and regression, the machine learning model is trained by using the training data marked with the center position of the optic disc and macula and the size of the optic disc, so that it can predict these messages.

As a preferred embodiment, the training process uses the following loss function:

Loss=Lp+λLwh,

Among them, Lp represents the difference between the predicted macular center position and optic disc center position and the macular center position and optic disc center position in the training data, Lwh represents the difference between the predicted optic disc size and the optic disc size in the training data, λ is the weight, 0<λ <1. In a specific embodiment, the value of λ is 0.1.

Further, in the above formula

Among them, Fmap(x, y) is the value of the pixels in the feature map extracted during the prediction process, Heatmap(x, y) is the value of the pixels in the heat map, and N is the number of pixels in the feature map , x, y are the coordinates of the pixel point, and Ω is the set of pixel positions. In the process of model training, the heatmap (heat map) is used as the target of the feature map generated by the convolutional network, that is, the feature map Fmap (feature map) generated by the convolutional network is as close as possible to the heatmap through backpropagation during model training.

Further, in the above formula

in is the predicted disc size, S _d is the disc size in the training data, and the disc size includes the length information and/or width information of the disc label frame.

Furthermore, in this embodiment, the heat map is obtained in the following manner:

Where (x _m0 , y _m0 ) is the center position of the macula in the training data, (x _d0 , y _d0 ) is the center position of the optic disc in the training data, σ is the size information of the optic disc in the training data, (x, y) is the heatmap Corresponding to the coordinate position of the pixel point.

σ ² =wh/4, w and h are the width and height of the optic disc area in the training data respectively, and σ is equivalent to the radius of the optic disc.

The present invention also provides a fundus data prediction model training device, including: at least one processor; and a memory connected to the at least one processor in communication; wherein, the memory stores instructions that can be executed by the one processor , the instructions are executed by the at least one processor, so that the at least one processor executes the above fundus data prediction model training method.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

Apparently, the above-mentioned embodiments are only examples for clear description, rather than limiting the implementation. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. And the obvious changes or changes derived therefrom are still within the scope of protection of the present invention.

Claims (10)

1. A method for predicting fundus data, comprising:

acquiring a fundus image;

predicting a predicted macular center position, a disc center position, and a disc size in the fundus image using a machine learning model that generates a thermodynamic diagram in a prediction process, the macular center position and the disc center position being determined from pixel values of the thermodynamic diagram.

2. The method of claim 1, wherein the thermodynamic diagram is a feature diagram of a last layer output of a neural network used to extract fundus image features in the machine learning model.

3. The method according to claim 1 or 2, wherein predicting the central position of the macula lutea and the central position of the optic disc in the fundus image using a machine learning model comprises:

acquiring two peaks in the thermodynamic diagram;

determining that the two peak values correspond to two pixel points in the fundus image;

and determining the central position of the macula lutea and the central position of the optic disc according to the pixel values of the two pixel points.

4. A method for training a model for predicting eye fundus data, comprising:

acquiring training data including a fundus image in which a macular center position, a disc center position, and a disc region are marked, wherein a marking content of the disc region is used to determine a disc size;

the machine learning model is trained using the training data so as to predict a central position of the macula lutea, a central position of the optic disc, and a size of the optic disc from the input fundus image.

5. The method of claim 4, wherein the training process uses the following loss function:

Loss＝Lp+λLwh，

where Lp represents the difference between the predicted macular center position, the disk center position, and the macular center position, the disk center position in the training data, Lwh represents the difference between the predicted disk size and the disk size in the training data, λ is a weight, and 0 < λ < 1.

6. The method of claim 5, wherein:

the method comprises the steps of obtaining a feature diagram, extracting the feature diagram from a database, extracting the feature diagram from the database, and extracting the feature diagram from the database, wherein Fmap (x, y) is the value of a pixel point in the feature diagram extracted in the prediction process, Heatmap (x, y) is the value of the pixel point in the thermodynamic diagram, N is the number of the pixel points in the feature diagram, x and y are coordinates of the pixel point, and omega is the set of the pixel point positions in the feature diagram and the thermodynamic diagram.

7. The method of claim 6, wherein:

wherein (x)_m0,y_m0) To train the macular center position in the data, (x)_d0,y_d0) σ is the disc size information in the training data, and (x, y) is the coordinates of the pixel points in the thermodynamic diagram.

8. The method of claim 5, wherein:

whereinFor predicted disc size, S_dThe size of the optic disc in the training data comprises the length information and/or the width information of the optic disc labeling frame.

9. An apparatus for predicting eye fundus data, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the one processor to cause the at least one processor to perform the fundus data prediction method of any of claims 1-3.

10. An eye fundus data prediction model training device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the one processor to cause the at least one processor to perform the fundus data prediction model training method of any of claims 4-8.