CN114663421B - Retina image analysis system and method based on information migration and ordered classification - Google Patents

Abstract

The present invention proposes an intelligent retinal image analysis system based on information migration and orderly classification, including: a fundus image preprocessing module, a fundus image blood vessel segmentation network module based on information migration, and an intelligent analysis and prediction module based on orderly classification, wherein the image The preprocessing module is based on the visual image enhancement algorithm, and multi-algorithms work together to effectively enhance the image quality; the fundus image blood vessel segmentation network based on information migration effectively realizes retinal blood vessel segmentation; the intelligent analysis and prediction module combined with ordered classification accurately predicts the retinal state change level ;Effective and fast intelligent analysis of retinal images.

Description

Retinal image analysis system and method based on information migration and ordered classification

Technical Field

The present invention belongs to the field of artificial intelligence image processing, and specifically relates to a retinal image intelligent analysis system based on information migration and orderly classification.

Background Art

Fundus images are fundus structures captured by a fundus camera, mainly including the retina, choroid, macula, and optic nerve. Retinal image acquisition refers to the process of using acquisition equipment to collect retinal image data, involving issues such as imaging equipment and imaging systems. The acquired retinal fundus images are usually divided into two types according to different imaging principles: fluorescent fundus images and conventional source fundus images.

In the process of collecting retinal images, the read-in retinal images are affected by factors such as light brightness, distance, image collection angle, and the inherent phase difference limitations of the human eye. Usually, the collected retinal images have a lot of noise, some of which also contain lesions. The image pixels have low contrast, local illumination is uneven, the blood vessels are complex and densely distributed, there are multiple texture backgrounds, and the image quality is not high.

In the image processing process of the prior art, image processing is generally performed based on neural networks. Due to the quality of fundus image acquisition and the lack of universality of conventional algorithms, the retinal blood vessel segmentation of current fundus images is inaccurate, which in turn increases the constraints on the practical application of intelligent analysis of fundus images.

Summary of the invention

The purpose of the present invention is to address the problems of the prior art. This application proposes a retinal image intelligent analysis system based on information migration and ordered classification, which performs image preprocessing enhancement, then performs retinal vascular segmentation based on the information migration fundus image segmentation network, and finally uses a neural network combined with ordered classification for intelligent analysis and prediction to achieve the purpose of intelligent analysis of retinal images.

The technical solution of this application is as follows:

A retinal image intelligent analysis system based on information migration and ordered classification, comprising a fundus image preprocessing module, a fundus image blood vessel segmentation network module and a fundus image intelligent analysis and prediction module;

The fundus image preprocessing module performs image preprocessing on the collected fundus image, with the purpose of improving image quality and highlighting useful features;

The fundus image vascular segmentation network module performs vascular segmentation on the preprocessed fundus image based on information migration to obtain a vascular image, wherein a guiding network and a bionic network are defined in the fundus image vascular segmentation network module;

The fundus image intelligent analysis and prediction module predicts state changes of the blood vessel image based on ordered classification.

The fundus image preprocessing module includes an image enhancement submodule, a mask generation submodule and a multi-scale linear filter submodule;

The image enhancement submodule performs dual-tree complex wavelet and improved top-hat transform on the collected original fundus image to obtain an image-enhanced fundus image; the image enhancement submodule effectively improves the quality of retinal images, increases vascular contrast, and highlights characteristic elements (vascular structure, optic disc, macula);

The dual-tree complex wavelet transform (DT-CWT) overcomes the defects of the commonly used discrete wavelet transform. When the corresponding wavelet basis approximately satisfies the Hilbert transform relationship, the dual-tree complex wavelet transform can reduce the translation sensitivity in the usual real wavelet transform, improve the directional selectivity, and retain the detail information while effectively improving the image quality. After the original fundus image is subjected to the dual-tree complex wavelet transform, the fundus image after the complex wavelet transform is subjected to the improved top-hat transform.

The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an opening operation fundus image, when the transformed fundus image is subtracted from the opening operation fundus image, the data with the transformed grayscale value remains unchanged, and the unchanged data is the subtraction result;

The improved top-hat transformation of the present application is that when the fundus image after complex wavelet transformation is subtracted from the fundus image after opening operation, the data with changed grayscale value remains unchanged, and the unchanged data is the subtraction result. After the improved top-hat transformation, the image grayscale difference is significantly increased, and some edges with small amplitude changes can also be effectively protected;

The mask generation submodule extracts the field of view based on the spatial brightness information, converts the image-enhanced fundus image (a color image, an image after local detail enhancement and image quality improvement through complex wavelet transform and top-hat transform) from the RGB format to the YIQ format, sets a segmentation threshold, extracts the surrounding black field of view, obtains the useful information area through the corrosion operation, separates the fundus area from the background area, obtains the mask image, and multiplies the mask image and the image-enhanced fundus image to obtain the fundus area image, including the following steps:

Step 1: Convert the enhanced fundus image from RGB format to YIQ format:

The above formula (1) obtains the three components of the fundus image in YIQ format;

Step 2: Set the segmentation threshold, extract the surrounding black field of view, and obtain the region of interest through corrosion operation;

The mask image is obtained as:

Among them, "1" represents the background border image, and "0" represents the blood vessels of the eyeball; Y is the brightness information of the image, which is equal to the grayscale value of the brightness component of the image (the Y component of the fundus image in YIQ format), M(x′,y) is the extracted background border, and x′,y represent the pixel coordinates;

The multiscale linear filter submodule is based on the multiscale linear filter of the Hessian matrix, sets parameters according to the grayscale values and characteristic values of the blood vessels in the fundus area image, eliminates noise after filtering, further highlights the blood vessel features, and obtains a filtered blood vessel feature image.

The training process of the fundus image blood vessel segmentation network module includes the following steps:

Step 201, using the filtered vascular feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the vascular segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every A iterations, saving the weights with segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network;

Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in 201, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the loss function as a constraint condition during the fitting process;

Step 203, update the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction of decreasing the loss function value through the back propagation method, and jump to step 202 for iterative training. Perform a segmentation accuracy test on the validation set every A iterations, save the weights with accuracy greater than the set segmentation threshold, and after the iterative training is completed, select the weight with the highest segmentation accuracy as the final optimal weight of the bionic network.

In step 202, the guided codec matrix and the bionic codec matrix are G∈R ^m×n′ , G∈R ^m×n′ is generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, the output feature map of the encoder layer is F ¹ ∈R ^h×w×m , where h, w, m are the height, width and number of channels of the F ¹ feature map, and the output feature map of the decoder layer is F ² ∈R ^h×w×n′ , where h, w, n′ are the height, width and number of channels of the F ² feature map, respectively. The guided codec matrix and the bionic codec matrix G∈R ^m×n′ are calculated as:

Wherein, s=1, ..., h, t=1, ..., w, x and W represent the weights of the input image and the guidance/bionic network respectively; _Ga,b (x; W) represents the ath row and bth column of the guidance codec matrix or the bionic codec matrix.

In step 202, residual similarity is extracted by a multi-scale residual similarity collection module, and the multi-scale residual similarity collection module uses a similar volume to collect context information, specifically including the following steps:

For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is:

P _j ′＝P _j ×P _center (4)

其中d代表自定义区域的大小，H和W′分别表示特征向量的高和宽，获取相应的残差相似度

然后将残差相似度

相加求和，第i个特征向量的最终残差相似度为

Where j represents the coordinate of the d×d region. For each pixel in the vascular feature image after image filtering, a local representation is obtained, and then the local representations are connected along the channel dimension to obtain the residual similarity of the i-th feature vector

Where d represents the size of the custom region, H and W′ represent the height and width of the feature vector respectively, and the corresponding residual similarity is obtained.

Then the residual similarity

Add and sum, the final residual similarity of the i-th eigenvector is

引导残差相似度为

仿生网络生成的仿生编解码矩阵

仿生网络生成的仿生残差相似度为和

i＝1，...，n；所述信息迁移任务的损失函数为：The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

i=1,...,n; the loss function of the information migration task is:

^Among them, _Wt is the guided network weight ^{, Ws is the bionic network weight, GiT represents the encoding and decoding matrix of the i-th eigenvector of the guided network, GiS} _{represents the encoding} and decoding matrix of the i-th eigenvector of the bionic network, n represents the number of eigenvectors; _λi and _βi represent the weight factors of the corresponding loss terms, and N represents the number of data points.

The fundus image intelligent analysis and prediction module based on ordered classification predicts the state change of the segmented vascular image, and sequentially divides the ordered multi-classification problem into a number of binary classification problems;

The loss function for ordered classification is:

(7)

代表第c个样本相对于第t个二元分类任务的输出，

代表第c个样本的第t个二元分类任务的真实标签，

代表第t个二元分类任务中第c个样本的权重，W^t表示第t个二元分类任务分类器的参数，x_c代表第c个输入向量，

表示概率模型。Where N represents the number of data points, T represents the binary classification task, and ^γt represents the weight of the t-th binary classification task.

represents the output of the c-th sample relative to the t-th binary classification task,

represents the true label of the t-th binary classification task for the c-th sample,

represents the weight of the cth sample in the tth binary classification task, ^Wt represents the parameters of the tth binary classification task classifier, _xc represents the cth input vector,

Represents a probability model.

A retinal image intelligent analysis method based on information migration and ordered classification comprises the following steps:

Step 1: Preprocess the acquired fundus images to improve image quality and highlight useful features;

Step 2, performing blood vessel segmentation on the fundus image based on information migration; performing blood vessel segmentation on the preprocessed fundus image to obtain a blood vessel image;

Step 3: predicting the state change of the vascular image based on the ordered classification. This embodiment accurately predicts the stage of fundus image lesions.

Step 1 specifically includes the following steps:

S101, performing dual-tree complex wavelet and improved top-hat transform on the collected original fundus image to obtain an enhanced fundus image; step S101 effectively improves the retinal image quality, increases the vascular contrast, and highlights the characteristic elements (vascular structure, optic disc, macula);

The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an opening operation fundus image; when the transformed fundus image is subtracted from the opening operation fundus image, the data whose grayscale value is transformed remains unchanged, and the unchanged data is the subtraction result.

S102, extracting the field of view based on the spatial brightness information, converting the image-enhanced fundus image from the RGB format to the YIQ format, setting a segmentation threshold, extracting the surrounding black field of view, obtaining the useful information area through an erosion operation, separating the fundus area from the background area, obtaining a mask image, and multiplying the mask image and the image-enhanced fundus image to obtain a fundus area image;

S103, a multi-scale linear filter based on the Hessian matrix is used to set parameters according to the grayscale values and characteristic values of the blood vessels in the fundus area image, and the noise is eliminated after filtering to further highlight the blood vessel features and obtain a filtered blood vessel feature image.

Step 2 specifically includes the following steps:

Step 201, using the filtered vascular feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the vascular segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every A iterations, saving the weights with segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network;

Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in step 201, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the loss function as a constraint condition during the fitting process;

Step 203, updating the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction where the loss function value becomes smaller through the back propagation method, and jumping to step 202 for iterative training, performing a segmentation accuracy test on the validation set every A iterations, saving the weights with accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the final optimal weight of the bionic network;

In step 202, the guided codec matrix and the bionic codec matrix are G∈R ^m×n′ , G∈R ^m×n′ is generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, the output feature map of the encoder layer is F ¹ ∈R ^h×w×m , where h, w, m are the height, width and number of channels of the F ¹ feature map, and the output feature map of the decoder layer is F ² ∈R ^h×w×n′ , where h, w, n′ are the height, width and number of channels of the F ² feature map, respectively. The guided codec matrix and the bionic codec matrix G∈R ^m×n′ are calculated as:

Wherein, s=1, ..., h, t=1, ..., w, x and W represent the weights of the input image and the guidance/bionic network respectively; _Ga,b (x; W) represents the ath row and bth column of the guidance codec matrix or the bionic codec matrix;

In step 202, residual similarity is extracted by a multi-scale residual similarity (MRS) collection module, and the multi-scale residual similarity (MRS) collection module uses a similarity volume to collect context information, specifically including the following steps:

For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is:

P _j ′＝P _j ×P _center (4)

其中d代表自定义区域的大小，H和W′分别表示特征向量的高和宽，分别获取相应的残差相似度

将残差相似度

相加求和，第i个特征向量的最终残差相似度为

Where j represents the coordinate of the d×d region. For each pixel in the vascular feature image after image filtering, a local representation is obtained, and then the local representations are connected along the channel dimension to obtain the residual similarity of the i-th feature vector

Where d represents the size of the custom region, H and W′ represent the height and width of the feature vector, respectively, and the corresponding residual similarity is obtained.

The residual similarity

Add and sum, the final residual similarity of the i-th eigenvector is

引导残差相似度为

仿生网络生成的仿生编解码矩阵

仿生网络生成的仿生残差相似度为和

信息迁移任务的损失函数为：The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

The loss function of the information transfer task is:

Where _Wt is the ^guide network weight, _Ws is the bionic network weight, _GiT represents the encoding and decoding matrix of the i-th eigenvector of the guide network, _GiS represents the encoding and decoding matrix of the i-th eigenvector of the bionic network, ⁿ represents the number of eigenvectors (number of encoding and decoding matrices); _λi and _βi represent the weight factors of the loss term corresponding to the i-th eigenvector, and N represents the number of data points;

Step S3 predicts the state change of the segmented blood vessel image, and sequentially divides the ordered multi-classification problem into several binary classification problems;

The loss function for ordered classification is:

代表第c个样本相对于第t个二元分类任务的输出，

代表第c个样本的第t个二元分类任务的真实标签，

代表第t个二元分类任务中第c个样本的权重，W^t表示第t个二元分类任务，x_c代表第c个输入向量，

表示概率模型最终的分类结果依据每一个二分类任务的分类结果来进行整合判断。(7) where N represents the number of data points, T represents the binary classification task, and ^γt represents the weight of the t-th binary classification task.

represents the output of the c-th sample relative to the t-th binary classification task,

represents the true label of the t-th binary classification task for the c-th sample,

represents the weight of the cth sample in the tth binary classification task, ^Wt represents the tth binary classification task, _xc represents the cth input vector,

It indicates that the final classification result of the probability model is integrated and judged based on the classification results of each binary classification task.

Compared with the prior art, the present invention has the following beneficial effects:

The present invention discloses a retinal image intelligent analysis system based on information migration and ordered classification for image preprocessing enhancement, then performs retinal blood vessel segmentation based on an information migration fundus image segmentation network, and finally uses a neural network combined with ordered classification for intelligent analysis and prediction to achieve intelligent analysis of retinal images.

The visual image enhancement algorithm of the image preprocessing module of the present invention uses multiple algorithms to work together to effectively enhance image quality; the fundus image vascular segmentation network based on information migration effectively realizes retinal vascular segmentation; the intelligent analysis and prediction module combined with ordered classification accurately predicts and effectively and quickly realizes intelligent analysis of retinal images.

The fundus image preprocessing module of the present application includes an image enhancement submodule, a mask generation submodule and a multi-scale linear filter submodule, which improves the contrast of blood vessels, highlights characteristic elements, overcomes the defects of the commonly used discrete wavelet transform, and retains detail information while effectively improving the image quality; the image grayscale difference is significantly increased, and some edges with small amplitude changes are effectively protected.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG1 is a schematic diagram of the overall system architecture of the retinal image intelligent analysis system based on information migration and orderly classification of the present invention;

FIG2 is a schematic diagram of the overall structure of a fundus image blood vessel segmentation network module based on information migration in the present invention;

FIG. 3 is a mask image of this embodiment.

DETAILED DESCRIPTION

The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The core idea of the present invention is to design a suitable algorithm for image preprocessing and enhancement according to the characteristics of the fundus image itself, and then perform retinal vascular segmentation based on the information migration fundus image segmentation network, and finally use a neural network combined with ordered classification for intelligent analysis and prediction to achieve the purpose of intelligent analysis of retinal images.

The overall process of the retinal image intelligent analysis system based on information migration and orderly classification of the present invention is shown in Figure 1. A retinal image intelligent analysis system based on information migration and orderly classification includes a fundus image preprocessing module, a fundus image blood vessel segmentation network module, and a fundus image intelligent analysis prediction module. This embodiment also includes a front-end display module;

The fundus image preprocessing module performs image preprocessing on the collected fundus images, aiming to improve the image quality and highlight the useful features;

The fundus image vascular segmentation network module performs vascular segmentation on the preprocessed fundus image based on information migration to obtain a vascular image. The fundus image vascular segmentation network module defines two networks: a guide network and a bionic network. The fundus image vascular segmentation network module defines important information of the guide network and transfers the knowledge information extracted from the guide network to the bionic network.

The fundus image intelligent analysis and prediction module predicts the state change of the vascular image based on the ordered classification. In this embodiment, the progression of diabetes is an ordered process, from mild lesions to moderate and then to severe lesions, and the staged prediction of the fundus image state change is accurately achieved;

The front-end display module reads the corresponding data from the database, displays the processing results of each algorithm sub-module to the user and interacts with the user.

In this embodiment, data set collection is first performed, and 1,000 outpatient or inpatient type 2 diabetes patients diagnosed from January 2020 to December 2022 are selected. After exclusion and routine examination, a unified specially trained diabetes specialist nurse uses a non-mydriasis fundus camera to take fundus photos of the subjects and obtain a unique fundus image.

The fundus image preprocessing module includes an image enhancement submodule, a mask generation submodule and a multi-scale linear filter submodule;

The image enhancement submodule performs dual-tree complex wavelet and improved top-hat transform on the collected original fundus image to obtain an image-enhanced fundus image; the image enhancement submodule effectively improves the quality of retinal images, increases vascular contrast, and highlights characteristic elements (vascular structure, optic disc, macula);

The dual-tree complex wavelet transform (DT-CWT) is designed to overcome the defects of the commonly used discrete wavelet transform. When the corresponding wavelet basis approximately satisfies the Hilbert transform relationship, the dual-tree complex wavelet transform can reduce the translation sensitivity in the usual real wavelet transform, improve the directional selectivity, and retain the detail information while effectively improving the image quality. After the original fundus image is subjected to the dual-tree complex wavelet transform, the fundus image after the complex wavelet transform is subjected to the improved top-hat transform.

The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an opening operation fundus image; when the transformed fundus image is subtracted from the opening operation fundus image, the data whose grayscale value is transformed remains unchanged, and the unchanged data is the subtraction result.

The morphological top-hat transformation is performed by combining opening and closing operations, which are derived from dilation and erosion operations. The traditional top-hat transformation takes the original image minus the result of the opening operation as the final result, but this will make the final result darker overall, and some darker edges cannot be displayed. The improved top-hat transformation of the present application is that when the fundus image after the complex wavelet transformation is subtracted from the fundus image after the opening operation, the data whose grayscale value has been transformed remains unchanged, and the unchanged data is the subtraction result. After the improved top-hat transformation, the image grayscale difference is significantly increased, and some edges with small amplitude changes can also be effectively protected;

After dual-tree complex wavelet transform and improved top-hat transform, the fundus image with local detail enhancement and image quality improvement is obtained;

The mask generation submodule performs field extraction based on spatial brightness information, converts the image-enhanced fundus image (a color image, an image after local detail enhancement and image quality improvement through complex wavelet transform and top-hat transform) from RGB format to YIQ format, sets a segmentation threshold, extracts the surrounding black field of view, obtains the useful information area through corrosion operation, separates the fundus area from the background area, obtains a mask image (as shown in FIG. 3 ), and multiplies the mask image and the image-enhanced fundus image to obtain a fundus area image.

After image enhancement, fundus image analysis needs to obtain ROI (region of interest), that is, fundus area image, so that the influence of pixels outside the ROI area can be effectively avoided in subsequent processing, reducing the complexity of calculation. The mask image can define the ROI and useless areas.

The mask image is usually figuratively referred to as the "field of view". The original fundus image is used to generate a mask image of the same size to separate the fundus area and the background area. In order to accurately extract the mask image (fundus image field of view), the present invention proposes a field of view extraction method based on spatial brightness information. The field of view extraction method based on spatial brightness information separates brightness information and chromaticity information on the basis of an image in YIQ format, and then extracts the surrounding black area by selecting a segmentation threshold. Different from the RGB three-color channels, in the YIQ format, Y refers to the brightness information of the image, I refers to the color change from orange to cyan, and Q refers to the color change from purple to yellow-green. Since it contains dual information of brightness and color, the brightness component in the image is separated and extracted, including the following steps:

Step 1: Convert the enhanced fundus image from RGB format to YIQ format:

The above formula (1) can obtain the three components of the image in YIQ format;

Step 2: Set the segmentation threshold, extract the surrounding black field of view, and obtain the region of interest through corrosion operation;

The selection of segmentation threshold is the basis of segmentation extraction. For a single segmentation threshold, the grayscale value of the pixel is divided into two categories by the segmentation threshold, one is the grayscale value greater than the segmentation threshold, and the other is the grayscale less than the segmentation threshold. The Y component refers to the brightness information of the image. The Y component histogram of the fundus image in YIQ format is a bimodal state, including two parts of high pixels, and the pixel values at both ends are 0. Separation is performed based on the characteristics of the Y component histogram;

After a series of experiments, this embodiment sets the segmentation threshold to 50 to extract and separate the black area to obtain a mask image:

Among them, "1" represents the background border image, and "0" represents the eyeball blood vessels; Y is the brightness information of the image, which is equal to the grayscale value of the image brightness component (the Y component of the fundus image in YIQ format), M(x′, y) is the extracted background border, x′, y represent the pixel coordinates, if the Y component of the pixel in the x′th row and yth column is greater than 50, the point is considered to be the background area;

The multiscale linear filter submodule is based on the multiscale linear filter of the Hessian matrix, and sets parameters according to the grayscale value and characteristic value of the blood vessels in the fundus area image, eliminates noise after filtering, further highlights the blood vessel features, and obtains a filtered blood vessel feature image;

The Hessian matrix can enhance specific structures (point structures/line structures) in the image, thereby extracting target features and eliminating other useless noise information.

In a two-dimensional image, the Hessian matrix is a two-dimensional positive definite matrix with two eigenvalues and corresponding eigenvectors. The two eigenvalues represent the anisotropy of the image change in the direction indicated by the two eigenvectors. If an ellipse is formed using the eigenvectors and eigenvalues, then the ellipse marks the anisotropy of the image change. The point structure in the image is isotropic, while the linear structure is anisotropic. Therefore, the Hessian matrix characteristics are used to construct a filter to enhance the linear structure in the fundus area image, that is, the vascular structure, while filtering out the point structure, that is, the noise point.

As shown in Figure 2, the fundus image vascular segmentation network module performs vascular segmentation on the vascular feature image after preprocessing and filtering of the fundus image based on information migration. The fundus image vascular segmentation network module performs vascular segmentation based on information migration. The fundus image vascular segmentation network module includes two segmentation networks: a guidance network and a bionic network. Information migration refers to the transfer of knowledge information extracted from the pre-trained guidance network model to the bionic network. The training process of the fundus image vascular segmentation network module includes the following steps:

Step 201, using the filtered blood vessel feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the blood vessel segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every 100 iterations, saving the weights with a segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network (saving the optimal weight means that during the iterative training process, the network will be tested on the validation set every 100 iterations, and the one with the highest segmentation accuracy is called the optimal weight);

Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in A10, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the L2 loss function as a constraint condition during the fitting process;

Step 203, update the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction where the loss function value becomes smaller through the back propagation method, and jump to 202 for iterative training. Every 100 iterations, perform a segmentation accuracy test on the validation set, save the weights with an accuracy greater than the set segmentation threshold, and after the iterative training is completed, select the weight with the highest segmentation accuracy as the final optimal weight of the bionic network.

In the process of neural network encoding and decoding (the solution to the overall problem of segmentation network is a process of encoding and then decoding), the flow of information includes the method of solving the problem. Transferring the network's problem-solving method as knowledge information to the bionic network can enable it to achieve better generalization.

In step 202, the guided codec matrix and the bionic codec matrix are G∈R ^m×n′ , G∈R ^m×n′ (m*n′ matrix, G matrix is generated by the feature maps on both sides) generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, and the output feature map of the encoder layer is F ¹ ∈R ^h×w×m , where h, w, m are the height, width and number of channels of the F ¹ feature map (the height, width and number of channels of the feature map image of this layer of the guided network), and the output feature map of the decoder layer is F ² ∈R ^h×w×n′ , where h, w, n′ represent the height, width and number of channels of the F ² feature map, respectively. The guided codec matrix and the bionic codec matrix G∈R ^m×n′ are calculated as follows:

Wherein, s=1, ..., h, t=1, ..., w, x and W represent the weights of the input image and the guided/bionic network (guided network or bionic network), respectively; _Ga,b (x; W) represents the a-th row and b-th column of the guided codec matrix or the bionic codec matrix (the codec matrix is an m*n′ high-dimensional matrix with a-th row and b-th column).

In step 202, residual similarity is extracted by a multi-scale residual similarity (MRS) collection module, and the multi-scale residual similarity (MRS) collection module is used to collect feature information of the captured local area. The multi-scale residual similarity (MRS) collection module uses a similar volume to collect context information, and specifically includes the following steps:

For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is:

P _j ′＝P _j ×P _center (4)

其中d代表自定义区域的大小，H和W′分别表示特征向量的高和宽，中心像素周围像素点的重要性随着距离衰减而降低，因此，所述多尺度残差相似度收集模块，就是设定不同值的d，分别获取相应的残差相似度

然后将残差相似度

相加求和，第i个特征向量的最终残差相似度为

然后选取不同的d值(d＝3、5、7)将对应的残差相似度

(d＝3，5，7，不同的d值大小体现多尺度)相加求和，得到第i个特征向量的最终残差相似度为

(前述的

是输入的特征向量的残差相似度，

是输出的特征向量的残差相似度)。Where j represents the coordinate of the d×d region. For each pixel in the vascular feature image after image filtering, a local representation is obtained, and then the local representations are connected along the channel dimension to obtain the residual similarity of the i-th feature vector

Where d represents the size of the custom area, H and W′ represent the height and width of the feature vector respectively. The importance of the pixels around the central pixel decreases with the distance attenuation. Therefore, the multi-scale residual similarity collection module sets different values of d to obtain the corresponding residual similarities.

Then the residual similarity

Add and sum, the final residual similarity of the i-th eigenvector is

Then select different d values (d = 3, 5, 7) to convert the corresponding residual similarity

(d = 3, 5, 7, different d values reflect multi-scale) add and sum, and the final residual similarity of the i-th eigenvector is obtained as

(The aforementioned

is the residual similarity of the input feature vector,

is the residual similarity of the output feature vector).

引导残差相似度为

仿生网络生成的仿生编解码矩阵

仿生网络生成的仿生残差相似度为和

信息迁移使得仿生编解码矩阵和仿生残差相似度分别趋近于引导编解码矩阵和引导残差相似度。所述信息迁移任务的损失函数为：The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

Information migration makes the bionic codec matrix and bionic residual similarity approach the guided codec matrix and guided residual similarity respectively. The loss function of the information migration task is:

^Among them, _Wt is the guided network weight ^{, Ws is the bionic network weight, GiT represents the encoding and decoding matrix of the i-th eigenvector of the guided network, GiS} _{represents the encoding} and decoding matrix of the i-th eigenvector of the bionic network, n represents the number of eigenvectors; _λi and _βi represent the weight factors of the corresponding loss terms, and N represents the number of data points.

The fundus image intelligent analysis and prediction module based on ordered classification predicts the state change of the segmented vascular image. It is worth noting that this embodiment is an experiment conducted with fundus images of diabetic patients as samples, and this application is applicable to image processing of all fundus images;

The basic principle of ordered multi-classification is that there is an order relationship between dependent variables, and the ordered multi-classification problem is divided into multiple simple binary classification problems in turn. In the staging problem of diabetic retinopathy studied in the present embodiment, according to the severity of the state change, it is divided into 5 stages in total. Normal (stage 0), mild NPDR (stage 1), moderate NPDR (stage 2), severe NPDR (stage 3), PDR (stage 4). When using ordered classification for classification, this problem is split into 4 binary classification problems, namely (0 vs 1+2+3+4), (0+1 vs 2+3+4), (0+1+2 vs 3+4) and (0+1+2+3 vs 4).

This embodiment aims at the problem of diabetic retinopathy staging. If the traditional convolutional neural network is used to solve it, it is a multi-classification problem. After multiple layers of convolution, pooling, and full connection layers, a softmax regression can be used to complete the classification task. However, this does not take into account the characteristics of the lesion staging itself. The network structure mainly refers to GoogLeNet, and the last layer of the network is set to ordered classification. Its loss function is as follows:

代表第c个样本相对于第t个任务的输出，

代表真实标签。

代表第t个二元分类任务中每个样本的权重，后续随着网络的训练在优化。w^t表示第t个分类器所具有的参数，xc代表第c个输入向量、

表示概率模型。对于每一个二分类任务，我们得到的输出结果都为0或者1。最终的分类结果会依据每一个二分类任务的分类结果来进行整合判断。具体在本任务中，将4个二分类任务的分类结果进行求和，即可得到最终的疾病的分期预测.Where N represents the number of data points, T represents the binary classification task, and ^γt represents the weight of each binary classification task.

represents the output of the cth sample relative to the tth task,

Represents the true label.

represents the weight of each sample in the t-th binary classification task, which is subsequently optimized as the network is trained. ^{w t} represents the parameters of the t-th classifier, xc represents the c-th input vector,

Represents a probability model. For each binary classification task, the output result we get is 0 or 1. The final classification result will be integrated and judged based on the classification results of each binary classification task. Specifically in this task, the classification results of the four binary classification tasks are summed up to get the final disease stage prediction.

The front-end user interface module displays the corresponding results of each stage to the user, and the system after training can realize intelligent analysis of fundus images.

As shown in FIG1 , a retinal image intelligent analysis method based on information migration and ordered classification includes the following steps:

Step 1: Preprocess the acquired fundus images to improve image quality and highlight useful features;

Step 2, performing blood vessel segmentation on the fundus image based on information migration; performing blood vessel segmentation on the preprocessed fundus image to obtain a blood vessel image;

Step 3: predict the state change of the vascular image based on the ordered classification.

Step 1 specifically includes the following steps:

S101, performing dual-tree complex wavelet and improved top-hat transform on the collected original fundus image to obtain an enhanced fundus image; step S101 effectively improves the retinal image quality, increases the vascular contrast, and highlights the characteristic elements (vascular structure, optic disc, macula);

The dual-tree complex wavelet transform (DT-CWT) is designed to overcome the defects of the commonly used discrete wavelet transform. When the corresponding wavelet basis approximately satisfies the Hilbert transform relationship, the dual-tree complex wavelet transform can reduce the translation sensitivity in the usual real wavelet transform, improve the directional selectivity, and retain the detail information while effectively improving the image quality. After the original fundus image is subjected to the dual-tree complex wavelet transform, the fundus image after the complex wavelet transform is subjected to the improved top-hat transform.

The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an opening operation fundus image, when the transformed fundus image is subtracted from the opening operation fundus image, the data with the transformed grayscale value remains unchanged, and the unchanged data is the subtraction result;

The morphological top-hat transformation is performed by combining opening and closing operations, which are derived from dilation and erosion operations. The traditional top-hat transformation takes the original image minus the result of the opening operation as the final result, but this will make the final result darker overall, and some darker edges cannot be displayed. The improved top-hat transformation of the present application is that when the fundus image after the complex wavelet transformation is subtracted from the fundus image after the opening operation, the data whose grayscale value has been transformed remains unchanged, and the unchanged data is the subtraction result. After the improved top-hat transformation, the image grayscale difference is significantly increased, and some edges with small amplitude changes can also be effectively protected;

After dual-tree complex wavelet transform and improved top-hat transform, the fundus image with local detail enhancement and image quality improvement is obtained;

S102, extracting the field of view based on the spatial brightness information, converting the image-enhanced fundus image (a color image, an image after local detail enhancement and image quality improvement through complex wavelet transform and top-hat transform) from RGB format to YIQ format, setting a segmentation threshold, extracting the surrounding black field of view, obtaining the useful information area through corrosion operation, separating the fundus area and the background area, obtaining a mask image, and multiplying the mask image and the image-enhanced fundus image to obtain a fundus area image.

After image enhancement, fundus image analysis needs to obtain ROI (region of interest), that is, fundus area image, so that the influence of pixels outside the ROI area can be effectively avoided in subsequent processing, reducing the complexity of calculation. The mask image can define the ROI and useless areas.

The mask image is usually figuratively referred to as the "field of view". The original fundus image is used to generate a mask image of the same size to separate the fundus area and the background area. In order to accurately extract the mask image (fundus image field of view), the present invention proposes a field of view extraction method based on spatial brightness information. The field of view extraction method based on spatial brightness information separates brightness information and chromaticity information on the basis of an image in YIQ format, and then extracts the surrounding black area by selecting a segmentation threshold. Different from the RGB three-color channels, in the YIQ format, Y refers to the brightness information of the image, I refers to the color change from orange to cyan, and Q refers to the color change from purple to yellow-green. Because it contains dual information of brightness and color, the brightness component in the image is separated and extracted;

Mask image extraction specifically includes the following steps:

Step 1: Convert the enhanced fundus image from RGB format to YIQ format:

The above formula (1) can obtain the three components of the image in YIQ format;

Step 2: Set the segmentation threshold, extract the surrounding black field of view, and obtain the region of interest through corrosion operation;

The selection of segmentation threshold is the basis of segmentation extraction. For a single segmentation threshold, the grayscale value of the pixel is divided into two categories by the segmentation threshold, one is the grayscale value greater than the segmentation threshold, and the other is the grayscale less than the segmentation threshold. The Y component refers to the brightness information of the image. The Y component histogram of the fundus image in YIQ format is a bimodal state, including two parts of high pixels, and the pixel values at both ends are 0. Separation is performed based on the characteristics of the Y component histogram;

After a series of experiments, this embodiment sets the segmentation threshold to 50 to extract and separate the black area to obtain a mask image:

Among them, "1" represents the background border image, and "0" represents the blood vessels of the eyeball; Y is the brightness information of the image, which is equal to the grayscale value of the image brightness component (the Y component of the fundus image in YIQ format), M(x',y) is the extracted background border, x',y represent the pixel coordinates, if the Y component of the pixel in the x'th row and yth column is greater than 50, the point is considered to be the background area;

S103, a multi-scale linear filter based on the Hessian matrix, setting parameters according to the grayscale values and characteristic values of the blood vessels in the fundus area image, eliminating noise after filtering, further highlighting the blood vessel features, and obtaining a filtered blood vessel feature image;

The Hessian matrix can enhance specific structures (point structures/line structures) in the image, thereby extracting target features and eliminating other useless noise information.

In a two-dimensional image, the Hessian matrix is a two-dimensional positive definite matrix with two eigenvalues and corresponding eigenvectors. The two eigenvalues represent the anisotropy of the image change in the direction indicated by the two eigenvectors. If an ellipse is formed using the eigenvectors and eigenvalues, then the ellipse marks the anisotropy of the image change. The point structure in the image is isotropic, while the linear structure is anisotropic. Therefore, the Hessian matrix characteristics are used to construct a filter to enhance the linear structure in the fundus area image, that is, the vascular structure, while filtering out the point structure, that is, the noise point.

Step 2 performs blood vessel segmentation on the filtered blood vessel feature image of the preprocessed fundus image based on information migration. The fundus image blood vessel segmentation network module includes two segmentation networks: a guided network and a bionic network. Information migration refers to the migration of knowledge information extracted from the pre-trained guided network model to the bionic network. Step 2 includes the following steps:

Step 201, using the filtered blood vessel feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the blood vessel segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every 100 iterations, saving the weights with a segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network (saving the optimal weight means that during the iterative training process, the network will be tested on the validation set every 100 iterations, and the one with the highest segmentation accuracy is called the optimal weight);

The overall data set will be divided into training set, validation set and test set, and the guiding network and bionic network use the same data set;

Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in step 201, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the L2 loss function as a constraint condition during the fitting process;

Step 203, update the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction where the loss function value becomes smaller through the back propagation method, and corresponding parameters (parameters include the encoding and decoding matrix parameters, residual similarity parameters and segmentation network parameters of the bionic network), and jump to step 202 for iterative training. Every 100 iterations, perform a segmentation accuracy test on the validation set, save the weights with an accuracy greater than the set segmentation threshold, and after the iterative training is completed, select the weight with the highest segmentation accuracy as the final optimal weight of the bionic network.

In step 202, the guided codec matrix and the bionic codec matrix are G∈Rm×n′, G∈Rm×n′ (m*n′ matrix, G matrix is generated by the feature maps on both sides) are generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, and the output feature map of the encoder layer is ^F1∈Rh × ^w ×m, where h, w, m are the height, width and number of channels of the F1 feature map (the height, width and number of channels of the feature map image of this layer of the guided network), and the output feature map of the decoder layer is ^F2∈Rh × ^w ×n′, where h, w, n′ are the height, width and number of channels of the ^F2 feature map, respectively. The guided codec matrix and the bionic codec matrix G∈Rm×n′ are calculated as follows:

Wherein, s=1, ..., h, t=1, ..., w, x and W represent the weights of the input image and the guided/bionic network (guided network or bionic network), respectively; _Ga,b (x; W) represents the a-th row and b-th column of the guided codec matrix or the bionic codec matrix (the codec matrix is an m*n′ high-dimensional matrix with a-th row and b-th column).

In step 202, residual similarity is extracted by a multi-scale residual similarity (MRS) collection module, and the multi-scale residual similarity (MRS) collection module is used to collect feature information of the captured local area. The multi-scale residual similarity (MRS) collection module uses a similar volume to collect context information, and specifically includes the following steps:

For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is:

P _j ′＝P _j ×P _center (4)

其中d代表自定义区域的大小，H和W′，分别表示特征向量的高和宽，中心像素周围像素点的重要性随着距离衰减而降低，因此，所述多尺度残差相似度收集模块，选取不同的d值(d＝3、5、7)将对应的残差相似度

(d＝3，5，7，不同的d值大小体现多尺度)相加求和，得到第i个特征向量的最终残差相似度为

(前述的

是输入的特征向量的残差相似度，

是输出的特征向量的残差相似度)。Where j represents the coordinate of the d×d region. For each pixel in the vascular feature image after image filtering, a local representation is obtained, and then the local representations are connected along the channel dimension to obtain the residual similarity of the feature vector of the i-th input

Where d represents the size of the custom region, H and W′ represent the height and width of the feature vector, respectively. The importance of the pixels around the central pixel decreases with the distance decay. Therefore, the multi-scale residual similarity collection module selects different d values (d = 3, 5, 7) to convert the corresponding residual similarities

(d = 3, 5, 7, different d values reflect multi-scale) add and sum, and the final residual similarity of the i-th eigenvector is obtained as

(The aforementioned

is the residual similarity of the input feature vector,

is the residual similarity of the output feature vector).

引导残差相似度为

仿生网络生成的仿生编解码矩阵

仿生网络生成的仿生残差相似度为和

信息迁移使得仿生编解码矩阵和仿生残差相似度分别趋近于引导编解码矩阵和引导残差相似度。所述信息迁移任务的损失函数为：The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

Information migration makes the bionic codec matrix and bionic residual similarity approach the guided codec matrix and guided residual similarity respectively. The loss function of the information migration task is:

^Among them, _Wt is the guided network weight, _Ws is the bionic network weight, _GiT represents the encoding and decoding matrix ^{of the i-th eigenvector of the guided network, Gis} _represents the encoding and decoding matrix of the i-th eigenvector of the bionic network, n represents the number of eigenvectors; _λi and _βi represent the weight factors of the corresponding loss terms, and N represents the number of data points.

Step S3 predicts the state change of the segmented blood vessel image, and sequentially divides the ordered multi-classification problem into several binary classification problems;

Specifically, for M (M represents the need to predict the state change output as M categories, in the specific embodiment, M = 5) classification problems, when using ordered classification for classification, this problem is split into M-1 binary classification problems, namely (0 vs 1 + 2 + 3 + ... + M-1), (0 + 1 vs 2 + 3 + ... + M-1), ... and (0 + 1 + 2 + 3 ... + M-2 vs M-1). According to the characteristics between the variables, ordered classification is introduced in the network structure of fundus image intelligent analysis prediction (the network refers to the ordered classification of fundus image intelligent analysis is a classification network), and the loss function of ordered classification is:

代表第c个样本相对于第t个二元分类任务的输出，

代表第c个样本的第t个二元分类任务的真实标签，

代表第t个二元分类任务中第c个样本的权重，随着网络的训练在优化。W^t表示第t个二元分类任务分类器的参数(分类器的参数，具体包含网络层数，滤波器大小)，x_c代表第c个输入向量，

表示概率模型。最终的分类结果依据每一个二分类任务的分类结果来进行整合判断。Where N represents the number of data points, T represents the binary classification task, and ^γt represents the weight of the t-th binary classification task.

represents the output of the c-th sample relative to the t-th binary classification task,

represents the true label of the t-th binary classification task for the c-th sample,

represents the weight of the cth sample in the tth binary classification task, which is optimized as the network is trained. ^Wt represents the parameters of the classifier for the tth binary classification task (the parameters of the classifier, including the number of network layers and filter size), _xc represents the cth input vector,

Represents a probability model. The final classification result is integrated and judged based on the classification results of each binary classification task.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

Those skilled in the art will appreciate that the modules or units or groups of the devices in the examples disclosed herein may be arranged in the device as described in the embodiment, or alternatively may be located in one or more devices different from the devices in the example. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or groups in the embodiments may be combined into one module or unit or group, and in addition they may be divided into a plurality of submodules or subunits or subgroups. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and/or processes or units are mutually exclusive. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present invention, or some aspects or portions of the method and apparatus of the present invention may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a floppy disk, a CD-ROM, a hard disk drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present invention.

In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

By way of example and not limitation, computer readable media include computer storage media and communication media. Computer readable media include computer storage media and communication media. Computer storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery media. Combinations of any of the above are also included within the scope of computer readable media.

As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved, and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and didactic purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims (2)

1. A retinal image intelligent analysis system based on information migration and ordered classification, characterized by comprising a fundus image preprocessing module, a fundus image blood vessel segmentation network module and a fundus image intelligent analysis and prediction module; The fundus image preprocessing module performs image preprocessing on the collected fundus image; The fundus image vascular segmentation network module performs vascular segmentation on the preprocessed fundus image based on information migration to obtain a vascular image, wherein a guiding network and a bionic network are defined in the fundus image vascular segmentation network module; The fundus image intelligent analysis and prediction module predicts the state change of the blood vessel image based on the ordered classification; The fundus image preprocessing module includes an image enhancement submodule, a mask generation submodule and a multi-scale linear filter submodule; The image enhancement submodule performs dual-tree complex wavelet and improved top-hat transformation on the collected original fundus image to obtain an image-enhanced fundus image; The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an open operation fundus image, when the fundus image after the dual-tree complex wavelet transform is subtracted from the open operation fundus image, the data with the changed gray value remains unchanged, and the unchanged data is the subtraction result; The mask generation submodule extracts the field of view based on the spatial brightness information, converts the image-enhanced fundus image from the RGB format to the YIQ format, sets the segmentation threshold, extracts the surrounding black field of view, obtains the useful information area through the corrosion operation, separates the fundus area from the background area, obtains the mask image, and multiplies the mask image and the image-enhanced fundus image to obtain the fundus area image, specifically including the following steps: Step 1: Convert the enhanced fundus image from RGB format to YIQ format:

Formula (1) obtains the three components of the fundus image in YIQ format; Step 2: Set the segmentation threshold, extract the surrounding black field of view, and obtain the region of interest through corrosion operation; The mask image is obtained as:

Among them, "1" represents the background border image, and "0" represents the blood vessels of the eyeball; Y is the brightness information of the image, which is equal to the grayscale value of the brightness component of the image, M(x′,y) is the extracted background border, and x′,y represent the pixel coordinates; The multiscale linear filter submodule is based on the multiscale linear filter of the Hessian matrix, and sets parameters according to the grayscale value and characteristic value of the blood vessels in the fundus area image, eliminates noise after filtering, and obtains the filtered blood vessel characteristic image; The training process of the fundus image blood vessel segmentation network module includes the following steps: Step 201, using the filtered vascular feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the vascular segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every A iterations, saving the weights with segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network; Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in step 201, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the loss function as a constraint condition during the fitting process; Step 203, updating the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction where the loss function value becomes smaller through the back propagation method, and jumping to step 202 for iterative training, performing a segmentation accuracy test on the validation set every A iterations, saving the weights with accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the final optimal weight of the bionic network; In step 202, the guided codec matrix and the bionic codec matrix are G∈R ^m×n′ , G∈R ^m×n′ is generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, the output feature map of the encoder layer is F ¹ ∈R ^h×w×m , where h, w, m are the height, width and number of channels of the F ¹ feature map, and the output feature map of the decoder layer is F ² ∈R ^h×w×n′ , where h, w, n′ are the height, width and number of channels of the F ² feature map, respectively. The guided codec matrix and the bionic codec matrix G∈R ^m×n′ are calculated as:

Wherein, s = 1, ..., h, t = 1, ..., w, x and W represent the weights of the input image and the guided/bionic network respectively; _Ga,b (x; W) represents the ath row and bth column of the guided codec matrix or the bionic codec matrix; Step 202 specifically includes the following steps: For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is: P _j ′＝P _j ×P _center ⑷ Wherein, in step 202, j represents the coordinate of the d×d region, and a local representation is obtained for each pixel in the vascular feature image after the image is filtered, and then the local representations are connected along the channel dimension to obtain the residual similarity of the i-th feature vector

Where d represents the size of the region, H and W′ represent the height and width of the feature vector respectively, and the corresponding residual similarity is obtained.

Select different d values and convert the corresponding residual similarity

Add and sum, and the final residual similarity of the i-th eigenvector is

The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

i＝1,...,n; the loss function of the information transfer task is:

Among them, _Wt is the guided network weight, _Ws is the bionic network weight,

represents the encoding and decoding matrix of the i-th eigenvector of the guided network,

represents the encoding and decoding matrix of the i-th eigenvector of the bionic network, n represents the number of eigenvectors; λ _i and β _i represent the weight factors of the corresponding loss terms, and N represents the number of data points; The fundus image intelligent analysis and prediction module based on ordered classification predicts the state change of the segmented vascular image, and sequentially divides the ordered multi-classification problem into a number of binary classification problems; The loss function for ordered classification is:

Where N represents the number of data points, T represents the binary classification task, and ^γt represents the weight of the t-th binary classification task.

represents the output of the c-th sample relative to the t-th binary classification task,

represents the true label of the t-th binary classification task for the c-th sample,

represents the weight of the cth sample in the tth binary classification task, ^Wt represents the parameters of the tth binary classification task classifier, _xc represents the cth input vector,

Represents a probability model. 2. A retinal image intelligent analysis method based on information migration and ordered classification, characterized by comprising the following steps: Step 1, performing image preprocessing on the collected fundus image; Step 2, performing blood vessel segmentation on the fundus image based on information migration; performing blood vessel segmentation on the preprocessed fundus image to obtain a blood vessel image; Step 3, predicting the state change of the vascular image based on the ordered classification; Step 1 specifically includes the following steps: S101, performing dual-tree complex wavelet transform and improved top-hat transform on the collected original fundus image to obtain an image-enhanced fundus image; The improved top-hat transform specifically includes the following steps: performing an opening operation on the fundus image after the dual-tree complex wavelet transform to obtain an opening operation fundus image, when the transformed fundus image is subtracted from the opening operation fundus image, the data with the transformed grayscale value remains unchanged, and the unchanged data is the subtraction result; S102, extracting the field of view based on the spatial brightness information, converting the image-enhanced fundus image from the RGB format to the YIQ format, setting a segmentation threshold, extracting the surrounding black field of view, obtaining the useful information area through an erosion operation, separating the fundus area from the background area, obtaining a mask image, and multiplying the mask image and the image-enhanced fundus image to obtain a fundus area image; S103, a multi-scale linear filter based on the Hessian matrix, setting parameters according to the grayscale values and characteristic values of the blood vessels in the fundus area image, eliminating noise after filtering, and obtaining a filtered blood vessel characteristic image; Step 2 specifically includes the following steps: Step 201, using the filtered vascular feature image and the segmentation label as the common input of the guide network and the bionic network, training the guide network to perform the vascular segmentation task, during the iterative training of the guide network, performing a segmentation accuracy test on the validation set after every A iterations, saving the weights with segmentation accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the optimal weight of the guide network; Step 202, during the iterative training of the bionic network, the guide network loads the guide network optimal weights saved in step 201, generates a guide encoding and decoding matrix and a guide residual similarity for the filtered vascular feature image, the bionic network generates a corresponding bionic encoding and decoding matrix and a bionic similarity matrix for the filtered vascular feature image, the bionic network fits the segmentation label, the guide encoding and decoding matrix and the guide residual similarity, and uses the loss function as a constraint condition during the fitting process; Step 203, updating the bionic encoding and decoding matrix, bionic residual similarity parameters and segmentation network parameters of the bionic network in the direction where the loss function value becomes smaller through the back propagation method, and jumping to step 202 for iterative training, performing a segmentation accuracy test on the validation set every A iterations, saving the weights with accuracy greater than the set segmentation threshold, and after the iterative training is completed, selecting the weight with the highest segmentation accuracy as the final optimal weight of the bionic network; In step 202, the guided codec matrix and the bionic codec matrix are G∈R ^m×n′ , G∈R ^m×n′ is generated by the feature maps of the guided/bionic encoder and the guided/bionic decoder, the output feature map of the encoder layer is F ¹ ∈R ^h×w×m , where h, w, m are the height, width and number of channels of the F ¹ feature map, and the output feature map of the decoder layer is F ² ∈R ^h×w×n′ , where h, w, n′ are the height, width and number of channels of the F ² feature map, respectively. The guided codec matrix and the bionic codec matrix G∈R ^m×n′ are calculated as:

Wherein, s = 1, ..., h, t = 1, ..., w, x and W represent the weights of the input image and the guided/bionic network respectively; _Ga,b (x; W) represents the ath row and bth column of the guided codec matrix or the bionic codec matrix; Step 202 specifically includes the following steps: For the i-th feature vector Y ⁽ⁱ⁾ , the similarity value P _j ′ is calculated by element-wise multiplication between each central pixel P _center and the pixels P _j in its adjacent d×d region. The formula is: P _j ′＝P _j ×P _center ⑷ Where j represents the coordinate of the d×d region. For each pixel in the vascular feature image after image filtering, a local representation is obtained, and then the local representations are connected along the channel dimension to obtain the residual similarity of the feature vector of the i-th input

Where d represents the size of the custom region, H and W′ represent the height and width of the feature vector, respectively, and the corresponding residual similarity is obtained.

The residual similarity

Add and sum, the final residual similarity of the i-th eigenvector is

The guided encoding and decoding matrix generated by the guided network is

The guided residual similarity is

Bionic codec matrix generated by bionic network

The similarity of the bionic residual generated by the bionic network is and

i＝1,...,n; the loss function of the information transfer task is:

Where _Wt is the weight of ^the guided network ^, _Ws is the weight of the bionic network, _GiT represents the encoding and decoding matrix of the i-th eigenvector of the guided network, _GiS represents the encoding and decoding matrix of the i-th eigenvector of the bionic network, n represents the number of eigenvectors; _λi and _βi represent the weight factors of the corresponding loss terms, and N represents the number of data points; Step S3 predicts the state change of the segmented blood vessel image, and sequentially divides the ordered multi-classification problem into several binary classification problems; The loss function for ordered classification is:

Where T represents the binary classification task, ^γt represents the weight of the t-th binary classification task,

represents the output of the c-th sample relative to the t-th binary classification task,

represents the true label of the t-th binary classification task for the c-th sample,

represents the weight of the cth sample in the tth binary classification task, ^Wt represents the parameters of the tth binary classification task classifier, _xc represents the cth input vector,

Represents a probability model. The final classification result is integrated and judged based on the classification results of each binary classification task.

Images