CN115457058A - An Image Segmentation Method Based on Uncertainty-Guided Deep Learning Strategy and Its Application - Google Patents

Abstract

The invention discloses an image segmentation method based on an uncertainty-guided deep learning strategy and its application, which includes the following steps: (1) Roughly segment the fundus image based on the existing deep learning network, and obtain the rough image corresponding to the optic cup region; Segmentation results; (2) use morphological operations to process the rough image segmentation results to obtain a possible area of the cup boundary, that is, the potential boundary area; (3) use traditional Gaussian filtering to smooth the potential boundary area, The uncertainty image corresponding to the target boundary can be obtained; (4) Based on the boundary uncertainty image and the original image, retrain the selected deep learning network and perform fine segmentation of the fundus cup, which can be achieved without changing the network structure. Get more accurate object extraction performance. The optic cup segmentation experiment based on the public REFUGE fundus image data shows that the present invention can significantly improve the image segmentation performance of the existing deep learning network, and greatly improve the extraction accuracy of the optic cup in the fundus image.

Description

An image segmentation method based on an uncertainty-guided deep learning strategy and its application

technical field

The present invention specifically relates to an image segmentation method and application based on an uncertainty-guided deep learning strategy.

Background technique

Image segmentation is a technology that divides the entire image into several independent local areas according to the gray distribution of pixels and tissue contrast. This technology can not only be used in the understanding and analysis of images, the detection and location of lesions and the measurement and evaluation of morphological characteristics, but also is a key preprocessing step in many image processing tasks, so it plays a very important role in image processing. In order to accurately extract the required image regions, various image segmentation algorithms have been proposed and divided into unsupervised segmentation and supervised segmentation.

The unsupervised segmentation algorithm usually distinguishes the target of interest from the irrelevant background according to the color and gray distribution of the image, the shape and position relationship between different tissue structures, and realizes the accurate extraction of the target area. This type of algorithm usually has the characteristics of simple operation and fast operation, and can effectively process images with relatively good imaging quality, but it is easily disturbed by image artifacts or noise, making it difficult to process images with severe imaging artifacts, noise, or weak images. Extract regions of interest in images of phenomena such as tissue contrast. In addition, the performance of this type of algorithm is often closely related to personal experience, because the relevant parameters in the algorithm depend on the individual, and inappropriate parameter values and changing imaging conditions will seriously degrade the image segmentation performance of the algorithm, making it incapable of performing large-scale tasks. High-quality processing of clinical images at scale.

Most supervised segmentation algorithms rely on manual annotation information to assist in the detection of image feature information and the extraction of objects of interest. The use of manual annotation information enables this type of algorithm to effectively alleviate the impact of image artifacts or noise on segmentation performance, making it have better target extraction performance than unsupervised segmentation algorithms. Currently, the segmentation algorithm based on deep learning is a key research direction in the field of supervised segmentation; this type of algorithm can perform end-to-end image segmentation and obtain extremely high accuracy, and the representative segmentation network is U-Net. This network is widely used in various image processing and obtains reasonable segmentation performance, but it is difficult to deal with the boundary area of the target and has a large boundary detection error. This is because U-Net downsamples the image multiple times, which will lead to a significant reduction in image resolution, blurring of object boundaries, and loss of a large amount of texture information. In order to improve the segmentation performance of U-Net, a large number of improved algorithms have been proposed, such as Attention U-Net, BiO-Net and U-Net++ networks. These improved networks obtain a large amount of target-related feature information by using different network structures, thereby reducing the information loss caused by multiple downsampling; however, the design of the network structure is often very complicated and requires a lot of professional knowledge as support.

Contents of the invention

Aiming at the deficiencies in the existing technology, the purpose of the present invention is to provide an image segmentation method and application based on an uncertainty-guided deep learning strategy, and realize the rough segmentation of the target of interest with the help of the existing image segmentation network. A boundary uncertainty image is constructed on the basis, and then the segmentation network is retrained based on the boundary uncertainty image and the original image to perform fine segmentation of the target area, thereby improving the segmentation performance of the existing segmentation network.

To achieve the above object, the present invention provides the following technical solutions:

An image segmentation method based on an uncertainty-guided deep learning strategy, which includes:

1) Roughly segment the image based on the image segmentation network, and perform preliminary extraction of the target of interest according to the medical image to be segmented and the corresponding manual annotation information;

2) Extract the boundary area from the result of rough segmentation;

3) Use Gaussian filtering to smooth the binary image corresponding to the potential boundary area, so that the binary image is converted into the required boundary uncertainty image,

4) Based on the boundary uncertainty image obtained in step 3), integrate the boundary uncertainty image and the original image to be segmented, and then input the integration result and the manual annotation information corresponding to the original image into the current image segmentation network to perform Training of deep learning models.

The segmentation network is a U-Net segmentation network.

In step 2), the rough segmentation results of the target of interest are respectively expanded and eroded through morphological operations. In the expansion and erosion operations, their structural elements are all set to a circular area of 25*25, and the rough segmentation results can be obtained corresponding dilated and shrunk versions, followed by pixel-based subtraction of the dilated and eroded results, resulting in an approximately ring-shaped potential boundary region centered on the coarse segmentation boundary.

The segmentation strategy adopted in step 4) is as follows:

a) Train existing deep learning networks directly based on boundary uncertainty images;

b) Integrate the original image and the boundary uncertainty image and use it to train the existing deep learning network;

c) With the help of the boundary uncertainty image, first remove the background area in the original image; then integrate it with the boundary uncertainty image for the training of the deep learning network.

An application of an image segmentation method based on the above uncertainty-guided deep learning strategy, where the uncertainty-guided deep learning method is applied to fundus cup segmentation.

It includes the following steps:

1) Use the U-Net segmentation network to perform a rough segmentation of the cup area in the fundus image;

2) The rough segmentation result in step 1) roughly points out the area where the target of interest may appear, and the rough segmentation result of the target of interest is expanded and eroded by morphological operations, and the enlarged and reduced segmentation results can be obtained; then Perform a pixel-based subtraction process on the two to obtain an approximately circular potential boundary area;

3) With the help of traditional Gaussian filtering, the binary image corresponding to the potential boundary area is smoothed, so that the binary image is converted into the required boundary uncertainty image;

4) Based on the boundary uncertainty image obtained in step 3), perform fine segmentation of the target area and retrain the current segmentation network.

A storage medium, where instructions are stored, and when a computer reads the instructions, the computer is made to execute the above image segmentation method based on the uncertainty-guided deep learning strategy.

The beneficial effect of the present invention is to provide a new network training strategy and use the existing segmentation network to perform high-quality segmentation of the target of interest from coarse to fine.

Description of drawings

Fig. 1 is a schematic diagram of an uncertainty-guided deep learning strategy in the present invention.

Figure 2 is the fundus cup segmentation data used to verify U-Net under the proposed learning strategy; from left to right are a fundus image to be segmented, a local optic disc region image, and the cup correspondence in the public REFUGE dataset Annotated by hand.

Figure 3 is the rough segmentation result of the optic cup based on the U-Net network and its corresponding potential boundary area image and boundary uncertainty image.

Figure 4 shows the rough and fine segmentation results of the U-Net network and the information difference between them and manual annotations; their corresponding cup boundary curves are red, blue and white, respectively.

detailed description

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

It should be noted that all directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relationship between the components in a certain posture (as shown in the accompanying drawings). Relative positional relationship, movement conditions, etc., if the specific posture changes, the directional indication will also change accordingly.

As shown in the figure, the present invention provides an image segmentation method based on an uncertainty-guided deep learning strategy, which includes:

1) Roughly segment the image based on the image segmentation network, and perform preliminary extraction of the target of interest according to the medical image to be segmented and the corresponding manual annotation information;

2) Extract the boundary area from the result of rough segmentation;

3) Use Gaussian filtering to smooth the binary image corresponding to the potential boundary area, so that the binary image is converted into the required boundary uncertainty image,

4) Based on the boundary uncertainty image obtained in step 3), integrate the boundary uncertainty image and the original image to be segmented, and then input the integration result and the manual annotation information corresponding to the original image into the current image segmentation network to perform Training of deep learning models. Specifically, firstly, the boundary uncertainty image and the original image to be segmented are integrated according to different strategies, and then the integration result and the manual annotation corresponding to the original image are input into the selected image segmentation network to train the deep learning model . In model training, use the Dice function as the cost function, use the RMSprop algorithm to maximize the Dice function, and set the learning rate of the optimization algorithm to 0.001. Other parameters (such as batch size and epoch, etc.) depend on the hardware configuration of the specific training model.

The segmentation strategy adopted in step 4) is as follows:

a) Train existing deep learning networks directly based on boundary uncertainty images;

b) Integrate the original image and the boundary uncertainty image and use it to train the existing deep learning network;

c) With the help of the boundary uncertainty image, first remove the background area in the original image; then integrate it with the boundary uncertainty image for the training of the deep learning network.

The existing deep learning network (such as the classic U-Net) is usually difficult to effectively deal with the boundary area of the cup when performing the segmentation of the cup in the fundus image, resulting in a relatively large boundary detection and extraction error, which seriously restricts the practicality of the network model. application. The reasons for this boundary detection error mainly include the following aspects: (a) Most of the existing deep learning networks need to downsample the image to be segmented multiple times to speed up the extraction of feature information, reduce the network training time, and reduce the training process. Requirements for computer hardware equipment; these downsampling inevitably cause loss of image information in the boundary area and blurring of the target boundary; (b) the boundary area of the target is the transition area between the target and the background, and this area is relative to other areas of the target It often has more complex gray distribution characteristics and relatively fuzzy tissue boundaries, which makes the detection and processing of boundary areas usually a challenging task; (c) fundus images are extremely sensitive to conditions such as imaging equipment, imaging angle, and exposure intensity , this imaging characteristic causes fundus images to degrade the segmentation performance of deep learning networks more easily than medical images of other modalities.

Therefore, it is necessary to carry out targeted processing on the boundary area of the target to reduce the detection and extraction error of the boundary. In order to detect the boundary area of the target, the existing deep learning network can be used to initially obtain the possible boundary area. Based on this, the present invention performs a preliminary segmentation of the optic cup region in the fundus image based on the existing U-Net network. Taking the classic U-Net segmentation network as an example, the object of interest is executed according to the medical image to be segmented and the corresponding manual labeling information. The preliminary extraction of the optic cup region is obtained by the rough segmentation results.

The rough segmentation result of the target can be regarded as a highly condensed feature image of the image to be segmented. Although the image has a relatively large segmentation error in the boundary area, it roughly points out the area where the target of interest is located. According to the region to find the potential region of the target boundary. Once the potential boundary area is obtained, the deep learning network can be used to fine-tune the area, thereby effectively improving the overall segmentation result of the image and reducing the segmentation error of the boundary area. Based on this, the present invention respectively expands and erodes the rough segmentation results of the target of interest through the cv2.dilate and cv2.erode morphological operations in OpenCV. The corresponding enlarged and reduced versions of the rough segmentation results can be obtained, and then pixel-based subtraction is performed on the dilated and eroded results to obtain an approximate ring-shaped potential boundary area centered on the rough segmentation boundary. Since the rough segmentation result is a binary image (binary image), that is, the gray value of the pixel in the target area is 1, and the gray value of the pixel in the background area is 0; this leads to the potential area of the target boundary is also a binary image, thus This severely constrains deep learning networks to detect enough feature information from this region. In order to extract enough feature information from potential boundary regions, it needs to be processed as necessary.

Since the boundary corresponding to the rough segmentation result is very close to the manually marked boundary of the target of interest, an area of specified width can be extracted near the corresponding boundary of the rough segmentation result as the area where the target boundary may appear, that is, the potential boundary area. In order to obtain the potential area of the boundary, the present invention uses morphological operations to expand and corrode the rough segmentation results of the target of interest respectively (expansion and erosion operations use the same structural elements to ensure that the expansion and erosion results have a certain degree of accuracy on the coarse segmentation boundary. Symmetry) to obtain the enlarged and reduced version of the rough segmentation results; and then perform a pixel-based subtraction operation on the two processing results to obtain the desired approximate ring-shaped image area. This area gives a rough range where the target boundary may appear, and has a higher probability of occurrence on the center line of the area, and the farther away from the center line of the area, the lower the probability of appearance of the boundary. However, both the latent boundary region and the corresponding image of the coarse segmentation result are binary images (i.e., the grayscale of pixels in the target region is 1, and the grayscale of pixels in the background region is 0), which leads to them having less texture features , is not conducive to the feature detection of the deep learning network, so it is necessary to convert the potential boundary area into a gray-scale image with a specific gray-scale distribution and use it to guide the fine segmentation of the image.

The potential area of the target boundary is approximately ring-shaped. After comparing the area with the rough segmentation result and the corresponding manually labeled image of the target, it can be seen intuitively that: (a) the target boundary corresponding to the rough segmentation is near the center line of the ring; (b ) is farther away from the centerline of the ring, the lower the possibility of belonging to the target boundary, and vice versa. Therefore, it is necessary to design a suitable strategy to convert the binary image corresponding to the potential boundary region into a grayscale image with a unique grayscale distribution. The grayscale image has the largest grayscale value on the centerline of the potential boundary region and the smallest grayscale value at the edge of the region. Based on this, the present invention uses the traditional Gaussian filter to smooth the binary image corresponding to the potential boundary area, so that the binary image is converted into the required grayscale image, and the grayscale image of each pixel The gray value represents the possibility that it belongs to the boundary of the object, so it can be called a boundary uncertainty map (boundary uncertainty map).

Once the boundary uncertainty image corresponding to the target of interest is obtained, fine segmentation of the target area can be performed based on the image. Specific fine segmentation strategies may include the following: (a) directly train the selected depth based on the boundary uncertainty image learning network; (b) integrate the original image to be segmented and its corresponding boundary uncertainty image and use it for the training of the existing deep learning network; (c) use the boundary uncertainty image to first eliminate the original image to be segmented The irrelevant background far away from the potential boundary region; it is then integrated with the boundary uncertainty image for the training of the deep learning network. Based on the above several training strategies, retraining the selected deep learning network can effectively reduce the detection error of the target boundary and achieve fine segmentation of the target area. Compared with the traditional segmentation training strategy, the training strategy based on uncertainty guidance in the present invention has the following characteristics: (a) By performing two different network trainings, the high-quality extraction of the target of interest from coarse to fine can be achieved, effectively reducing the The processing error of the segmentation network on the target boundary; (b) Integrate deep learning technology with traditional Gaussian filtering to achieve rough positioning of the boundary area and exclusion of irrelevant backgrounds, which helps the deep learning network to learn specific image areas; (c) The present invention has better versatility, and can use existing segmentation networks, optimization algorithms, and cost functions to perform segmentation tasks of different types of images.

An application of an image segmentation method based on the above uncertainty-guided deep learning strategy, where the uncertainty-guided deep learning method is applied to fundus cup segmentation.

Referring to Figure 1, it includes the following steps:

Step 1. Evaluate the existing segmentation network (such as U-Net) in image segmentation, which can usually accurately extract the internal region of the target of interest, but it is difficult to effectively process the boundary region of the target, resulting in a relatively large boundary segmentation error. To this end, it is necessary to use the rough segmentation results to preliminarily determine the possible areas of the target boundary, and then use the potential boundary area to guide the segmentation of the target boundary.

Step 2, boundary area extraction based on rough segmentation results

The rough segmentation result of the U-Net network relatively roughly gives the possible area of the target of interest, and there is only a certain difference between the area and the manual labeling of the target in the position of the target boundary. Therefore, the present invention uses morphological operations to process the rough segmentation results and subtracts the processing results to obtain an approximately circular binary image region. This binary image area is where the target boundary is most likely to appear (that is, the potential boundary area image). Based on this, irrelevant image information far away from this area can be excluded to ensure that the segmentation network is more focused on processing the target boundary area.

Step 3, Design Boundary Uncertainty Image

Although the potential boundary area image gives the possible location information of the target boundary, it lacks the necessary texture information, which is not conducive to the detection and extraction of target-related features, because the potential boundary area image is a binary image. In order to improve the extraction of feature information, Gaussian filtering is used to process the image of potential boundary area, and the image of boundary uncertainty can be obtained. This image gives the probability that each pixel belongs to the boundary of the object.

Step 4, target fine segmentation based on uncertainty image

Integrating the boundary uncertainty image and the original fundus image and retraining the U-Net network can achieve better segmentation performance, because the boundary uncertainty image can greatly compress the interference of the irrelevant background, thus improving to a certain extent Segmentation Network's Processing Potential for Object Boundaries.

1. Simulation conditions:

The present invention uses the Keras open source deep learning library and the public REFUGE data set to perform accurate extraction of the optic cup region in the fundus image on the Windows 10 64bit Intel(R) Core(TM) i9-10920X CPU @ 3.50GHz3.50 GHz RAM 32GB platform.

2. Simulation content and results

This simulation experiment uses the existing U-Net network to perform the extraction of the cup area in the fundus image. The experimental results are shown in Figure 4:

Figure 4 uses the U-Net network to extract objects of interest from coarse to fine under the deep learning strategy to be designed; the rough segmentation, fine segmentation, and the corresponding cup boundary curves corresponding to manual annotation are red and blue respectively and white; According to these boundary information, it can be found that U-Net can achieve more accurate cup segmentation under the proposed learning strategy.

The present invention also provides a storage medium, and instructions are stored in the storage medium, and when a computer reads the instructions, the computer is made to execute the above image segmentation method based on the uncertainty-guided deep learning strategy.

The embodiments should not be regarded as limiting the present invention, but any improvement based on the spirit of the present invention should be within the protection scope of the present invention.

Claims (7)

1. An image segmentation method based on an uncertainty-guided deep learning strategy is characterized by comprising the following steps of: it includes:

1) Roughly segmenting the image based on an image segmentation network, and performing preliminary extraction of an interest target according to the medical image to be segmented and corresponding manual annotation information;

2) Carrying out boundary region extraction on the result of the rough segmentation;

3) Smoothing the binary image corresponding to the potential boundary region by Gaussian filtering to convert the binary image into a required boundary uncertainty image,

4) Integrating the boundary uncertain image and the original image to be segmented based on the boundary uncertain image obtained in the step 3), and then inputting the integrated result and manual annotation information corresponding to the original image into a current image segmentation network to train a deep learning model.

2. The image segmentation method based on the uncertainty-guided deep learning strategy of claim 1, characterized in that: the segmentation network is a U-Net segmentation network.

3. The image segmentation method based on the uncertainty-guided deep learning strategy of claim 1, characterized in that: and 2) respectively performing expansion and corrosion processing on the rough segmentation result of the target of interest through morphological operation, setting structural elements of the rough segmentation result to be 25-by-25 circular areas in the expansion and corrosion operation to obtain expanded and reduced versions corresponding to the rough segmentation result, and then performing pixel-based subtraction processing on the expansion and corrosion result to obtain an approximately annular potential boundary area taking the rough segmentation boundary as the center.

4. The image segmentation method based on the uncertainty-guided deep learning strategy of claim 1, characterized in that: the segmentation strategy adopted in step 4) is as follows:

a) Training an existing deep learning network directly based on a boundary uncertainty image;

b) Integrating the original image and the boundary uncertainty image and training the existing deep learning network by using the original image and the boundary uncertainty image;

c) With the help of the image with uncertain boundary, firstly eliminating a background area in the original image; and then integrating the image with the boundary uncertainty image for training the deep learning network.

5. An application of the image segmentation method based on the uncertainty-based guided deep learning strategy of any one of the claims 1 to 4 is characterized in that: the uncertainty-guided deep learning method is applied to fundus cup segmentation.

6. Use according to claim 4, characterized in that: which comprises the following steps:

1) Performing rough segmentation of an optic cup region in the fundus image by adopting a U-Net segmentation network, and performing preliminary extraction of an interest target according to a medical image to be segmented and corresponding manual labeling information;

2) The rough segmentation result in the step 1) roughly indicates the possible area of the interest target, and the rough segmentation result of the interest target is respectively subjected to expansion and corrosion treatment by adopting morphological operation, so that expanded and contracted segmentation results can be obtained; then, carrying out pixel-based subtraction processing on the two to obtain an approximate annular potential boundary area;

3) Smoothing the binary image corresponding to the potential boundary region by means of traditional Gaussian filtering to convert the binary image into a required boundary uncertainty image;

4) Integrating the boundary uncertain image and the original image to be segmented based on the boundary uncertain image obtained in the step 3), and then inputting the integrated result and manual annotation information corresponding to the original image into a current image segmentation network to train a deep learning model.

7. A storage medium, wherein instructions are stored in the storage medium, and when the instructions are read by a computer, the instructions cause the computer to execute the image segmentation method based on the uncertainty-guided deep learning strategy according to any one of claims 1 to 4.

Images