CN112070726B - Deep learning-based grape embryo slice image processing method and device - Google Patents

Abstract

The invention discloses a grape embryo image processing method and device based on deep learning, belongs to the field of medical image detection of grape embryo in the technical field of medical images, and is used for solving the problem of low efficiency of clinical diagnosis detection of grape embryo in the prior art. The invention acquires a scanned grape embryo slice map under a microscope, inputs the scanned grape embryo slice map into a villus network and an oedema network to obtain a slice villus tag map and a slice oedema tag map of the scanned grape embryo slice map, and finally obtains a slice oedema distribution map. According to the invention, through a villus network and an oedema network, two different pathological features of the villus and oedema can be subjected to image processing to obtain a section oedema distribution map, and the distribution map is visually displayed to a clinician so as to intuitively obtain the section oedema regional distribution situation.

Description

A method and device for processing hydatidiform mole slice images based on deep learning

Technical Field

The present invention belongs to the field of medical imaging technology, and relates to a medical imaging detection of hydatidiform mole, and in particular to a method for processing hydatidiform mole slice images using a deep learning method.

Background Art

Hydatidiform mole (HM) refers to a grape-shaped blister-like fetal mass formed by the placenta after pregnancy. Babies with hydatidiform mole often die or develop malformations, and very few full-term babies are born. Under normal circumstances, 10% to 20% of hydatidiform moles will develop into malignant hydatidiform mole and choriocarcinoma. These cancers can metastasize through blood groups and pose a life-threatening threat to the patient if not treated in time. Therefore, early pathological diagnosis of hydatidiform mole is of great significance to every pregnant woman with the disease.

In the prior art, there are two main ways to detect and screen hydatidiform mole. The first is to manually observe the sections under a microscope, and the second is to detect genes related to hydatidiform mole.

In the first method, pathologists generally use 5*10x and 10*10x microscopes to observe multiple sections of the patient, and then make a comprehensive diagnosis based on experience and the morphology of the tissue cells in the sections. The diagnosis of hydatidiform mole is mainly made by observing the characteristics of the villi in the sections. The pathological characteristics of the sections are mainly villous trophoblastic hyperplasia and interstitial edema inside the villi.

Pathologists in gynecological hospitals need to spend a lot of time every day diagnosing diseases such as hydatidiform mole, which have a lower risk factor than tumors. Most of these patients are not sick, but this takes up a lot of the pathologists' working time.

However, there are currently about 15,000 pathologists in China, resulting in a large talent gap and low detection efficiency. In addition, since the diagnosis of clinical hydatidiform mole is mainly performed by doctors through manual screening of sections, it is difficult to guarantee accuracy, especially for hydatidiform mole before 12 weeks. Since the hydatidiform mole has not reached maturity, the lesion is not fully developed, and the tissue morphology is similar to that of normal hydatidiform mole sections, making it difficult to distinguish. This results in an extremely low clinical diagnostic accuracy of less than 50%.

In the second method, the invention patent with application number 201310027715.1 and titled Gene chip, detection reagent and kit for detecting NLRP7 gene discloses that by detecting NLRP7 gene SNPs related to hydatidiform mole, it is of great significance to achieve clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention, and can be widely used for clinical efficient screening of high-risk populations for hydatidiform mole. This invention patent constructs a gene chip detection system for screening high-risk populations for NLRP7 gene polymorphisms related to hydatidiform mole. The gene chip includes a solid phase carrier and an oligonucleotide probe synthesized on the carrier. The detection reagent includes a gene chip and 18 pairs of PCR primers for amplifying each SNPs in the sample. The kit includes a detection reagent, a negative control sample and a positive control sample. This invention patent can quickly and accurately detect each related SNPs site of NLRP7 gene in clinical samples, which is of great significance for clinical diagnosis of hydatidiform mole and early screening of high-risk populations and early preventive intervention.

Although there is a necessity for screening hydatidiform mole by testing genes, screening hydatidiform mole by testing the NLRP7 gene not only increases the testing steps of the test kit, but also involves the production of chips, reagents and test kits, which greatly increases the screening cost. Its application scope in the clinical diagnosis of hydatidiform mole is very limited and is not easy to promote and apply.

Based on the above-mentioned current situation of small number of pathologists, low efficiency and low accuracy of manual screening of slices by physicians, high cost and long cycle of genetic testing screening, it is necessary to develop a complete set of methods and devices from automatically acquiring images with a microscope to generating distribution maps of pathological characteristics such as edema and hyperplasia, so as to assist clinicians in screening cases more efficiently.

Clinical pathologists mainly judge whether it is a hydatidiform mole based on pathological features such as villous stromal edema, diffuse hyperplasia of trophoblastic cells at the villous margins, and information such as the patient's menopausal duration and pregnancy history. Among them, the pathological feature of villous stromal edema is the more critical basis for diagnosis.

Summary of the invention

The purpose of the present invention is to provide a method and device for processing hydatidiform mole slice images based on deep learning, so as to solve the problem of low efficiency of clinical diagnosis and detection of hydatidiform mole caused by manual observation of pathological characteristics of sliced tissue in the prior art by obtaining the distribution of villi and edema of hydatidiform mole.

The technical solution adopted by the present invention is as follows:

A method for processing hydatidiform mole slice images based on deep learning, the steps are as follows:

S1, placing the HE-stained hydatidiform mole slice on the microscope stage, focusing the microscope, and obtaining a scan of the hydatidiform mole slice under the microscope;

S2, cutting the scanned image of the hydatidiform mole slice into blocks to obtain scanned image block 1, inputting the scanned image block 1 into the villus network a-net to obtain a block villus label map of the hydatidiform mole slice, and fusing all the block villus label maps to obtain a slice villus label map;

Obtain the second scanned image slice, input the second scanned image slice into the edema network b-net, obtain the sliced edema label map of the hydatidiform mole slice, and fuse all the sliced edema label maps to obtain the sliced edema label map;

S3, obtaining the slice edema distribution map according to the slice edema label map;

Optionally, in step S2, the villus network a-net and the edema network b-net are both image segmentation networks UNet networks, and the networks include convolutional layers, pooling layers, upper convolutional layers, and union layers. The first block of the scanned image is passed through the villus network a-net to obtain a block villus label map, and the second block of the scanned image is passed through the edema network b-net to obtain a block edema label map.

Optionally, in step S2, the hydatidiform mole slice image is cut into blocks of size size at intervals s, and two adjacent blocks in the upper, lower, left and right directions overlap in area of size×(size-s), wherein the size size1 of the scanned image block one is a pixel size of 3000×3000 to 18000×18000, and the size size2 of the scanned image block two is a pixel size of 1500×1500 to 9000×9000.

Optionally, in step S2, the fusion of the cut edema label map to obtain the slice edema label map is to splice all the cut edema label maps according to the corresponding positions of the original cuts in the slice, and the overlapping parts of the upper, lower, left and right adjacent cuts are fitted according to the pixel mean of the overlapping parts of multiple cut images.

Optionally, in step S3, the villus region is obtained according to the slice villus label map, and the edema region outside the villus region in the slice edema label map is removed to obtain a slice edema distribution map;

The villous area and non-villous area refer to the areas with pixel values of 0 and 255 in the slice villous label map, and the edema area and non-edema area refer to the areas with pixel values of 0 and 255 in the slice edema label map. The pixel value of the edema area outside the corresponding villous area in the slice edema label map is changed to 255 to obtain the slice edema distribution map.

Optionally, in step S3, the obtained distribution map can be visualized and output. According to the hydatidiform mole slice image and the slice edema distribution map, an edema visualization slice image, a hyperplasia visualization slice image, and an edema area screenshot are obtained, and the edema visualization slice image, the edema area screenshot, and the hydatidiform mole slice classification result obtained in S5 are displayed and output;

The edema visualization slice image is obtained by marking the edema area in the hydatidiform mole slice image with a first color;

The edema region screenshot is obtained by cutting the slice edema distribution map into blocks according to size, and the block images corresponding to 1 to 10 blocks with the smallest block pixel mean are the edema region screenshots.

A hydatidiform mole slice image processing device based on deep learning, comprising:

A microscope to magnify the tiny structures in a section of the mole;

A slice scanning module, used for obtaining a scanned image of a hydatidiform mole slice under a microscope;

A sliced villus label map generation module is used to slice the hydatidiform mole slice scan image into blocks to obtain a scanned image block 1, input the scanned image block 1 into a villus network a-net to obtain a sliced villus label map of the hydatidiform mole slice, and fuse all the sliced villus label maps to obtain a sliced villus label map;

A slice edema label map generation module is used to slice the hydatidiform mole slice scan image into blocks to obtain a second scan image slice, input the second scan image slice into an edema network b-net to obtain a slice edema label map of the hydatidiform mole slice, and fuse all the slice edema label maps to obtain a slice edema label map;

A distribution map generating module, used for obtaining a slice edema distribution map according to a slice villus label map and a slice edema label map;

As described above, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

The present invention combines deep learning technology with hydatidiform mole slice pathology recognition technology, and inputs the hydatidiform mole slice scan image into the villus network a-net and the edema network b-net, so as to obtain the slice edema distribution map of the hydatidiform mole slice scan image. The slice edema distribution map can be directly output to the clinician to provide assistance to the clinician. The doctor can intuitively obtain the edema distribution situation according to the slice edema distribution map, thereby reducing the excessive reliance on the subjective analysis of the pathologist during the detection process.

In addition, deep training was performed on the villus network a-net and the edema network b-net. Through specific training methods, a more mature network was obtained to improve the accuracy of the slice edema distribution map.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a villus annotation example of a hydatidiform mole slice scan required by the present invention;

FIG2 is an example of edema annotation of a hydatidiform mole slice scan required by the present invention;

FIG3 is a hydatidiform mole slice scan image input by the edema network disclosed in the present invention, an actual annotated slice edema label image, and a network-predicted slice edema label image;

FIG4 is a flowchart of obtaining a training data set for annotating a hydatidiform mole slice scan image disclosed in the present invention;

FIG5 is an overall flow chart of hydatidiform mole slice image processing disclosed in the present invention;

Figure 6 is a diagram of the network structure of the villus network a-net, and the edema network b-net has the same structure as the villus network a-net;

Fig. 7 is a digital microscope slice scanning device;

DETAILED DESCRIPTION

All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and/or steps, can be combined in any manner.

Embodiment 1

A method for processing hydatidiform mole slice images based on deep learning, the steps are as follows:

S1, placing the hydatidiform mole slice on a microscope stage, focusing the microscope, and obtaining a scan image of the hydatidiform mole slice under the microscope;

In this step, first, the hydatidiform mole slice needs to be treated with HE staining, and then the hydatidiform mole slice after HE staining is placed on the stage of a digital microscope, and a digital microscope is selected as the microscope; secondly, the autofocus module performs autofocusing so that a clear hydatidiform mole slice can be seen in the field of view of the microscope; finally, the slice scanning module obtains a scanned image of the hydatidiform mole slice under the microscope.

S2, since the size of the obtained hydatidiform mole slice scan image is large, and the existing network input has requirements on the image size, it is necessary to segment the image input to the network, and segment the hydatidiform mole slice scan image into multiple groups of blocks suitable for different networks.

Cut the scanned image of the hydatidiform mole slice into blocks to obtain scanned image block 1, input the scanned image block 1 into the villus network a-net to obtain a block villus label map of the hydatidiform mole slice, and fuse all the block villus label maps to obtain a slice villus label map;

The hydatidiform mole slice scan image is cut into blocks to obtain the scan image cut block 2, and the scan image cut block 2 is input into the edema network b-net to obtain the cut edema label map of the hydatidiform mole slice, and all the cut edema label maps are fused to obtain the slice edema label map.

The slicing of hydatidiform mole slice image is to cut the slice into several slices of size with an interval of s. The two adjacent slices above, below, left and right have an area overlap of size×(size-s). The size of the scanned image slice one, size1, is 3000×3000 to 18000×18000 pixels, and the size of the scanned image slice two, size2, is 1500×1500 to 9000×9000 pixels.

The slice label map is obtained by fusing the slice label map by splicing all the slice label maps according to the corresponding positions of the original slices in the slice, and the overlapping parts of the upper, lower, left and right adjacent slices are fitted according to the pixel mean of the overlapping parts of multiple slice images.

The above-mentioned villus network a-net and edema network b-net have the same network structure, both of which include a convolutional layer, a pooling layer, an upper convolutional layer, and a union layer.

The first part of the network is the upper part, which is similar to the traditional convolutional network structure. It consists of a convolution layer and a pooling layer. Each time it passes through a pooling layer, there is a scale. There are a total of 5 scales including the original image scale, which is mainly used for image feature extraction. The second part is the lower part in Figure 6, which is the upsampling part. Each time it is upsampled, it is merged with the scale of the same number of channels corresponding to the feature extraction part. However, it must be trimmed to the same size before merging, which makes full use of the multi-scale image feature information and has a better recognition effect on multi-scale image features.

S3, obtaining the slice edema distribution map according to the slice edema label map;

There are many ways to generate a slice edema distribution map. The edema distribution can be directly marked according to the edema area in the slice edema label map to obtain the slice edema distribution map.

Since edema often occurs together with villi, in order to improve the detection accuracy, this embodiment also provides another generation method:

According to the slice villus label map, the villus area of the slice can be obtained. The villus area and non-villus area are respectively the areas with pixel values of 0 and 255 in the slice villus label map, and the edema area and non-edema area are respectively the areas with pixel values of 0 and 255 in the slice edema label map. The pixel value of the edema area outside the corresponding villus area in the slice edema label map is changed to 255 to obtain the slice edema distribution map, that is, the edema area outside the corresponding villus area in the slice edema label map is eliminated.

The villus network a-net and the edema network b-net mentioned above need to be trained. This embodiment also provides a network training method.

To train the network, you need training images to feed into the network and train it.

This embodiment combines the morphological characteristics of hydatidiform mole required for the actual diagnosis of hydatidiform mole, and through nearly one year of hydatidiform mole slice annotation training, hydatidiform mole slice annotation and annotation review, obtains the annotation results of 157 typical hydatidiform mole slice scan images. Each slice needs two kinds of annotations. The villous annotation is to circle the villous area of the slice, and the edema annotation is to circle the edema part in the villous area of the slice. Specific annotation samples can be seen in Figures 1 and 2.

The current bottleneck of medical imaging is the extreme lack of high-quality annotated data sets, and there is no good annotated data set for hydatidiform mole worldwide. The recognition rate of deep networks is based on a good data set, so first of all, standardized, reasonable, and strict annotation is required to obtain a good data set. The training data set acquisition process is shown in Figure 4, and the process includes:

Step 101: Develop a reliable annotation scheme

After several proposed annotation schemes were reviewed and confirmed by multiple professional clinicians, a final annotation scheme was determined, which is the aforementioned from villus annotation to edema annotation. Specific annotation samples can be seen in Figures 1 and 2. Villus annotation is the first step of screening and preliminary extraction, and edema annotation is to extract the edema area, that is, to annotate the villi with obvious edema and pool-like stroma.

Step 102: Training of labelers

Several annotators with relevant medical knowledge undergo several weeks of training, and the most suitable annotators are selected for subsequent annotation.

Step 103: Initial labeling by labelers

According to the labeling regulations, each labeler is responsible for about 80 scan slices and annotates the edema and villus for network training.

Step 104: Pathologist Review

The preliminary annotations obtained in step 103 are strictly reviewed by clinicians with long-term clinical experience and fed back to the annotators.

Step 105: Detailed annotation by annotators

The annotation review results are analyzed and modified, and the annotators unify the annotation standards to obtain the final annotation data set.

Step 106: Automatic network annotation

The network trained on the basis of a better annotated data set performs image semantic segmentation on the newly added slice scan images to obtain a new annotated data set. That is, the network automatically annotates the new slice scan images to expand the data set, and then trains a network with better robustness.

The training method of the fluff network a-net is:

The training image is cut into several training image blocks with size size1, and the training image blocks are input into the fluff network a-net. The actual segmentation output corresponding to each training image block is obtained by processing the fluff labeling file, and finally the fluff classification network a-net is trained;

The training method of the edema network b-net is:

The training image is cut into several training image blocks with a size of size2, and the training image blocks with villus areas are input into the edema network b-net. The edema segmentation blocks obtained from the edema annotation files of the training image blocks are output, and the slice edema distribution map is obtained. Finally, the edema network b-net is trained;

Embodiment 2

This embodiment also provides a hydatidiform mole slice image processing device based on deep learning, comprising:

A microscope to magnify the tiny structures in a section of the mole;

A slice scanning module, used for obtaining a scanned image of a hydatidiform mole slice under a microscope;

A slice edema label map generation module is used to slice the hydatidiform mole slice scan image into blocks to obtain a second scan image slice, input the second scan image slice into an edema network b-net to obtain a slice edema label map of the hydatidiform mole slice, and fuse all the slice edema label maps to obtain a slice edema label map;

A distribution map generating module, used for obtaining a slice edema distribution map according to a slice edema label map;

Additionally, it may include:

The sliced villus label map generation module is used to cut the hydatidiform mole slice scan image into blocks to obtain scan image block 1, input the scan image block 1 into the villus network a-net to obtain the sliced villus label map of the hydatidiform mole slice, and fuse all the sliced villus label maps to obtain the sliced villus label map.

Claims (10)

1. The grape embryo slice image processing method based on deep learning is characterized by comprising the following steps:

S1, cutting a grape embryo slice image into blocks to obtain a block I, inputting the block I into an image segmentation network 1 to obtain a block fluff label graph of a grape embryo slice, and fusing all the block fluff label graphs to obtain a slice fluff label graph;

s2, cutting the grape embryo slice image into blocks to obtain blocks II, inputting the blocks II into the image segmentation network 2 to obtain a block edema label graph of the grape embryo slice, and fusing all the block edema label graphs to obtain a block edema label graph;

s3, obtaining a villus region according to the slice villus tag map, removing an edema region except the villus region in the slice edema tag map, and finally obtaining a slice edema distribution map;

In the step S1, the cut piece fluff label images are fused to obtain a cut piece fluff label image, namely all the cut piece fluff label images are spliced according to the corresponding positions of the original cut pieces in the cut pieces, and overlapping parts of the adjacent cut pieces at the upper part, the lower part, the left part and the right part are fitted according to the pixel average value of the overlapping parts of a plurality of cut piece images;

in the step S2, the cut edema label images are fused to obtain a cut edema label image, namely all the cut edema label images are spliced according to the corresponding positions of the original cut pieces in the cut pieces, and the overlapping parts of the adjacent cut pieces at the upper part, the lower part, the left part and the right part are fitted according to the pixel average value of the overlapping parts of the plurality of cut piece images;

In step S3, the villus region and the non-villus region are regions with pixel values of 0 and 255 in the slice villus tag image, the edema region and the non-edema region are regions with pixel values of 0 and 255 in the slice edema tag image, and the pixel values of the edema region outside the corresponding villus region in the slice edema tag image are changed to 255, so as to obtain the slice edema distribution map.

2. A method for processing a grape embryo slice image based on deep learning as claimed in claim 1, wherein the grape embryo slice image is obtained by scanning with a scanner.

3. The method for processing the grape embryo slice image based on the deep learning as claimed in claim 1, wherein the image segmentation network 1 is a network for segmenting the villus region obtained through the data set training of the manually marked grape embryo villus cut-out label image, and the image segmentation network 2 is a network for segmenting the edema region obtained through the data set training of the manually marked grape embryo edema cut-out label image.

4. The method for processing the grape embryo slice image based on the deep learning as set forth in claim 1, comprising the steps of:

In the step S1, the grape embryo slice image dicing is to divide the slice into a plurality of dicing blocks with the size of size1 at an interval S1, wherein the upper, lower, left and right adjacent dicing blocks have the area of sizel × (sizel-S1) and overlap, and the size1 is the pixel size of 3000×3000-18000×18000;

In step S2, the grape embryo slice image dicing is to divide the slice into a plurality of dicing blocks with size of size2 at interval S2, and two adjacent dicing blocks with size2× (size 2-S2) overlap each other in the upper, lower, left and right directions, and size2 is a pixel size of 1500×1500 to 9000×9000.

5. The method for processing a grape embryo slice image based on deep learning as set forth in claim 1, wherein in steps S1 and S2, the image segmentation network 1 and the image segmentation network 2 are both convolutional network-based image segmentation networks.

6. Grape child section image processing device based on degree of depth study, characterized by, include:

the slice fluff label map generation module is used for cutting the grape embryo slice image into blocks to obtain a block I, inputting the block I into the image segmentation network 1 to obtain a block fluff label map of the grape embryo slice, and fusing all the block fluff label maps to obtain a slice fluff label map;

the slice edema label map generation module is used for slicing the grape embryo slice image to obtain a second slice, inputting the second slice into the image segmentation network 2 to obtain a slice edema label map of the grape embryo slice, and fusing all the slice edema label maps to obtain a slice edema label map;

The edema distribution map generation module is used for obtaining a slice edema distribution map according to the slice edema label map and the slice villus label map;

In the slice fluff label map generation module, the cut fluff label maps are fused to obtain a slice fluff label map, namely all the cut fluff label maps are spliced according to the corresponding positions of the original cut blocks in the slices, and overlapping parts of the adjacent cut blocks at the upper part, the lower part, the left part and the right part are fitted according to the pixel average value of overlapping parts of a plurality of cut block images;

In the slice edema label image generation module, the slice edema label images are fused to obtain a slice edema label image, namely all the slice edema label images are spliced according to the corresponding positions of the original slices in the slices, and overlapping parts of the adjacent slices are fitted according to the pixel average value of overlapping parts of a plurality of slice images;

In the edema distribution map generation module, the villus region and the non-villus region are respectively the regions with the pixel values of 0 and 255 in the slice villus tag image, the edema region and the non-edema region are respectively the regions with the pixel values of 0 and 255 in the slice edema tag image, and the pixel values of the edema region except the corresponding villus region in the slice edema tag image are changed into 255, so that the slice edema distribution map is obtained.

7. A deep learning based grape embryo slice image processing device as claimed in claim 6, wherein the grape embryo slice image is obtained by scanning with a scanner.

8. The deep learning-based grape embryo slice image processing apparatus as set forth in claim 6, wherein the image segmentation network 1 is a network for villus region segmentation obtained through training of a data set of manually labeled grape embryo villus cut-out label images, and the image segmentation network 2 is a network for edema region segmentation obtained through training of a data set of manually labeled grape embryo edema cut-out label images.

9. A deep learning-based grape embryo slice image processing device as defined in claim 6, comprising:

In the section fluff label graph generating module, a grape embryo section image is cut into a plurality of sections with size of size1 at interval s1, wherein two sections which are adjacent to each other at upper, lower, left and right are overlapped by sizel × (sizel-s 1), and size1 is 3000×3000-18000×18000 pixel size;

in the section edema label image generation module, the grape embryo section image dicing is to divide the section into a plurality of dicing blocks with size of size2 at interval s2, wherein two adjacent dicing blocks with size2× (size 2-s 2) on the upper, lower, left and right are overlapped, and size2 is 1500×1500-9000×9000 pixels.

10. The deep learning-based grape embryo slice image processing apparatus as set forth in claim 6, wherein the slice fluff label map generation module and the slice edema label map generation module are both image segmentation networks 1 and 2 based on convolution networks.