CN108852268A - A kind of digestive endoscopy image abnormal characteristic real-time mark system and method - Google Patents

Abstract

The invention discloses a real-time marking system and method for abnormal features of digestive endoscope images. The system includes an image acquisition module, an image preprocessing module, a model training module, an abnormality detection module and a marker display module; the model training module includes an image data set, The classification model training unit and the detection model training unit use the deep learning CNN classification model to obtain the classification information of suspicious gastric precancerous diseases, and use the deep learning CNN target detection model based on the regression method to quickly and accurately obtain the location of the lesion. Utilizing the present invention helps to effectively classify and detect the abnormal characteristics of the stomach under digestive endoscope, can reduce the missed diagnosis rate based on long-term and subjective diagnosis of doctors, and supports doctors to perform real-time analysis and real-time diagnosis during endoscopic examination. Display suspicious lesions under the endoscope, reduce the workload of doctors, and improve the efficiency of medical diagnosis.

Description

A system and method for real-time marking of abnormal features of digestive endoscopy images

technical field

The invention belongs to the field of medical data mining, in particular to a real-time marking system and method for abnormal features of digestive endoscope images.

Background technique

There are 405,000 new gastric cancer cases and 325,000 deaths each year in my country, accounting for 42.6% and 45.0% of the global total respectively. Reducing the incidence and mortality of gastric cancer in my country is an urgent public health problem. Clinical studies have shown that the prognosis of gastric cancer is closely related to the treatment effect. For patients with advanced gastric cancer (advanced gastric cancer, AGC), even if they receive surgery-based gastric cancer resection, the postoperative five-year survival rate of the patient is still lower than 30%, and the postoperative quality of life of the patient is low. A huge burden on the family and society. If patients receive endoscopic examination and treatment in time in the early stage of gastric cancer, the five-year survival rate is as high as 90%, and radical treatment can even be performed for early gastric cancer (EGC) under endoscopy. Therefore, early detection, early diagnosis, and early treatment of EGC are of great significance for reducing the incidence and mortality of gastric cancer and saving medical resources.

Precancerous diseases of the stomach refer to benign diseases of the stomach and are the main risk factors for gastric cancer, including chronic atrophic gastritis, gastric polyps, gastric ulcers, remnant stomach, and verrucous gastritis. As the main gastric precancerous lesion, atrophic gastritis has a cancer transformation rate of 8.6-13.8%, and it is 1.2-7.1% in my country. Existing research results have shown that regular monitoring of gastric precancerous lesions can make the detection rate of early gastric cancer more than 50%. Gastric ulcers have a 1-2% cancer rate. Therefore, convenient, cost-effective and effective monitoring should be carried out early for patients with precancerous diseases at risk of gastric cancer, so as to intervene and reduce the occurrence of gastric cancer.

At present, the monitoring of gastric precancerous lesions mainly uses ordinary optical endoscopic biopsy techniques. In endoscopic examination, the problem of narrow endoscopic field of view often brings inconvenience to doctors: for example, due to the limitation of field of view, doctors must repeatedly move the lens on the inner wall surface of the target organ to ensure that all lesions are found to avoid bringing Come to missed diagnosis. Therefore, it is very necessary to develop a reliable and fast assisting doctor to perform endoscopic examination, especially in the face of large amount of data, based on the real-time marking system of abnormal features of digestive endoscopic images, which can be used to assist doctors in early cancer screening.

Deep learning is a method based on representation learning of data in machine learning. Convolutional Neural Network (CNN) is a deep learning feed-forward artificial neural network, which is widely used in image classification, segmentation and target detection. . CNN is widely used in image feature extraction and classification detection, especially CNN for object detection based on candidate regions. Its recognition accuracy has been continuously improved, but due to the requirement of millions of model training parameters, the time performance is low, and it is difficult to meet the requirements of The demand for real-time detection of gastroscope images in the gastroscope video scene.

Contents of the invention

The invention provides a real-time marking system and method for abnormal features of digestive endoscope images. Effective classification and detection of gastric precancerous diseases under digestive endoscopy can reduce the missed diagnosis rate based on doctors' long-term and subjective diagnosis, and support doctors to analyze and display suspicious endoscopic diseases in real time during endoscopic examination. focus, reduce the workload of doctors, and improve the efficiency of medical diagnosis.

A real-time marking system for abnormal features of digestive endoscope images, comprising a computer system, the computer system comprising:

The image acquisition module acquires the video stream of the conventional white light endoscope of the stomach input in real time through the endoscope image system, eliminates invalid frames, and screens effective key frames;

An image preprocessing module, which is used to perform image enhancement processing on the filtered key frames;

The model training module is used to perform model training on endoscopic images and lesion location information of early gastric precancerous diseases to obtain a detection model;

The abnormal detection module is used to input the pre-processed image sequence into the trained detection model to obtain the classification information and focus location information of gastric precancerous diseases;

The marker display module marks the positioning information on the picture sequence, and maps the markers to the original input video stream of the conventional white light endoscope of the stomach, and displays the video stream of the gastric endoscope in real time.

Abnormal features of digestive endoscopy images mainly include: gastric polyps, gastric ulcers, gastric erosions, and gastric atrophy.

The described image acquisition module uses the K-means algorithm to randomly select k objects from the data objects as initial clustering centers when screening effective key frames, and classifies the data objects according to the minimum distance criterion. Through the clustering algorithm, the image The images in the library are divided into k categories.

When doing gastroscopy, the doctor will perform various operations in the patient's stomach, such as inflating, deflating, flushing with water, biopsy calibration, etc. These operations will affect the image quality of the gastroscope to varying degrees, and it will be effective to remove or ignore these images Reducing the amount of calculation and using the K-means algorithm to filter key frames can greatly speed up the efficiency of key frame extraction.

The image enhancement processing in the image preprocessing module includes: image normalization, invalid pixel clipping, image smoothing, image sharpening and image scaling.

Described model training module comprises:

The image data set is used to store the classification data set and detection data set based on digestive endoscopy; the classification data set contains endoscopic images of various early gastric precancerous diseases, and the detection data set contains lesion location information;

Classification model training unit, for carrying out the training of deep learning CNN classification model to described classification data set;

The detection model training unit is used to use the CNN classification model as a pre-training model, and use the detection data set as a training set to perform the training of the deep learning CNN target detection model based on the regression method.

Before the image data set is used to train the CNN model, data amplification is performed through translation transformation, mirror flip or random cropping. This step can further enhance the generalization ability of CNN and improve the training effect of the model

When the classification model training unit is carrying out deep learning CNN classification model training, the method of reinforcement learning is introduced to improve the accuracy of network classification, the layer-by-layer scale normalization layer is used to regularize the network, and the dropout method is adopted to alleviate the problem of over-fitting Problem, the ReLU activation function is used to avoid the gradient disappearance problem, and the training efficiency of the classification model is improved through transfer learning.

The detection model obtained after the training of the model training module includes a classification model, an auxiliary network structure and a target detection model; the classification model is based on the VGG-16 framework and is used to obtain classification information; the auxiliary network structure is used to extract picture features; The target detection model is used to obtain lesion location information.

The present invention also provides a method for real-time marking of abnormal features of digestive endoscope images, comprising the following steps:

(1) Input the endoscopic image and lesion location information of early gastric precancerous diseases into the model training module for training to obtain a detection model;

(2) Obtain the regular white-light endoscopic video stream of the subject's stomach by using the endoscopic image system equipment;

(3) Carry out key frame extraction after analyzing the video stream of the gastroscope, remove invalid frames, and screen to obtain effective video image sequences of the gastroscope;

(4) input effective video image sequence to image preprocessing module, carry out image preprocessing;

(5) Input the image sequence after image preprocessing into the detection model, and the detection model outputs the classification result and classification confidence of suspicious gastric precancerous diseases, and simultaneously outputs the lesion location coordinates and location reliability;

(6) Mark the effective image sequence according to the lesion location coordinates, and map the markers to the original gastroscope video stream at the same time;

(7) The marked gastroscope video stream is displayed on the monitor in real time for the doctor to observe and confirm the diagnosis.

The beneficial effects of the present invention are:

1. Provides a computer-aided classification and detection technology for abnormal features in digestive endoscopy, which can be applied to effectively detect suspicious lesions for gastric precancerous diseases, find the type of lesions, and monitor the specific location of lesions. Accurate positioning helps to find tiny lesions and avoid missing lesions.

2. The method of the present invention is based on a deep learning convolutional neural network, which has a certain degree of invariance to the geometric transformation, deformation, and illumination of the image to be detected, and has a good effect of feature classification, and can scan the entire image to be detected with a relatively small calculation cost. Detection images, for digestive endoscopy big data image processing, this method can efficiently complete end-to-end classification and detection of clinical endoscopy image data.

3. The method of the present invention provides a means for real-time auxiliary clinical detection. Even if the traditional deep learning model of CNN target detection based on candidate regions has a high accuracy rate, it is difficult to achieve the calculation speed of real-time detection, and often needs to be sacrificed. The detection time is shortened at the expense of detection accuracy. The method of the present invention reduces the amount of calculation under the premise of ensuring the detection accuracy. Effective frames are extracted from the endoscopic video stream, invalid frames are eliminated, and the image sequence is maximized to highlight the features through image preprocessing. information, and adopt a deep learning target detection method based on regression, so that it can adapt to the needs of real-time processing and real-time display. Clinicians can observe the results of classification and detection in real time during the process of digestive endoscopy without waiting for a long time , can effectively speed up the diagnosis process, and realize early detection and early treatment of lesions.

4. This technology is based on the image sequence of digestive endoscopy with big image data, which can help relieve doctors' high-intensity and long-term film reading work, avoid doctors' subjective judgment errors caused by work intensity and working hours, reduce doctors' workload, and improve medical treatment. Efficiency of diagnostic work.

Description of drawings

Fig. 1 is a structural block diagram of a system for real-time marking of abnormal features of digestive endoscope images according to the present invention;

Fig. 2 is a flow chart of the method for marking abnormal features of digestive endoscope images in real time according to the present invention;

Fig. 3 is an effective frame image of the stomach under a digestive endoscope according to an embodiment of the present invention;

Fig. 4 is an invalid frame image of the stomach under a digestive endoscope according to an embodiment of the present invention;

Fig. 5 is the original image of gastric polyps in gastric precancerous diseases under digestive endoscopy according to an embodiment of the present invention;

6 is an image of gastric polyps in gastric precancerous diseases under digestive endoscopy after image enhancement processing according to an embodiment of the present invention;

Fig. 7 is a localization map of gastric polyp lesions in gastric precancerous diseases under digestive endoscopy according to an embodiment of the present invention.

Detailed ways

The technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

As shown in Figure 1, a real-time marking system for abnormal features of digestive endoscopy images, the system runs on a computer system, including:

The image acquisition module acquires the video stream of the conventional white light endoscope of the stomach input in real time through the endoscope image system, eliminates invalid frames, and screens effective key frames. The K-means clustering method is used to screen key frames. Through this clustering algorithm, the images in the image database can be divided into k categories, and the prior classification processing can be completed to speed up the efficiency of key frame extraction.

The image preprocessing module is used to perform image preprocessing on the image sequence to achieve image enhancement. The image sequence comes from the image acquisition module. The preprocessing process includes: image normalization, invalid pixel cropping, image smoothing, image sharpening, and image scaling.

The image data set is used to store image data sets based on digestive endoscopy. The data sets include classification data sets and detection data sets. The classification data sets contain endoscopic images of various early gastric precancerous diseases, and the detection data sets contain lesion locations. Information, when the image data set is used to train the CNN model, data amplification is performed through but not limited to translation transformation, mirror flip, random cropping, etc., to further enhance the generalization ability of CNN and improve the training effect of the model.

The model training module includes a classification model training unit and a detection model training unit, which are used to perform model training on the endoscopic images of early gastric precancerous diseases and lesion location information in the image data set to obtain a detection model.

Wherein, the classification model training unit is used to carry out the training of the deep learning CNN classification model on the classification data set, and introduces the method of reinforcement learning to adjust the parameters of the network model, including but not limited to the parameters with a small proportion in the network structure Reinforcement learning mode of zeroing and repeated iterative training to improve the accuracy of network classification; optimize the network structure, including but not limited to using layer-by-layer scale normalization layer (Batch Normalization, BN) to regularize the network, adopt The Dropout method is enhanced to alleviate the problem of over-fitting, and the ReLU activation function is used to avoid the problem of gradient disappearance; the training efficiency of the classification model is improved through transfer learning, that is, before training the classification model, it will be based on the VGG-16 classification framework (different from VGG -16 original version, the modified full name is VGG_ILSVRC_16_layers_fc_reduced classification framework) The model trained on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) natural image set is used as a pre-training model to speed up the modeling of the classification model progress and improve its classification performance.

The detection model unit is used to use the CNN classification model trained by the classification model unit as a pre-training model, and use the detection data set as a training set to carry out the training of the deep learning CNN target detection model based on the regression method. In the classification model The auxiliary network structure and detection layer are added on the basis of CNN. The auxiliary network structure uses the basic units of CNN, namely the convolutional layer and the pooling layer, as the basic units to further extract high-dimensional features in the picture. The auxiliary network structure has a total of 8 layers. Using the same The network structure optimization method of the above classification model optimizes the auxiliary network structure. The underlying detection layer consists of two parts. On the one hand, rectangular boxes of different proportions are introduced into the extracted features (in order to adapt to targets of different shapes and proportions). The existing abnormal feature targets are regressed and positioned, and the rectangular frame adapts to the proportion of the target under the function of the regression algorithm and selects it as a frame. During the training process, these rectangular frames are corrected by the training data set. On the other hand, the fusion The image features extracted from the convolutional layer in the above-mentioned basic classification model and the convolutional layer of the auxiliary network structure predict the classification confidence of the image selected by the rectangular frame, and the classification result is corrected by the training data set during the training process , using the training picture set contained in the detection gallery to carry out the training of the deep learning CNN target detection model based on the regression method. With the repeated iterative training of the network model, the detection model unit can finally generate objects that can correctly classify and detect abnormal features. For image frames, the detection model unit includes but is not limited to adding a detection method based on a feature pyramid, connecting high-level features with low resolution and high semantic information to low-level features with high resolution and low semantic information, so that the features at all scales are There are rich semantic information, including more semantic information to make the detection more accurate, including removing the fully connected layer to greatly improve the calculation speed, and the regression method is based on the single shot multibox detector (Single Shot MultiBox Detector) deep learning neural network framework.

The anomaly detection module is used to input the pre-processed image sequence into the trained detection model to obtain the classification information and location information of the abnormal features of the stomach.

The marker display module is used to integrate the output classification information of gastric precancerous diseases and lesion location information. The information is output by the lesion detection module. The marker display unit marks the location information on the picture sequence, and the location coordinate information The target lesion is marked in the form of a rectangular frame, and the mark is mapped to the original input video stream of the conventional white light endoscope of the stomach, and the video stream of the gastric endoscope is displayed in real time.

As shown in Figure 2, a real-time marking method for abnormal features of digestive endoscopy images, comprising the following steps:

(1) Obtain the regular white-light endoscopic video stream of the subject's stomach by using the endoscopic image system equipment;

(2) After the gastroscope video stream is parsed, key frames are extracted, invalid frames are removed, and the effective video image sequence of the gastroscope is obtained by screening;

As shown in FIG. 3 , it is an effective frame image of the stomach under the digestive endoscope; as shown in FIG. 4 , it is an invalid frame image of the stomach rinsed with water under the digestive endoscope. The original images of gastric precancerous diseases under the digestive endoscope are captured by the endoscope probe of the endoscope image system equipment of the medical institution, and are transmitted to the computer-aided digestive endoscope early cancer identification and screening system of the present invention in real time through the interface. The key frame extracting unit extracts the valid frames therein, and the content information expressed by a video can be expressed by its key frame,

The K-means algorithm randomly selects k objects from a large number of data objects as the initial clustering center, and classifies the data objects according to the minimum distance criterion. Through this clustering algorithm, the images in the image library are divided into k categories, and the prior Classification processing to speed up the efficiency of key frame extraction. For the useless frames of the stomach rinsed with water under the digestive endoscope, this kind of operation will affect the image quality of the gastroscope to some extent. If such images are removed or ignored, it will reduce a lot. Calculations.

(3) Perform image preprocessing on the gastroscope effective video image sequence. As shown in Figure 5, it is the original image of gastric polyps in gastric precancerous diseases under digestive endoscopy. The polyp image of the original image after image preprocessing has undergone image normalization, invalid pixel cropping, image smoothing, image sharpening, and image scaling. After a series of image preprocessing, useless part of the information is removed, and the polyp image after image enhancement processing A more accurate classification and positioning effect can be obtained, and the image is scaled to a size (300*300) matching the input image size of the lesion unit detected by the model, as shown in FIG. 6 .

(4) Input the image sequence after image preprocessing into the detection model. The detection model consists of a classification model plus an auxiliary network structure and a detection layer. The classification model is based on the VGG-16 framework, and the auxiliary network structure further extracts image features. The detection layer Get the classification result and return the position of the target, add the detection method based on the feature pyramid, and connect the high-level features with low resolution and high semantic information and the low-level features with high resolution and low semantic information. The detection model outputs the classification results and classification confidence of suspicious gastric precancerous diseases, and at the same time outputs the lesion location coordinates and location reliability;

(5) The system marks the effective image sequence according to the lesion location coordinates. The marks are shown in Figure 7, and the polyp lesions in the image are accurately located with eye-catching rectangular frame marks, and the marks are mapped to the original gastroscope video stream.

(6) The marked gastroscope video stream is displayed on the monitor in real time for the doctor to observe and confirm the diagnosis. The monitor displays the classification and detection information of the gastroscope image sequence frame by frame.

Claims (8)

1. a kind of digestive endoscopy image abnormal characteristic real-time mark system, including computer system, which is characterized in that the calculating Machine system includes：

Image collection module is obtained the stomach Conventional white endoscopic video stream of input in real time by endoscopic images system, picked Except wherein invalid frame, effective key frame is screened；

Image pre-processing module, for carrying out image enhancement processing to the key frame filtered out；

Model training module is obtained for carrying out model training to disease endoscope figure before early stage stomach cancer and lesions position information To detection model；

Abnormality detection module, for the image sequence for passing through image preprocessing to be input among the detection model of training completion, Obtain the classification information and location information of stomach off-note；

Display module is marked, location information is made a mark in sequence of pictures, and label is mapped to original input stomach In Conventional white endoscopic video stream, and real-time display is carried out to the described stomach scope video flowing.

2. digestive endoscopy image abnormal characteristic real-time mark system according to claim 1, which is characterized in that the figure K object is randomly choosed from data object using K-means algorithm when screening effective key frame as just as obtaining module Beginning cluster centre classifies to data object according to minimum distance criterion, by the clustering algorithm, the image in image library It is divided into k class.

3. digestive endoscopy image abnormal characteristic real-time mark system according to claim 1, which is characterized in that described image Image enhancement processing includes in preprocessing module：Image normalization, inactive pixels cutting, image smoothing, image sharpening and image Scaling.

4. digestive endoscopy image abnormal characteristic real-time mark system according to claim 1, which is characterized in that the mould Type training module includes：

Image data set, for storing categorized data set and detection data collection based on digestive endoscopy；Categorized data set includes more Disease endoscope figure before kind early stage stomach cancer, detection data collection include lesions position information；

Disaggregated model training unit, for carrying out the training of deep learning CNN disaggregated model to the categorized data set；

Detection model training unit is used for using CNN disaggregated model as pre-training model, using the detection data collection as training Collection, carries out the training of the deep learning CNN target detection model based on homing method.

5. digestive endoscopy image abnormal characteristic real-time mark system according to claim 4, which is characterized in that described image Data set is for carrying out data amplification by translation transformation, mirror image switch or random cropping before training CNN model.

6. digestive endoscopy image abnormal characteristic real-time mark system according to claim 4, which is characterized in that the classification Model training unit introduces the method for intensified learning when carrying out deep learning CNN disaggregated model training to improve network class Accuracy the problem of over-fitting is alleviated using Dropout method, adopted using layer-by-layer dimension normalization layer to network regularization Gradient disappearance problem is avoided with ReLU activation primitive, is promoted by training effectiveness of the transfer learning mode to disaggregated model.

7. digestive endoscopy image abnormal characteristic real-time mark system according to claim 4, which is characterized in that the model The detection model obtained after training module training includes disaggregated model, auxiliary network structure and target detection model；The classification Model is based on VGG-16 frame, for obtaining classification information；The auxiliary network structure is for extracting picture feature；The target Detection model is for obtaining location information.

8. a kind of side for carrying out digestive endoscopy image abnormal characteristic real-time mark using any one of the claim 1-7 system Method, which is characterized in that include the following steps：

(1) disease endoscope figure before early stage stomach cancer and lesions position information input model training module are trained, are obtained Detection model；

(2) subject's stomach Conventional white endoscopic video stream is obtained using endoscopic images system equipment；

(3) key-frame extraction will be carried out after the parsing of gastroscope video flowing, rejects invalid frame, screening obtains stomach endoscope effective video Image sequence；

(4) effective video image sequence is input to image pre-processing module, carries out image preprocessing；

(5) image sequence after image preprocessing is input in detection model, detection model exports disease before suspicious stomach cancer Lesion target positions coordinate, while exporting in positioning coordinate classification of diseases result and classification confidence before suspicious stomach cancer；

(6) it is marked in effective image sequence according to lesion localization coordinate, while label is mapped to former gastroscope video flowing In；

(7) real-time display diagnoses labeled gastroscope video flowing over the display so that doctor observes confirmation.