CN111091562B - A method and system for measuring the size of gastrointestinal lesions - Google Patents

Abstract

The invention discloses a method and system for measuring the size of digestive tract lesions. The method includes: acquiring an endoscopic image of the digestive tract including lesions and a reference object; identifying the reference object contained therein; calculating the relationship between the reference object and the digestive tract mucosa The width of the intersection; combined with the size of the junction between the reference object and the digestive tract mucosa obtained from the real experiment, the corresponding relationship between the image pixels and the actual size was obtained; the lesion area included in the digestive tract endoscopy image was extracted; The corresponding relationship between the actual sizes is obtained to obtain the actual size of the lesion. The present invention can realize the accurate measurement of the size of the lesion during the operation of the endoscope without resorting to additional equipment and without prolonging the operation time.

Description

A method and system for measuring the size of gastrointestinal lesions

technical field

The invention belongs to the technical field of artificial intelligence, and in particular relates to a method and system for measuring the size of gastrointestinal lesions.

Background technique

The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

Using electronic endoscopy for diagnosis, it is difficult to accurately measure the size of the lesion before removing the lesion for biopsy. The measurement of the lesion size under the endoscope is related to the decision-making of endoscopic treatment and data analysis. Due to the lack of sensors, endoscopy cannot measure polyp size as easily as ultrasound. In clinical practice, most endoscopists estimate polyp size by visual inspection or open biopsy forceps. However, studies have confirmed that endoscopists often cannot accurately estimate polyp size in vitro, which directly affects the choice of treatment decisions. At present, there are many methods to measure lesions. According to the inventor's knowledge, the currently established methods include delivering reference objects into the body, such as delivering a special measurement reference object, projecting a laser beam from an external laser device, etc.; or purchasing a special three-dimensional stereoscopic endoscope. Establishing a spatial coordinate system and obtaining the lesion size through three-dimensional reconstruction technology, etc., but these methods more or less require additional equipment or endoscopic operation time, and the measurement accuracy needs to be improved.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a method and system for measuring the size of gastrointestinal lesions, which can achieve the size of the lesions during the endoscopic operation without the aid of additional equipment and without prolonging the operation time. accurate measurement.

To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:

A method for measuring the size of gastrointestinal lesions, comprising the following steps:

Obtain endoscopic images of the digestive tract including lesions and reference objects;

identify the references contained therein;

calculating the width of the intersection of the reference and the mucous membrane of the digestive tract;

Combined with the size of the reference object obtained in the real experiment and the size of the junction of the alimentary canal mucosa, the corresponding relationship between the image pixels and the actual size is obtained;

Extract the lesion area included in the endoscopic image of the digestive tract;

According to the corresponding relationship between the image pixels and the actual size, the actual size of the lesion is obtained.

Further, the reference object included in the identification image adopts a pre-built reference object identification model; the model construction method includes:

Obtaining training data, the training data is the alimentary tract endoscope sample image that has been pre-marked with reference to the outline of the object;

performing mask processing on the sample images in the training data;

Using the masked training data, the Mask R-CNN model is trained to obtain the reference object recognition model.

Further, calculating the width of the intersection of the reference object and the alimentary canal mucosa includes:

Perform corner detection on the contour of the identified reference object;

The width of the intersection of the reference and the mucous membrane of the digestive tract is calculated from the detected corner points and the outline of the reference.

Further, calculating the width of the intersection of the reference object and the alimentary canal mucosa includes:

Identify the corner point on one side of the junction between the reference object and the alimentary canal mucosa according to the outline of the reference object;

Take a plurality of points on the contour line of the opposite side, and perform straight line fitting based on the plurality of points to obtain the fitted straight line on the opposite side;

The identified corner points are projected onto the fitted straight line, and the distance between the obtained projection point and the corner point is the width of the intersection of the reference object and the alimentary canal mucosa.

Further, an edge extraction algorithm is used to extract the lesion area included in the image.

Further, obtaining the actual size of the lesion includes:

Extract the circumscribed rectangle of the lesion area, and obtain the number of pixels corresponding to the length and width of the circumscribed rectangle;

According to the corresponding relationship between the image pixels and the actual size, the actual size of the length and width of the circumscribed rectangle of the lesion is obtained.

Further, the reference object is a jetted forward water column or a biopsy forceps.

One or more embodiments provide a system for measuring the size of gastrointestinal lesions, comprising:

A reference object image acquisition module, which acquires an endoscopic image of the digestive tract including the lesion and the reference object;

A reference object identification module to identify the reference objects contained therein;

The pixel and actual size correspondence relationship calculation module calculates the size of the junction between the reference object and the alimentary canal mucosa; Combined with the size of the reference object and the junction of the alimentary canal mucosa obtained in the real experiment, the difference between the image pixel and the actual size is obtained. Correspondence;

The lesion extraction module extracts the lesion area included in the gastrointestinal endoscopy image;

The lesion size calculation module obtains the actual size of the lesion according to the corresponding relationship between the image pixels and the actual size.

One or more embodiments provide an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implements the gastrointestinal lesion when the processor executes the program Size measurement method.

One or more embodiments provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for measuring the size of a gastrointestinal lesion.

One or more of the above technical solutions have the following beneficial effects:

In the method for measuring the size of the lesion provided by the present invention, the forward water column or biopsy forceps in the endoscope is used as the reference standard for lesion measurement, and the convolutional neural network is used to identify the size of the junction of the forward water column or biopsy forceps and the alimentary canal mucosa as the measurement scale , the size of the lesion is calculated according to the diameter of the projection surface of the lesion in the image of the reference object. The present invention can complete the measurement of the size of the lesion during the normal endoscopic operation without the aid of additional equipment, without prolonging the operation time, and is economical and operable. It is strong and provides a basis for the selection of treatment methods for lesions.

Description of drawings

The accompanying drawings that form a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.

1 is a flow chart of a method for measuring the size of gastrointestinal lesions in one or more embodiments of the present invention;

Figure 2a is an image of a gastroscope including a biopsy forceps in one or more embodiments of the present invention;

Figure 2b is a gastroscopic image including an anterior water column in one or more embodiments of the present invention;

Fig. 3 is the identification effect diagram of the forward water column in one or more embodiments of the present invention;

4 is a contour image of a reference object generated in one or more embodiments of the present invention;

5 is a schematic diagram of the junction between the reference object and the gastric mucosa in one or more embodiments of the present invention;

6 is a gastroscopic image including a lesion obtained in one or more embodiments of the present invention;

7 is a schematic diagram of a circumscribed rectangle of a lesion area in one or more embodiments of the present invention;

FIG. 8 is a schematic diagram of a dimensioned lesion area in one or more embodiments of the present invention.

Detailed ways

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

Embodiments of the invention and features of the embodiments may be combined with each other without conflict.

Example 1

As shown in Figure 1, the present embodiment discloses a method for measuring the size of a digestive tract lesion, taking the measurement of the lesion size on the gastric mucosa as an example, comprising the following steps:

Step 1: During the procedure using the endoscope, acquire a gastroscopic image including the lesion and reference object.

Since the distance between the gastroscope lens and the gastric mucosa will change during gastroscopy, in order to ensure that the lesion and the reference object are at the same distance from the lens, when the physician finds the lesion, the gastroscope image containing the lesion and the reference object is collected. The reference can be a jet of forward water or a biopsy forceps. Since both the water jet and biopsy forceps are auxiliary means or tools frequently used by physicians in the process of gastroscopy, they are used as reference objects, and the acquisition of gastroscopic images containing both lesions and reference objects will not significantly prolong the examination process. In the normal examination process to complete.

The electronic gastroscope is used to reflect the patient's examination situation on the display screen, and the gastroscope image with the measurement standard reference object is stored as data, so as to obtain the target gastroscope image to be detected.

The distortion of the electronic gastroscope lens will cause errors in the measurement of lesions under the gastroscope, so it is necessary to correct the distortion of the electronic gastroscope lens. In this embodiment, a specific template is used to calculate the mapping relationship between the ideal effect map and the distortion map, and obtain the distortion change from it. equation to rectify the image.

Step 2: Based on the trained gastroscope image reference object recognition model, the gastroscope image collected by the electronic gastroscope is detected, and the outline of the marked reference object is obtained, as shown in Figure 3-4.

Among them, the contour detection model for training gastroscope image reference objects specifically includes:

(1) Obtain a sample gastroscopic image containing a reference object, and perform contour annotation on the reference object in the sample gastroscopic image to obtain a training sample.

(2) Perform image processing on the sample gastroscopic image to obtain a sample gastroscopic image data set.

Wherein, the image processing includes: generating a mask image, bicubic interpolation scaling processing of 4x4 pixel neighborhood. The image reduced by mask processing can retain as many image details as possible, which is beneficial to the training of the network.

(3) according to described sample gastroscope image data set, train MASK-RCNN convolutional neural network, obtain the reference object recognition model that recognizes gastroscope image trained.

The sample gastroscopic image data set is divided into a sample gastroscopic image training set, a sample gastroscopic image testing set and a sample gastroscopic image verification set according to the ratio of 6:2:2, and input them into the reference object recognition model of the gastroscopic image to be trained. In the embodiment of the present invention, the reference object recognition model is obtained by training a convolutional neural network. The convolutional neural network includes a plurality of convolutional layers, and each convolutional layer is superimposed with several optional convolution kernels to improve the convolutional performance. performance of neural networks. After the sample gastroscopic image data set undergoes a series of convolution and pooling operations of the convolutional neural network, the image features of the sample gastroscopic image are extracted, and according to the image features, it is judged whether the image of the sample gastroscopic image contains the water column or biopsy forceps to be detected. The convolutional neural network is tested and verified through the sample gastroscope image test set and the sample gastroscope image verification set. If the preset conditions are met, the trained reference object recognition model is obtained.

Step 3: Calculate the width of the intersection of the reference object and the mucous membrane of the digestive tract.

Among them, for the water column, the width of the intersection of the water column and the mucous membrane of the alimentary canal is the diameter of the contact surface where the water column is sprayed to the mucous membrane of the alimentary canal.

3.1: First, find the corner set with the largest eigenvalue in the contour image, which is implemented by the Shi-Tomasi corner detection algorithm.

3.2: Obtain the corner points at both ends of the junction between the reference object and the digestive tract mucosa. Since the angle of the water jet is fixed, the right corner can be identified more accurately, but the left corner is not easy to identify accurately, so first identify the right corner at the junction of the reference object and the mucosa, and then start from the reference object. Take 5 points on the left side of the contour, use the minimum binarization fitting straight line algorithm for these 5 points, and fit the straight line equation of the left contour line (y=kx+d), find the right corner point in The projection point on the left contour line, the right corner point and the projection point are considered to be the corner points at the junction between a reference object such as a water column or biopsy forceps and the gastrointestinal mucosa, and the distance between the two points is calculated as the reference object and the digestive tract. The width of the mucosal intersection, as shown in Figure 5.

Combined with the size of the junction of the reference object obtained from the real experiment and the digestive tract mucosa, the corresponding relationship between the image pixels and the actual size is obtained. For example, according to the real experiment, it is concluded that the diameter of the contact surface between the water column and other reference objects and the alimentary canal mucosa is 1mm. According to the two contact points obtained in step 4, the actual diameter of the two contact points is considered to be 1mm.

Step 4: Extract the lesion area from the endoscopic image of the digestive tract.

The acquired gastroscopic image containing the lesion is shown in Figure 6, and the contour image of the lesion is obtained according to the Canny boundary extraction algorithm, and the extracted lesion contour is shown in Figure 7. The size of the rectangular area can be obtained from the position of the contour area.

Step 5: Compare the diameter of the reference object to obtain the size of the lesion in the image.

According to the following formula: the distance between two points at the junction in the image/the distance between the length and width of the lesion area=the actual length of the junction/the actual length of the lesion area.

Step 6: According to the corresponding relationship between the image pixels and the actual size, the actual size of the lesion is obtained, as shown in FIG. 8 .

Embodiment 2

The purpose of this embodiment is to provide a system for measuring the size of digestive tract lesions.

A system for measuring the size of gastrointestinal lesions, comprising:

A reference object image acquisition module, which acquires an endoscopic image of the digestive tract including the lesion and the reference object;

A reference object identification module to identify the reference objects contained therein;

The pixel and actual size correspondence relationship calculation module calculates the size of the junction between the reference object and the alimentary canal mucosa; Combined with the size of the reference object and the junction of the alimentary canal mucosa obtained in the real experiment, the difference between the image pixel and the actual size is obtained. Correspondence;

The lesion extraction module extracts the lesion area included in the gastrointestinal endoscopy image;

The lesion size calculation module obtains the actual size of the lesion according to the corresponding relationship between the image pixels and the actual size.

Embodiment 3

The purpose of this embodiment is to provide an electronic device.

An electronic device, comprising a memory, a processor and a computer program stored on the memory and running on the processor, the processor implements the following steps when executing the program, including:

Obtain endoscopic images of the digestive tract including lesions and reference objects;

identify the references contained therein;

Calculate the size of the junction between the reference object and the alimentary canal mucosa;

Combined with the size of the reference object obtained in the real experiment and the size of the junction of the alimentary canal mucosa, the corresponding relationship between the image pixels and the actual size is obtained;

Extract the lesion area included in the endoscopic image of the digestive tract;

The lesion area contained therein is extracted, and the actual size of the lesion is obtained according to the corresponding relationship between the image pixels and the actual size.

Embodiment 4

The purpose of this embodiment is to provide a computer-readable storage medium.

A computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the following steps are performed:

Obtain endoscopic images of the digestive tract including lesions and reference objects;

identify the references contained therein;

Calculate the size of the junction between the reference object and the alimentary canal mucosa;

Combined with the size of the reference object obtained in the real experiment and the size of the junction of the alimentary canal mucosa, the corresponding relationship between the image pixels and the actual size is obtained;

Extract the lesion area included in the endoscopic image of the digestive tract;

According to the corresponding relationship between the image pixels and the actual size, the actual size of the lesion is obtained.

The steps involved in the second, third and fourth embodiments above correspond to the method embodiment 1, and the specific implementation can refer to the relevant description part of the embodiment 1. The term "computer-readable storage medium" should be understood to include a single medium or multiple media including one or more sets of instructions; it should also be understood to include any medium capable of storing, encoding or carrying for use by a processor The executed instruction set causes the processor to perform any of the methods of the present invention.

The above one or more embodiments have the following technical effects:

In the method for measuring the size of the lesion provided by the present invention, the forward water column or biopsy forceps in the endoscope is used as the reference standard for lesion measurement, and the convolutional neural network is used to identify the size of the junction of the forward water column or biopsy forceps and the alimentary canal mucosa as the measurement scale , the size of the lesion is calculated according to the thickness change of the reference object in the image, the present invention does not require additional equipment, the measurement of the size of the lesion can be completed during the normal endoscopic operation, and the operation time is not prolonged, with good economy and operability. It provides a basis for the selection of treatment methods for lesions.

Those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by the computing device, so that they can be stored in a storage device. The device is executed by a computing device, or they are separately fabricated into individual integrated circuit modules, or multiple modules or steps in them are fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Although the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, they do not limit the scope of protection of the present invention. Those skilled in the art should understand that on the basis of the technical solutions of the present invention, those skilled in the art do not need to pay creative work. Various modifications or deformations that can be made are still within the protection scope of the present invention.

Claims (8)

1. a method for measuring the size of digestive tract lesions, is characterized in that, comprises the following steps: Obtain endoscopic images of the digestive tract including lesions and reference objects; identify the references contained therein; calculating the width of the intersection of the reference and the mucous membrane of the digestive tract; Combined with the size of the reference object obtained in the real experiment and the size of the junction of the digestive tract mucosa, the corresponding relationship between the image pixels and the actual size is obtained; Extract the lesion area included in the endoscopic image of the digestive tract; Comparing the diameter of the reference object, the size of the lesion in the image is obtained; according to the corresponding relationship between the image pixels and the actual size, the actual size of the lesion is obtained; The calculation method of the width of the intersection of the reference object and the mucous membrane of the digestive tract includes: Perform corner detection on the contour of the identified reference object; Calculate the width of the intersection of the reference object and the alimentary canal mucosa according to the detected corner points and the contour of the reference object: identify the corner point on one side of the junction of the reference object and the alimentary canal mucosa according to the contour of the reference object; on the contour line on the opposite side Take a plurality of points, perform straight line fitting based on the plurality of points, and obtain the fitted straight line on the opposite side; project the identified corner points onto the fitted straight line, and obtain a distance between the projected point and the corner point The distance is the width of the intersection of the reference object and the mucous membrane of the digestive tract. 2. A method for measuring the size of digestive tract lesions as claimed in claim 1, wherein the reference object included in the identification image adopts a pre-built reference object identification model; the model construction method comprises: acquiring training data, where the training data is a sample image of a digestive tract endoscope that has been pre-marked with reference to the contour of the object; performing mask processing on the sample images in the training data; Using the masked training data, the Mask R-CNN model is trained to obtain a reference object recognition model. 3 . The method for measuring the size of a gastrointestinal lesion according to claim 1 , wherein an edge extraction algorithm is used to extract the lesion area. 4 . 4. The method for measuring the size of a gastrointestinal lesion according to claim 1, wherein obtaining the actual size of the lesion comprises: Extract the circumscribed rectangle of the lesion area, and obtain the number of pixels corresponding to the length and width of the circumscribed rectangle; According to the correspondence between the image pixels and the actual size, the actual size of the length and width of the circumscribed rectangle of the lesion is obtained. 5 . The method for measuring the size of digestive tract lesions according to claim 1 , wherein the reference object is a jetted forward water column or a biopsy forceps. 6 . 6. A system for measuring the size of digestive tract lesions, comprising: A reference object image acquisition module, which acquires an endoscopic image of the digestive tract including the lesion and the reference object; A reference object identification module to identify the reference objects contained therein; The pixel and actual size correspondence relationship calculation module calculates the size of the junction between the reference object and the alimentary canal mucosa; Combined with the size of the reference object and the junction of the alimentary canal mucosa obtained in the real experiment, the difference between the image pixel and the actual size is obtained. Correspondence; The lesion extraction module extracts the lesion area included in the gastrointestinal endoscopy image; The lesion size calculation module compares the diameter of the reference object to obtain the size of the lesion in the image, and obtains the actual size of the lesion according to the correspondence between the image pixels and the actual size; The calculation of the size of the intersection of the reference object and the mucous membrane of the digestive tract includes: Perform corner detection on the contour of the identified reference object; Calculate the width of the intersection of the reference object and the alimentary canal mucosa according to the detected corner points and the contour of the reference object: identify the corner point on one side of the junction of the reference object and the alimentary canal mucosa according to the contour of the reference object; on the contour line on the opposite side Take multiple points, perform straight line fitting based on the multiple points, and obtain the fitted straight line on the opposite side; project the identified corner points onto the fitted straight line, and obtain a distance between the projected point and the corner point The distance is the width of the intersection of the reference object and the mucous membrane of the digestive tract. 7. An electronic device, comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1-5 when the processor executes the program The method for measuring the size of gastrointestinal lesions described in item. 8 . A computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method for measuring the size of a gastrointestinal lesion according to any one of claims 1 to 5 is implemented.

Images