CN112686865A - 3D view auxiliary detection method, system, device and storage medium - Google Patents

Abstract

The present invention relates to the technical field of medical auxiliary detection, in particular to a 3D view auxiliary detection method, system, device and storage medium. The method includes: acquiring a left-view image and a right-view image of the area to be detected in real time; performing lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame respectively; After image feature fusion, 2D-3D conversion is performed to obtain a 3D view of the area to be detected. The entire image processing process is fast and real-time. 3D annotation is performed to obtain a 3D lesion annotation view, and the 3D lesion annotation view is displayed in real time. Doctors wearing special 3D glasses can clearly observe the lesions in the detection area in real time, and the real-time display of 3D views makes the observation effect better.

Description

A 3D view auxiliary detection method, system, device and storage medium

technical field

The present invention relates to the technical field of medical auxiliary detection, in particular to a 3D view auxiliary detection method, system, device and storage medium.

Background technique

The endoscope can enter the patient's body through the natural orifice of the human body or through a minimally invasive surgical wound, providing doctors with a clear and stable high-quality picture to complete the operation. 3D endoscope is a new type of stereo imaging endoscope, which can directly reflect the depth of field characteristics of the observation area, which is conducive to diagnosis.

In 3D endoscopy-assisted diagnosis, clinicians use an endoscope to observe the patient's internal conditions to determine the diagnosis result. However, manual analysis has the following obvious defects: (1) It is not accurate enough, doctors can only rely on experience to identify, and it is easy to cause misdiagnosis due to the lack of quantitative standards; (2) There will inevitably be errors in human eyesight and visual fatigue. ; (3) Massive image information is prone to misdiagnosis.

Traditional Computer Aided Diagnosis (CAD) uses medical image processing technology combined with computer analysis and calculation to assist in the discovery of lesions. Although it can solve some of the problems of manual analysis, it still needs to extract features manually, which has poor real-time performance and is prone to missed diagnosis.

SUMMARY OF THE INVENTION

The main technical problem to be solved by the present invention is that the existing medical image processing technology has poor real-time performance when using artificial feature extraction to assist in finding lesions.

A 3D view aided detection method, comprising:

Obtain the left-view image and right-view image of the area to be detected in real time;

Performing lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame, respectively;

2D-3D conversion is performed after feature fusion of the left-view image and the right-view image to obtain a 3D view of the area to be detected;

According to the first lesion labeling frame and the second lesion labeling frame, perform 3D labeling on the position of the lesion in the 3D view to obtain a 3D lesion labeling view;

The 3D lesion annotation view is displayed in real time.

In an embodiment, the obtaining a 3D lesion annotation view by performing 3D annotation on the position of the lesion in the 3D view according to the first lesion annotation frame and the second lesion annotation frame includes:

Calculate the degree of association between the first lesion annotation frame and the second lesion annotation frame, and if the association degree reaches a preset interval, associate the first lesion annotation frame and the second lesion annotation frame to obtain a 3D annotation frame;

The 3D annotation frame is marked in the 3D view to obtain the 3D lesion annotation view, and the area where the 3D annotation frame is located is the lesion area.

In an embodiment, the performing lesion detection on the left-view image and the right-view image to obtain a first lesion annotation frame and a second lesion annotation frame respectively includes:

Inputting the left-view image and the right-view image into the pre-trained first lesion detection model and the second lesion detection model, respectively, to obtain the first lesion annotation frame and the second lesion annotation frame; The label box and the second lesion label box represent the lesion area in the left-view image and the right-view image, respectively.

In one embodiment, the first lesion detection model and the second lesion detection model are both obtained by training the following methods:

Obtain normal image datasets and lesion image datasets as training sample sets;

scaling, rotating, flipping and changing the brightness of the images in the training sample set to expand the training sample set;

The expanded training sample set is input into the initial YOLO network model for multiple training to obtain the first lesion detection model and the second lesion detection model.

A 3D view aided detection system, comprising:

The image acquisition module is used to acquire the left-view image and right-view image of the area to be detected in real time;

a lesion detection module, configured to perform lesion detection on the left-view image and right-view image to obtain a first lesion labeling frame and a second lesion labeling frame, respectively;

A view conversion module, configured to perform 2D-3D conversion after feature fusion of the left-view image and the right-view image to obtain a 3D view of the area to be detected;

a 3D labeling unit, configured to perform 3D labeling on the position of the lesion in the 3D view according to the first lesion labeling frame and the second lesion labeling frame to obtain a 3D lesion labeling view;

The real-time display module is used to display the 3D lesion annotation view in real time.

In an embodiment, the 3D labeling unit includes an association module and a labeling module:

The association module is used to calculate the degree of association between the first lesion annotation frame and the second lesion annotation frame, and if the association degree reaches a preset interval, the first lesion annotation frame and the second lesion annotation frame are calculated. The association gets the 3D callout frame;

The labeling module is configured to label the 3D labeling frame in the 3D view to obtain the 3D lesion labeling view, and the area where the 3D labeling frame is located is the lesion area.

In an embodiment, the lesion detection module is provided with a pre-trained first lesion detection model and a second lesion detection model;

The performing lesion detection on the left-view image and the right-view image to obtain the first lesion labeling frame and the second lesion labeling frame respectively includes:

Inputting the left-view image and the right-view image into the pre-trained first lesion detection model and the second lesion detection model, respectively, to obtain the first lesion annotation frame and the second lesion annotation frame; The label box and the second lesion label box represent the lesion area in the left-view image and the right-view image, respectively.

In one embodiment, it also includes:

The sample set acquisition module is used to obtain the normal image data set and the lesion image data set as training sample sets;

a preprocessing module for scaling, rotating, flipping and changing the brightness of the images in the training sample set to expand the training sample set;

The training module is used for obtaining the first lesion detection model and the second lesion detection model by using the expanded training sample set to train two initial YOLO network models.

An auxiliary detection device, comprising:

3D endoscope for real-time acquisition of left-view and right-view images of the area to be detected;

a processor, configured to perform lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame respectively; perform feature fusion on the left-view image and the right-view image to perform 2D-3D Convert to obtain a 3D view of the area to be detected; according to the first lesion annotation frame and the second lesion annotation frame, perform 3D annotation on the position of the lesion in the 3D view to obtain a 3D lesion annotation view;

A display for displaying the 3D lesion annotation view in real time.

A computer-readable storage medium comprising a program executable by a processor to implement the method as described above.

According to the 3D view auxiliary detection method, system and device according to the above-mentioned embodiments, the method includes: acquiring a left-view image and a right-view image of a region to be detected in real time; performing lesion detection on the left-view image and the right-view image to obtain a first lesion label, respectively frame and the second lesion labeling frame; the left-view image and the right-view image are fused and then 2D-3D converted to obtain a 3D view of the area to be detected. The whole image processing process is fast and has good real-time performance; according to the first lesion labeling frame and the second lesion labeling frame, perform 3D labeling on the position of the lesion in the 3D view to obtain the 3D lesion labeling view, and display the 3D lesion labeling view in real time. Doctors wearing special 3D glasses can clearly observe the lesions in the detection area in real time, and the real-time display of 3D views makes the observation effect better.

Description of drawings

1 is a flowchart of an auxiliary detection method according to an embodiment of the present application;

2 is a flowchart of a method for training a lesion detection model according to an embodiment of the present application;

3 is a framework diagram of an auxiliary detection method according to an embodiment of the present application;

4 is a schematic structural diagram of a lesion detection model according to an embodiment of the present application;

FIG. 5 is a schematic diagram of the position of a labeling frame according to an embodiment of the present application;

FIG. 6 is a schematic diagram of constructing a 3D view for a binocular view according to an embodiment of the present application;

7 is a schematic diagram of the principle of constructing a 3D annotation frame according to a first lesion annotation frame and a second lesion annotation frame according to an embodiment of the present application;

8 is a structural block diagram of an auxiliary detection system according to an embodiment of the application;

FIG. 9 is a structural block diagram of an auxiliary detection apparatus according to an embodiment of the present application.

Detailed ways

The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments have used associated similar element numbers. In the following embodiments, many details are described so that the present application can be better understood. However, those skilled in the art will readily recognize that some of the features may be omitted under different circumstances, or may be replaced by other elements, materials, and methods. In some cases, some operations related to the present application are not shown or described in the specification, in order to avoid the core part of the present application from being overwhelmed by excessive description, and for those skilled in the art, these are described in detail. The relevant operations are not necessary, and they can fully understand the relevant operations according to the descriptions in the specification and general technical knowledge in the field.

Additionally, the features, acts, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can also be exchanged or adjusted in order in a manner obvious to those skilled in the art. Therefore, the various sequences in the specification and drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary order unless otherwise stated, a certain order must be followed.

The serial numbers themselves, such as "first", "second", etc., for the components herein are only used to distinguish the described objects, and do not have any order or technical meaning.

In this application, the two parallel binocular lenses that use 3D endoscope to detect the left -view images and right -view images of the area to detect the area, and the lesions detection of the left -view image and right -view image are performed. Second lesion labeling frame; the first lesion labeling frame and the second lesion labeling frame are used to represent the lesion area in the left-view image and the right-view image respectively, and then the first lesion labeling frame and the second lesion labeling frame are associated to obtain the final Marking the frame, in this way, the left-view image and the right-view image are separately detected for lesions, which reduces the amount of calculation, improves the detection efficiency, and makes the real-time detection results better. As long as it is detected that there is a lesion in one of the left-view image and right-view image, the left-view image and the right-view image are fused, and then 2D-3D conversion is performed to obtain a 3D view of the area to be detected. After the second lesion annotation frame is associated, the corresponding 3D annotation frame in the 3D view is obtained, and the 3D annotation frame is marked in the 3D view to obtain the 3D lesion annotation view. The location of the 3D annotation frame represents the lesion area, and then the 3D lesion is marked. The view is displayed in 3D, and doctors or experts wearing special 3D glasses can observe the three-dimensional view of the detection area in real time and clearly, so that the displayed lesion information is more clear, which is convenient for doctors to view, and plays a certain role in the diagnosis or operation of doctors. effect.

In addition, after verification by the inventor, if the lesion area is directly detected on the 3D image, and then the lesion area is marked in 3D, the amount of calculation is very large, and it is difficult to perform real-time processing. After the lesion area is detected, 2D-3D conversion is performed to obtain a 3D view of the area to be detected, and then the lesion area is marked on the 3D view, so that the amount of calculation will be greatly reduced, the lesion detection speed will be faster, and the 3D view can be achieved. The real-time display is better for the auxiliary diagnosis of doctors.

Further, in the present application, a trained neural network model is used to detect lesions on the left-view image and the right-view image to obtain the first lesion labeling frame and the second lesion labeling frame, so that the lesion position obtained in this way is more accurate and robust. Better sex and strong generalization ability.

Example 1:

Please refer to FIG. 1 and FIG. 3 , this embodiment provides a 3D view auxiliary detection method, which includes:

Step 101: acquiring the left-view image and the right-view image of the area to be detected in real time;

Step 102: Perform lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame, respectively;

Step 103: Perform feature fusion of the left-view image and the right-view image and then perform 2D-3D conversion to obtain a 3D view of the area to be detected;

Step 104: according to the first lesion annotation frame and the second lesion annotation frame, perform 3D annotation on the position of the lesion in the 3D view to obtain a 3D lesion annotation view;

Step 105: Display the 3D lesion annotation view in real time.

Wherein, in step 101, in this embodiment, a binocular 3D electronic endoscope is used to collect the left-view image and the right-view image of the area to be detected. That is, dual digital images, namely left-view images and right-view images of the 3D electronic endoscope, are obtained through photoelectric conversion through the dual camera wafers of the 3D electronic endoscope that are parallel to the optical axis.

In another embodiment, before the lesion detection is performed on the left-view image and the right-view image, the left-view image and the right-view image need to be preprocessed, for example, the image size, resolution, etc. are normalized. . Then, the feature information related to the lesion in the left-view image and the right-view image is detected respectively, that is, the area corresponding to the lesion is obtained. In this embodiment, the lesion detection is performed on the left-view image and the right-view image to obtain the first lesion annotation frame and the second lesion respectively. The lesion labeling frame, the first lesion labeling frame represents the lesion area in the left-view image, and the second labeling frame represents the lesion area in the right-view image.

Wherein, in this embodiment, a trained neural network model is used for lesion detection. Specifically, in order to improve the speed of lesion detection, in this embodiment, the left-view image and the right-view image are respectively input into the pre-trained first lesion detection model and the second lesion detection model, and the first lesion annotation frame and the second lesion detection model are obtained respectively. The second lesion annotation frame; the first lesion annotation frame and the second lesion annotation frame respectively represent the lesion area in the left-view image and the right-view image.

Since the image acquisition of the 3D electronic endoscope is a dual digital image obtained by photoelectric conversion through a dual camera wafer with parallel optical axes, the left and right images have the same application scene and the same size, so the first lesion detection model and the second lesion detection model based on the YOLO algorithm are constructed. When the lesion detection model is used, the same detection model is constructed to improve the speed of image transmission and detection, and reduce the proportion of space.

In this embodiment, a lesion detection model suitable for 3D endoscope is constructed, which is obtained by training the network structure based on the YOLO detection algorithm. The YOLO detection algorithm directly regresses the categories and borders on the feature layer, and uses the regression method to make The framework is not as complex as other network structures, and the detection speed is fast, which is suitable for application in video streams. Therefore, when the 3D endoscope detects the lesion image in the diagnosis, the detection model based on the YOLO algorithm can be used to meet the requirements of real-time detection.

In this embodiment, the first lesion detection model and the second lesion detection model based on the YOLO algorithm are constructed, and multi-scale features are extracted for fusion training based on the YOLO structure. The specific network structure is shown in FIG. 4 . This method can solve the problem of less use of shallow network features in target detection and low detection accuracy of small lesion targets in endoscopic images, so that the detection accuracy of the lesion detection model can be improved while the calculation speed is guaranteed.

Specifically, as shown in FIG. 2 , the first lesion detection model and the second lesion detection model in this embodiment are both obtained by training the following methods:

Step 201: Acquire a normal image data set and a diseased image data set as a training sample set; the images in the training sample set are obtained by a monocular endoscope, and are analyzed by a doctor or an expert to form a normal image set and a diseased image set.

Step 202: scaling, rotating, flipping and changing the brightness of the images in the training sample set to expand the training sample set; to make the data set images of each lesion category equal, the lesion type in this embodiment includes lesions and no lesions; This prevents overfitting during training of the model. In other embodiments, other preprocessing and image enhancement methods may also be used to expand the training sample set.

Step 203 : Input the expanded training sample set into the initial YOLO network model and train it for multiple times to obtain a first lesion detection model and a second lesion detection model. The first lesion detection model and the second lesion detection model in this embodiment have the same structure, and can be used to detect whether there is a lesion in the input image and to mark the area of the lesion in the form of a labeling frame.

For example, training the first lesion detection model based on the initial YOLO network model includes the following steps:

1) Divide the images used for training in the processed training sample set into S×S grids;

2) Input the training image into the YOLO network structure to extract features, and extract three different scale features at the same time, and fuse high-level abstract features and shallow features to detect the final lesion target;

3) Simultaneously train and predict the multi-label Bounding Box in each grid, and predict targets of different sizes and types in each label;

4) Simultaneously predict in each grid that the label box predicted by the grid contains the detection target, and predicts the probability of the category in the label box. Therefore, in this embodiment, a class-specific confidence score (class-specific confidence score) is calculated. ) is used as the discriminant value of whether there is lesion information in the grid. Classification Confidence Multiplies the classification information predicted by each grid and the confidence of the label box prediction to obtain the classification confidence of each label box, which is calculated as follows:

是预测出来的标注框与实际物体边界框的交并比值，Pr(Object)用于判断是否有目标病变信息在此网格中，Pr(Class_i)是用于判断目标病变信息的分类情况。in,

is the intersection ratio of the predicted label box and the actual object bounding box, Pr(Object) is used to determine whether there is target lesion information in this grid, and Pr(Class _i ) is used to determine the classification of target lesion information.

5) Then set the classification confidence threshold of each annotation box. If it is less than the threshold, the corresponding annotation box is discarded, and the non-maximum value suppression (NMS) processing is applied to the annotation box that meets the threshold condition to obtain the final prediction result.

The NMS algorithm is used to achieve this effect: first, find the box with the highest confidence from all the annotation boxes, and then calculate the IOU of it and the remaining annotation boxes one by one, if its value is greater than a set threshold (the coincidence is too high) , then remove the frame; then repeat the above process for the remaining frames until all frames are processed.

6) The final prediction can obtain the classification probability of the target lesion's labeling box, the location information of the labeling box, and the confidence of the labeling box. The prediction of the position size, type, confidence and other information of the annotation frame is trained by the loss function.

The loss function is composed of four parts, which are to calculate the loss of the center coordinates of the predicted annotation frame, and to calculate relative to the actual center position in the prediction and training;

为训练中的中心位置。Among them, in the calculation of the center position loss of the above formula, λ _C is a given constant, (x _i , y _i ) is the predicted center position,

is the center position during training.

Calculate the loss related to the width and height of the predicted annotation box, and calculate the square root of its height and width;

为1，预测有效；否则为

为0，预测无效。Among them, in the calculation of the loss of the width w and height h of the label box, if there is lesion information in the grid cell i, then

is 1, the prediction is valid; otherwise,

0, the prediction is invalid.

Calculate the loss of the confidence of the label box, and predict and punish the confidence level of the intersection between the label box and the actual lesion target;

为第i个预测标注框与实际目标交叉处的置信度。

为得到的最低置信度预测惩罚。In the calculation of the confidence loss of the label box in the above formula, c _i represents the confidence value of the ith prediction box,

is the confidence of the intersection of the i-th predicted annotation box and the actual target.

Prediction penalty for the lowest confidence obtained.

Calculate the loss for the classification, and calculate the squared error of the classification in the labeled box and the actual classification.

In the classification calculation loss in the above formula, pi is the classification percentage result of the _i -th prediction box.

Wherein, in step 103, if only one of the left-view image and the right-view image is detected to have a lesion, it is determined that there is a lesion in the current to-be-detected area, and then the left-view image and the right-view image are feature-fused to perform 2D to 3D The conversion obtains a 3D view of the area to be detected. Specifically, the depth information of the image needs to be acquired when performing 2D to 3D conversion. In this embodiment, depth information can be obtained according to the left-view image and right-view image parallel to the optical axis, and then a 3D view can be obtained through spatial calculation according to parallax, camera parameters, and depth information.

Wherein, in step 104, the correlation degree calculation is performed on the first lesion annotation frame and the second lesion annotation frame, and if the correlation degree reaches a preset interval, the first lesion annotation frame and the second lesion annotation frame are associated to obtain a 3D annotation frame , for example, the correlation degree is between 0.7-1, then associate the first lesion annotation frame with the second lesion annotation frame to obtain a 3D annotation frame; mark the 3D annotation frame in the 3D view to obtain a 3D lesion annotation view, where the 3D annotation frame is located The area is the lesion area. If the correlation degree does not reach the preset interval, it means that the first lesion labeling box and the second lesion labeling box may not mark the same lesion target. It should be noted that, in the actual lesion feature detection, multiple lesion labeling boxes may appear in both the left-view image and the right-view image, each labeling a different lesion target. A labeling frame marks erosion, and the first lesion labeling frame and the second lesion labeling frame in this application refer to the labeling of the same lesion target in the two figures, and the first lesion labeling frame and the Whether the second lesion labeling box marks the lesion target in the same lesion area, if the correlation degree does not reach the preset interval, the model continues to perform the correlation degree matching calculation until the correlation degree reaches the preset interval. 20 times), the correlation degree calculated by the model can never reach the preset range, then the two annotation boxes are not associated with the operation, and the 3D annotation box is not output at the same time. If a certain annotation frame only appears in one of the left-view image or the right-view image, the above-mentioned requirement of the degree of correlation reaching the preset interval also cannot be satisfied, and the 3D annotation frame is also not output at this time.

For example, the horizontal disparity and depth information can be obtained according to the coordinate positions of the first lesion annotation frame and the second lesion annotation frame in the left-view image and the right-view image, so as to obtain the 3D annotation frame through spatial coordinate transformation calculation. The 3D annotation frame is in The 3D solid area marked in the 3D view is the lesion area.

Wherein, the coordinates of the first lesion labeling frame and the second lesion labeling frame in this embodiment are obtained by the following method, which specifically includes: extracting the coordinates of the first lesion labeling frame and the second lesion labeling frame in the left-view image and the right-view image. The depth information is obtained through the parameter prediction of the first lesion annotation frame and the second lesion annotation frame in the YOLO network, and the prediction parameters of each lesion annotation frame (ie, the coordinates of the annotation frame) include:

1) The local coordinate offset values x, y of the target center position of the callout box relative to the grid position;

2) The width and height ratios w and h of the annotation frame relative to the entire image;

3) Confidence degree c (under the condition of whether each grid contains the target of lesion detection, the accuracy information of its detection frame);

Therefore, the coordinate position of the lesion annotation frame relative to the entire image can be calculated by the offset value of the center position of the detection frame and the relative width and height ratio. The depth information is calculated by the coordinates of the corner points, and then the three-dimensional projection of the corresponding coordinates is realized, so as to obtain the frame information in the three-dimensional display of the endoscope. The coordinates of the callout box are shown in Figure 5 below.

Therefore, the local coordinate offset values x, y of the target center position of the annotation frame relative to the grid position and the width and height ratios w and h of the annotation frame relative to the entire image have the following relationship between the annotation frame and the entire image :

In the above formula, (a _x , a _y ) is the coordinate position of the center of the annotation frame in the entire detection image, (m, n) is the coordinate position of the upper left corner of the annotation frame, C _x , C _y are the grid sizes, a, b is the size of the annotation frame, and u and v are the size of the entire image.

Therefore, the position coordinates of the center point of the first lesion annotation frame and the center point of the second lesion annotation frame in the whole image, as well as the center point of the first lesion annotation frame and the size of the second lesion annotation frame can be obtained by the above formula, so as to obtain The position coordinates of the corner points of the labeling box can be obtained to obtain the center point of the first lesion labeling box and the coordinates of the second lesion labeling box.

The depth information of the first lesion labeling frame and the depth information of the second lesion labeling frame are mainly obtained by the following methods, which specifically include: The axes are parallel, as shown in Figure 6, the optical center points of the left and right imaging wafers are OL and OR, and the projection points of the object point P in the stereoscopic space scene on the imaging planes of the two imaging wafers are (x _I , f) and (x _r , f), f is the focal length of the imaging wafer, and the distance between the optical centers of the two wafers is b. Therefore, according to the above and Figure 6, the relationship between the parallax d and the depth z can be known. Therefore, the 3D image information can be constructed according to the information of the two projection points in the left-view image and the right-view image, and then the 3D annotation constructed according to this principle can be used. frame.

According to the position of the center point and the position coordinates of each corner point of the first lesion annotation frame and the second lesion annotation frame obtained by the above calculation, while calculating the parallax and depth map for three-dimensional display according to the left-view image and right-view image, calculate the left-view image. The parallax of each corresponding corner point and the corresponding center point in the image and the right-view image, and then calculate the depth value according to the camera wafer parameter focal length, the binocular wafer baseline distance and the parallax of the corresponding annotation frame, so as to obtain the depth map, which can reflect the annotation. The position of each pixel in the box from the camera wafer. Therefore, the three-dimensional image can be displayed through spatial calculation, and the lesion information can be marked in the 3D view of the 3D endoscope by the 3D annotation frame.

Finally, the binocular image can be seen through the polarizer. The polarized glasses use the principle that light has a "vibration direction" to decompose the original image collected. By dividing the image into two groups of vertically polarized light and horizontally polarized light, The polarized glasses type needs to add a phase retardation film on the LCD panel to generate orthogonal polarization directions. Using the depolarization of the polarized glasses, the left and right eyes receive images with different polarization directions, and after synthesis, a three-dimensional image that can be observed by the human eye is formed. .

For example, the annotation and display of the 3D annotation frame can be realized by the following methods. The two-dimensional detection frame of the left-view image and the right-view image can be directly mapped to a 3D frame, and the size of the two-dimensional detection frame of the left-view image and the right-view image can be regressed. to reproject.

For example, let the size information of the 3D detection frame be f={x, y, z, θ}, and use 6 eigenvalues to correlate the relationship between the 2D annotation frame and the 3D annotation frame in the process of 3D frame projection, that is, U _l1 , V _t , U _p , V _b , U _l2 , U _r , which represent the left, top, right, and bottom borders of the two-dimensional labeling frame of the left-view image and the left, top, and bottom borders of the two-dimensional detection frame of the right-view image, respectively. right border.

The calculation of other feature correlation values (such as depth information) is the same as the above, so it is not listed one by one, where θ is the included angle between the labeling frame and the imaging wafer. These multivariate equations are solved by the Gauss-Newton method. The three-dimensional 3D annotation frame is mapped through the position and size information of the two-dimensional annotation frame as shown in FIG. 7 .

Wherein, before the first lesion labeling frame and the second lesion labeling frame in the left-view image and the right-view image in this embodiment are projected to the three-dimensional to form the 3D labeling frame, the first lesion labeling frame and the second lesion labeling frame are firstly labelled. The purpose of calculating the degree of association is to determine whether the first lesion annotation frame and the second lesion annotation frame are annotation frames of the same target. Therefore, it is necessary to calculate the degree of association between the first lesion annotation frame and the second lesion annotation frame. If the degree of association is less than The default value is that the two callout boxes are the callout boxes of the same target. The degree of association may be the degree of feature matching of the two images, and the degree of association may also be the degree of coordinate coincidence of the first lesion labeling frame and the second lesion labeling frame (which may also be understood as the degree of coordinate matching). For example, assuming that the left and right view images are detected respectively, the output detection frame information is x _1i , x _2i , and the output categories of the first lesion annotation frame and the second lesion annotation frame are y ₁ , y ₂ , and the categories here include the lesion area. and non-diseased areas. So the squared distance between x1 and _x2 is _:

d _W =(x ₁ -x ₂ )T _W (x ₁ -x ₂ )

In the above formula, W is the parameter matrix W in the Mahalanobis distance. The structural similarity between the first lesion annotation frame and the second lesion annotation frame can be obtained through metric association learning, so the metric function is:

In the above formula, p is the detection frame corresponding to two images, p=2; s _{i, j} ∈ [0, 1], if y ₁ =y ₂ then s _{i, j} =1, otherwise s _{i, j} =0. In this embodiment, a metric function threshold is reasonably set to determine the degree of association between the first lesion labeling frame and the second lesion labeling, and then three-dimensional mapping is performed on the two labeling frames.

Through the auxiliary detection method in this embodiment, based on the left-view image and right-view image collected by an image acquisition module (such as a binocular 3D endoscope), two trained detection models can detect whether a lesion occurs in real time and quickly, In addition, the lesion image is further converted into a 3D image in real time, and the lesion area is marked in 3D in the 3D image and displayed in real time, which is convenient for doctors to wear 3D glasses to observe the detection area, and plays a certain auxiliary role in the doctor's diagnosis.

Embodiment 2:

This embodiment provides a 3D view auxiliary detection system, as shown in FIG. 8 , which includes: an image acquisition module 301 , a lesion detection module 302 , a view conversion module 303 , a 3D labeling unit 304 , and a real-time display module 305 . The image acquisition module 301 is used to acquire the left-view image and the right-view image of the area to be detected in real time; the lesion detection module 302 is used to perform lesion detection on the left-view image and the right-view image to obtain the first lesion annotation frame and the second lesion respectively. Labeling frame; the view conversion module 303 is used to perform 2D-3D conversion after the feature fusion of the left-view image and the right-view image to obtain a 3D view of the area to be detected; the 3D labeling unit 304 is used to label the frame according to the first lesion and the second lesion. Labeling box, the location of the lesion is 3D labeled in the 3D view to obtain the 3D lesion labeling view; the real-time display module 305 is configured to display the 3D lesion labeling view in real time. The specific method for converting a two-dimensional image into a 3D view is the same as that in the first embodiment, and details are not repeated here.

Among them, the image acquisition module 301 in this embodiment adopts a binocular 3D endoscope, and two parallel lenses can acquire a left-view image and a right-view image at the same time. The real-time display module 305 is a display screen used to display the 3D lesion annotation view in real time. The display effect is like a 3D movie. The doctor can clearly observe the lesion information in the area to be detected by wearing the adapted 3D glasses.

The 3D labeling unit 304 includes an association module 3041 and a labeling module 3042: the association module 3041 is used to calculate the association degree between the first lesion annotation frame and the second lesion annotation frame. The labeling frame is associated with the second lesion labeling frame to obtain a 3D labeling frame; the labeling module 3042 is used to label the 3D labeling frame in the 3D view to obtain a 3D lesion labeling view, and the area where the 3D labeling frame is located is the lesion area. The specific correlation calculation method and the 3D labeling method are also the same as those in the first embodiment, and will not be repeated here.

The lesion detection module 302 is provided with a pre-trained first lesion detection model and a second lesion detection model; the left-view image and the right-view image are respectively input into the pre-trained first lesion detection model and the second lesion detection model. In the model, a first lesion annotation frame and a second lesion annotation frame are obtained respectively; the first lesion annotation frame and the second lesion annotation frame respectively represent the lesion area in the left-view image and the right-view image.

Further, the auxiliary detection system of this embodiment further includes: a sample set acquisition module 306, a preprocessing module 307, and a training module 308; the sample set acquisition module 306 is used to acquire a normal image data set and a diseased image data set as training sample sets; The preprocessing module 307 is used to scale, rotate, flip and change the brightness of the images in the training sample set to expand the training sample set; the training module 308 is used to train the two initial YOLO network models using the expanded training sample set. The first lesion detection model and the second lesion detection model are described.

Through the auxiliary detection system of this embodiment, based on the left-view image and right-view image collected by the binocular 3D endoscope, two trained detection models can detect whether a lesion occurs in real time and quickly, and further detect the lesion in real time. The image is converted into a 3D image, and the lesion area is marked in 3D in the 3D image, and the marked 3D lesion annotation view is displayed on the display screen in real time, which is convenient for doctors to wear 3D glasses to observe the detection area, which plays a certain role in the doctor's diagnosis. Supporting role.

Embodiment three:

This embodiment provides a 3D view auxiliary detection device, as shown in FIG. 9 , which includes: a 3D endoscope 401 , a processor 402 and a display 403 , wherein the 3D endoscope 401 is used to acquire a left-view image of an area to be detected in real time and the right-view image; the processor 402 is configured to perform lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame respectively; the left-view image and the right-view image are fused after feature fusion to perform 2D- 3D conversion to obtain a 3D view of the area to be detected; according to the first lesion annotation frame and the second lesion annotation frame, 3D annotation is performed on the position of the lesion in the 3D view to obtain a 3D lesion annotation view; the display 403 is used to display the 3D lesion in real time. Lesion Annotation View.

Through the auxiliary detection device of this embodiment, based on the left-view image and right-view image collected by the binocular 3D endoscope, two trained detection models can detect whether a lesion occurs in real time and quickly, and further detect the lesion in real time. The image is converted into a 3D image, and the lesion area is marked in 3D in the 3D image, and the marked 3D lesion annotation view is displayed on the display screen in real time, which is convenient for doctors to wear 3D glasses to observe the detection area, which plays a certain role in the doctor's diagnosis. Supporting role.

Embodiment 4:

This embodiment provides a computer-readable storage medium, which includes a program, and the program can be executed by a processor to implement the auxiliary detection method provided in the first embodiment.

The above specific examples are used to illustrate the present invention, which are only used to help understand the present invention, and are not intended to limit the present invention. For those skilled in the art to which the present invention pertains, according to the idea of the present invention, several simple deductions, modifications or substitutions can also be made.

Claims (10)

1. a 3D view auxiliary detection method, is characterized in that, comprises: Obtain the left-view image and right-view image of the area to be detected in real time; Performing lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame, respectively; 2D-3D conversion is performed after feature fusion of the left-view image and the right-view image to obtain a 3D view of the area to be detected; According to the first lesion labeling frame and the second lesion labeling frame, perform 3D labeling on the position of the lesion in the 3D view to obtain a 3D lesion labeling view; The 3D lesion annotation view is displayed in real time. 2 . The auxiliary detection method according to claim 1 , wherein the 3D lesion is obtained by 3D marking the position of the lesion in the 3D view according to the first lesion labeling frame and the second lesion labeling frame. 3 . Callout views include: Calculate the degree of association between the first lesion annotation frame and the second lesion annotation frame, and if the association degree reaches a preset interval, associate the first lesion annotation frame and the second lesion annotation frame to obtain a 3D annotation frame; The 3D annotation frame is marked in the 3D view to obtain the 3D lesion annotation view, and the area where the 3D annotation frame is located is the lesion area. 3. The auxiliary detection method according to claim 2, wherein said performing lesion detection on said left-view image and right-view image to obtain a first lesion labeling frame and a second lesion labeling frame respectively comprises: Inputting the left-view image and the right-view image into the pre-trained first lesion detection model and the second lesion detection model, respectively, to obtain the first lesion annotation frame and the second lesion annotation frame; The label box and the second lesion label box represent the lesion area in the left-view image and the right-view image, respectively. 4. auxiliary detection method as claimed in claim 3, is characterized in that, described first lesion detection model and second lesion detection model are all obtained by following method training: Obtain normal image datasets and lesion image datasets as training sample sets; scaling, rotating, flipping and changing the brightness of the images in the training sample set to expand the training sample set; The expanded training sample set is input into the initial YOLO network model for multiple training to obtain the first lesion detection model and the second lesion detection model. 5. A 3D view auxiliary detection system, characterized in that, comprising: The image acquisition module is used to acquire the left-view image and right-view image of the area to be detected in real time; a lesion detection module, configured to perform lesion detection on the left-view image and right-view image to obtain a first lesion labeling frame and a second lesion labeling frame, respectively; A view conversion module, configured to perform 2D-3D conversion after feature fusion of the left-view image and the right-view image to obtain a 3D view of the area to be detected; a 3D labeling unit, configured to perform 3D labeling on the position of the lesion in the 3D view according to the first lesion labeling frame and the second lesion labeling frame to obtain a 3D lesion labeling view; The real-time display module is used to display the 3D lesion annotation view in real time. 6. auxiliary detection system as claimed in claim 5, is characterized in that, described 3D labeling unit comprises associative module and labeling module: The association module is used to calculate the degree of association between the first lesion annotation frame and the second lesion annotation frame, and if the association degree reaches a preset interval, the first lesion annotation frame and the second lesion annotation frame are calculated. The association gets the 3D callout frame; The labeling module is used for labeling the 3D labeling frame in the 3D view to obtain the 3D lesion labeling view, and the area where the 3D labeling frame is located is the lesion area. 7. The auxiliary detection system according to claim 5, wherein the lesion detection module is provided with a pre-trained first lesion detection model and a second lesion detection model; The performing lesion detection on the left-view image and the right-view image to obtain the first lesion labeling frame and the second lesion labeling frame respectively includes: Inputting the left-view image and the right-view image into the pre-trained first lesion detection model and the second lesion detection model, respectively, to obtain the first lesion annotation frame and the second lesion annotation frame; The label box and the second lesion label box represent the lesion area in the left-view image and the right-view image, respectively. 8. The auxiliary detection system of claim 5, further comprising: The sample set acquisition module is used to obtain the normal image data set and the lesion image data set as training sample sets; a preprocessing module for scaling, rotating, flipping and changing the brightness of the images in the training sample set to expand the training sample set; The training module is used for obtaining the first lesion detection model and the second lesion detection model by using the expanded training sample set to train two initial YOLO network models. 9. A 3D view auxiliary detection device, characterized in that, comprising: 3D endoscope for real-time acquisition of left-view and right-view images of the area to be detected; a processor, configured to perform lesion detection on the left-view image and the right-view image to obtain a first lesion labeling frame and a second lesion labeling frame respectively; perform feature fusion on the left-view image and the right-view image to perform 2D-3D Converting to obtain a 3D view of the area to be detected; 3D labeling the position of the lesion in the 3D view according to the first lesion labeling frame and the second lesion labeling frame to obtain a 3D lesion labeling view; A display for displaying the 3D lesion annotation view in real time. 10. A computer-readable storage medium, characterized by comprising a program executable by a processor to implement the method according to any one of claims 1-4.

Images