TWI817848B - Method for obtaining three-dimensional vein information based on binocular stereo vision - Google Patents

Abstract

A method for obtaining three-dimensional vein information based on binocular stereo vision. The method includes the following steps: photographing an arm of a patient to obtain a first infrared light image and a second infrared light image respectively; according to the first infrared light image The light image and the second infrared light image use a vein segmentation model to obtain multiple pieces of first vein position information and multiple pieces of second vein position information; based on the first vein position information and the second vein position information , obtain a first target vein position information, and a second target vein position information; and obtain a target vein three-dimensional information based on the first target vein position information, the second target vein position information, and an external parameter matrix. Medical staff correctly grasp the depth of injection to increase the chance of successful injection.

Description

Method for obtaining three-dimensional vein information based on binocular stereo vision

The present invention relates to a method of establishing a 3D model for computer graphics, and in particular, to a method of obtaining three-dimensional vein information based on binocular stereo vision.

Intravenous therapy (IV) is a medical method that injects liquid substances such as blood, medicinal solutions, and nutritional solutions directly into the veins. Since there are many superficial subcutaneous tissues in the human body, the intravenous injection process will be affected by factors such as small blood vessels, fat layer thickness, and skin pigment content. As a result, medical staff cannot find blood vessels during intravenous injection, so intravenous injection is difficult. Quite big.

However, even if the blood vessels can be clearly displayed in the arm, only two-dimensional information of the blood vessels can be obtained, and medical staff still cannot correctly grasp the depth of the injection. Therefore, experienced medical staff may not be able to successfully inject in one go. Failure of intravenous injection is not only It will increase the patient's pain and increase the frequency of medical disputes.

Therefore, the object of the present invention is to provide a method that can obtain three-dimensional intravenous data. A method for obtaining three-dimensional vein information based on binocular stereo vision.

Therefore, the method for obtaining three-dimensional vein information based on binocular stereo vision of the present invention is suitable for obtaining three-dimensional vein information of an arm of a patient, and is implemented by an imaging system, which includes a first infrared light photography module. A set, a second infrared light photography module, and a processing module electrically connected to the first infrared light photography module and the second infrared light photography module. The method includes the following steps.

The first infrared light photography module and the second infrared light photography module respectively capture the patient's arm to obtain a first infrared light image and a second infrared light image respectively.

The processing module uses a vein segmentation model for obtaining the location of veins in an infrared image based on the first infrared image and the second infrared image to obtain a plurality of strokes corresponding to the first infrared image. First vein position information of a plurality of veins, and a plurality of second vein position information respectively corresponding to the veins in the second infrared light image.

The processing module obtains first target vein position information corresponding to one of the target veins in the first infrared light image based on the first vein position information and the second vein position information, and a Second target vein position information corresponding to the target vein in the second infrared image.

The processing module is based on the first target vein position information, the second target vein position information, and an external parameter matrix related to the second infrared light photography module. array to obtain a target vein three-dimensional information related to the target vein.

The effect of the present invention is that: the processing module uses the vein segmentation model to obtain the first vein position information and the second vein position information based on the first infrared image and the second infrared image, and based on the The first target vein position information and the second target vein position information are obtained from the first target vein position information and the second target vein position information, and then based on the first target vein position information, the second target vein position information, and the external The parameter matrix can obtain the three-dimensional information of the target vein, allowing medical staff to correctly grasp the depth of injection to increase the probability of successful injection.

1:Imaging system

11: The first infrared light shooting module

12: The second infrared light shooting module

13: Processing module

100:arm

21~24: Equipment initialization program

31~36: Procedure for obtaining three-dimensional vein information

351~353: Sub-steps

361~362: Sub-steps

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: Figure 1 is a schematic diagram illustrating an implementation of the method for obtaining three-dimensional vein information based on binocular stereo vision of the present invention. An example of an imaging system; Figure 2 is a flowchart of an equipment initialization procedure of an embodiment of a vein three-dimensional information acquisition method based on binocular stereoscopic vision of the present invention; Figure 3 is a flowchart of an equipment initialization procedure of the present invention based on binocular stereoscopic vision One embodiment of the visual vein three-dimensional information acquisition method is a vein three-dimensional information acquisition procedure; Figure 4 is a flow chart to assist in explaining the sub-steps of step 35 in Figure 3; and Figure 5 is a flow chart to assist in explaining the sub-steps of step 36 in Figure 3 steps.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are designated with the same numbering.

Referring to Figure 1, an imaging system 1 is used to implement a method for obtaining three-dimensional vein information based on binocular stereo vision according to an embodiment of the present invention. The imaging system 1 includes a first infrared light shooting module 11, a focal length and The first infrared light photography module 11 is the same as the second infrared light photography module 12, and a processing module 13 that is electrically connected to the first infrared light photography module 11 and the second infrared light photography module 12.

In this embodiment, the first infrared light photography module 11 and the second infrared light photography module 12 are binocular cameras. The first infrared light photography module 11 and the second infrared light photography module 12 are The field of view (FOV) is, for example, 110 degrees, the focal length is, for example, 3 mm, the shooting distance is, for example, less than 20 cm, and the wavelength range is, for example, 780 nanometers to 940 nanometers. The first infrared light shooting module 11 and the The image captured by the second infrared light photography module 12 is a binocular image, but is not limited to this.

This embodiment of the method for obtaining three-dimensional vein information based on binocular stereo vision of the present invention includes an equipment initialization program and a three-dimensional vein information obtaining program.

Referring to Figures 1 and 2, the following describes the initial procedure of the equipment of this embodiment of the method for obtaining three-dimensional vein information based on binocular stereo vision of the present invention.

In step 21, the first infrared light shooting module 11 and the second infrared light The photography module 12 captures a target object respectively to obtain a plurality of first infrared light target images and a plurality of second infrared light target images.

It is worth noting that in this embodiment, the first infrared light photography module 11 captures, for example, 12 images of the first infrared light target, and the second infrared light photography module 12 captures, for example, 12 images of the second infrared light target. object image, but not limited to this.

In step 22, the first infrared light photography module 11 performs internal and external parameter correction based on the first infrared light target images, and the second infrared light photography module 12 performs internal and external parameter correction based on the second infrared light target images. Calibrate internal and external parameters.

In step 23, the corrected first infrared light photographing module 11 and the second infrared light photographing module 12 respectively photograph the target object to obtain a plurality of first infrared light corrected images and a plurality of second infrared light photographs. Infrared light corrected image.

It is worth noting that in this embodiment, the corrected first infrared light photography module 11 takes, for example, 12 first infrared light corrected images, and the corrected second infrared light photography module 12 takes, for example, 12 pictures. The second infrared light corrects the image, but is not limited to this.

In step 24 , the processing module 13 obtains the external parameter matrix based on the first infrared light correction images and the second infrared light correction images and an internal parameter matrix related to the second infrared light photography module 12 . parameter matrix.

其中，K為該內部參數矩陣，f為該第二紅外光拍攝模組12的焦距，[R _B ｜T _B ]為該外部參數矩陣，R _B 為旋轉(rotation)矩陣，T _B 為位移(translation)矩陣，(

,

,

)為該目標物在一世界坐標系的座標，(

,

)為該目標物在一對應該第二紅外光拍攝模組12的第二成像平面座標系的座標。 The external parameter matrix is obtained by the following formula:

Among them, K is _the internal parameter matrix, f is the focal length of the second infrared light shooting module 12, [ RB | T _B ] is the external parameter matrix, R _B is the rotation matrix, and T _B is the displacement ( translation) matrix, (

,

,

) is the coordinate of the target object in a world coordinate system, (

,

) is the coordinate of the target object corresponding to the second imaging plane coordinate system of the second infrared light photography module 12 .

及

，並根據該第一紅外光拍攝模組11的焦距以相似三角形獲得該目標物在拍攝時的

。對於每一第二紅外光校正影像，該處理模組13根據該第二紅外光校正影像，獲得(

,

)。最後根據每一第一紅外光校正影像及第二紅外光校正影像所獲得的(

,

,

)、(

,

)及已知的該第二紅外光拍攝模組12的焦距，以線性回歸求得該外部參數矩陣。 Specifically, the processing module 13 uses a first infrared light photography module coordinate system related to the first infrared light photography module 11 as the world coordinate system, that is, the focus of the first infrared light photography module 11 as the origin of the world coordinate system. For each first infrared light correction image, the processing module 13 obtains the image of the target object when shooting based on the first infrared light correction image.

and

, and according to the focal length of the first infrared light shooting module 11, a similar triangle is used to obtain the image of the target object during shooting.

. For each second infrared light correction image, the processing module 13 obtains (

,

). Finally, (

,

,

), (

,

) and the known focal length of the second infrared light photography module 12, use linear regression to obtain the external parameter matrix.

Referring to Figures 1 and 3, the following describes the procedure for obtaining three-dimensional vein information according to this embodiment of the method for obtaining three-dimensional vein information based on binocular stereo vision of the present invention.

In step 31, the first infrared light photography module 11 and the second infrared light photography module 12 respectively photograph an arm 100 of a patient to obtain a first infrared light image and a second infrared light image respectively. image.

In step 32 , the processing module 13 uses an arm angle model for obtaining the arm angle in the binocular image to obtain an angle related to the arm 100 based on the first infrared image and the second infrared image. information.

It is worth noting that in this embodiment, the arm angle model is established by the processing module 13 using a machine learning algorithm based on multiple arm angle training data, where each arm angle training data includes two binoculars. image, and an angle annotation. The machine learning algorithm is, for example, Convolutional Neural Network (CNN), Residual neural network (ResNet), Fully Convolutional Network (Fully Convolutional Network) , FCN), or U-Net and other neural network algorithms, but are not limited to this.

In step 33, the processing module 13 determines whether the angle information meets a predetermined condition. When it is determined that the angle information meets the predetermined condition, the process proceeds to step 34; and when it is determined that the angle information does not meet the predetermined condition, steps 31 to 33 are repeated.

It is worth noting that in this embodiment, the angle information includes a deflection Angle (deflection angle), the predetermined condition is that the deflection angle is less than a threshold, the threshold is, for example, 90 degrees, but is not limited to this.

It should be noted that the processing module 13 obtains the angle information and determines whether the angle information meets the predetermined conditions to confirm whether the patient's arm 100 is correctly placed on the shooting platform. In other implementations, it may not be necessary. Confirm whether the patient's arm 100 is correctly placed on the imaging platform, that is, steps 32 and 33 are omitted and step 34 is performed directly.

In step 34 , the processing module 13 uses a vein segmentation model for obtaining vein positions in an infrared image based on the first infrared image and the second infrared image to obtain a plurality of strokes corresponding to the third infrared image. First vein position information of a plurality of veins in an infrared light image, and a plurality of second vein position information respectively corresponding to the veins in the second infrared light image.

It is worth noting that in this embodiment, each first vein position information includes a plurality of pixel positions of the corresponding vein in the first infrared image, and each second vein position information includes a corresponding vein at a plurality of pixel positions in the first infrared image. Multiple pixel positions of the second infrared image. The vein segmentation model is established by the processing module 13 using the machine learning algorithm based on multiple vein segmentation training data, where each vein segmentation training data includes an infrared light vein image marked with at least one vein area.

In step 35, the processing module 13 obtains a pair corresponding to the first infrared light image based on the first vein position information and the second vein position information. First target vein position information of one of the target veins of the isovenous vein, and second target vein position information corresponding to the target vein in the second infrared light image.

Referring to Figure 4, step 35 includes the following sub-steps: In sub-step 351, the processing module 13 obtains a plurality of records corresponding to the first veins based on the first vein position information and the second vein position information. The first vein characteristic information of the position information, and a plurality of second vein characteristic information respectively corresponding to the second vein position information.

In sub-step 352, the processing module 13 obtains the target vein from the veins based on the first vein characteristic information and the second vein characteristic information.

It is worth noting that in this embodiment, each first vein characteristic information includes a first vein aperture, a first vein blood flow direction, and a first vein blood flow direction normal vector. Each second vein characteristic information includes a second vein aperture, a second vein blood flow direction, and a second vein blood flow direction normal vector. The first vein aperture and the second vein aperture are the widths of the corresponding veins in the edge transverse direction or the edge longitudinal direction. For each vein, the processing module 13 obtains a center of the edge transverse direction or edge longitudinal coordinates of the vein, and then uses the center to an end point of the vein to obtain the first venous blood flow direction and the second venous blood flow direction. flow direction, after obtaining the first venous blood flow direction normal vector perpendicular to the first venous blood flow direction, and the second venous blood flow direction normal vector perpendicular to the second venous blood flow direction, according to the first The venous aperture, the first venous blood flow direction, the normal vector of the first venous blood flow direction, the second venous aperture, the second venous blood flow direction The vein blood flow direction and the normal vector of the second vein blood flow direction are used to estimate a vein depth of the vein using, for example, a Gaussian Mixture Model, and finally the target vein is obtained based on the vein depths corresponding to the veins. But it is not limited to this.

In sub-step 353, the processing module 13 obtains the first target vein position information and the second target vein position information corresponding to the target vein.

For example, in step 34, the processing module 13 obtains a first vein position information A corresponding to a vein A, a first vein position information B corresponding to a vein B, and the corresponding A first vein position information C of a vein C, and a second vein position information A corresponding to the vein A, a second vein position information B corresponding to the vein B, and a second vein position information B corresponding to the vein B are obtained from the second infrared image. A second vein location information C that should be vein C.

In sub-step 351, the processing module 13 respectively determines the first vein position information A, the first vein position information B, the first vein position information C, the second vein position information A, and the second vein position information. Information B, and the second vein position information C, obtain a first vein characteristic information A, a first vein characteristic information B, a first vein characteristic information C, a second vein characteristic information A, and a second vein characteristic Information B, and a second vein characteristic information C.

In sub-step 352, the processing module 13 obtains a vein depth A according to the first vein characteristic information A and the second vein characteristic information A, and obtains a vein depth A according to the first vein characteristic information B and the second vein characteristic information B. A vein depth B, according to the first The vein characteristic information C and the second vein characteristic information C obtain a vein depth C. Among them, the vein depth B is the best. Therefore, the processing module 13 selects the vein B as the target vein from the veins A, B, and C, and in sub-step 353, the processing module 13 selects the vein B as the target vein. The first vein position information B is used as the first target vein position information, and the second vein position information B is used as the second target vein position information.

In step 36, the processing module 13 obtains a target vein three-dimensional information related to the target vein based on the first target vein position information, the second target vein position information, and the external parameter matrix. The target vein three-dimensional information is The information includes target vein depth information related to the target vein.

It should be particularly noted that in the initialization process of the device in this embodiment, the first infrared light photography module 11 and the second infrared light photography module 12 perform internal and external parameter correction, so the external parameter matrix has Corrected, in other embodiments, the device initialization procedure may be skipped and the vein three-dimensional information acquisition procedure may be directly performed, then the external parameter matrix has not been corrected.

Referring to FIG. 5 , step 35 includes the following sub-steps: In sub-step 361 , the processing module 13 obtains a plurality of locations corresponding to the first infrared based on the first target vein position information and the second target vein position information. The first target vein coordinates of the first imaging plane coordinate system of the light photography module 11 and a plurality of second target vein coordinates located corresponding to the second imaging plane coordinate system of the second infrared light photography module 12 .

It is worth noting that in this embodiment, the processing module 13 performs a feature matching (Feature Matching) and an image stitching (image stitching) on the first target vein position information and the second target vein position information. , and a block detection to obtain the first target vein coordinates and the second target vein coordinates, wherein the feature point matching, the image stitching, and the block detection are based on scale-invariant feature transformation (Scale-invariant feature transform, SIFT) algorithm, but is not limited to this.

In sub-step 362, the processing module 13 obtains the vein depth information based on the first target vein coordinates, the second target vein coordinates, and the external parameter matrix.

其中，

為每一靜脈深度，(

,

)為每一第一目標靜脈座標，(

,

)為每一第二目標靜脈座標，f為該第一紅外光拍攝模組11及該第二紅外光拍攝模組12的焦距，

、

、

、

、

、

、

、

為該外部參數矩陣的參數。 It is worth noting that the processing module 13 obtains the vein depth information according to the following formula:

in,

For each vein depth, (

,

) is the coordinate of each first target vein, (

,

) is the coordinate of each second target vein, f is the focal length of the first infrared light photography module 11 and the second infrared light photography module 12,

,

,

,

,

,

,

,

are the parameters of this external parameter matrix.

It should be noted again that after obtaining the three-dimensional information of the target vein, the processing module 13 can use the object tracking algorithm to follow up the first infrared photography module 11 And the image captured by the second infrared light photography module 12 tracks the target vein.

To sum up, the method of obtaining three-dimensional vein information based on binocular stereo vision of the present invention uses the processing module 13 to determine whether the angle information meets the predetermined conditions according to the angle information, so as to confirm whether the patient's arm 100 is correct. Place it on the shooting platform, and use the vein segmentation model to obtain the first vein position information and the second vein position information based on the first infrared light image and the second infrared light image through the processing module 13, and based on The first vein position information and the second vein position information obtain the first target vein position information and the second target vein position information, and then based on the first target vein position information, the second target vein position information, and the The external parameter matrix obtains the three-dimensional information of the target vein, allowing medical personnel to correctly grasp the depth of injection to improve the probability of successful injection, so the purpose of the present invention can indeed be achieved.

However, the above are only examples of the present invention and should not be used to limit the scope of the present invention. All simple equivalent changes and modifications made based on the patent scope of the present invention and the content of the patent specification are still within the scope of the present invention. within the scope covered by the patent of this invention.

31~36: Procedure for obtaining three-dimensional vein information

Claims (10)

A method of obtaining three-dimensional vein information based on binocular stereo vision, suitable for obtaining three-dimensional vein information of an arm of a patient, is implemented by an imaging system, which includes a first infrared light shooting module, a a second infrared light photography module, and a processing module electrically connected to the first infrared light photography module and the second infrared light photography module. The method includes the following steps: the first infrared light photography module and the The second infrared light photography module captures the patient's arm respectively to obtain a first infrared light image and a second infrared light image; the processing module uses the first infrared light image and the second infrared light image. Light image, using a vein segmentation model for obtaining the position of veins in an infrared light image, obtaining first vein position information of multiple strokes corresponding to multiple veins in the first infrared light image, and multiple strokes respectively corresponding to the first vein position information in the first infrared light image. The second vein position information of the veins in the second infrared image; the processing module obtains a corresponding position information in the first infrared image based on the first vein position information and the second vein position information The first target vein position information of one of the target veins, and the second target vein position information corresponding to the target vein in the second infrared image; and the processing module is based on the first target vein. The position information, the second target vein position information, and an external parameter matrix related to the second infrared light photography module are used to obtain a target vein three-dimensional information related to the target vein. Acquisition of three-dimensional vein information based on binocular stereo vision as described in request 1 The method further includes the following steps between the steps of obtaining the first infrared light image and the second infrared light image, and the steps of obtaining the first vein position information and the second vein position information: the processing module According to the first infrared image and the second infrared image, an arm angle model used to obtain the arm angle in the binocular image is used to obtain angle information related to the arm; the processing module determines whether the angle information Meets a predetermined condition; and when it is determined that the angle information does not meet the predetermined condition, the processing module repeats the steps of obtaining the first infrared light image and the second infrared light image, the steps of obtaining the angle information, and determining the The step of whether the angle information meets the predetermined condition until the angle information meets the predetermined condition; wherein, when it is determined that the angle information meets the predetermined condition, the first vein position information and the second vein position information are obtained steps. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 2, wherein the angle information includes a deflection angle, and the predetermined condition is that the deflection angle is less than a threshold. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 2 further includes the following steps before obtaining the angle information: the processing module uses a machine learning algorithm based on multiple arm angle training data , to establish the arm angle model, each arm angle training data includes two binocular images and an angle label. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 1, wherein the step of obtaining the first target vein position information and the second target vein position information includes the following sub-steps: The processing module obtains a plurality of pieces of first vein characteristic information corresponding to the first vein position information and a plurality of pieces of first vein characteristic information respectively corresponding to the second vein based on the first vein position information and the second vein position information. The second vein characteristic information of the position information; the processing module obtains the target vein from the veins based on the first vein characteristic information and the second vein characteristic information; and the processing module obtains the target vein correspondence The first target vein position information and the second target vein position information. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 5, wherein in the sub-step of obtaining the first vein characteristic information and the second vein characteristic information, each first vein characteristic information It includes a first venous aperture, a first venous blood flow direction, and a first venous blood flow direction normal vector, and each second vein characteristic information includes a second venous aperture, a second venous blood flow direction, and a The second venous blood flow direction normal vector, for each vein, the processing module is based on the first venous aperture, the first venous blood flow direction, the first venous blood flow direction normal vector, the second venous aperture, the The second venous blood flow direction and the normal vector of the second venous blood flow direction are used to estimate a vein depth of the vein, and then the target vein is obtained based on the vein depths corresponding to the veins. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 1, wherein the following steps are further included before obtaining the first infrared light image and the second infrared light image: the first infrared light photography The module and the second infrared light shooting module respectively capture a target object to obtain multiple first infrared light target object images. image, and a plurality of second infrared light target images; the first infrared light photography module performs internal and external parameter correction based on the first infrared light target images, and the second infrared light photography module performs internal and external parameter correction based on the first infrared light target images. The two infrared light target images are calibrated for internal and external parameters; the corrected first infrared light shooting module and the second infrared light shooting module respectively shoot the target object to obtain multiple first infrared light corrected images, and a plurality of second infrared light correction images; and the processing module is based on the first infrared light correction images, the second infrared light correction images, and an internal parameter matrix related to the second infrared light shooting module , obtain the external parameter matrix. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 1, wherein in the step of obtaining the three-dimensional information of the target vein, the three-dimensional information of the target vein includes target vein depth information related to the target vein. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 8, wherein the step of obtaining the three-dimensional information of the target vein includes the following sub-steps: the processing module is based on the first target vein position information and the second The target vein position information is obtained by obtaining a plurality of first target vein coordinates located in a first imaging plane coordinate system corresponding to the first infrared light shooting module, and a plurality of second target vein coordinates located in a corresponding position corresponding to the second infrared light shooting module. The second target vein coordinates of the imaging plane coordinate system; and the processing module obtains the target vein depth information based on the first target vein coordinates, the second target vein coordinates, and the external parameter matrix News. The method for obtaining three-dimensional vein information based on binocular stereo vision as described in claim 1 further includes the following steps before obtaining the first vein position information and the second vein position information: the processing module is based on multiple The vein segmentation training data uses a machine learning algorithm to establish the vein segmentation model. Each vein segmentation training data includes an infrared light vein image marked with at least one vein area.

Images