KR102631241B1 - Method for generating a three-dimensional blood vessel model from two dimensional angiographic images using deep learning - Google Patents

Abstract

The present invention relates to a method for generating a three-dimensional blood vessel model from two-dimensional X-ray angiography images using machine learning.
The method of generating a 3D blood vessel model from a 2D X-ray angiography image of a patient using a computer according to the present invention includes receiving a pair of 2D X-ray angiography images taken at different angles. A step of using a machine learning program to generate a pair of images showing the entrance and exit of major blood vessels for each image from the pair of two-dimensional X-ray angiography images taken at different angles. Includes. Additionally, by processing the image showing the inlet and outlet of the pair of major blood vessels, the major blood vessel area from the inlet to the outlet can be automatically segmented. In addition, each center line of the pair of segmented major blood vessel regions is obtained, corresponding points are obtained for the center lines of the pair of segmented major blood vessel regions, the diameter of the blood vessel region at each corresponding point is obtained, and a back projection algorithm is used. Using , the coordinates in three-dimensional space and the diameter in three-dimensional space for the corresponding point of the center line of the pair of segmented major blood vessel regions can be obtained.

Description

Method for generating a three-dimensional blood vessel model from two dimensional angiographic images using deep learning}

The present invention relates to a method for generating a three-dimensional blood vessel model from two-dimensional X-ray angiography images using machine learning. More specifically, it relates to a method of generating a three-dimensional blood vessel model by automatically extracting at least two feature points from an angiography image using machine learning.

Coronary artery diseases (CAD), which are caused by narrowing of blood vessels, are one of the major causes of death (Non-patent Document 1). The severity of coronary artery stenosis is primarily assessed by examining the cross-sectional area inside the blood vessel on medical imaging. However, it is known that the blood flow flowing through the coronary artery is not necessarily related to the visually observed shape of the blood vessel (Non-patent Documents 2, 3).

One of the methods of determining coronary blood flow is to insert a pressure sensor into the coronary artery and measure the drop in blood pressure along the stenosis area (Non-patent Document 4). The drop in blood pressure is expressed as fractional flow reserve (FFR), which is defined as the ratio of pressure before and after the stenosis of the coronary artery (Non-patent Document 4). The narrow flow reserve can be calculated by analytical or numerical methods (Non-patent Documents 5 - 8). A 3D model reconstructing the inner volume of a blood vessel is used to calculate the FFR value, and therefore the accuracy of the 3D model is important in the numerical calculation of the FFR.

X-ray coronary angiography (XCA) images are one of the medical images used for three-dimensional reconstruction of coronary arteries. X-ray coronary angiography images are widely used in clinical practice because they provide relatively clear boundaries of the coronary arteries. However, X-ray coronary angiography images are two-dimensional images, and it is not easy to reconstruct a three-dimensional shape using two-dimensional images taken at different angles. The back-projection method is known to be a frequently used method for reconstructing a three-dimensional shape using a two-dimensional X-ray coronary angiography image (Non-patent Document 9). In the back projection method, arbitrary points of the three-dimensional model are determined by triangulation (Non-Patent Document 10), and the triangulation method links the center lines of blood vessels in two-dimensional X-ray coronary angiography images taken at different angles. do. However, the difficulty in reconstructing the volume inside a three-dimensional blood vessel in this method is segmenting the blood vessel region in two-dimensional It determines the relationship. Although epipolar constraints are commonly used to determine correspondence relationships (Non-Patent Documents 11, 12), it is known that epipolar constraints do not always result in unique correspondence relationships (13, 14). This is because overlap and reduction of two-dimensional images occur due to the relative direction and position of blood vessels with respect to the imaging device.

1. Virani, S.S., Alonso, A., Benjamin, E.J., Bittencourt, M.S., Callaway, C.W., Carson, A.P., Chamberlain, A.M., Chang, A.R., Cheng, S., Delling, F.N., Djousse, L., Elkind. , M.S.V., Ferguson, J.F., Fornage, M., Khan, S.S., Kissela, B.M., Knutson, K.L., Kwan, T.W., Lackland, D.T., Lewis, T.T., Lichtman, J.H., Longenecker, C.T., Loop, M.S., Lutsey; P.L., Martin, S.S., Matsushita, K., Moran, A.E., Mussolino, M.E., Perak, A.M., Rosamond, W.D., Roth, G.A., Sampson, U.K.A., Satou, G.M., Schroeder, E.B., Shah, S.H., Shay, C.M. , Spartano, N.L., Stokes, A., Tirschwell, D.L., VanWagner, L.B., Tsao, C.W., American Heart Association Council on, E., Prevention Statistics, C., Stroke Statistics, S.: Heart Disease and Stroke Statistics-2020 Update: A Report From the American Heart Association. Circulation 141, e139-e596 (2020) 2. Toth, G., Hamilos, M., Pyxaras, S., Mangiacapra, F., Nelis, O., De Vroey, F., Di Serafino, L., Muller, O., Van Mieghem, C., Wyffels, E., Heyndrickx, G.R., Bartunek, J., Vanderheyden, M., Barbato, E., Wijns, W., De Bruyne, B.: Evolving concepts of angiogram: fractional flow reserve discordances in 4000 coronary stenoses. Eur Heart J 35, 2831-2838 (2014) 3. Park, S.J., Kang, S.J., Ahn, J.M., Shim, E.B., Kim, Y.T., Yun, S.C., Song, H., Lee, J.Y., Kim, W.J., Park, D.W., Lee, S.W., Kim, Y.H. , Lee, C.W., Mintz, G.S., Park, S.W.: Visual-functional mismatch between coronary angiography and fractional flow reserve. JACC Cardiovasc Interv 5, 1029-1036 (2012) 4. Corcoran, D., Hennigan, B., Berry, C.: Fractional flow reserve: a clinical perspective. Int J Cardiovasc Imaging 33, 961-974 (2017) 5. Lee, K.E., Lee, S.H., Shin, E.S., Shim, E.B.: A vessel length-based method to compute coronary fractional flow reserve from optical coherence tomography images. Biomed Eng Online 16, 83 (2017) 6. Lee, K.E., Ryu, A.J., Shin, E.S., Shim, E.B.: Physiome approach for the analysis of vascular flow reserve in the heart and brain. Pflugers Arch 469, 613-628 (2017) 7. Lee, K.E., Kim, G.T., Lee, J.S., Chung, J.H., Shin, E.S., Shim, E.B.: A patient-specific virtual stenotic model of the coronary artery to analyze the relationship between fractional flow reserve and wall shear stress. Int J Cardiol 222, 799-805 (2016) 8. Kwon, S.S., Chung, E.C., Park, J.S., Kim, G.T., Kim, J.W., Kim, K.H., Shin, E.S., Shim, E.B.: A novel patient-specific model to compute coronary fractional flow reserve. Prog Biophys Mol Biol 116, 48-55 (2014) 9. Cimen, S., Gooya, A., Grass, M., Frangi, A.F.: Reconstruction of coronary arteries from X-ray angiography: A review. Med Image Anal 32, 46-68 (2016) 10. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2004) 11. Delaere, D., Smets, C., Suetens, P., Marchal, G., Van de Werf, F.: Knowledge-based system for the three-dimensional reconstruction of blood vessels from two angiographic projections. Med Biol Eng Comput 29, NS27-36 (1991) 12. Dumay, A.M., Reiber, J.C., Gerbrands, J.J.: Determination of optimal angiographic viewing angles: basic principles and evaluation study. IEEE TransMed Imaging 13, 13-24 (1994) 13. Zhang, Z., Deriche, R., Faugeras, O., Luong, Q.-T.: A robust technique formatting two uncalibrated images through the recovery of the unknown epipolar geometry. Artif Intell 78, 87-119 (1995) 14. Cong, W., Yang, J., Ai, D., Chen, Y., Liu, Y., Wang, Y.: Quantitative Analysis of Deformable Model-Based 3-D Reconstruction of Coronary Artery From Multiple Angiograms. IEEE Trans Biomed Eng 62, 2079-2090 (2015) 15. Frangi, A.F., Niessen, W.J., Vincken, K.L., Viergever, M.A.: Multiscale vessel enhancement filtering. In: Wells, W.M., Colchester, A., Delp, S. L. (ed.) Medical Image Computing and Computer-Assisted Intervention - MICCAI'98, vol. 1496, pp. 130-137. Springer Verlag, Berlin, Germany (1998) 16. Van Uitert, R., Bitter, I.: Subvoxel precise skeletons of volumetric data based on fast marching methods. Med Phys 34, 627-638 (2007) 17. Chung, J.H., Lee, K.E., Nam, C.W., Doh, J.H., Kim, H.I., Kwon, S.S., Shim, E.B., Shin, E.S.: Diagnostic Performance of a Novel Method for Fractional Flow Reserve Computed from Noninvasive Computed Tomography Angiography (NOVEL-FLOW Study). Am J Cardiol 120, 362-368 (2017)

The purpose of the present invention is to provide a new method for generating a three-dimensional blood vessel model using a pair of two-dimensional X-ray angiography images taken at different angles. In particular, the present invention solves the problem of overlap and reduction of two-dimensional images due to the relative direction and position of blood vessels with respect to the imaging device.

The present invention provides a method for determining the entrance and exit of blood vessels in an angiography image using a machine learning method. The present invention provides a new method for segmenting blood vessel regions in two-dimensional X-ray coronary angiography images taken at different angles and automatically determining corresponding points with respect to the center line of each segmented two-dimensional blood vessel region. Additionally, the present invention provides a method of determining corresponding points between center lines of each segmented two-dimensional blood vessel region using a template model.

The method of generating a 3D blood vessel model from a 2D X-ray angiography image of a patient using a computer according to the present invention includes receiving a pair of 2D X-ray angiography images taken at different angles. and generating a pair of images showing the entrance and exit of major blood vessels for each image from the pair of two-dimensional X-ray angiography images taken at different angles using a machine learned program; , processing the image showing the inlet and outlet of the pair of major blood vessels to segment the major blood vessel region from the inlet to the outlet, and calculating the center line of each of the pair of segmented major blood vessel regions, Obtaining corresponding points with respect to the center line of a pair of segmented major blood vessel regions, calculating the diameter of the blood vessel region at each corresponding point, and calculating the corresponding points of the center line of the pair of segmented major blood vessel regions using a back projection algorithm. It includes the step of calculating the coordinates in three-dimensional space and the diameter in three-dimensional space.

In some embodiments, the machine learned program is a deep learning network learned with a first labeled image including major blood vessels and a background and a second labeled image including the inlet and outlet of the major blood vessels.

Additionally, in some embodiments, the major blood vessels may include the left anterior descending coronary artery (LAD), left circumflex coronary artery (LCX), and right coronary artery (RCA).

In addition, in some embodiments, the step of finding a corresponding point with respect to the center line of the pair of segmented major blood vessel regions involves determining the major blood vessel regions using a template for the major blood vessels previously generated using a 3D CT image. It is desirable to find the corresponding point with respect to the center line.

According to the present invention, a new method for generating a three-dimensional blood vessel model from a pair of two-dimensional X-ray angiography images taken at different angles is provided.

In particular, a method of generating a pair of images showing the entrance and exit of major blood vessels for each image from a pair of two-dimensional X-ray angiography images taken at different angles using a machine learning program is provided. do. Therefore, it is possible to automatically create a 3D blood vessel model using a pair of 2D X-ray angiography images.

According to the present invention, a new method of generating a three-dimensional blood vessel model from a pair of two-dimensional X-ray angiography images using a pre-generated three-dimensional blood vessel template is provided. Using a template model, it is possible to automatically obtain corresponding points of the center lines of major blood vessels in two-dimensional X-ray angiography images taken at approximately different angles.

Figure 1 is a schematic diagram of machine learning image data
Figure 2 is a schematic diagram of the machine learning network
Figure 3 is an explanatory diagram showing the state in which the inlet and outlet of major blood vessels are output by inputting a two-dimensional X-ray angiography image into a machine learning network.
Figure 4 is a schematic diagram explaining how to create a template
Figure 5 is an explanatory diagram of matching the template and the shooting angle of the angiography image.
Figure 6 is a schematic diagram illustrating a method of calculating the corresponding point of the center line in two-dimensional X-ray angiography images taken at different angles.
Figure 7 is a schematic diagram showing a three-dimensional blood vessel model generated by the method according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

How to determine the entrance and exit of the internal areas of major blood vessels

A deep learning method is applied to automatically determine the inlet and outlet positions of major blood vessels in two-dimensional X-ray coronary angiography images taken at different angles. The major blood vessels are the left anterior descending coronary artery (LAD), left circumflex coronary artery (LCX), and right coronary artery (RCA). To apply the deep learning method, first create a learning image to be used in the deep learning network. Learning images are created by labeling coronary angiography images. Figure 1 shows a labeled image. As shown in Figure 1, labeling uses a learning image (first labeling image) that divides a two-dimensional X-ray coronary angiography image into two classes (background and major blood vessels) and seven classes (background, major blood vessels, Prepare learning images (second labeling images) divided into main vessel inlet, main vessel outlet, secondary vessel outlet, branch, and catheter. For the entrance of major blood vessels, the end of the catheter was labeled in the image before the contrast agent was injected, and for the outlet of the major blood vessel, the location where the contrast agent was administered was labeled in the image corresponding to the end of one heartbeat cycle in the image after the contrast agent was injected.

As a deep running network, U-net with an encoder (Resnet) and decoder as shown in Figure 2 was used. To learn the deep learning network, learning was performed using 293 learning images for each major blood vessel, that is, 293 learning images for LAD, 204 learning images for LCX, and 206 learning images for RCA. In order to learn a deep running network for outputting the inlet and outlet of major blood vessels, two class learning images in which major blood vessels and background were labeled were first trained, and seven class learning images in which the inlet and outlet of major blood vessels were labeled were additionally trained. . The deep running network learned in this way predicts and outputs the entrance and exit only for the major blood vessel image pixels that were first learned.

Figure 3 shows the results in which the inlet and outlet of major blood vessels are automatically output when a test image is input to the deep running network learned as above. In the test image, it can be seen that the inlet is marked at the end of the catheter, and the outlet is only marked at the end of the major blood vessel.

Two-dimensional vessel segmentation for X-ray coronary angiography images.

Once the inlet and outlet of the major blood vessel are determined for the two-dimensional X-ray coronary angiography image, segmentation is performed from the inlet to the outlet of the major blood vessel region. A Frangi filter is applied as a method of segmenting the area where the entrance and exit are marked in the image into which the contrast agent is injected (Non-patent Document 15). A fast-marching algorithm was applied to determine the center line of the segmented two-dimensional blood vessel region (Non-patent Document 16). The fast marching algorithm is applied to find the shortest path from the entrance to the exit using a velocity weight proportional to the distance to the nearest wall. The diameter of a blood vessel is obtained by averaging the distance from a point on each center line to two boundaries in the perpendicular direction. The diameter obtained from the segmented two-dimensional image is converted to the diameter of the three-dimensional model that is later reconstructed.

Determination of registration point of 2D image centerline using coronary artery template model

To overcome the problem of vasoconstriction that occurs when determining the registration point using two-dimensional X-ray coronary angiography images taken at different angles, a coronary artery template model is used.

First, create template models for major blood vessels such as LAD, LCX, and RCA. The template model used cardiac CT image volume data used in the research results conducted in Non-Patent Document 17. Figure 4 shows examples of models created using cardiac CT volume image data. In FIG. 4, the template model obtains a three-dimensional model of the coronary artery and major blood vessels by segmenting 10 CT image data and averages them to create a template. Specifically, the process of creating a template is as follows. First, the scale of the 10 3D coronary artery models is converted to have a length of 1, and then a plurality of points (for example, 100) are generated at equal intervals along the center line. Next, the position of the starting point for each major blood vessel is moved in parallel to match, and the average coordinate of the points created along the center line is calculated to create the center line of the template model.

Next, the template model is used to obtain the center line registration point to reconstruct the three-dimensional shape of the main blood vessel with a pair of two-dimensional images segmented from a pair of two-dimensional X-ray coronary angiography images taken at different angles. Explain how.

First, the template model is rotated to the same angle as the shooting angle when angiography is performed on the segmented two-dimensional image. Figure 5 shows an example in which the template is rotated to the same angle as the photographed angle.

Next, the distance from the starting point is calculated for each point created along the center line using the two-dimensional coordinates of the rotated template model (example in FIG. 5) and then expressed as a ratio to the total length.

The matching point in the angiography image is selected using the ratio of the same point on the center line to the total length calculated in two-dimensional coordinates for the two shooting angles. Figure 6 is an example of such a selection method. At this time, the starting and ending points in the angiography image are the same identifiable points used when creating the template model. A three-dimensional model is created using the matching point determined in this way. Figure 7 is an example of a three-dimensional model created by determining the matching point in this way.

Creation of 3D models of major blood vessels

By segmenting a pair of two-dimensional Non-patent document 9).

Briefly, each center line point in three-dimensional space is determined from the geometric relationship between the position of the X-ray source at two angles and the coordinates of the two-dimensional image at two angles. First, an imaginary line is formed between the X-ray source and the two-dimensional image at the first imaging angle. Next, the closest point to the position corresponding to the projection of the two-dimensional image taken at a different angle is searched on the virtual line to determine three-dimensional coordinates. Next, after determining the 3D coordinates for the matching points of the center line, the diameters of blood vessels calculated in the 2D image segmentation step for the matching points of the center line are converted into 3D diameters.

Claims (4)

A method of generating a three-dimensional blood vessel model from a patient's two-dimensional X-ray angiography image using a computer,
receiving a pair of two-dimensional X-ray angiography images taken at different angles;
Generating a pair of images showing the entrance and exit of major blood vessels for each image from the pair of two-dimensional X-ray angiography images taken at different angles using a machine learned program;
Processing the image showing the inlet and outlet of the pair of major blood vessels to segment the major blood vessel area from the inlet to the outlet;
Obtaining the center line of each of the pair of segmented major blood vessel regions;
Obtaining corresponding points with respect to the center line of the pair of segmented major blood vessel regions and calculating the diameter of the blood vessel region at each corresponding point;
Comprising the step of obtaining coordinates and diameters in three-dimensional space for corresponding points of the center lines of the pair of segmented major blood vessel regions using a back projection algorithm,
The step of finding a corresponding point with respect to the center line of the pair of segmented major blood vessel regions includes obtaining a corresponding point with respect to the center line of the major blood vessel region using a template for the major blood vessel previously created using a 3D CT image. Method for generating three-dimensional vascular models from dimensional x-ray angiography images. According to paragraph 1,
The machine learned program is a deep learning network learned with a first labeling image including major blood vessels and background and a second labeling image including the inlet and outlet of major blood vessels. How to create a dimensional vascular model. According to paragraph 1,
A method of generating a 3D blood vessel model from a 2D X-ray angiography image where the major blood vessels include the left anterior descending coronary artery (LAD), left circumflex coronary artery (LCX), and right coronary artery (RCA). delete