KR101988531B1 - Navigation system for liver disease using augmented reality technology and method for organ image display - Google Patents

Abstract

In order for the surgical navigation system to display an organ image on the organ to be operated on, the 3D medical image is generated from a 2D medical image of the organ of the patient on which the first ROI is displayed. Reflect the second region of interest corresponding to the first region of interest in the generated three-dimensional medical image, filter the three-dimensional medical image based on the threshold calculated based on the three-dimensional medical image, and display the long-term image in which the second region of interest is reflected. Create Then, the first ROI coincides with the second ROI, and the organ image is matched to the organ to be operated to display the organ image on the organ.

Description

Navigation system for liver disease using augmented reality technology and method for organ image display}

The present invention relates to a liver lesion surgery navigation system and a long-term image display method using augmented reality technology.

In general, when doctors plan a patient's surgery plan before entering the surgery, the surgery plan is referred to two-dimensional medical images such as CT (Computed Tomographic) or MRI (Magnetic Resonance Imaging) information. Stand up. In this case, it is difficult to match the two-dimensional medical image with the location of the lesion or the relationship with blood vessels according to the location of the lesion in most patients' organs. There are many limitations in determining the location of lesions and the distribution of surrounding blood vessels.

In order to solve this problem, a technique for providing a 3D image by providing CT or MRI pictures has been studied. Even if the surgical plan is made using medical images made with 3D augmented reality images, doctors using the surgical navigation system may damage unwanted parts if they do not accurately recognize the surgical instruments and the long-term depth of the patient. There is.

Accordingly, the present invention provides a surgical navigation system and a long-term image display method for providing an image of the internal organs, such as the location of the internal organ lesions of the patient and the distribution of surrounding blood vessels using augmented reality technology.

Surgical navigation system which is one feature of the present invention for achieving the technical problem of the present invention,

3D medical image information is generated from the 2D medical image information of a patient's surgery target organ, the generated 3D medical image information, a threshold calculated from the 3D medical image information, an external region of interest and a patient posture Based on the information, the image matching unit for generating an organ image to match the surgery target organ, and the display unit for projecting and displaying the organ image generated by the image matching unit on the actual organ of the patient.

The image matching unit may include an information receiver configured to receive 2D medical image information about the organ to be operated on, and generate the 3D medical image information from the 2D medical image information received by the information receiver, and generate the 3D medical image. A 3D image generating unit for calculating the threshold value from the information, a plurality of first ROIs displayed on the organ to be operated on, and a second ROI to be reflected in the 3D medical image information from the acquired first ROI A region of interest setting unit for calculating a region, and generating a long-term image by filtering the 3D medical image by the threshold value, and a second region of interest included in the generated organ image and a first region of interest displayed on the target organ It may include a long-term image generating unit for matching to match.

The organ image may include blood vessels and lesions of the surgery target organ, depth information of the blood vessels and lesions, and the second region of interest, and the threshold value may be a gray scale value calculated from the 3D medical image.

The organ image generator may receive posture information of the patient and correct the 3D medical image reflecting a second ROI based on the posture information.

The long-term image generator may be configured to rotate and size the second ROI based on the first ROI in order to match the second ROI of the organ image with the first ROI displayed on the organ to be operated. The change value and the position change value can be calculated.

The display unit may include an organ image receiving unit for receiving an organ image generated by the organ image generating unit, and an image providing unit configured to display the received organ image to be projected onto the surgical target organ.

In another aspect of the present invention for achieving the technical problem of the present invention, a navigation navigation system for displaying a long-term image on the organ to be operated,

Generating a 3D medical image from a 2D medical image of a target organ of a patient displaying a first ROI, and reflecting a second ROI corresponding to the first ROI to the generated 3D medical image Generating a long-term image reflecting a second ROI by filtering the 3D medical image based on a threshold calculated based on the 3D medical image, and matching the first ROI with the second ROI. And matching the organ image to the surgery target organ to display the organ image on the surgery target organ.

The reflecting of the second ROI may include obtaining posture information extracted from a posture image of a patient input from the outside, and correcting a 3D medical image reflecting the second ROI based on the acquired posture information. It may include a step.

Before generating the 3D medical image, receiving a 2D medical image of the organ to be operated on, and generating the 3D medical image from the received 2D medical image

The 2D medical image may include a plurality of pieces of information of an organ shape, a lesion location, a lesion size, a peripheral organ location, a peripheral organ structure, a vascular structure, and a blood vessel location, using either MRI or CT. It may include.

The displaying may include a rotation angle and a size of the second ROI based on the first ROI in order to match the second ROI of the organ image with the first ROI displayed on the organ to be operated. The method may further include calculating a change value and a position change value.

According to the present invention, the internal organ state of the patient can be directly confirmed by the augmented reality image, so that it can be used as a clear data in the planning of the surgery, thereby reducing the occurrence of medical accidents, reducing the operation time, and the burden on the patient's surgery. Can be reduced.

In addition, it can be distributed as educational software to medical and other majors, not directly for surgery and surgery, thereby reducing the cost of education.

1 is an illustration of a surgical navigation system according to an embodiment of the present invention.
2 is a structural diagram of a surgical navigation system according to an embodiment of the present invention.
3 is a flowchart illustrating a long-term image providing method using a surgical navigation system according to an embodiment of the present invention.
4 is an exemplary diagram of an organ image provided by a surgical navigation system according to an exemplary embodiment of the present invention.
5 is an exemplary diagram of an ROI set for each organ according to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, a surgical navigation system and a stereoscopic image providing method using the same according to an embodiment of the present invention will be described with reference to the accompanying drawings.

 1 is an illustration of a surgical navigation system according to an embodiment of the present invention.

As shown in Figure 1, by matching a three-dimensional stereoscopic organ image to the organ of the patient, the operator (hereinafter referred to as a doctor) to allow the operator to directly check the internal structure of the organ, such as lesions or vessels during surgery The navigation system 100 includes an image matching unit 110 and a display unit 120.

The image matching unit 110 generates a 3D medical image based on a 2D medical image such as CT or MRI of a patient's organ. The image matching unit 110 generates a 3D medical image based on posture information that the patient takes when the surgery is performed. To this end, the image matching unit 1100 will be described using an example of acquiring posture information from a posture taken by the patient. However, when the posture information taken when the patient is undergoing surgery is fixed, the image matching unit 1100 is previously described. Posture information may be stored.

The 3D medical image generated by the image matching unit 110 includes all the external appearance of organs, internal vascular tissues, lesion tissues, peripheral organs, and positions and structures of organs. Since the image matching unit 110 generates the 3D medical image based on the 2D medical image and the posture information, the method may be performed in various ways, and thus, the image matching unit 110 is not limited to any one embodiment.

The image matching unit 110 also receives a preset number of points around the actual organ input from the outside as a land mark for the organ. When the doctor displays an arbitrary position or a predetermined position so as to easily locate the organ without damaging the organ, such as a dye, during surgery, the image matching unit 110 receives the plurality of displayed positions as the ROI. .

The image matching unit 110 sets a region of interest received in the generated 3D medical image. The image matching unit 110 generates a long-term image to be matched to an actual organ by filtering the 3D medical image in which the ROI is set using a threshold value calculated in advance. Then, the generated organ image is provided to the display unit 120 so that the surgeon doctor can check the stereoscopic organ image matched with the actual organ through the display unit 120. At this time, since the region of interest is also displayed on the organ image, the region of interest is positioned to coincide with the region of interest displayed on the organ, and then the organ image is displayed on the organ.

The display unit 120 matches and displays the organ image transmitted from the image matching unit 110 on the organ position of the patient, so that the doctor can easily check the internal state of the organ of the patient. The display unit 120 may be implemented in the form of a head mounted device (HMD) and a non-head mounted device (HMD) that can be worn by a doctor, and the embodiment of the present invention will be described by using an HMD as an example. .

The structure of the surgical navigation system 100 described above will be described in detail with reference to FIG.

2 is a structural diagram of a surgical navigation system according to an embodiment of the present invention.

As shown in FIG. 2, the surgical navigation system 100 includes an image matching unit 110 for generating an organ image and a display unit 120 for displaying the organ image in a form matched to an organ to be operated on a patient. .

The image matching unit 110 includes an information receiver 111, a 3D image generator 112, an ROI setting unit 113, and a long term image generator 114. The display unit 120 includes a long term image receiving unit 121 and an image providing unit 122.

First, the information receiver 111 receives a 2D medical image of a patient's surgery target organ. Types of two-dimensional medical images include CT or MRI, and the embodiment of the present invention does not limit the shape of the two-dimensional medical images. Two-dimensional medical images include various information such as organ appearance, organ location, lesion location, lesion size, peripheral organ location, peripheral organ structure, vascular structure of the organ under operation, vascular location, etc. have.

The 3D image generator 112 generates the 2D medical image received by the information receiver 111 as a 3D medical image through a preprocessing process, a synthesis process, a background projection process, and a postprocessing process. Here, the generated 3D medical image includes the same information included in the 2D medical image, such as the size of the target organ, the location of the organ, the position with the surrounding tissue, the location of the lesion and the size of the lesion, and the location of the blood vessel.

That is, the 3D image generator 112 removes noise from an input 2D medical image of an organ to be operated, improves image quality, and performs a preprocessing operation such as bias correction to produce a 3D medical image. Generate preprocessed medical images in a state suitable for conversion. The 3D image generator 112 synthesizes the preprocessed medical image and projects the background to generate a 3D medical image.

Through the three-dimensional medical image generated by the three-dimensional image generator 112, the doctor can determine the overall appearance of the organ to be operated, the internal vascular tissue, the lesion tissue, the position of the peripheral organ or organ, the structure of the peripheral organ or organ, and the like. have. The 3D image generating unit 112 uses position information indicating how deep the lesion in the surgical organ, the internal vascular tissue of the surgical organ, and the like are located at the depth of the surface of the surgical organ using a 3D coordinate system. It is generated and included in the 3D medical image.

Since the 3D image generating unit 112 preprocesses the 2D medical image, synthesizes it, projects the background, and extracts the depth information using the 3D coordinate system and provides the same together, it can be performed through various methods. The embodiment of the present invention is not limited to any one method.

The 3D image generator 112 calculates a threshold value to be used for filtering by the organ image generator 114 to be described later from the generated 3D medical image. Here, the threshold used for filtering means a specific value of a gray scale representing a 3D medical image.

For example, in two-dimensional medical images, CT images, bones appear bright and other tissues (epithelial, connective, muscular, neural) appear dark. If the brightness of blood vessels and the brightness of lesions are seen as the brightness between the brightness of bones and the brightness of other tissues, then the brightness of bones and other tissues, blood vessels, lesions, etc. can be expressed numerically.

Therefore, the 3D image generator 112 may set a threshold value of the gray scale based on a value corresponding to the brightness of the blood vessel and the lesion. In the exemplary embodiment of the present invention, the 3D image generator 112 calculates the threshold value as an example, but is not necessarily limited thereto. In addition, the method of calculating the threshold value from the 3D medical image is not limited to any one method.

The ROI setting unit 113 receives an ROI input from the outside (hereinafter, referred to as a “first ROI” for convenience of description). In the embodiment of the present invention, the region of interest is described as being location information of three points displayed on the organ to be operated on as an example, but is not necessarily limited thereto.

The ROI setting unit 113 generates the first ROI, which is three points displayed on the organ to be operated, as a second ROI by applying a triangulation method. The second region of interest is information to be included and set in the 3D medical image, and is information generated so that three points marked as the first region of interest may be displayed on the 3D medical image in the same manner.

That is, when the doctor leaves an arbitrary mark on three points on the patient's surgery target organ during surgery, the ROI setting unit 113 acquires the location information of the position where the corresponding mark remains as the first ROI. Since the ROI setting unit 113 obtains the display of the three points left in the organ as the first ROI in various ways, the embodiment of the present invention is not limited to any one method.

Here, the three points will be described as an example in which the deformation by the lesion is less around the organ to be operated, and the doctor selects and displays three rigid structures having a predetermined strength or more. An example of the doctor setting the region of interest will be described first with reference to FIG. 5.

5 is an exemplary diagram of an ROI set for each organ according to an exemplary embodiment of the present invention.

As shown in FIG. 5, when the doctor sets the region of interest for the middle of the organ, the doctor may set three points as the region of interest by setting the aorta, the gallbladder, and the context. Using another example of the kidney, the doctor may set the vein, the first position of the plurality of ribs, and the second position of the ribs at three points.

In another organ, the cardiac example is used. The region of interest set by the doctor may be a relative vein, a pulmonary artery, and an inferior vena cava. In the case of the uterus, the left ovary, the right ovary, and the abdominal vena cava may be set.

In the exemplary embodiment of the present invention, the doctor has indicated that the region of interest is set by displaying three points on the organ during surgery, but the region of interest may be set in advance according to the type of organ.

Meanwhile, referring to FIG. 2, the long-term image generator 114 sets the second ROI generated by the ROI setting unit 113 to the 3D medical image generated by the 3D image generator 112. do. The organ image generating unit 114 corrects the organ image based on the posture information extracted based on the posture of the patient taking the 3D medical image in which the second ROI is set.

To this end, the organ image generating unit 114 receives an image of the posture of the patient from the outside, and extracts the posture information of the organ whose shape is modified according to the posture. This is because the shape, position, and size of organs may vary depending on the patient's posture. Since the long-term image generator 114 receives the posture image of the patient, extracts the posture information from the posture image, and corrects the 3D medical image by using the posture information, the method may be performed in various ways. The embodiment of the present invention is not limited to any one method.

The organ image generator 114 filters the 3D medical image in which the various organ information and the second ROI are set based on the threshold value calculated by the 3D image generator 112 to generate the organ image. In the organ image generated by filtering the 3D medical image, only the internal structure of the organ to be operated, that is, the blood vessel and the lesion, and the second region of interest remain, and the other tissues (eg, bone, muscle, etc.) are deleted. As a result, only blood vessels and lesions of the target organ may be displayed on the actual organ.

That is, if the organ image generator 114 deletes the lower value and the higher value than the calculated threshold value through filtering, the bone image and the dark image tissue disappear, and only the blood vessel and the lesion image corresponding to the middle image remain. .

The organ image generator 114 matches the organ image modified according to the posture information of the patient with the actual organ. When the organ image is matched to the actual organ, the organ image generator 114 matches the first region of interest displayed on the actual organ and the second region of interest displayed on the organ image.

When the long term image generator 114 matches the first region of interest with the second region of interest, how long the long term image generator 114 rotates the second region of interest based on the first region of interest (rotation angle), Calculate how much the position changes (position change value) and how much the size changes (size change value). Then, the calculated rotation angle, position change value, and size change value are reflected in the second region of interest so that the organ image is matched to the actual target organ.

Since the long-term image generator 114 calculates the rotation angle, the position change value, and the size change value of the second ROI based on the first ROI, various embodiments of the present disclosure may be described in detail. Omit. In addition, the method of matching the changed organ image to the organ to be operated may also be performed in various ways, and thus, detailed description thereof will be omitted.

Meanwhile, the organ image receiver 121 of the display unit 120 receives the organ image matched to the organ to be operated by the organ image generator 112.

The image provider 122 projects and displays the organ image received by the organ image receiver 121 on an actual target organ. The organ image, which is projected onto the target organ and displayed, shows only the organ structure, lesion location, and lesion size along with the three-dimensional coordinate system.

The long-term image projected onto the surgical target organ provided by the image provider 122 is displayed while being fixed to the surgical target organ while the surgery is in progress. To this end, if the posture of the patient is changed or the target organ moves, the long-term image reflecting the changed posture or movement degree is continuously provided.

A method of providing an organ image by using the surgical navigation system described above will be described with reference to FIG. 3.

3 is a flowchart illustrating a long-term image providing method using a surgical navigation system according to an embodiment of the present invention.

As shown in FIG. 3, the surgical navigation system 100 receives a 2D medical image such as a CT or an MRI of an organ of a patient. In operation S100, a 3D medical image is generated based on the received 2D medical image.

At the same time, the surgical navigation system 100 calculates a threshold to be used for later filtering based on the generated 3D medical image. In the process of generating a two-dimensional medical image as a three-dimensional stereoscopic image, the surgical navigation system 100 also extracts various organ information such as organ structure or location of the patient, organ internal vascular structure, lesion location or lesion size.

The surgical navigation system 100 obtains location information on three marks marked by a doctor on an organ to be operated by a patient as a first region of interest. In operation S110, a second ROI is calculated from the acquired first ROI.

That is, when a doctor marks a point at three points using a dye or the like on a patient's surgery target organ, the doctor acquires location information on the points as the first ROI. This is to ensure that the matching is performed using the ROI in the process of matching the organ image generated from the 3D medical image to the target organ. The second ROI is calculated to apply the first ROI to the 3D medical image.

After calculating the second ROI, the surgical navigation system 100 extracts posture information from the patient posture image of the patient (S120). Since the method of collecting the patient posture image or the method of extracting the posture information from the collected patient posture image may be performed in various ways, the embodiment of the present invention is not limited to any one method.

The surgical navigation system 100 generates a corrected long-term image based on the 3D medical image generated in step S100 and the threshold value calculated in step S110, the second region of interest calculated in step S110, and the posture information extracted in step S120 (S130). ).

That is, the surgical navigation system 100 corrects the 3D medical image displaying the second ROI based on posture information extracted from the patient posture image. The surgical navigation system 100 generates a long-term image by filtering the corrected 3D medical image using the threshold value calculated in step S100 (S140). In the filtered organ image, internal structures of the organ to be operated, i.e., blood vessels and lesions, and a second region of interest remain, and other tissues (e.g., bones, muscles, etc.) are deleted, and the blood vessels on the actual organs are deleted. Only and lesions can be displayed.

The surgical navigation system 100 matches and displays the long-term image generated in operation S140 with the actual image based on the region of interest (S150). That is, the surgical navigation system 100 matches the first region of interest displayed on the actual organ with the second region of interest displayed on the organ image, and displays the organ image on the target organ.

An example of an organ image projected on an operation target organ provided by the surgical navigation system described above will be described with reference to FIG. 4.

4 is an exemplary diagram of an organ image provided by a surgical navigation system according to an exemplary embodiment of the present invention.

As shown in (a) of FIG. 4, assuming that surgery is performed on the liver while the patient is lying in the forward position, the doctor sets three regions of interest in the liver of the patient. The surgical navigation system 100 generates a three-dimensional organ image based on the two-dimensional organ image of the patient, and as shown in (b) of FIG. 4, the surgeon navigation system 100 also displays a region of interest displayed directly by the doctor on the three-dimensional organ image. Mark to reflect.

The organ image is generated as shown in FIG. 4B by reflecting the 3D organ image in the same manner as the shape of the liver which can be confirmed in the posture taken by the patient. The region of interest reflected in the long-term image is matched to match the region of interest indicated by the liver of the patient shown in FIG. 4A.

On the basis of this, when the doctor performs the actual surgery, the patient's liver is shown through the display unit 120, that is, the HMD-type equipment, and the organ shown in (b) on the actual liver of the patient shown in FIG. The image is projected so that doctors can easily determine the size of the blood vessels or lesions in the liver and the location of the lesions.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (13)

In surgical navigation systems,
A 3D medical image is generated by using 2D medical image information of a patient's surgery target organ, the generated 3D medical image, a value corresponding to a gray scale representing the 3D medical image, and the surgery An image matching unit configured to generate an organ image to be matched with the target organ based on the first ROI and the patient posture information, which are location information of three points displayed on the target organ, and
A display unit for projecting and displaying the organ image generated by the image matching unit on the actual organ of the patient
Including,
The image matching unit,
An information receiver configured to receive 2D medical image information of the organ to be operated on;
A 3D image generating unit generating the 3D medical image using the 2D medical image information received by the information receiving unit and calculating a value corresponding to the gray scale from the generated 3D medical image;
A region of interest setting unit for obtaining a plurality of first regions of interest displayed on the surgery target organ, and calculating a second region of interest that is the same region as the first region of interest and is reflected and provided in the 3D medical image;
The 3D medical image is filtered to a value corresponding to the gray scale to generate an organ image, and to match the second region of interest included in the generated organ image with the first region of interest displayed on the target organ to match. Long term image generator
Surgical navigation system comprising a.
delete The method of claim 1,
The organ image may include blood vessels and lesions of the organ to be operated on, depth information about the blood vessels and lesions, and the second region of interest. The method of claim 3,
The long term image generator,
Receiving posture information of the patient,
And a navigation navigation system for correcting the 3D medical image reflecting the second ROI based on the posture information. The method of claim 4, wherein
The long term image generator,
In order to match and match the second ROI of the organ image with the first ROI displayed on the organ to be operated, the rotation angle, the size change value, and the position change value of the second ROI based on the first ROI. Surgical navigation system to calculate the. The method of claim 1,
The 2D medical image information includes a plurality of pieces of information such as organ appearance, lesion location, lesion size, peripheral organ location, peripheral organ structure, vascular structure and blood vessel location. The method of claim 1,
The display unit,
A long term image receiver configured to receive a long term image generated by the long term image generator, and
Image providing unit for displaying the received organ image to be projected on the surgery target organ
Surgical navigation system comprising a. The method of claim 7, wherein
The display unit,
Surgical navigation system implemented in the form of a head-mounted device (HMD) or non-head-mounted device (HMD) wearable by a doctor. In the method of the surgical navigation system to display an organ image on the organ to be operated,
Generating a 3D medical image from the 2D medical image of the target organ of the patient displaying the first ROI,
Reflecting a second region of interest corresponding to the first region of interest on the generated 3D medical image;
Generating the organ image reflecting the second region of interest by filtering the 3D medical image based on a value corresponding to a gray scale representing the 3D medical image, and
Matching the first region of interest with a second region of interest, matching the organ image to the organ to be operated, and displaying the organ image onto the surgical organ;
Long term image display method comprising a. The method of claim 9,
The organ image may include blood vessels and lesions of the organ to be operated, depth information about the vessels and lesions, and the second region of interest. The method of claim 9,
Reflecting the second region of interest,
Obtaining posture information extracted from a posture image of a patient input from outside; and
Correcting the 3D medical image reflecting the second ROI based on the acquired posture information
Long term image display method comprising a. The method of claim 9,
Before generating the 3D medical image,
Receiving a 2D medical image of the organ to be operated on, and
Generating the 3D medical image from the received 2D medical image
Including,
The 2D medical image is either MRI or CT, and includes a plurality of pieces of information of an organ shape, a lesion position, a lesion size, a peripheral organ position, a peripheral organ structure, a vascular structure, and a blood vessel position of the target organ. Display method. The method of claim 9,
The displaying step,
In order to match and match the second ROI of the organ image with the first ROI displayed on the organ to be operated, the rotation angle, the size change value, and the position change value of the second ROI based on the first ROI. Step to calculate
The long-term image display method further comprising.

Images