JP2016202351A - Medical support system, medical support method, image processing apparatus, control method and control program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a medical support system which allows a user to easily recognize the state of a region in the actual time inside a living body of a patient.SOLUTION: A medical support system comprises: a region model generation unit 110 which generates data of a three-dimensional region model on the basis of a tomographic image group of a patient; a first feature point extraction unit 120 which extracts position information of a first feature point having a prescribed feature; an ultrasonic model generation unit 130 which accumulates ultrasonic data acquired by using an ultrasonic probe and position and direction data of the ultrasonic probe by associating them with each other and generates data of a three-dimensional ultrasonic model by using the ultrasonic data and position and direction data; a second feature point extraction unit 140 which extracts position information of a second feature point having a prescribed feature from the three-dimensional ultrasonic model; a deformation unit 150 which deforms a three-dimensional region model so as to overlap the plurality of first feature points with the plurality of second feature points; and a display unit 160 which displays a projection image of the three-dimensional region model deformed by the deformation unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for supporting medical treatment by displaying a three-dimensional image in a patient body.

  In the above technical field, Patent Document 1 discloses a technique for displaying a CT or MRI tomographic image whose cross section coincides with an ultrasonic image. Patent Document 2 discloses a technique for displaying the display position of an ultrasonic image on a CT or MRI tomographic image. Patent Document 3 discloses a technique for displaying a CT or MRI three-dimensional image whose cross section substantially matches a three-dimensional ultrasonic image.

JP 2008-188193 A JP 2012-223416 A JP 2013-066332 A

  However, the technique described in the above document aligns the ultrasonic image and the CT or MRI image and displays the same cross-sectional image so that it can be referred to, but recognizes the state of the part in the patient's living body in real time. It was difficult to do.

  The objective of this invention is providing the technique which solves the above-mentioned subject.

In order to achieve the above object, a medical support system according to the present invention includes:
A part model generation means for generating data of a three-dimensional part model based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
Ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired are stored in association with each other, and the ultrasound data and the position and Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using the orientation data;
Second feature point extracting means for extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
Deformation means for deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
Display means for displaying a projection image of the three-dimensional part model deformed by the deformation means;
Is provided.

In order to achieve the above object, a medical support method according to the present invention comprises:
A site model generation step for generating data of a 3D site model based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A display step of displaying a projected image of the three-dimensional part model deformed in the deformation step;
including.

In order to achieve the above object, an image processing apparatus according to the present invention provides:
Receiving means for receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
Accumulation means for accumulating the ultrasonic data acquired using an ultrasonic probe for the patient and the position and orientation data of the ultrasonic probe from which the ultrasonic data was acquired;
An ultrasonic model generating means for generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
Second feature point extracting means for extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
Deformation means for deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
Transmitting means for transmitting a projection image of the three-dimensional part model deformed by the deforming means to a display means;
Is provided.

In order to achieve the above object, a method for controlling an image processing apparatus according to the present invention includes:
A receiving step of receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmission step of transmitting to the display means a projection image of the three-dimensional region model deformed in the deformation step;
including.

In order to achieve the above object, a control program for an image processing apparatus according to the present invention provides:
A receiving step of receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmission step of transmitting to the display means a projection image of the three-dimensional region model deformed in the deformation step;
Is executed on the computer.

  According to the present invention, it is possible to easily recognize the state of a part in a patient's living body in real time.

It is a block diagram which shows the structure of the medical assistance system which concerns on 1st Embodiment of this invention. It is a figure which shows the process outline | summary of the medical assistance system which concerns on 2nd Embodiment of this invention. It is a figure which shows the outline | summary of the extraction process of the 1st feature point group and 2nd feature point group which concern on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the medical assistance system which concerns on 2nd Embodiment of this invention. It is a sequence diagram which shows the operation | movement procedure of the medical assistance system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the division area table which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the ultrasonic image table which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the 2nd feature point table which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the function structure of the display image generation part which concerns on 2nd Embodiment of this invention. It is a figure explaining the process of the division area connection part which concerns on 2nd Embodiment of this invention. It is a figure which shows the example of the structure of the division | segmentation area | region connection table which concerns on 2nd Embodiment of this invention, and the information for alignment. It is a figure which shows the structure of the display data generation table which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the image processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the procedure of the feature point extraction process which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the procedure of the area division process which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the process outline | summary of the medical assistance system which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the structure of the medical assistance system which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the structure of the medical assistance system which concerns on 4th Embodiment of this invention. It is a block diagram which shows the function structure of the image processing apparatus which concerns on 4th Embodiment of this invention. It is a block diagram which shows the structure of the medical assistance system which concerns on 5th Embodiment of this invention. It is a figure which shows the various structural examples of the medical assistance system of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the constituent elements described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention only to them. The “part” used in this specification represents the whole or a part of the inside of the living body. For example, from a wide range such as the lower abdomen, internal organs such as the lung, liver, stomach, and the lesion part thereof, blood vessels, It is a word that expresses a narrow area such as a lymph line or a part of a nerve.

[First Embodiment]
A medical support system 100 as a first embodiment of the present invention will be described with reference to FIG. The medical support system 100 is a system that supports medical treatment by displaying a three-dimensional site model in a patient.

  As shown in FIG. 1, the medical support system 100 includes a part model generation unit 110, a first feature point extraction unit 120, an ultrasonic model generation unit 130, a second feature point extraction unit 140, and a deformation unit 150. , And display unit 160. The part model generation unit 110 generates data of the three-dimensional part model 111 based on the tomographic image group of the patient 101. The first feature point extraction unit 120 extracts position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model 111. The ultrasonic model generation unit 130 stores the ultrasonic data 104 acquired for the patient 101 using the ultrasonic probe 103 and the position and orientation data of the ultrasonic probe 103 that acquired the ultrasonic data 104 in association with each other. Then, using the ultrasonic data 104 and the position and orientation data, data of the three-dimensional ultrasonic model 131 is generated. The second feature point extraction unit 140 extracts position information of a plurality of second feature points having a predetermined feature from the three-dimensional ultrasonic model 131. The deforming unit 150 deforms the three-dimensional part model 111 so that the plurality of first feature points and the plurality of second feature points overlap. The display unit 160 displays the projection image 160 of the three-dimensional part model deformed by the deformation unit 150.

  The deforming unit 150 divides the three-dimensional part model 111 into a plurality of three-dimensional regions each including at least one first feature point, a plurality of first feature points, and a plurality of first feature points. The display unit 160 includes a divided region connecting unit 152 that adjusts the position of each of the plurality of three-dimensional regions so that the two feature points overlap each other and connects the plurality of three-dimensional regions 153 that have been adjusted to each other. A plurality of three-dimensional regions 161 connected by the divided region connecting unit 152 are displayed as a three-dimensional part model.

  According to the present embodiment, the three-dimensional site model based on the tomographic image group is converted and displayed in correspondence with the three-dimensional ultrasonic image acquired by the ultrasonic probe, so that the real time inside the living body of the patient can be obtained. The state of the part can be easily recognized.

[Second Embodiment]
Next, a medical support system according to the second embodiment of the present invention will be described. The medical support system according to the present embodiment extracts the first feature point of the three-dimensional blood vessel model included in the target region of the three-dimensional region model data generated from the CT image or the MRI image, Divide into regions containing feature points. Further, the second feature point of the three-dimensional blood vessel model included in the target region of the ultrasonic image data acquired by the ultrasonic probe is extracted. Then, the three-dimensional part model data is transformed and the three-dimensional part model is transformed by aligning and connecting the divided regions so that the three-dimensional position of the first feature point matches the three-dimensional position of the second feature point. Then, the deformed three-dimensional part model is displayed so as to overlap the patient's living body. The detailed deformation of the three-dimensional part model may be further performed by non-rigid deformation or the like.

  In addition, in order to display the image so as to overlap the patient's living body, the optical marker fixed to the patient's living body is imaged, and three-dimensional site model data having a position, shape and size based on the position and orientation of the optical marker is generated To do. The optical marker is provided as a magnetic sensor with an optical marker. An accurate position and orientation of the ultrasonic probe are detected from the magnetic sensor fixed to the ultrasonic probe and the magnetic sensor fixed to the living body. The three-dimensional part model is displayed on an optical see-through type head mounting display (hereinafter, HMD: Head Mounted Display). In this embodiment, the optical marker and the magnetic sensor are fixed to the patient's living body. However, when the abdomen is opened during the operation, the optical marker and the magnetic sensor are directly fixed to the organ to be operated. Thus, a more accurate position can be calculated.

《Medical support system》
With reference to FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 3, the structure and operation | movement of the medical assistance system of this embodiment are demonstrated.

(System overview)
FIG. 2A is a diagram showing an outline of the medical support system 200 according to the present embodiment.

  In the medical support system 200, for example, a three-dimensional part model or three-dimensional part represented by three-dimensional model data in an STL (Standard Triangulated Language) format generated from a CT image 210 that is one of computed tomography images. A first feature point group 240 arranged on the three-dimensional coordinates representing the features of the model is extracted from the three-dimensional blood vessel model extracted from the model. The first feature point group 240 is not limited as long as it represents a characteristic position of the biological tissue in the three-dimensional model data. For example, the number of branches, the branch vector, or the overall pattern at the branch point of the blood vessel. A unique point may be used as the first feature point. Next, a division process 260 into a region including at least one, preferably at least three adjacent first feature points from the first feature point group 240 is performed. Note that it is desirable that the first feature point is on the division plane or division point of the area because the division areas can be easily connected.

  On the other hand, in the examination room or the operating room 230, from the time-series two-dimensional ultrasonic image group 220 acquired in real time by the ultrasonic probe 203 whose position is determined using a position detection instrument (for example, a magnetic sensor), or A three-dimensional blood vessel model is generated from the three-dimensional ultrasonic image, and a second feature point group 250 arranged on the three-dimensional coordinates representing the model features is extracted. The second feature point group 250 is not limited as long as it represents a characteristic position of the living tissue in the three-dimensional ultrasound image data, but the number of branches at the branch point of the blood vessel, similar to the first feature point, It is desirable that a branching vector or a point unique to the entire pattern be the second feature point. Here, the position detection instrument of the ultrasonic probe 203 is preferably a magnetic sensor that is not blocked, rather than an optical marker that is blocked by a doctor or the like during examination or surgery.

  In the medical support system 200, the first feature point and the second feature point are three-dimensionally matched, and the position of the divided region is adjusted so that the first feature point group 240 overlaps the second feature point group 250. As a result, the three-dimensional model data is transformed. Note that the deformation of the three-dimensional model data may further include non-rigid deformation of the divided regions and the connected three-dimensional part model.

  Next, in the medical support system 200, based on the positional relationship between the living body and the three-dimensional part model based on the three-dimensional image data, which is measured using a position detecting instrument fixed on the living body 201 in real time, The three-dimensional part model 270 deformed corresponding to the ultrasonic image of time is adjusted so as to be inside the living body and displayed in three dimensions. An optical see-through type HMD is desirable for the display unit for performing the three-dimensional display of the three-dimensional part model 270, and the doctor visually observes the actual living body ahead of the three-dimensional part model on the display screen of the HMD. Here, the position detecting instrument on the living body is preferably a magnetic sensor with an optical marker for obtaining the positional relationship between the living body and the display of the three-dimensional site model with high accuracy.

  FIG. 2B is a diagram showing an outline of extraction processing of the first feature point group 240 and the second feature point group 250 according to the present embodiment. In FIG. 2B, the extraction of the first feature point group 240 will be described as a representative, but the extraction of the second feature point group 250 is the same.

  First, a center line (core line) 242 of each blood vessel image is extracted from the acquired three-dimensional blood vessel image 241. Next, feature points 243 such as branch points and inflection points of the plurality of core wires 242 are extracted and used for the dividing process 260.

(System configuration)
FIG. 2C is a block diagram illustrating a configuration of the medical support system 200 according to the present embodiment.

  The medical support system 200 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 280, and a diagnosis / surgical instrument group disposed in an examination room or an operating room 230.

  The CT apparatus 211 acquires a computer tomography image of the patient 201 in advance. Note that it is desirable to convert the acquired three-dimensional CT image generated for the target organ into three-dimensional data such as the STL format. Although there is no limitation on the range of the three-dimensional data, it is desirable that the unit is a target organ unit of the patient 201. The three-dimensional model generation apparatus 212 generates a three-dimensional part model including, for example, a three-dimensional blood vessel model from the computer tomography image group from the CT apparatus 211 and passes it to the image processing apparatus 280. In the image processing apparatus 280 in FIG. 2C, the CT apparatus 211 acquires a three-dimensional part model based on the computer tomography image group. However, even with an MRI apparatus, the three-dimensional part model is based on another computer tomography apparatus. It may be. The data of the three-dimensional site model is not limited to the STL format as long as it represents the configuration of the living tissue inside the living body of the patient 201 or the configuration of the organ or affected part. In this embodiment, the image processing apparatus 280 acquires the data of the three-dimensional part model generated by the external three-dimensional model generation apparatus 212. However, the image processing apparatus 280 is the three-dimensional model generation apparatus. A configuration including 212 functions may be adopted.

  On the other hand, the examination / surgical instrument group in the examination room or the operating room 230 includes an ultrasonic probe 203, a magnetic sensor 231, a magnetic sensor 293 with an optical marker, a Web camera 294, and an HMD 295. Note that the Web camera 294 and the HMD 295 are integrated as an optical see-through HMD. The ultrasonic probe 203 acquires an ultrasonic image from the patient 201 during real-time examination or surgery. The magnetic sensor 231 is disposed on the ultrasonic probe 203 and detects the position of the ultrasonic probe 203. The magnetic sensor 293 with an optical marker is disposed on the patient 201, detects the position of the optical marker by the magnetic sensor, and the optical marker is used to determine the positional relationship between the living body of the patient 201 and the web camera 294. The web camera 294 images the optical marker of the magnetic sensor 293 with an optical marker, and determines the positional relationship between the living body of the patient 201 and the web camera 294. A magnetic sensor may be attached to the HMD in order to measure the positional relationship between the living body of the patient 201 and the Web camera 294. The HMD 295 displays a three-dimensional model represented by three-dimensional data deformed so as to match the ultrasonic image at the position of the patient 201 viewed by the doctor 202.

  The image processing device 280 transforms the three-dimensional part model represented by the three-dimensional data so as to match the ultrasonic image acquired by the ultrasonic probe 203 whose position and orientation are determined by the detection signal of the magnetic sensor 231. Such deformation includes a dividing process for dividing the three-dimensional region model into regions based on feature points such as blood vessel branch points, and a connecting process for connecting the divided regions so as to match the feature points of the ultrasonic image. . Next, the image processing apparatus 280 determines the position and orientation of the patient 201 and the position of the patient 201 and the HMD 295 based on the detection signal of the magnetic sensor 293 with the optical marker and the image data of the Web camera 294 that has captured the optical marker. Calculate the relationship. Note that the positional relationship between the Web camera 294 and the HMD 295 is fixed as an integrated HMD. Then, the image processing device 280 generates three-dimensional display data so that the deformed three-dimensional part model is at an accurate position inside the living body of the patient 201 and causes the HMD 295 to display the three-dimensional display data. The merit of fixing the sensor to the living body in this way is that the three-dimensional position can be accurately estimated even when the living body moves.

(Operation procedure)
FIG. 3 is a sequence diagram showing an operation procedure of the medical support system 200 according to the present embodiment.

  In step S301, the CT apparatus 211 acquires a computer tomography image group of the patient 201. In step S <b> 302, the three-dimensional model generation apparatus 212 extracts a part such as the liver or its blood vessel from the computer tomography image group, generates three-dimensional part model data for three-dimensional display, and transmits the three-dimensional part model data to the image processing apparatus 280. . In step S303, the image processing device 280 receives the three-dimensional part model data from the three-dimensional model generation device 212. In step S305, the image processing apparatus 280 extracts a first feature point group based on a three-dimensional part model such as a three-dimensional blood vessel model. In step S307, the image processing device 280 divides and holds the three-dimensional part model into three-dimensional regions including at least one first feature point.

  In step S <b> 309, the magnetic sensor 231 acquires the position and orientation of the ultrasonic probe 203 and transmits it to the image processing device 280, and the ultrasonic probe 203 acquires the ultrasonic signal at that time and transmits it to the image processing device 280. In step S <b> 311, the image processing apparatus 280 accumulates ultrasonic two-dimensional data based on a plurality of successive position information and ultrasonic signals, and then generates, for example, three-dimensional ultrasonic model data indicating a blood vessel model. To do. Next, in step S313, the image processing device 280 extracts a second feature point group based on the three-dimensional ultrasonic model data indicating the blood vessel model.

  Further, in step S315, the image processing apparatus 280 converts the position of the divided region generated from the three-dimensional part model so that the first feature point group matches the second feature point group. Then, the image processing apparatus 280 connects the divided areas whose positions have been converted. In order to accurately connect the divided regions, it is desirable to include the first feature point on the surface, ridgeline, or vertex of the divided region.

  In step S317, the magnetic sensor of the magnetic sensor 293 with the optical marker acquires the position and orientation of the living body of the placed patient 201 and transmits the acquired position and orientation to the image processing apparatus 280. In step S319, the Web camera 294 images the optical marker of the magnetic sensor 293 with the optical marker, and transmits the imaging data of the optical marker to the image processing device 280. In step S321, the image processing apparatus 280 calculates the positional relationship (distance and orientation) between the living body and the HMD 295 that is the display unit from the imaging data of the optical marker.

  The position of the ultrasonic probe 203 can be detected more accurately by the magnetic sensor 293 with an optical marker and the magnetic sensor 231 of the ultrasonic probe 203. In other words, an accurate position can be detected by placing an optical marker on the ultrasonic probe 203, but it is a condition that the optical marker is not hidden from the view of the camera. In the present embodiment, the magnetic sensor 231 is arranged on the ultrasonic probe 203 so that the ultrasonic probe 203 can be freely moved. However, since accuracy is lowered as compared with the optical marker, the accuracy is ensured by arranging the magnetic sensor 293 with the optical marker in the living body 201. Therefore, when a magnetic sensor is attached to the HMD 295, the positional relationship between the HMD 295 and the ultrasonic probe 203 can be detected, so that there is no need for imaging with a camera.

  In step S323, the image processing apparatus 280 generates display data for displaying the three-dimensional part model deformed by the connection of the divided areas as if it is inside the living body of the patient 201, and transmits the display data to the HMD 295. In step S325, the HMD 295 displays the three-dimensional site model as if inside the living body of the patient 201 based on the received display data.

<< Functional configuration of image processing apparatus >>
FIG. 4 is a block diagram illustrating a functional configuration of the image processing apparatus 280 according to the present embodiment.

  The image processing device 280 includes a three-dimensional part model reception unit 411, a first feature point extraction unit 412 and a divided region generation unit 424. The image processing apparatus 280 includes a communication control unit 413, a magnetic signal receiving unit 414, an ultrasonic probe position detecting unit 415, an ultrasonic signal receiving unit 416, an ultrasonic model data generating unit 417, and a second feature. A point extraction unit 418. Further, the image processing device 280 includes a magnetic signal receiving unit 419, a patient position detecting unit 420, an optical marker image receiving unit 421, an HMD position calculating unit 422, a display image generating unit 423, and a display image transmitting unit 425. . The display image generation unit 423 includes a divided region matching unit 431, a divided region connection unit 432, and an HMD image generation unit 433.

  The three-dimensional part model reception unit 411 receives a three-dimensional part model such as a blood vessel model generated from a computer tomography image group in the external three-dimensional model generation device 212. The reception of the three-dimensional part model may be performed via a storage medium or the communication control unit 413. Note that when the image processing apparatus 280 generates three-dimensional data such as STL data from a tomographic image, the image processing apparatus 280 includes an STL data generation unit. The first feature point extraction unit 412 extracts a first feature point from a three-dimensional part model such as a blood vessel model. The divided region generation unit 424 divides the three-dimensional part model represented by STL data or the like into three-dimensional small regions including the first feature points that are close to each other. Each divided area includes at least one first feature point. In order to perform position conversion by matching with the second feature point of the ultrasonic image of the divided area, it is desirable that each divided area includes at least three first feature points. Furthermore, in order to facilitate the connection of the divided areas, it is desirable to divide the first feature points so that they are on the surface, ridge line, and vertex of each divided area.

  The communication control unit 413 exchanges information with each instrument included in the diagnosis / surgical instrument group 290. The communication with the examination / surgical instrument group in the examination room or the operating room 230 may be wired or wireless, but is preferably wireless communication. The magnetic signal receiving unit 414 receives a magnetic detection signal from the magnetic sensor 231 disposed in the ultrasonic probe 203 via the communication control unit 413. The ultrasonic probe position detection unit 415 detects the position and orientation of the ultrasonic probe 203 based on the magnetic detection signal from the magnetic sensor 231. Specifically, detection of the position and orientation by the magnetic sensor 231 based on the magnetic detection signal is performed by a controller in a position detection system including a magnetic field generator (not shown), a controller unit (not shown), and the magnetic sensor 231. The direction and coordinates are calculated and transmitted by the unit. In the case where the magnetic sensor 231 has a storage unit for detecting the position and orientation from the magnetic detection signal, data (for example, coordinates and vectors) indicating the position and orientation may be received from the magnetic sensor 231. The ultrasonic signal receiving unit 416 receives an ultrasonic signal from the ultrasonic probe 203 via the communication control unit 413. The ultrasonic signal receiving unit 416 may receive ultrasonic image data when the ultrasonic probe 203 has a function of generating ultrasonic image data from the ultrasonic signal.

  The ultrasonic model data generation unit 417 corresponds to the ultrasonic signal received by the ultrasonic signal reception unit 416 in accordance with the position and orientation of the ultrasonic probe 203 detected by the ultrasonic probe position detection unit 415. Generate ultrasonic model data. The ultrasonic model data generation unit 417 generates three-dimensional ultrasonic model data by connecting each cross-sectional image acquired by the ultrasonic probe 203 in three dimensions. In the present embodiment, a structure such as a blood vessel is extracted from each cross-sectional image acquired by the ultrasonic probe 203 and connected in three dimensions to generate three-dimensional ultrasonic model data. The second feature point extraction unit 418 extracts second feature points such as blood vessel branching from the three-dimensional ultrasound model data such as the blood vessel model generated by the ultrasound model data generation unit 417.

  The magnetic signal receiving unit 419 receives a magnetic detection signal from the magnetic sensor with optical marker 293 via the communication control unit 413. In addition, when the magnetic sensor 293 with an optical marker has a storage unit for detecting the position and orientation from the magnetic detection signal, data (for example, coordinates and vectors) indicating the position and orientation may be received. The patient position detection unit 420 detects the position and orientation of the patient 201 based on the magnetic detection signal from the magnetic sensor 293 with an optical marker. Specifically, the position and orientation detection by the magnetic sensor 293 with an optical marker based on the magnetic detection signal is a position detection system including a magnetic field generator (not shown), a controller unit (not shown), and a magnetic sensor 293. In the configuration, the controller unit calculates and transmits the direction and coordinates.

  The optical marker image reception unit 421 receives a captured image of the optical marker of the magnetic sensor with optical marker 293 captured by the Web camera 294 installed in the HMD 295 via the communication control unit 413. The HMD position calculation unit 422 calculates the positional relationship between the HMD 295 and the patient 201 based on the captured image of the optical marker received by the optical marker image reception unit 421. As described above, if a magnetic sensor for position determination is arranged on the HMD 295, the positional relationship between the HMD 295 and the patient 201 may not be calculated based on the captured image of the optical marker.

  The divided region matching unit 431 of the display image generation unit 423 matches the first feature point included in each divided region generated by the divided region generation unit 424 with the second feature point extracted by the second feature point extraction unit 418. Then, each divided area is moved and rotated so that the feature points of each other match. Then, the divided region connecting unit 432 connects the divided regions with the first feature points, and generates a three-dimensional part model represented by STL data and the like that is transformed to match the three-dimensional ultrasonic image. To do. Next, the HDM image generation unit 433 performs three-dimensional display at the HMD position calculated by the HMD position calculation unit 422 so that the deformed three-dimensional part model overlaps the position of the patient 201 detected by the patient position detection unit 420. Image data is generated. That is, a three-dimensional image is generated that calculates how the three-dimensional part model that overlaps the living body of the patient 201 looks when observed from the measured position of the HMD 295. The display image transmission unit 425 transmits the image data overlapping the position of the patient 201 generated by the HMD image generation unit 433 to be displayed on the HMD 295 via the communication control unit 413.

(Division area table)
FIG. 5 is a diagram showing a configuration of the divided region table 500 according to the present embodiment. The divided region table 500 holds a first feature point extracted from a part model data such as a three-dimensional blood vessel model from a CT image, which is one of computed tomography images, and a divided region including the first feature point. It is a table. Note that the configuration of the divided region table 500 is not limited to FIG. FIG. 5 shows an example in which the branching location of the blood vessel is the first feature point. However, the present invention is not limited to this, and any location that represents a unique feature may be used.

  The divided region table 500 is generated from CT data of a plurality of slices, and the received three-dimensional region model data 501 such as the STL format, the three-dimensional blood vessel model data 502 included in the three-dimensional region model data 501, and the blood vessel core line are displayed. The extracted blood vessel core line 503 is stored. In this example, in order to abstract and consider universal features, feature points are extracted based on the blood vessel core line 503. If the features of the feature points are invariable so that they can be matched, the thickness of the blood vessel is extracted. The ratio may be taken into consideration. The received three-dimensional site model data 501 may be an STL file in which only blood vessels are extracted.

  The divided region table 500 includes, as the first feature point data 504 of this example, branching IDs, branching feature data, and three-dimensional position data of feature points from the pattern of the blood vessel core line 503 as branch points. Remember. As the branching feature data, at least one of a branching number 511, a branching vector 512 indicating the start point and direction of each branch, a branching pattern 513 obtained by patterning the entire branching, and other 514 such as a blood flow direction is stored. The branching feature data may be classified in advance as the feature ID 515. The first feature point data 504 is not limited to FIG. Data that increases reliability in matching can be added, and data that decreases reliability can be excluded.

  The divided area table 500 stores a divided area ID 505 that identifies a divided area including the first feature point, and divided area data 506. Here, the divided region data 506 includes blood vessel data other than the first feature point, shape data of the divided region, centroid data, and the like. In FIG. 5, the divided region includes three first feature points or four first feature points as an example, but the number of first feature points included in the divided region is illustrated. There is no limitation. In addition, the adjacent divided regions are generated so as to have the first feature point common to the surface, the ridgeline, and the vertex, thereby facilitating the connection process.

(Ultrasonic image table)
FIG. 6A is a diagram showing a configuration of the ultrasonic image table 610 according to the present embodiment. The ultrasonic image table 610 is used to generate three-dimensional ultrasonic model data considering the position and orientation of the ultrasonic probe 291 from the ultrasonic signal received from the ultrasonic probe 291.

  The ultrasound image table 610 is associated with the ultrasound image data ID 611, patient information 612 including the patient ID and the affected area, and the acquisition date (time stamp) 613, and acquisition parameters 614 corresponding to the format of the ultrasound probe 203. The received ultrasonic image data 615 processed by the acquisition parameter 614 is stored.

  The ultrasonic image table 610 includes a magnetic sensor relative signal 616 from the magnetic sensor 231 and the magnetic sensor 293, and a three-dimensional position of the ultrasonic probe indicating the position and orientation of the ultrasonic probe 203 detected from the magnetic sensor relative signal 616. Data 617 is stored. Here, the position and orientation of the ultrasonic probe 203 is detected based on the magnetic sensor relative signal 616 from the magnetic sensor 231 and the magnetic sensor 293 because the living body moves only by the position information of the ultrasonic probe 203 by the magnetic sensor 231. This is because the three-dimensional position sometimes shifts. Therefore, the position and orientation of the ultrasonic probe 203 are recorded as a relative position and direction from the magnetic sensor 293 attached to the living body.

  The ultrasonic image table 610 stores three-dimensional ultrasonic image data 618 obtained by performing three-dimensional coordinate conversion on the received ultrasonic image data 615 corresponding to the position and orientation of the ultrasonic probe 203. Furthermore, 3D ultrasound model data 619 of the patient's organ unit or diseased part unit connected to the 3D ultrasound image data 618 corresponding to each scan is generated and stored.

(Second feature point table)
FIG. 6B is a diagram showing a configuration of the second feature point table 620 according to the present embodiment. The second feature point table 620 is a table that holds feature points extracted from the ultrasound image data 618. The configuration of the second feature point table 620 is not limited to FIG. 6B. 6B shows an example in which the branching location of the blood vessel is a feature point. However, the feature point is not limited to this, and any location that represents a unique feature may be used. However, the feature is common to the first feature point. desirable.

  The second feature point table 620 stores the blood vessel part extracted image data 621 in association with the ultrasound image data ID 611 and the ultrasound image data 618 of the ultrasound image table 610. The blood vessel part extraction is performed by binarization and contour extraction of the ultrasonic image data 618 and selection of an annular contour.

  The second feature point table 620 stores blood vessel model data 622 extracted by connecting the blood vessel part extraction image data 621 and a blood vessel core line 623 from which a blood vessel core line is extracted. In this example, in order to abstract and consider universal features, feature points are extracted from the blood vessel core line 623. However, if the features of the feature points are invariable so that they can be matched, You may consider a ratio etc.

  The second feature point table 620 includes, as the second feature point data 624 of this example, a branch ID from the pattern of the blood vessel core line 623 as a feature point, branch feature data, and a three-dimensional position of the feature point. Data. Since the branching feature data is the same as that in FIG. 5, a detailed description thereof is omitted.

<Display image generator>
FIG. 7A is a block diagram illustrating a functional configuration of the display image generation unit 423 according to the present embodiment.

  The display image generation unit 423 includes a three-dimensional part model acquisition unit 701, a divided region acquisition unit 702, a three-dimensional part model arrangement unit 703, a divided region arrangement unit 704, a second feature point data acquisition unit 705, and a division. A region matching unit 431. Here, the three-dimensional part model arrangement unit 703 and the divided region arrangement unit 704 correspond to the divided region connection unit 432. The display image generation unit 423 includes a patient position data acquisition unit 707, a patient position model generation unit 708, an HMD position data acquisition unit 709, and an HMD image generation unit 433.

  The three-dimensional part model acquisition unit 701 acquires a three-dimensional part model including the three-dimensional blood vessel model received by the three-dimensional part model reception unit 411. The divided area acquisition unit 702 acquires the divided area generated by the divided area generation unit 424. The three-dimensional part model arrangement unit 703 adjusts and arranges the three-dimensional part model data acquired by the three-dimensional part model acquisition unit 701 in association with the arrangement of the divided regions. The divided region placement unit 704 arranges and arranges the divided regions so that the divided regions match the three-dimensional ultrasonic image by matching the first feature points and the second feature points included in the divided regions. The second feature point data acquisition unit 705 acquires the second feature point data extracted from the second feature point extraction unit 418. The divided region matching unit 431 matches the first feature point and the second feature point included in the divided region, and the first feature point data of the position-adjusted divided region matches the second feature point data. The three-dimensional part model placement unit 703 and the divided region placement unit 704 are deformed by position adjustment. Note that the deformation by position adjustment includes parallel movement, rotational movement, or size change while maintaining the shape of the divided area. Furthermore, a shape change that matches the feature points after the arrangement of the divided regions may be included.

  The patient position data acquisition unit 707 acquires patient position data detected from the patient position detection unit 420. The patient position model generation unit 708 generates a three-dimensional model that overlaps the patient position from the three-dimensional part model whose position is adjusted so as to match the ultrasound image in the divided region connection unit 432. The HMD position data acquisition unit 709 acquires the HMD position data calculated from the HMD position calculation unit 422. In response to the HMD position data, the HMD image generation unit 710 generates an HMD image that is three-dimensionally displayed on the HMD 295 so as to be superimposed on the patient's living body from the model generated by the model generation unit 708 of the patient position. The data is output to the image transmission unit 425.

(Processing of display image generator)
FIG. 7B is a diagram for explaining the processing 720 of the divided region connecting unit 432 according to the present embodiment. Note that the processing 720 of the divided region connecting unit 432 is not limited to FIG. 7B. Any processing can be applied as long as the three-dimensional part model is deformed by adjusting the position of the divided region so that the first feature point included in the divided region matches the second feature point.

  The process 720 of the divided region connecting unit 432 performs a first feature point set and a first set of divided regions from a set 721 of the divided region data divided from the initial three-dimensional part model and the first feature point data included in the divided regions. It includes conversion to three-dimensional part model data 722 in which divided regions are combined by performing size change, three-dimensional position conversion, and direction conversion by matching with two feature points. In addition, the process 720 of the divided region connecting unit 432 includes conversion from the three-dimensional part model data 722 obtained by combining the divided regions into the size adjustment data 723 that matches the position of the living body.

  The conversion from the set 721 of the divided area data and the first feature point data included in the divided area to the three-dimensional part model data 722 that combines the divided areas is performed by dividing the divided area data by the divided area matching 724 with the second feature point data 726. This is realized by repeating the position adjustment and connection of the regions. For example, the position is adjusted to the divided area of the size and orientation closest to the 3D ultrasonic model by the least square distance between the first feature point and the second feature point having the same feature, but other methods can also be adopted. It is.

  Next, the conversion from the three-dimensional part model data 722 obtained by combining the divided areas into the size adjustment data 723 according to the position of the living body is superimposed on the living body position 727 acquired by the patient position data acquiring unit 707. The size of the three-dimensional part model data 722 combined with is changed (725).

  FIG. 7C is a diagram illustrating an example 740 of the configuration and alignment information of the divided region connection table 730 according to the present embodiment.

  The divided region connection table 730 is used in the divided region connecting unit 432 to connect the divided regions and generate a three-dimensional part model that matches the three-dimensional ultrasonic image.

  The divided area connection table 730 stores the divided area ID 505, the divided area data 506, and the first feature point data 504 of the divided area table 500 of FIG. The divided region connection table 730 stores the current value 731 of the changed size, direction, and position of the divided region, and the changed position 732 of the first feature point data changed based on the current value 731. Then, the divided region connection table 730 stores divided region connection positions 734 based on the second feature point data 733 that matches the change position 732 of the first feature point data and the second feature point data 733 that matches. .

  In addition, the alignment information example 740 is a position to be a position alignment target in an alignment target part (organ), and determines a marker or sensor arrangement position and a scanning position by an ultrasonic probe. Used for.

  The alignment information example 740 stores the alignment target position 742 and the condition 743 of the target position in association with the alignment target portion 741.

  Note that non-rigid deformation that compensates for a match may be combined during position adjustment of the divided region and the connection process, or after the connection process. In such a case, even if the optimal solution is obtained by performing non-rigid deformation using a transformation matrix based on the material parameters, the displacement of the distance between the first feature point and the second feature point having the same feature is simply corrected for deformation. It may be a system to do. As the material parameter, a general internal non-rigid parameter may be used, but it is desirable to measure and use the non-rigid parameter for each patient or the affected part.

  Furthermore, other methods for matching feature points included in the non-rigid material may be employed. For example, a method may be used in which deformation of non-rigid material is accumulated as learning information and used. When the three-dimensional model is an organ unit such as the liver, stomach, or pancreas, it is possible to learn a large number of modified examples of these organs and express patient-specific deformation with a small number of parameters.

  Further, a method may be used in which control points including feature points are arranged in a fine grid pattern in a three-dimensional part and non-rigid deformation is performed between the control points. Or, using non-rigid body deformation (FFD: Free-form deformation) using B-Spline function, spatially smooth and irregular deformation (fluctuation) is estimated with high accuracy, and region-based alignment is performed. The super-resolution method used may be used.

  Further, as the non-rigid deformation, affine transformation or transformation using a higher order polynomial can be applied. Conformity evaluation of a three-dimensional model includes a method based on a sum of squared difference (SSD), a method based on normalized cross correlation (NCC), a mutual information amount or a normalized mutual information amount. (MI: Mutual information) is considered.

  In addition, an image having anatomical information obtained by MRI or X-ray CT and an image having functional information such as PET or SPECT are superposed, and the superposition technique is also used for non-rigid deformation. Applicable.

(Display data generation table)
FIG. 8 is a diagram showing a configuration of the display data generation table 800 according to the present embodiment. The display data generation table 800 is a table used for displaying on the HMD 295 a three-dimensional site model that matches the ultrasound image data acquired from the patient 201 in real time so as to overlap the patient 201.

  The display data generation table 800 stores ultrasonic probe position information 802, patient position information 803, and HMD position information 804 in association with a time stamp 801 indicating real time. The ultrasonic probe position information 802 includes detection data of the magnetic sensor 292 and position data of the ultrasonic probe 291. The patient position information 803 includes detection data of the magnetic sensor 293 with an optical marker and position data of the patient 201. The HMD position information 804 includes image data of the optical marker of the magnetic sensor with optical marker 293 and HMD position data.

  The display data generation table 800 stores HMD display information 805 generated with reference to the position information. The HMD display information 805 includes 3D part model data generated by referring to the position information, right-eye HMD image data, and left-eye HMD image data.

<< Hardware configuration of image processing apparatus >>
FIG. 9 is a block diagram showing a hardware configuration of the image processing apparatus 280 according to the present embodiment.

  In FIG. 9, a CPU 910 is a processor for arithmetic control, and implements a functional component of the image processing apparatus 280 of FIG. 4 by executing a program. The ROM 920 stores fixed data and programs such as initial data and programs. Further, the communication control unit 413 communicates with the examination / surgical instrument group in the examination room or the operating room 230 and other devices via the network. Note that the number of CPUs 910 is not limited to one, and may be a plurality of CPUs or may include a GPU for image processing. Further, it is desirable that the communication control unit 413 has a CPU independent of the CPU 910 and writes or reads transmission / reception data in an area of the RAM 940. In addition, it is desirable to provide a DMAC for transferring data between the RAM 940 and the storage 950 (not shown). Further, the input / output interface 960 preferably includes a CPU independent of the CPU 910 and writes or reads input / output data to / from the area of the RAM 940. Accordingly, the CPU 910 recognizes that the data has been received or transferred to the RAM 940 and processes the data. Further, the CPU 910 prepares the processing result in the RAM 940 and leaves the subsequent transmission or transfer to the communication control unit 413, the DMAC, or the input / output interface 960.

  The RAM 940 is a random access memory that the CPU 910 uses as a work area for temporary storage. In the RAM 940, an area for storing data necessary for realizing the present embodiment is secured. The three-dimensional part model data 501 is three-dimensional part model data including a three-dimensional STL format three-dimensional blood vessel model generated and received from CT data. The first feature point data 504 is first feature point data extracted from the three-dimensional part model data 501. The divided area data 506 includes the first feature point data 504, and is divided area data divided from the three-dimensional part model data 501. The ultrasonic model data 619 is three-dimensional image data including a three-dimensional blood vessel image generated from an ultrasonic signal detected by the ultrasonic probe 203 in real time. The second feature point data 624 is second feature point data extracted from the ultrasonic model data 619. The divided region matching table 730 is a table for connecting the divided regions by the divided region matching unit 431 shown in FIG. 7C by matching the first feature point and the second feature point included in the divided region. The HMD position data 804 is position data of the HMD 295 calculated based on the optical marker image captured by the Web camera 294. The display image data 805 is data generated in order to display a three-dimensional display on the HMD 295 by superimposing a three-dimensional part model on the patient 201 based on divided regions connected by adjusting positions. Each data in the RAM 940 may be stored as a table as shown in FIGS. 5, 6A, 6B, 7C, and 8 described above.

  The storage 950 stores a database, various parameters, or the following data or programs necessary for realizing the present embodiment. The feature point extraction algorithm 951 is an algorithm for extracting the first feature point or the second feature point. The area division algorithm 952 is an algorithm for dividing the three-dimensional part model into small areas including the first feature points. The divided region matching algorithm 953 is an algorithm for matching a divided region including the first feature point with the second feature point.

  The storage 950 stores the following programs. The image processing program 955 is a program that controls the entire image processing apparatus 280. The feature point extraction module 956 is a module that extracts the first feature point or the second feature point according to the feature point extraction algorithm 951. The area dividing module 957 is a module that generates a divided area including the first feature point in accordance with the area dividing algorithm 952. The divided region matching module 958 is a module that adjusts and connects the divided regions so that the first feature points included in the divided regions match the second feature points according to the divided region matching algorithm 953. The display image generation module 959 is a module that generates a display image that is three-dimensionally displayed on the HMD 295 so that the STL data representing the deformed three-dimensional part model is inside the living body of the patient 201.

  The input / output interface 960 interfaces input / output data with input / output devices. If necessary, a display unit 961 and an operation unit 962 are connected to the input / output interface 960. When the diagnosis / surgical instrument group is connected by wire, the ultrasonic probe 203 with the magnetic sensor 231, the magnetic sensor 293 with the optical marker, the Web camera 294, and the HMD 295 included in the diagnosis / surgical instrument group are included in the input / output interface 960. Connected to. The input / output interface 960 may be connected to a voice input / output unit or a GPS position determination unit, if necessary.

  Note that the RAM 940 and the storage 950 in FIG. 9 do not show programs and data related to general-purpose functions and other realizable functions that the image processing apparatus 280 has.

<< Processing procedure of image processing apparatus >>
FIG. 10 is a flowchart showing a processing procedure of the image processing apparatus 280 according to the present embodiment. This flowchart is executed by the CPU 910 in FIG. 9 using the RAM 940, and implements the functional components shown in FIGS. 4 and 7A.

  In step S <b> 1001, the image processing device 280 receives 3D site model data including a 3D blood vessel model from the external 3D model generation device 212. In addition, when generating STL data from CT data in the image processing apparatus 280, after receiving CT data and generating STL data, three-dimensional site model data is generated. In step S1003, the image processing device 280 executes a first feature point extraction process based on the three-dimensional part model data. In step S <b> 1005, the image processing apparatus 280 executes region division processing on a three-dimensional part model expressed in, for example, the STL format into small regions including the first feature points.

  In step S1007, the image processing apparatus 280 acquires the ultrasonic image data from the ultrasonic probe 203 and the magnetic sensor positions (ultrasonic probe positions) from the magnetic sensor 231 and the magnetic sensor 293. In step S1009, the image processing apparatus 280 accumulates the position of each ultrasonic probe 203 and ultrasonic image data, and generates three-dimensional ultrasonic model data. In step 1011, the image processing device 280 executes a second feature point extraction process based on the three-dimensional ultrasonic model data.

  In step S1013, the image processing apparatus 280 performs matching between the first feature point and the second feature point included in the divided region, and determines whether or not the first feature point of the divided region matches the second feature point. To do. If the first feature point of the divided region does not match the second feature point, the image processing apparatus 280 adjusts the first feature point by adjusting the position of the divided region in step S1015. In the case of divided area data in the STL format, the STL data is converted. In this way, matching between the first feature point and the second feature point of the divided region is repeated.

  If the first feature point of the divided area matches the second feature point, the image processing apparatus 280 connects the divided area to another divided area in step S1017. In step S1019, the image processing apparatus 280 determines whether the connection of the small areas divided in step S1005 has been completed. If the connection of the divided areas has not been completed, the image processing apparatus 280 returns to step S1013 and continues the position adjustment of the divided areas and the connection process for the next divided area.

  When the connection of the divided areas is completed, a three-dimensional part model corresponding to the three-dimensional ultrasonic model has been generated. Therefore, in step S1021, the image processing device 280 performs magnetism from the magnetic sensor with optical marker 293 disposed on the patient 201. The sensor position (patient position) is acquired and adjusted so that the three-dimensional part model deformed by the division process and the connection process is displayed inside the living body of the patient 201. Next, the image processing apparatus 280 acquires an optical marker image of the magnetic sensor 293 with an optical marker from the Web camera 294 in step S1023. Then, the image processing device 280 calculates the positional relationship between the patient 201 and the HMD 295 from the optical marker image, and displays the display data of the HMD 295 so that the three-dimensional part model deformed by the division process and the connection process overlaps the patient's visual observation. Is generated and output.

  In step S1025, the image processing apparatus 280 determines whether or not the display of the three-dimensional part model inside the living body of the patient 201 is completed. If the display of the three-dimensional part model is not completed, the image processing apparatus 280 returns to step S1021 and repeats the display of the three-dimensional part model aligned with the ultrasonic image inside the living body of the patient 201 in real time. For example, when the shape of the liver is deformed, another scan is performed using ultrasonic waves to perform new alignment. That is, if the shape of the liver does not change, observation of the liver is continued without scanning with ultrasound.

(Feature point extraction process)
FIG. 11 is a flowchart showing a procedure of feature point extraction processing (S1003 / S1011) according to the present embodiment.

  In step S <b> 1111, the image processing apparatus 280 acquires target three-dimensional model data. In the case of the first feature point extraction process (S1003), the target 3D model data is 3D part model data, and in the case of the second feature point extraction process (S1011), the target 3D model data is Ultrasonic model data. In step S1113, the image processing apparatus 280 extracts blood vessel model data from the acquired three-dimensional model data. In the case of the first feature point extraction process (S1003), step S1113 is not necessary if blood vessel model data has been acquired in advance.

  In step S1115, the image processing apparatus 280 extracts a blood vessel core line based on the blood vessel model data, and generates an abstracted blood vessel pattern. Next, in step S1117, the image processing apparatus 280 extracts a branch point of the blood vessel as a feature point based on the blood vessel core line pattern. Then, in step S1119, the image processing apparatus 280 stores the set of branch point feature data and their three-dimensional positions so that they can be compared.

(Area division processing)
FIG. 12 is a flowchart showing the procedure of the area division processing (S1005) according to the present embodiment.

  In step S1201, in this example, the image processing apparatus 280 divides the three-dimensional part model into a region including at least three branch points in the blood vessel core as first feature points. Next, the image processing apparatus 280 assigns an identifier (ID) to each divided region in step S1203. In step S1205, the image processing apparatus 280 stores the first feature point set in association with the divided region ID.

  In the present embodiment, the display of the deformed three-dimensional part model on the optical see-through HMD is shown. However, the deformed three-dimensional part model is superimposed on the patient's biological image captured by the Web camera 294. Then, it may be displayed on a video-through HMD, a monitor, or a portable terminal.

  According to this embodiment, since the three-dimensional part model deformed corresponding to the three-dimensional ultrasonic model is displayed superimposed on the patient's living body, the state of the part in the patient's living body in real time It can be recognized at an accurate position.

[Third Embodiment]
Next, a medical support system according to a third embodiment of the present invention will be described. Compared with the second embodiment, the medical support system according to the present embodiment deforms a three-dimensional site model obtained from a CT image or the like corresponding to an ultrasonic image and superimposes it on an internal image observed by a laparoscope It is different in that it is displayed. Since other configurations and operations are the same as those of the second embodiment, the same configurations and operations are denoted by the same reference numerals, and detailed description thereof is omitted.

《Configuration of medical support system》
With reference to FIG. 13A and FIG. 13B, a structure and operation | movement of the medical assistance system of this embodiment are demonstrated.

(System overview)
FIG. 13A is a block diagram illustrating an outline of processing of the medical support system 1300 according to the present embodiment. In FIG. 13A, the same components as those in FIG. 2A are denoted by the same reference numerals, and description thereof is omitted.

  In the medical support system 1300, a rod-shaped ultrasonic probe 1303 is inserted into the body, and an ultrasonic image around the site observed by the laparoscope 1391 is acquired. In addition, a three-dimensional deformed corresponding to the observation image of the laparoscope 1391 and the real-time ultrasonic image using a position detection instrument fixed to the laparoscope 1391 observing the inside of the living body in real time in the operating room 1330 The positional relationship with the part model is measured. Then, based on the measured positional relationship, the three-dimensional region model is adjusted and superimposed on the observation image of the laparoscope 1391 to display a three-dimensional display 1370. In FIG. 13A, in the three-dimensional display 1370, the blood vessel model 1371 is shown as an easy-to-understand part model superimposed on a stippled liver image 1372 that is an observation image of the laparoscope 1391. The internal results are displayed superimposed. Note that the display unit that superimposes on the observation image of the laparoscope 1391 and displays the three-dimensional region model three-dimensionally is preferably a portable terminal, a monitor, or a video see-through HMD. Here, a magnetic sensor fixed to the tip of a laparoscope 1391 is used as the position detection instrument. In order to obtain the positional relationship between the observation image and the display of the three-dimensional site model with high accuracy, it is desirable to fix the optical marker to the organ.

(System configuration)
FIG. 13B is a block diagram illustrating a configuration of a medical support system 1300 according to the present embodiment. In FIG. 13B, the same reference numerals are assigned to the same components as in FIG. 2B, and the description thereof is omitted.

  The medical support system 1300 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 1380, and a diagnosis / surgical instrument group in an examination room or an operating room 1330.

  The examination / surgical instrument group in the examination room or operating room 1330 includes a rod-shaped ultrasonic probe 1303, a magnetic sensor 1331 fixed to the ultrasonic probe, a laparoscope 1391, a magnetic sensor 1392 fixed to the laparoscope, It has a monitor 1370 and a magnetic sensor 1393 fixed to a living body or a target organ. The magnetic sensor 1331 and the magnetic sensor 1393 can stably detect the position of the ultrasonic probe 1303. The laparoscope 1391 acquires in-vivo images from the patient 201 during real-time examination or surgery. The magnetic sensor 1392 is disposed at the tip position of the laparoscope 1391 and detects the observation position of the laparoscope 1391. The magnetic sensor 1392 and the magnetic sensor 1393 can stably detect the position of the laparoscope 1391.

  The image processing device 1380 divides the three-dimensional part model represented by the three-dimensional data into regions and matches the three-dimensional ultrasonic image acquired by the ultrasonic probe 1303 whose position and orientation are determined by the detection signal of the magnetic sensor 1331. It deforms by connecting to. Then, the image processing device 1380 generates three-dimensional display data in which the deformed three-dimensional part model is superimposed on the observation image acquired by the laparoscope 1391 and displays the data on the monitor 1370. A laparoscopic treatment tool 1373 operated by the operator 202 is observed by the laparoscope 1391 and displayed on the monitor 1370.

  Although only the operator 202 is shown in FIG. 13B, in actual laparoscopic surgery, the operation is performed by a plurality of humans such as a scopist who operates the laparoscope and an operator or assistant who operates the ultrasonic probe. become. In addition, a three-dimensional part model matched with the observation image acquired by the laparoscope 1391 may be superimposed and displayed on the video see-through HMD.

<< Functional configuration of image processing apparatus >>
FIG. 14 is a block diagram showing a functional configuration of the image processing apparatus 1380 according to this embodiment. In FIG. 14, the same functional components as those in FIG. 4 are denoted by the same reference numerals, and description thereof is omitted.

  The magnetic signal receiving unit 1419 receives a magnetic detection signal from the magnetic sensor 1392 disposed in the laparoscope 1391 via the communication control unit 413. The laparoscope position detector 1420 detects the position and orientation of the laparoscope 1391 based on the magnetic detection signal from the magnetic sensor 1392 and the magnetic detection signal from the magnetic sensor 1393. Note that in the case where the magnetic sensor 1392 includes a storage unit for detecting the position and orientation from the magnetic detection signal, data (for example, coordinates and vectors) indicating the position and orientation may be received. The laparoscopic image receiving unit 1421 receives an observation image from the laparoscope 1391 via the communication control unit 413. Note that the laparoscopic image receiving unit 1421 may directly receive laparoscopic image data when the laparoscope 1391 has a function of generating laparoscopic image data. The magnetic signal receiving unit 1422 receives a magnetic detection signal from the magnetic sensor 1393 disposed in the living body 201 or a target organ in the living body via the communication control unit 413.

  The generation model superimposing unit 1433 of the display image generating unit 1423 matches the real-time three-dimensional ultrasound model by dividing the three-dimensional part model represented in the STL format and the like so that the feature points correspond to each other. An image is generated by superimposing the three-dimensional site model deformed to the observation image acquired by the laparoscope 1391. Then, the generated model superimposing unit 1433 transmits the generated superimposed image from the display image transmitting unit 425 to the monitor 1370, the portable terminal, or the video see-through HMD for display.

  According to this embodiment, since the three-dimensional region model deformed corresponding to the three-dimensional ultrasound image is superimposed on the in-vivo image acquired by the laparoscope, the region in real time inside the patient's body is displayed. The state can be accurately recognized with an image inside the living body.

[Fourth Embodiment]
Next, a medical support system according to the fourth embodiment of the present invention will be described. The medical support system according to the present embodiment is different from the second embodiment in that a three-dimensional part model deformed corresponding to the three-dimensional ultrasonic model is displayed on a sheet display on a patient's living body. Since other configurations and operations are the same as those of the second embodiment, the same configurations and operations are denoted by the same reference numerals, and detailed description thereof is omitted.

《Configuration of medical support system》
FIG. 15 is a block diagram illustrating a configuration of a medical support system 1500 according to the present embodiment. In FIG. 15, the same components as those in FIG. 2B or FIG. 13B are denoted by the same reference numerals, and description thereof is omitted.

  The medical support system 1500 includes a CT apparatus 211, a three-dimensional model generation apparatus 212, an image processing apparatus 1580, and a diagnosis / surgical instrument group in an examination room or an operating room 1530.

  The examination / surgical instrument group in the examination room or operating room 1530 includes an ultrasonic probe 203, a magnetic sensor 231 fixed to the ultrasonic probe, a magnetic sensor 293 with an optical marker, a video camera 1594, a sheet display 1595, Have The video camera 1594 captures an image of the optical marker of the magnetic sensor with optical marker 293 arranged on the sheet display 1595. A sheet display 1595 arranged on the surface of the patient 201 displays a three-dimensional model represented by three-dimensional data deformed to match the three-dimensional ultrasonic model at an accurate position inside the living body.

  The image processing apparatus 1580 matches the three-dimensional site model represented by the three-dimensional data with the three-dimensional ultrasonic model obtained by accumulating the ultrasonic signal from the ultrasonic probe 203 and the position information of the ultrasonic probe 203. It deforms as follows. Then, the image processing apparatus 1580 generates three-dimensional display data so that the deformed three-dimensional part model is at an accurate position inside the living body of the patient 201 and displays the three-dimensional display on the sheet display 1595.

  In the present embodiment, since the sheet display 1595 is the same as the living body surface of the patient 201, the image processing apparatus 1580 may detect the positions and orientations of the patient 201 and the sheet display 1595 from the captured image of the optical marker. It is not necessary to calculate the positional relationship between the sheet display 1595 and the sheet display 1595.

<< Functional configuration of image processing apparatus >>
FIG. 16 is a block diagram illustrating a functional configuration of the image processing apparatus 1580 according to the present embodiment. In FIG. 16, the same functional components as those in FIG. 4 or FIG.

  The sheet position calculation unit 1622 calculates the accurate position of the sheet display 1595 based on the captured image of the optical marker received by the optical marker image reception unit 421.

  The sheet image generation unit 1633 of the display image generation unit 1623 calculates the sheet display calculated by the sheet position calculation unit 1622 so that the three-dimensional part model based on the SLT data adjusted by the position of the patient 201 detected by the patient position detection unit 420 overlaps. A display image for three-dimensional display at a position is generated. The display image transmission unit 1625 transmits the image data that overlaps the position of the patient 201 generated by the sheet image generation unit 1633 to be displayed on the sheet display 1595 via the communication control unit 413.

  Note that the configuration and operation of the sheet image generation unit 1633 are the same as those in the second embodiment, except that the display destination of the three-dimensional part model is changed from the HMD 295 to the sheet display 1595, and thus detailed description thereof is omitted here. However, in this example, since the sheet display 1595 is deformed non-rigidly, a deformation operation of the sheet-shaped non-rigid body is necessary, and an existing technique can be used.

  According to the present embodiment, since the deformed three-dimensional part model is displayed on the sheet display on the patient's living body, the state of the part in the real time inside the patient's living body is usually accurately determined without considering the line of sight or the like. Can be recognized in position.

[Fifth Embodiment]
Next, a medical support system according to a fifth embodiment of the invention will be described. The medical support system according to the present embodiment is different from the third embodiment in that an image obtained by superimposing a deformed three-dimensional part model on an observation image of the laparoscope 1391 is displayed on a communication terminal via a network. . Since other configurations and operations are the same as those of the third embodiment, the same configurations and operations are denoted by the same reference numerals, and detailed description thereof is omitted.

《Configuration of medical support system》
FIG. 17 is a block diagram illustrating a configuration of a medical support system 1700 according to the present embodiment. In FIG. 17, the same components as those in FIGS. 2B, 13B, and 15 are denoted by the same reference numerals, and description thereof is omitted.

  In FIG. 17, the image processing device 1780 displays a three-dimensional part model superimposed on the observation image of the laparoscope 1391 as in the image processing device of FIG. 13B. At the same time, the image processing device 1780 generates an image in which the three-dimensional region model is superimposed on the observation image captured by the laparoscope 1391 and distributes it to other doctors and users via the network 1790. A notebook PC 1791, a portable terminal 1792 such as a smartphone or a tablet, a monitor 179i, a server 179n, or the like may be connected to the distribution destination.

  According to the present embodiment, other doctors and third parties can monitor the observation image of the laparoscope inside the living body of the patient outside the examination room or the operating room, and can support more accurate medical care.

[Other Embodiments]
In the above embodiment, only a limited configuration example of the medical support system of the present invention is shown. FIG. 18 is a diagram showing various configuration examples 1800 of the medical support system of the present invention. The present invention is not limited to the example of FIG. 18, but the scope of application of the present invention is clear from FIG.

  The configuration example 1800 in FIG. 18 includes a three-dimensional part model 1801 to be displayed in a deformed form, a three-dimensional ultrasonic model 1802 used for positioning the three-dimensional part model 1801, and a part model matching method (positioning method). 1803, a target 1804 on which a three-dimensional part model 1801 is superimposed, and a display unit 1805 are shown. These can be combined independently to achieve better alignment and overlay.

  Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, a system or an apparatus in which different features included in each embodiment are combined in any way is also included in the scope of the present invention.

  In addition, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention can also be applied to a case where a control program that realizes the functions of the embodiments is supplied directly or remotely to a system or apparatus. Therefore, in order to realize the functions of the present invention on a computer, a program installed in the computer, a medium storing the program, and a WWW (World Wide Web) server that downloads the program are also included in the scope of the present invention. . In particular, at least a non-transitory computer readable medium storing a program for causing a computer to execute the processing steps included in the above-described embodiments is included in the scope of the present invention.

Claims (15)

A part model generation means for generating data of a three-dimensional part model based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
Ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired are stored in association with each other, and the ultrasound data and the position and Ultrasonic model generation means for generating data of a three-dimensional ultrasonic model using the orientation data;
Second feature point extracting means for extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
Deformation means for deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
Display means for displaying a projection image of the three-dimensional part model deformed by the deformation means;
A medical support system comprising: The deformation means includes
A divided region generating means for dividing the three-dimensional part model into a plurality of three-dimensional regions each including at least one first feature point;
Divided region connection that adjusts the position of each of the plurality of three-dimensional regions so that the plurality of first feature points and the plurality of second feature points overlap each other, and connects the plurality of three-dimensional regions that have been adjusted to each other. Means,
Have
The medical support system according to claim 1, wherein the display unit displays the plurality of three-dimensional regions connected by the divided region connecting unit as the three-dimensional part model. The plurality of first feature points are feature points of a three-dimensional blood vessel model included in a target region of the three-dimensional site model, and the plurality of second feature points are included in the target region of the three-dimensional ultrasound model. It is a feature point of the included 3D blood vessel model,
The medical support system according to claim 1, wherein the display unit displays the three-dimensional blood vessel model as the three-dimensional site model. Imaging means for imaging the living body and a first position detection instrument for detecting the position and orientation of the living body or the position and orientation of the living body image;
First adjusting means for adjusting the position, orientation and size of the three-dimensional part model deformed by the deforming means based on the imaged position and orientation of the first position detecting instrument;
Further comprising
The medical support system according to any one of claims 1 to 3, wherein the display unit displays the deformed and adjusted three-dimensional part model so as to overlap the living body or the living body image of the patient.   The medical support system according to claim 4, wherein the first position detection instrument is a magnetic sensor with an optical marker or an optical marker. A laparoscope for imaging the inside of the living body, to which a second position detection instrument is fixed;
Second adjustment for adjusting the position, orientation and size of the three-dimensional part model deformed by the deformation means based on the position and orientation of the second position detecting instrument and the position and orientation of the ultrasonic probe Means,
Further comprising
The medical support system according to any one of claims 1 to 3, wherein the display unit displays the deformed and adjusted three-dimensional site model so as to overlap an in-vivo image captured by the laparoscope.   The medical support system according to claim 6, wherein the second position detection instrument is a magnetic sensor.   The medical support system according to any one of claims 1 to 7, further comprising an ultrasonic probe position detection unit configured to detect a position and an orientation of a third position detection instrument fixed to the ultrasonic probe.   The medical support system according to claim 8, wherein the third position detection instrument is an optical marker or a magnetic sensor.   The display means includes an optical see-through head mounted display that displays the deformed and adjusted three-dimensional part model and the living body so as to overlap each other, or the deformed and adjusted three-dimensional part model and the living body image. The medical support system according to any one of claims 1 to 9, wherein the medical support system is a video see-through type head-mounted display that displays the images so as to overlap each other.   The medical support system according to any one of claims 1 to 9, wherein the display unit is a sheet display that displays the deformed and adjusted three-dimensional site model and the living body or the living body image so as to overlap each other. A site model generation step for generating data of a 3D site model based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A display step of displaying a projected image of the three-dimensional part model deformed in the deformation step;
Including medical support methods. Receiving means for receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
First feature point extraction means for extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
Accumulation means for accumulating the ultrasonic data acquired using an ultrasonic probe for the patient and the position and orientation data of the ultrasonic probe from which the ultrasonic data was acquired;
An ultrasonic model generating means for generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
Second feature point extracting means for extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
Deformation means for deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
Transmitting means for transmitting a projection image of the three-dimensional part model deformed by the deforming means to a display means;
An image processing apparatus comprising: A receiving step of receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmission step of transmitting to the display means a projection image of the three-dimensional region model deformed in the deformation step;
A method for controlling an image processing apparatus. A receiving step of receiving data of a three-dimensional site model generated based on a tomographic image group of a patient;
A first feature point extracting step of extracting position information of a plurality of first feature points having a predetermined feature from the three-dimensional part model;
An accumulation step of associating and accumulating in the accumulation means the ultrasound data acquired using an ultrasound probe for the patient and the position and orientation data of the ultrasound probe from which the ultrasound data was acquired;
An ultrasonic model generation step of generating data of a three-dimensional ultrasonic model using the ultrasonic data and the position and orientation data;
A second feature point extracting step of extracting position information of a plurality of second feature points having the predetermined feature from the three-dimensional ultrasonic model;
A deforming step of deforming the three-dimensional site model so that the plurality of first feature points and the plurality of second feature points overlap;
A transmission step of transmitting to the display means a projection image of the three-dimensional region model deformed in the deformation step;
Control program for an image processing apparatus that causes a computer to execute

Images